Re: git on small systems

[matthew sporleder <msporleder%gmail.com@localhost>]

On Tue, Jan 6, 2015 at 10:13 PM, matthew sporleder <msporleder%gmail.com@localhost> wrote:
> On Tue, Jan 6, 2015 at 8:22 PM, matthew sporleder <msporleder%gmail.com@localhost> wrote:
>> On Tue, Jan 6, 2015 at 11:09 AM, matthew sporleder <msporleder%gmail.com@localhost> wrote:
>>> On Tue, Jan 6, 2015 at 11:08 AM, Justin Cormack
>>> <justin%specialbusservice.com@localhost> wrote:
>>>> On Tue, Jan 6, 2015 at 3:55 PM, matthew sporleder <msporleder%gmail.com@localhost> wrote:
>>>>> I did some testing over the last little bit for git on small systems.
>>>>> Please correct anything weird you see; I'm not a git master.
>>>>>
>>>>> I am testing on my VPC that looks like this:
>>>>>
>>>>> NetBSD vc136-15.vc.panix.com 6.0.2 NetBSD 6.0.2 (PANIX-VC) #0: Sun May
>>>>> 19 03:41:56 EDT 2013
>>>>> root%juggler.panix.com@localhost:/misc/obj/misc/devel/netbsd/6.0.2/src/sys/arch/amd64/compile/PANIX-VC
>>>>> amd64
>>>>
>>>> I am guessing that a 32 bit machine would use a noticeable amount
>>>> less, but the shallow clone looks excellent anyway. Since recent git
>>>> you can push from shallow clones which helps.
>>>>
>>>> Justin
>>>
>>> I wouldn't hold your breath, but if I get some time I'll try to get my
>>> rpi booting and see what I can accomplish.
>>
>> So someone just gave me a login to an rpi (128 MB config)
>>
>> the "shallow" clone worked without any trouble.  I'm actually pretty
>> excited to try this on pkgsrc.
>>
>> Full clone, even over http, was killed by the UVM killer (not the
>> weird signal I was getting on my 512 box).
>>
>> I am currently trying some more styles of clones on my 512 amd64 and
>> the rpi reconfigured for 256 but my predictions are this:
>>
>> Even the smallest systems can easily track necessary branches for
>> security fixes, developing patches, etc using the shallow clone
>> methods described earlier.
>>
>> A small-ish system like mine: 512MB + some swap + some git tuning can
>> get a full history clone.  I think this is probably the lowest
>> reasonable config for any kind of "normal" development (with the full
>> history).
>>
>> 1GB of memory is comfortable and probably doesn't require much tuning,
>> but I wouldn't mind seeing some testing.
>>
>> NetBSD should offer git over http/https* because it appears to be more
>> memory-friendly.  I am also testing an ssh-based checkout (git:// was
>> not finishing).
>>
>> * NetBSD does not ship with mozilla rootcerts so https requires some
>> extra stuff.
>>
>>
>> ---
>>
>> Anyway I hope this helped give some facts around the "small systems"
>> concerns of git.
>>
>> I also think there are some server-side options that can help the
>> clients use less memory (smaller pack sizes or something like that) at
>> the expense of disk space.  I'd be interested to hear any ideas on
>> that for when we are testing netbsd.org hosted git mirrors.
>
> for the record: the ssh clone also failed on my 512 system.
>
> I would suggest we do a weekly git bundle or git archive to handle
> this clone overhead issue, which is similar to offering rsync/sup/etc


After some even more aggressive tuning* I am able to use ssh or http
clone of full history on my 512 + 256 amd64 system.

I am also able to do http full clone on a 256 ram + 256 swap rpi, but
not ssh/git://

*
[pack]
        windowMemory = 1m
        packSizeLimit = 1m
        deltaCacheSize = 1m
        deltaCacheLimit = 10
        packSizeLimit = 1m
        threads = 1
[core]
        packedGitWindowSize = 1m
        packedGitLimit = 1m
        deltaBaseCacheLimit = 1m
        compression = 0
        loosecompression = 0
        bigFileThreshold = 10m
[http]
        sslVerify = false
[transfer]
        unpackLimit = 10

---

So if you can come up with 512MB somehow you should be able to clone
and work with the full git history over http.

Again, you can do a --depth 1 clone on pretty much any system.  This
is also much faster.

I am interested to try an rsync and test pull performance, etc, but
that will have to wait until we have our own servers setup.

Any more suggestions?  I'm pretty much finished with this line of testing.