Re: git on small systems

[matthew sporleder <msporleder%gmail.com@localhost>]

Sorry to revive this old thread but after reading some news about
Microsoft and other big players moving *GIANT* repos to git (always
with special considerations) I thought it would be worth testing our
own larger repo.

Without any of the special tuning  (well I still use threads = 1) and
the latest git  (2.11.1) I am able to clone full src with a peak
between 350 - 400MB memory usage over ssh, which is higher memory than
http.

This does appear to be a moderate improvement over our initial testing
two years ago.

Matt





On Wed, Jan 7, 2015 at 8:58 PM, matthew sporleder <msporleder%gmail.com@localhost> wrote:
> On Tue, Jan 6, 2015 at 10:13 PM, matthew sporleder <msporleder%gmail.com@localhost> wrote:
>> On Tue, Jan 6, 2015 at 8:22 PM, matthew sporleder <msporleder%gmail.com@localhost> wrote:
>>> On Tue, Jan 6, 2015 at 11:09 AM, matthew sporleder <msporleder%gmail.com@localhost> wrote:
>>>> On Tue, Jan 6, 2015 at 11:08 AM, Justin Cormack
>>>> <justin%specialbusservice.com@localhost> wrote:
>>>>> On Tue, Jan 6, 2015 at 3:55 PM, matthew sporleder <msporleder%gmail.com@localhost> wrote:
>>>>>> I did some testing over the last little bit for git on small systems.
>>>>>> Please correct anything weird you see; I'm not a git master.
>>>>>>
>>>>>> I am testing on my VPC that looks like this:
>>>>>>
>>>>>> NetBSD vc136-15.vc.panix.com 6.0.2 NetBSD 6.0.2 (PANIX-VC) #0: Sun May
>>>>>> 19 03:41:56 EDT 2013
>>>>>> root%juggler.panix.com@localhost:/misc/obj/misc/devel/netbsd/6.0.2/src/sys/arch/amd64/compile/PANIX-VC
>>>>>> amd64
>>>>>
>>>>> I am guessing that a 32 bit machine would use a noticeable amount
>>>>> less, but the shallow clone looks excellent anyway. Since recent git
>>>>> you can push from shallow clones which helps.
>>>>>
>>>>> Justin
>>>>
>>>> I wouldn't hold your breath, but if I get some time I'll try to get my
>>>> rpi booting and see what I can accomplish.
>>>
>>> So someone just gave me a login to an rpi (128 MB config)
>>>
>>> the "shallow" clone worked without any trouble.  I'm actually pretty
>>> excited to try this on pkgsrc.
>>>
>>> Full clone, even over http, was killed by the UVM killer (not the
>>> weird signal I was getting on my 512 box).
>>>
>>> I am currently trying some more styles of clones on my 512 amd64 and
>>> the rpi reconfigured for 256 but my predictions are this:
>>>
>>> Even the smallest systems can easily track necessary branches for
>>> security fixes, developing patches, etc using the shallow clone
>>> methods described earlier.
>>>
>>> A small-ish system like mine: 512MB + some swap + some git tuning can
>>> get a full history clone.  I think this is probably the lowest
>>> reasonable config for any kind of "normal" development (with the full
>>> history).
>>>
>>> 1GB of memory is comfortable and probably doesn't require much tuning,
>>> but I wouldn't mind seeing some testing.
>>>
>>> NetBSD should offer git over http/https* because it appears to be more
>>> memory-friendly.  I am also testing an ssh-based checkout (git:// was
>>> not finishing).
>>>
>>> * NetBSD does not ship with mozilla rootcerts so https requires some
>>> extra stuff.
>>>
>>>
>>> ---
>>>
>>> Anyway I hope this helped give some facts around the "small systems"
>>> concerns of git.
>>>
>>> I also think there are some server-side options that can help the
>>> clients use less memory (smaller pack sizes or something like that) at
>>> the expense of disk space.  I'd be interested to hear any ideas on
>>> that for when we are testing netbsd.org hosted git mirrors.
>>
>> for the record: the ssh clone also failed on my 512 system.
>>
>> I would suggest we do a weekly git bundle or git archive to handle
>> this clone overhead issue, which is similar to offering rsync/sup/etc
>
>
> After some even more aggressive tuning* I am able to use ssh or http
> clone of full history on my 512 + 256 amd64 system.
>
> I am also able to do http full clone on a 256 ram + 256 swap rpi, but
> not ssh/git://
>
> *
> [pack]
>         windowMemory = 1m
>         packSizeLimit = 1m
>         deltaCacheSize = 1m
>         deltaCacheLimit = 10
>         packSizeLimit = 1m
>         threads = 1
> [core]
>         packedGitWindowSize = 1m
>         packedGitLimit = 1m
>         deltaBaseCacheLimit = 1m
>         compression = 0
>         loosecompression = 0
>         bigFileThreshold = 10m
> [http]
>         sslVerify = false
> [transfer]
>         unpackLimit = 10
>
> ---
>
> So if you can come up with 512MB somehow you should be able to clone
> and work with the full git history over http.
>
> Again, you can do a --depth 1 clone on pretty much any system.  This
> is also much faster.
>
> I am interested to try an rsync and test pull performance, etc, but
> that will have to wait until we have our own servers setup.
>
> Any more suggestions?  I'm pretty much finished with this line of testing.