Re: git copies of cvs modules available

[matthew sporleder <msporleder%gmail.com@localhost>]

On Sun, Nov 8, 2009 at 8:44 PM, Arnaud Lacombe <lacombar%gmail.com@localhost> 
wrote:
> Hi Thor,
>
> On Fri, Nov 6, 2009 at 8:27 PM, Thor Lancelot Simon <tls%panix.com@localhost> 
> wrote:
>> Any software that suddenly consumes gigabytes of
>> RAM because someone was so foolish as to use a particular network
>> transport
>>
> I've been digging in GIT sources today, and actually, it is more
> clever than you might think. Most of the memory you are seeing
> "consumed" is in fact pack mmap()'ed, but munmap()'ed lazily (ie. when
> a subsequent allocation fails). So as long as there is memory to be
> used, it uses it to store pack. Yes, it might definitively not have
> been designed to run in parallel with other services, but that should
> be tunable.
>
> For testing, I used a modified version of pack-objects, which is only
> traversing the commit list. On the pkgsrc tree, that operations
> consumes ~280MB, with ~150MB of mmap()'ed pack, 16MB of object hash
> (for 1182595 objects), 30MB for 623601 commit objects, 17MB for 558994
> tree objects. That live about 80MB used $ididntfigureoutwhereyet.
>


Hey, guys.  I don't know if the following proves much more than that
my old powerbook is slow, but I found that git definitely consumes a
lot of resources.  :)  I checked out the netbsd source tree over http
and then grabbed it in various ways on localhost.

On my powerbook (1GHz, 768MB RAM) I started out with a clone of the
netbsd source over http and then did the following measurements:

The summary is that local disk wins by a lot, ssh:// and git:// seem
to be so memory and cpu intensive that it takes me over 7 hours to
complete a clone.  After I performed a git-gc, that time was more than
cut in half.

Http uses less resources so it finished in just over an hour.

Also to contrast, I cloned the linux kernel in 30 minutes from
kernel.org and in 51 minutes over localhost.

Commands and times are below.  I encourage anyone with a more modern
computer to try the same.  (I also highly recommend a git-gc)

---

time git clone /Users/msporleder/Sites/netbsd.git/src/
Initialized empty Git repository in /Users/msporleder/Documents/nb.git/src/.git/
Checking out files: 100% (102657/102657), done.

real    7m7.914s
user    0m43.280s
sys     2m3.053s

----

time git clone --upload-pack /usr/local/git/bin/git-upload-pack
ssh://msporleder@localhost/Users/msporleder/Sites/netbsd.git/src/
Initialized empty Git repository in /Users/msporleder/Documents/nb.git/src/.git/
Password:
remote: Counting objects: 4394714, done.
remote: Compressing objects: 100% (1147983/1147983), done.
Receiving objects: 100% (4394714/4394714), 1.06 GiB | 440 KiB/s, done.
remote: Total 4394714 (delta 3270782), reused 4205930 (delta 3081999)
Resolving deltas: 100% (3270782/3270782), done.
Checking out files: 100% (102657/102657), done.

real    472m39.184s
user    25m17.742s
sys     9m22.667s

---

time git clone git://localhost/src/Initialized empty Git repository in
/Users/msporleder/Documents/nb.git/src/.git/
remote: Counting objects: 4394714, done.
remote: Compressing objects: 100% (1147983/1147983), done.
Receiving objects: 100% (4394714/4394714), 1.06 GiB | 524 KiB/s, done.
remote: Total 4394714 (delta 3270782), reused 4205930 (delta 3081999)
Resolving deltas: 100% (3270782/3270782), done.
Checking out files: 100% (102657/102657), done.

real    443m21.852s
user    24m18.869s
sys     8m37.478s

---

ime git clone http://localhost/~msporleder/nbgit/src/.git
Initialized empty Git repository in /Users/msporleder/Documents/nb.git/src/.git/
Getting alternates list for http://localhost/~msporleder/nbgit/src/.git
Getting pack list for http://localhost/~msporleder/nbgit/src/.git
Getting index for pack 0426c065a1cfda477f504fde528580890c6ba265
Getting index for pack 0593059f51d77311aa6c1437e18fe147894fbc5a
Getting index for pack 06ca93bc9ea75300aff4e1796aab0f1d848cd8c4
Getting index for pack 075af377bc83e6ded67d29978bfe0d0318323c54
Getting index for pack 0188fc4a76addd849f040cf17098ae964f6ee213
Getting index for pack 03a3795567a6b42473deb64dca07c686ee090972
Getting index for pack 09e76e533237d3483baa1208866d5f21c4f18c95
Getting index for pack 0d96f1a44703d4b1bac2820272094cb42d24e5f1
Getting index for pack 0e0b5eddca458c189aef350199b7a539be3b7403
Getting index for pack 0e72c5325d2484b2ef192fead448fc22cb657b44
...
walk 9f9f48b818ca084759832522de57b993076768ba
Checking out files: 100% (102657/102657), done.

real    71m22.482s
user    25m47.983s
sys     8m2.197s

-----------

After a git-gc

time git clone git://localhost/src/Initialized empty Git repository in
/Users/msporleder/Documents/nb.git/src/.git/
remote: Counting objects: 4394714, done.
remote: Compressing objects: 100% (959200/959200), done.
Receiving objects: 100% (4394714/4394714), 1.06 GiB | 1265 KiB/s, done.
remote: Total 4394714 (delta 3270782), reused 4394713 (delta 3270782)
Resolving deltas: 100% (3270782/3270782), done.
Checking out files: 100% (102657/102657), done.

real    198m33.599s
user    23m53.748s
sys     8m31.833s

----
time git clone 
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
Initialized empty Git repository in
/Users/msporleder/Documents/nb.git/linux-2.6/.git/
remote: Counting objects: 1362704, done.
remote: Compressing objects: 100% (221448/221448), done.
remote: Total 1362704 (delta 1132826), reused 1361636 (delta 1131845)
Receiving objects: 100% (1362704/1362704), 289.37 MiB | 256 KiB/s, done.
Resolving deltas: 100% (1132826/1132826), done.
Checking out files: 100% (30478/30478), done.

real    35m58.543s
user    12m31.478s
sys     3m19.580s

----
time git clone git://localhost/linux-2.6Initialized empty Git
repository in /Users/msporleder/Documents/nb.git/linux-2.6/.git/
remote: Counting objects: 1362704, done.
remote: Compressing objects: 100% (220467/220467), done.
Receiving objects: 100% (1362704/1362704), 289.41 MiB | 785 KiB/s, done.
remote: Total 1362704 (delta 1132826), reused 1362704 (delta 1132826)
Resolving deltas: 100% (1132826/1132826), done.
Checking out files: 100% (30478/30478), done.

real    51m32.263s
user    12m36.750s
sys     3m27.435s