Re: cvs-fast-export on NetBSD repo, first results (was Re: Why I'm working on a NetBSD conversion)

["Eric S. Raymond" <esr%thyrsus.com@localhost>]

Jeff Rizzo <riz%NetBSD.org@localhost>:
> Now, onto the details.  I have not yet created a git (or any other)
> repository, but I ran cvs-fast-export successfully on the NetBSD
> repo starting on Thursday.  It took over a day on a 12-core Xeon
> with 74GB of RAM running NetBSD 7.0_BETA.

> http://www.tastylime.net/netbsd/errors.src.6213.txt
>  
> This is good, in that it seems to be complete.  Bad, in that 35
> hours is a LONG time... perhaps we can speed it up with some
> judicious use of tmpfs.

That is *astonishingly* slow.  So far I have seen full benchmarks 
on three systems, and the times cluster around 5 hours. I think
we first need to figure out why your runs are taking seven times
longer.

I can tell you a couple of relevant things:

1. Maximum working set is about 18GB. Output stream size is about 33GB
   (that will go up faster over time, of course).

2. Multiple cores don't help. The only part of the process that can be
   parallelized is the first stages of CVS master analysis, and as a
   matter of observed fact that is completely swamped by the other
   stages.

3. On all theree NetBSD runs I've seen, compute time for the branch
   merge dominated the runtime.  So faster single-thread performance
   of the hardware matters a lot.

4. Cache width matters a lot too because the working set is so big.

One of my development partners just turned in a patch that cuts 
conversion time on my largest benchmark repo (which happens to
be historical Emacs CVS) by 20%. I'll run some tests and report.
-- 
		<a href="http://www.catb.org/~esr/";>Eric S. Raymond</a>