cvs-fast-export on NetBSD repo, first results (was Re: Why I'm working on a NetBSD conversion)

[Jeff Rizzo <riz%NetBSD.org@localhost>]

On 10/24/14 3:36 AM, Eric S. Raymond wrote:
> (As just posted to my blog)
>
> Some people on the NetBSD tech-repository list have <a
> href="http://mail-index.netbsd.org/tech-repository/tindex.html";>wondered</a>
> why I've been working on a full NetBSD repository conversion without a
> formal request from NetBSD's maintainers that I do so.

As mentioned, several folks are interested in this and have been working 
with the NetBSD repo in various forms.  The successful conversions prior 
to this have been done with Jörg Sonnenberger's fossil conversion tool.

> 1. Alan Barrett tried to run a full conversion of NetBSD using
> cvs-fast-export last December and failed (OOM).  He then engaged me
> and we spent significant effort trying to reduce the program's working
> set, but could not prevent OOM on either of the machines we were
> using.  Because Alan was willing to work on this at some length, I
> formed the idea that there was real demand for a full NetBSD
> conversion.

Yes, there is demand for it.  It's good to have more tools.
> This is my offer: I have the tools and the experience to get you to
> the changeset-oriented VCS of your choice. I can do a really good job,
> better than you'll ever get from mechanical mirroring or a batch
> converter, because I know all about common conversion artifacts and
> how to do things like lifting old version references and
> ignore-pattern files.
It will certainly be useful to have help in this process, wherever it 
winds up.  (Politics, for sure!)

> It looks like my tools are git-oriented because they rely on git
> fast-import streams as an interchange format, but I'm not advocating
> git per se - I'm urging you to <em>move somewhere with
> changesets</em>.  It's a messy job and it wants an expert like me on
> it, but it only has to be done once.  Afterwards, the quality of your
> developer experience and your future technical options with regard to
> what VCS you actually want to use will both greatly improve.

It only needs to be done _finally_ once.  For the time being, it needs 
to be done repeatedly until a decision is made on the final direction.

> Related technical point: the architectural insight behind my tools is
> that the git folks created something more generally useful than they
> understood when they defined import streams.  Having an editable
> transfer format that can be used to move content and metadata
> relatively seamlessly between VCSes is as important in the long term
> as the invention of the DVCS - possibly more so.
>
> cvs-fast-export emits a fast-import stream not because I'm a git
> partisan (I actually rather wish hg had won the mindshare war) but
> because that's how you get to a sufficiently expressive interchange
> format.

Now, onto the details.  I have not yet created a git (or any other) 
repository, but I ran cvs-fast-export successfully on the NetBSD repo 
starting on Thursday.  It took over a day on a 12-core Xeon with 74GB of 
RAM running NetBSD 7.0_BETA.  Here's what I've got. The script I used 
invokes cvs-fast-export like so:

name=src
(cd /cvsroot && find ${name} -name '*,v' | ${doit} cvs-fast-export -p -l 
${here}/errors.${name}.$$ >${here}/${name}.fi )

2014-10-23T17:03:20Z: Reading file list...done, 11419042.338KB in 303571 
files (19.291105sec)
2014-10-23T17:03:39Z: Analyzing masters...266151ignoring symbol magnum 
(FreeBSD RELENG_2_1_0 braindamage?)
2014-10-23T17:03:39Z: Analyzing masters...266488ignoring symbol magnum 
(FreeBSD RELENG_2_1_0 braindamage?)
2014-10-23T17:03:39Z: Analyzing masters...291009ignoring symbol magnum 
(FreeBSD RELENG_2_1_0 braindamage?)
2014-10-23T17:03:39Z: Analyzing masters...291014ignoring symbol magnum 
(FreeBSD RELENG_2_1_0 braindamage?)
2014-10-23T17:03:39Z: Analyzing masters...291016ignoring symbol magnum 
(FreeBSD RELENG_2_1_0 braindamage?)
2014-10-23T17:03:39Z: Analyzing masters...291030ignoring symbol magnum 
(FreeBSD RELENG_2_1_0 braindamage?)
2014-10-23T17:03:39Z: Analyzing masters...291031ignoring symbol magnum 
(FreeBSD RELENG_2_1_0 braindamage?)
2014-10-23T17:03:39Z: Analyzing masters...done, 2652245 revisions 
(2084.337021sec)
2014-10-23T17:38:23Z: Make DAG branch heads...303571 of 303571(100%)    
(0.871937sec)
2014-10-23T17:38:24Z: Sorting...done  (32.117227sec)
2014-10-23T17:38:56Z: Find branch parent relationships...done (27.125036sec)
2014-10-23T17:39:23Z: Merge common branches...0 of 303571(0%) load: 
1.00  cmd: cvs-fast-export 4394 [0x7f7ff6cfee96/0] 13670.37u 56.37s 99% 
9339624k
2014-10-23T17:39:23Z: Merge common branches...178 of 178(100%) 
(80001.703951sec)
2014-10-24T15:52:45Z: Compute tail values...done  (2.007626sec)
2014-10-24T15:52:47Z: Find tag locations...
cvs-fast-export: tag vmlocking2-base2 could not be assigned to a commit
2014-10-24T15:52:47Z: Find tag locations...done  (75.890870sec)
2014-10-24T15:54:03Z: Generating snapshots...done (7068.878553sec))
2014-10-24T17:51:52Z: Saving in fast order: done (36242.848553sec)
cvs-fast-export: no commitids before 2014-06-22T23:53:00Z.
       after parsing:   2103.628        0KB
  after branch merge:   82243.347       0KB
               total:   125555.132      0KB
286792 commits/37067.034M text at 2 commits/sec.
cvs-fast-export: 3022 warning(s).
cvs-fast-export done
1d11h6m32s
xs2:riz  /cvs-fast-export>

The error log is here:
http://www.tastylime.net/netbsd/errors.src.6213.txt


This is good, in that it seems to be complete.  Bad, in that 35 hours is 
a LONG time... perhaps we can speed it up with some judicious use of tmpfs.

+j