Re: misc/50216: USB, PCI ID data grow quadratically in the repository

[Robert Elz <kre%munnari.OZ.AU@localhost>]

The following reply was made to PR misc/50216; it has been noted by GNATS.

From: Robert Elz <kre%munnari.OZ.AU@localhost>
To: gnats-bugs%www.NetBSD.org@localhost
Cc: 
Subject: Re: misc/50216: USB, PCI ID data grow quadratically in the repository
Date: Fri, 27 Oct 2017 18:54:06 +0700

 I just got asked to take a look at this issue, and I totally agree with
 Joerg - these files should be generated as part of the build, not
 committed to CVS.
 
 Matt's objection to that was ...
 
 mrg%eterna.com.au@localhost said (back in Sep 2015):
   |  i would rather avoid this method, since it means "grep" can't find
   |  the symbol used in the source code. 
 
 and I understand that.   I see two ways that could be resolved.  First,
 the files (as well as being used in the build) could be installed
 somewhere (not in an include directory, somewhere like /usr/share/misc)
 so they'd be available ... but that means only after the system has been
 built and installed, so is probably not the solution that will satisfy.
 
 Or second, there are two files generated (for each *devs file).  That is,
 for pcidevs we get pcidevs.h and pcidevs_data.h
 
 The first one I don't think is too much of a problem (and a little tweaking
 might be able to reduce the churn in that one even more.)
 
 The _data.h file is the one that really causes the problem, and frankly
 I can't see anyone wanting to grep that (or even keep it visible more than
 a few seconds so it can be included in whatever #includes it.)  If
 for some reason you do, you can always generate it - obviously a method
 to do that is going to be needed.
 
 So, for now the solution I am going to suggest, at least as a first
 quick fix, is that we change the procedure to only generate and check in
 the namedevs.h files when namedevs is altered, and we fix the Makefiles
 to generate namedevs_data.h on the fly during compilation (in the obj
 directories).
 
 Then we cvs delete the namedevs_data.h files, and arrange with the admins to
 make them all read only in the repo, so there can be no more commits to them,
 ever, from any release - any of the older releases that wants to change the
 device lists will need to adopt the new procedure first. Then after NetBSD-8
 has faded from memory (years from now) we simply nuke them.
 
 I mean rm namedevs_data.h,v so they are simply gone!  Anyone wanting to
 checkout an older source tree after that will need to regenerate the files
 (we could make a script for that to make life easier) before building, using
 the appropriate devlist2h.awk scripts that match the distribution in question.
 
 I think this is a much better solution than looking for a more stable
 compression algorithm for the _data.h files (and then having to debug
 the code to extract data from it) where what we have now seems to work
 well enough ... but doing this would also make it much easier for someone
 with the desire to create better algorithms, and experiment.
 
 Any objections to this strategy?   [The nuking the files from the repo
 part is kind of an optional extra, just to reduce the repo size, it is
 not crucial to the rest of the plan.]
 
 If not, I'll start the process making all of this happen.   I'll also look
 at why we seem to need 15 devlist2h.awk files in the tree.  That seems
 a little excessive, I find it hard to believe that they're all so
 different from each other that every single one of them is needed.
 
 kre