Re: misc/50216: USB, PCI ID data grow quadratically in the repository

[christos%zoulas.com@localhost (Christos Zoulas)]

The following reply was made to PR misc/50216; it has been noted by GNATS.

From: christos%zoulas.com@localhost (Christos Zoulas)
To: gnats-bugs%NetBSD.org@localhost, misc-bug-people%netbsd.org@localhost, 
	gnats-admin%netbsd.org@localhost, netbsd-bugs%netbsd.org@localhost, 
	gson%gson.org@localhost (Andreas Gustafsson)
Cc: 
Subject: Re: misc/50216: USB, PCI ID data grow quadratically in the repository
Date: Fri, 27 Oct 2017 08:10:59 -0400

 On Oct 27, 11:55am, kre%munnari.OZ.AU@localhost (Robert Elz) wrote:
 -- Subject: Re: misc/50216: USB, PCI ID data grow quadratically in the reposi
 
 |  and I understand that.   I see two ways that could be resolved.  First,
 |  the files (as well as being used in the build) could be installed
 |  somewhere (not in an include directory, somewhere like /usr/share/misc)
 |  so they'd be available ... but that means only after the system has been
 |  built and installed, so is probably not the solution that will satisfy.
 |  
 |  Or second, there are two files generated (for each *devs file).  That is,
 |  for pcidevs we get pcidevs.h and pcidevs_data.h
 |  
 |  The first one I don't think is too much of a problem (and a little tweaking
 |  might be able to reduce the churn in that one even more.)
 |  
 |  The _data.h file is the one that really causes the problem, and frankly
 |  I can't see anyone wanting to grep that (or even keep it visible more than
 |  a few seconds so it can be included in whatever #includes it.)  If
 |  for some reason you do, you can always generate it - obviously a method
 |  to do that is going to be needed.
 |  
 |  So, for now the solution I am going to suggest, at least as a first
 |  quick fix, is that we change the procedure to only generate and check in
 |  the namedevs.h files when namedevs is altered, and we fix the Makefiles
 |  to generate namedevs_data.h on the fly during compilation (in the obj
 |  directories).
 |  
 |  Then we cvs delete the namedevs_data.h files, and arrange with the admins to
 |  make them all read only in the repo, so there can be no more commits to them,
 |  ever, from any release - any of the older releases that wants to change the
 |  device lists will need to adopt the new procedure first. Then after NetBSD-8
 |  has faded from memory (years from now) we simply nuke them.
 |  
 |  I mean rm namedevs_data.h,v so they are simply gone!  Anyone wanting to
 |  checkout an older source tree after that will need to regenerate the files
 |  (we could make a script for that to make life easier) before building, using
 |  the appropriate devlist2h.awk scripts that match the distribution in question.
 |  
 |  I think this is a much better solution than looking for a more stable
 |  compression algorithm for the _data.h files (and then having to debug
 |  the code to extract data from it) where what we have now seems to work
 |  well enough ... but doing this would also make it much easier for someone
 |  with the desire to create better algorithms, and experiment.
 |  
 |  Any objections to this strategy?   [The nuking the files from the repo
 |  part is kind of an optional extra, just to reduce the repo size, it is
 |  not crucial to the rest of the plan.]
 |  
 |  If not, I'll start the process making all of this happen.   I'll also look
 |  at why we seem to need 15 devlist2h.awk files in the tree.  That seems
 |  a little excessive, I find it hard to believe that they're all so
 |  different from each other that every single one of them is needed.
 
 I am happy with the plan to move the foo_data.h file to be
 auto-generated during the build, and cvs delete it from head.
 Perhaps it should build both files (foo.h foo_data.h) int the obj
 directory and check that the committed foo.h matches the newly
 generated one.
 
 I would not do anything at the repo level until that stabilizes;
 any read-only decision etc, can be deferred.
 
 As for merging them further, by all means. I tried and merged some
 of them a while ago, the problem is that they are not all the same
 :-)
 
 christos