On Mon, Jan 29, 2001 at 09:19:10AM -0500, Todd Vierling wrote:
> On Mon, 29 Jan 2001, Alistair Crooks wrote:
> 
> : > : I don't do it, but notice that the ".if make(...)"'s prevent the
> : > : dependency from being registered at install or package time, so it's
> : > : not so bad.
> : >
> : > We have to avoid using ".if make()" in bsd.pkg.mk.  There are outstanding
> : > complaints as to the ultra-recursion nature of bsd.pkg.mk.  We need to
> : > eliminate spots where the "target being made" is referenced, so that
> : > changing some of those recursions into dependencies will be possible.
> 
> : It's well-known that there are a number of ways to speed up pkgsrc, the most
> : popular of which seem to be (in order)
> :
> : 1. avoid using != command expansion where possible, passing any results
> : down through the environment
> : 2. cut down on the number of `cmd` invocations
> : 3. split bsd.pkg.mk itself into sub-files, and include only the sub-parts
> : of it which need to be included
> : 4. lots more radical micro-optimisations
> :
> : and I'd include your optimisation in (4), in lieu of any direct benchmarked
> : numbers.
> 
> No, eliminating recursion goes into (2) -- although its performance impact
> exceeds everything you have listed here.  You vastly underestimate the
> number of recursion points in bsd.pkg.mk, and the overhead involved in
> having another make(1) plus another /bin/sh invoked for every recursion.
> This optimization deserves its own category.

I have not vastly underestimated the number of recursion points, since
I have nowhere provided an estimate of their number. 

> Recursion in bsd.pkg.mk is the reason it takes *hours*, not minutes, to
> build README.html on ftp.netbsd.org; it also accounts for a nontrivial
> amount of time in a bulk build.

pkgsrc's aim is not to provide the means of producing README.html
files on ftp.netbsd.org.  It is to provide a simple vehicle whereby
people can download third party packages and have them "ready-to-go"
without too much work.  Please keeep that in mind when doing any
optimisation.

> About 2 years ago, I had a pet project that eliminated almost all the
> recursions and used proper make(1) dependencies (`newpkgsrc').  Rather than
> taking 10-20 seconds *on a nonloaded system* to go through a package with
> entirely null extract, patch, configure, and build targets, the modified
> bsd.pkg.mk took *2* seconds.  I'd consider that quite significant.
> 
> -- 
> -- Todd Vierling <tv@wasabisystems.com>  *  Wasabi NetBSD:  Run with it.
> -- NetBSD 1.5 now available on CD-ROM  --  http://www.wasabisystems.com/
> 

You seem to be pretty sure about your arguments - we can't possibly
take a decision until we see some real figures.  So carry forward your
work from two years ago, implement what you talk about above, and
forward the patches to this list.  We can then see the significance of
what you're talking about.

And, yes, I have a definite interest in speeding up pkgsrc - I have
two 40 MHz ss2s, a NeXT and two 486s in my collection.

Alistair