Speeding up the bulk build scan phase

[Thomas Klausner <wiz%netbsd.org@localhost>]

Hi!

If you're running your own bulk builds, I'll describe a method to
majorly speed up the 'scan' part of the build.

This is heavily based on jperkin's work and help, thank you, Jonathan!

The basic idea is to avoid forks. Most forks during the scan phase
compute the same values over and over again while scanning the whole
pkgsrc tree, for stuff that does not change.

Since both pbulk and bob use the same pbulk-index-item target, this
works for both of them.

1. Create a cache file for the variables used by the built-in detection

   There's a Makefile target in a dummy package in pkgtools/varcache:

       cd /usr/pkgsrc/pkgtools/varcache
       make show-varcache > /etc/varcache.mk
       echo .include "varcache.mk" >> /etc/mk.conf

   This also sets a couple more variables that rarely change:
   HOST_OSTYPE, machine endian-ness, pkgtools version, gcc version,
   location of the pkgsrc tree.

   (You will have to regenerate this when a built-in package is
   updated or one of the other mentioned characteristics changes. It's
   fast, so you could run it before every bulk build.)

2. Set some more variables

       NATIVE_PKG_TOOLS_BIN=/usr/sbin # NetBSD location
       NO_PKGTOOLS_REQD_CHECK=yes     # assume pkg tools are up-to-date
       SKIP_LICENSE_CHECK=yes 	      # do not check license validity
       _SRC_TOP_= 	              # we're not running from inside /usr/src
       UNPRIVILEGED_GROUP=builder     # use the appropriate group for your system
       UNPRIVILEGED_GROUPS=builder    # use the appropriate groups for your system
       UNPRIVILEGED_USER=pbulk 	      # use the appropriate user for your system

   Note: You might not want to set SKIP_LICENSE_CHECK, since this
   disables checking the license(s) against your ACCEPTABLE_LICENSES
   settings.

   If you're on NetBSD-current, you can also set

       _TOOLS_USE_PKGSRC.flex=no

   since -current has flex 2.6.4, which is new enough for everything
   in pkgsrc as of 20260111.

3. Set some variables in the environment

   These variables are computed before /etc/mk.conf is read, so they
   need be passed in from the environment - so perhaps add them to
   your shell startup files.

         export HOST_MACHINE_ARCH=`uname -m`
         export NATIVE_OPSYS=`uname -s`
         export NATIVE_OPSYS_VERSION=`uname -r | awk -F. '{major=int($1); minor=int($2); if (minor>=100) minor=99; patch=int($3); if (patch>=100) patch=99; printf "%02d%02d%02d", major, minor, patch}'`
         export NATIVE_OS_VERSION=`uname -r`

With these change (and my commits from yesterday) I got the scan phase
for a full NetBSD-current/pkgsrc-current bulk build using 'bob scan'
down to less than 20 minutes. I have no reliably "before" number, but
it was definitely over an hour.

Enjoy!
 Thomas