Re: Speeding up the bulk build scan phase

[Thomas Klausner <wiz%netbsd.org@localhost>]

Hi!

I've just documented all of these in the bulk build section of the
pkgsrc guide ("8.3. Speeding up bulk build scans").

Btw, if you are using varcache on NetBSD with native X, please make
sure to use the latest pkgtools/varcache, there was an important
bugfix a couple days ago.

Cheers,
 Thomas

On Mon, Jan 12, 2026 at 09:45:26AM +0100, Thomas Klausner wrote:
> Hi!
> 
> If you're running your own bulk builds, I'll describe a method to
> majorly speed up the 'scan' part of the build.
> 
> This is heavily based on jperkin's work and help, thank you, Jonathan!
> 
> The basic idea is to avoid forks. Most forks during the scan phase
> compute the same values over and over again while scanning the whole
> pkgsrc tree, for stuff that does not change.
> 
> Since both pbulk and bob use the same pbulk-index-item target, this
> works for both of them.
> 
> 1. Create a cache file for the variables used by the built-in detection
> 
>    There's a Makefile target in a dummy package in pkgtools/varcache:
> 
>        cd /usr/pkgsrc/pkgtools/varcache
>        make show-varcache > /etc/varcache.mk
>        echo .include "varcache.mk" >> /etc/mk.conf
> 
>    This also sets a couple more variables that rarely change:
>    HOST_OSTYPE, machine endian-ness, pkgtools version, gcc version,
>    location of the pkgsrc tree.
> 
>    (You will have to regenerate this when a built-in package is
>    updated or one of the other mentioned characteristics changes. It's
>    fast, so you could run it before every bulk build.)
> 
> 2. Set some more variables
> 
>        NATIVE_PKG_TOOLS_BIN=/usr/sbin # NetBSD location
>        NO_PKGTOOLS_REQD_CHECK=yes     # assume pkg tools are up-to-date
>        SKIP_LICENSE_CHECK=yes 	      # do not check license validity
>        _SRC_TOP_= 	              # we're not running from inside /usr/src
>        UNPRIVILEGED_GROUP=builder     # use the appropriate group for your system
>        UNPRIVILEGED_GROUPS=builder    # use the appropriate groups for your system
>        UNPRIVILEGED_USER=pbulk 	      # use the appropriate user for your system
> 
>    Note: You might not want to set SKIP_LICENSE_CHECK, since this
>    disables checking the license(s) against your ACCEPTABLE_LICENSES
>    settings.
> 
>    If you're on NetBSD-current, you can also set
> 
>        _TOOLS_USE_PKGSRC.flex=no
> 
>    since -current has flex 2.6.4, which is new enough for everything
>    in pkgsrc as of 20260111.
> 
> 3. Set some variables in the environment
> 
>    These variables are computed before /etc/mk.conf is read, so they
>    need be passed in from the environment - so perhaps add them to
>    your shell startup files.
> 
>          export HOST_MACHINE_ARCH=`uname -m`
>          export NATIVE_OPSYS=`uname -s`
>          export NATIVE_OPSYS_VERSION=`uname -r | awk -F. '{major=int($1); minor=int($2); if (minor>=100) minor=99; patch=int($3); if (patch>=100) patch=99; printf "%02d%02d%02d", major, minor, patch}'`
>          export NATIVE_OS_VERSION=`uname -r`
> 
> With these change (and my commits from yesterday) I got the scan phase
> for a full NetBSD-current/pkgsrc-current bulk build using 'bob scan'
> down to less than 20 minutes. I have no reliably "before" number, but
> it was definitely over an hour.
> 
> Enjoy!
>  Thomas