bob v0.9.0: dynamic scheduler, history, and lots more

[Jonathan Perkin <jperkin%pkgsrc.org@localhost>]

Hi!

Just in time for building 2026Q1, I've released the next version of bob.  
This one includes a big new feature: dynamic MAKE_JOBS and WRKOBJDIR.

As someone who has probably run the most number of bulk builds out of 
anyone, something I spend a lot of time on that I'd prefer not to is 
tuning MAKE_JOBS and WRKOBJDIR for each package.  Larger builds should 
get proportionally larger MAKE_JOBS, and builds that can do so should be 
built with WRKDIR on tmpfs.  However, this is an almost constant task to 
manage correctly, especially with builds outgrowing available tmpfs 
space and having to manually add them to e.g.:

  https://github.com/TritonDataCenter/pkgbuild/blob/master/include/wrkobjdir.mk

Bob now has a simple, dynamic scheduler, that will calculate everything 
for you automatically.  Here's an example that I'm using on my 16 CPU, 
64GB, NetBSD 10.1 host:

    dynamic = {
        jobs = 16,
        wrkobjdir = {
            tmpfs = "/var/tmp/work",
            disk = "/home/pbulk/work",
            threshold = "5G",
        },
    }

This works in tandem with a new feature: 'bob history'.  Every build is 
saved to a permanent history database, including CPU time usage for each 
pkgsrc stage, and WRKOBJDIR space requirement.

On first build, as bob doesn't know how big a package is or how many 
CPUs it should get, it will default to using disk-backed WRKOBJDIR, and 
will allocate approximately "dynamic.jobs / build_threads" for 
MAKE_JOBS.  Then for subsequent builds, MAKE_JOBS and WRKOBJDIR will be 
tuned automatically based on the history data.

As an example, I built meta-pkgs/bulk-large 3 times from scratch.  First 
with no history, second with just dynamic.jobs enabled, and then thirdly 
with dynamic.wrkobjdir enabled.  Here are the results:

  1. Initial build: static MAKE_JOBS=4 allocation
  Built 1662 in 17h 19m 19s (1662 succeeded, 0 cached, 0 failed, 0 skipped)

  2. Second build: dynamic.jobs
  Built 1662 in 14h 42m 58s (1662 succeeded, 0 cached, 0 failed, 0 skipped)

  3. Third build: dynamic.jobs and dynamic.wrkobjdir
  Built 1662 in 14h 22m 36s (1662 succeeded, 0 cached, 0 failed, 0 skipped)  

We can use the new 'bob history' command to see exactly how a particular 
package compared for each run:

  $ bob history -o timestamp,pkgname,outcome,make_jobs,wrkobjdir,duration ^llvm
  TIMESTAMP            PKGNAME         OUTCOME  MAKE_JOBS  WRKOBJDIR  DURATION
  2026-03-23 18:13:01  llvm-19.1.7nb1  success          6  tmpfs         1h37m
  2026-03-21 21:52:14  llvm-19.1.7nb1  success          6  -             1h46m
  2026-03-20 17:59:14  llvm-19.1.7nb1  success          4  -             2h30m

And we can use 'bob status' before a build to see what bob will assign 
for each build, in case you want to tweak any values beforehand:

  $ bob status -o pkgname,pkgpath,status,jobs,wrkobjdir gcc12
  PKGNAME               PKGPATH          STATUS   JOBS  WRKOBJDIR
  gcc12-12.5.0nb1       lang/gcc12       pending     6  disk
  gcc12-libs-12.5.0nb2  lang/gcc12-libs  pending     2  tmpfs

Hopefully this has given you a flavour of what's possible.  Of course 
there are many, many other changes, fixes, and improvements in v0.9.0:

  https://github.com/jperkin/bob/blob/main/CHANGES.md#version-090-2026-03-25

Thanks to wiz for testing many versions of main during this cycle and 
providing valuable feedback.

I would recommend re-creating any config.lua files from the updated 
examples, as there are a few fixes included in those too.

Happy building,

--
Jonathan Perkin                    pkgsrc.smartos.org
Open Source Complete Cloud   www.tritondatacenter.com