trip report from a first-time bob user: zero to a built package in 2h

[Greg Troxel <gdt%lexort.com@localhost>]

and not just a built package, but a huge email about it!

For a long time, I've been thinking I should set up pbulk, but I never
actually tried, as it always seemed like it would be more effort than I
wanted to spend -- which may or may not have been accurate.  I decided
to set up bob, given how easy it is said to be, and this is notes from
along the way, my one chance to experience it as a first-time user.  It
contains things that have been mentioned before, and I'm trying to be
brief.  I hope it's useful for improvements, and to others who may be
contemplating setting up bob.

This note is 217 lines, but it's a running commentary as I went.  It
might help you set up faster, and it might be better not to read it --
your call!

My chosen machine is NetBSD 11 amd64, actually a xen dom0, with packages
installed to support dom0 operations.  It has space in both UFS2 and
ZFS.

I installed from pkgtools/bob, uneventfully.

* usage

I see that bob has "usage" output, but no man page.  The usage talks
about "init" for first-time setup, and gives the config file path.

I noticed that the config file already existed, which seemed off from
the usage output.  I diffed it to /usr/pkg/share/exampls/bob, and it
matched (no surprise).  I ran bob init, and got an error that the file
exists.  So I moved it aside to a backup name, and re-ran bob init, and
the new filea matched the pre-installed file.  All of this is reasonable
modulo the usage text not matching the package.

* config file meta

I read the config file.  I knew lua in 2016 and mostly have forgotten,
but easy enough to cope with given knowing a bunch of other languages.

Some of the config file contents are active, and some are commented out.
I can't figure out why.  I can't figure out whether running bob with an
empty config file assumes the contents that are in the config file, or
runs with an empty config and thus mostly errors/does-nothing.   My
preference is always for default behaviors (but not paths) to be baked
in, and for configs to not set them unles they should be non-default,
leading to small config files.  I realize that's at odds with how this
probably needs to be, but it's perhaps part of why I felt unclear.

* option

I got the wrong idea from build_threads and scan_threads from the config
file; the docs say it's about processes not threads, and apparently one
thread per process.  There's no advice how how these values should
relate to hw.cpu for people that want max bob performance, or that want
to have a workstation not negatively impacted.  Suggestion: calls these
build_parallelism and scan_parallelism instead, if the point is to limit
the number of cpu-using disk-using threads, with the job/thread mix
being an unimportant detail.

* dynamic

The dynamic section is in the example, but it's not next in the docs.  I
suggest to have the example config and the docs in the same order.

I didn't understand dynamic from the example, beyond the general goal.
From the docs, I think that MAKE_JOBS and WRKOBJDIR are used from
mk.conf if dynamic is not defined.  If it is, then bob sets values based
on history/explanation.  I have ad hoc code to set WRKOBJDIR to a
big/slow place for some packages; this seems like it won't play well
with that and that's ok.

It's good that jobs is defined as total budget shared against all
simultaneous builds.  Perhaps add "# thread limit for union of build
jobs" to that line.

The wrkobjdir sub-table is confusing.  Must there be exactly 1, and if
so, why is it sub?   What does 0 or 3 mean?

threshold appears to be an amount for each package build, so one could
have build_threads * threshold usage.  I expected the same thing as
threads, a max resource usage limit across all simultaneous jobs.  I
wonder why it doesn't default to 50% of the free space in the given
filesystem, except that there would be complexities.

* pkgsrc

If best practice is to have an unprivileged user, then the package
should default to creating one.  "bob" is obviously not good, and we
don't have good practices for choosing, but I'd suggest either pkgbob or
_bob, as NetBSD base system users start with _ to get them out of normal
user namespace, and that's probably ok elsewhere.

I set pkgpaths to read from a file and put devel/m4 in it, my
examplar/benchmark package.

Might be later, but distfiles and binary packages aren't set in pkgsrc,
and I wonder if they are read from mk.conf, or need to be set someplace
else.

* sandboxes

This seemed like I'll need to pay attention but not mysterious.

* things I didn't understand at this point

** pkgsrc read only

Arguably pkgsrc should be mounted ro by default.  Comments say about
DISTDIR and PACKAGES but they don't address if just setting those in
mk.conf is enough, and how that interacts with sandbox building.  I'm
guessing I need to add mount lines for them.

** distfiles

I have distfiles and binary packages pointed elsewhere (one pkgsrc tree,
NFS on multiple builders).  I don't see what happens; maybe there's auto
null mounts.  Maybe not.

** mk.conf

I have /etc/mk.conf, a symlink into a file managed in CVS in my homedir.
That does an include of /etc/mk.conf.local.    I don't feel confident
that's going to be ok, and didn't find discussion about how system
mk.conf* is carried into sandboxes but reading the setup seems like
  - /etc is mapped, so mk.conf.local is ok
  - symlink to home is a problem

** /data/chroot

It seems obviously reasonable for me to "zfs create /tank0/bob" and use
that as sandbox location, and that I can maybe make it nodev but not
noexec.  Somewhere the rules should be explained.

** interaction with my linking scheme

To deal with pkgsrc on many machines, my shared conf sets
  PACKAGES?=               /links/packages
  DISTDIR?=                /links/distfiles
and on each machine /links is a dir with symlinks, in this case:
  lrwxr-xr-x  1 root  wheel  18 Jul  3  2023 distfiles -> ../tank0/distfiles
  lrwxr-xr-x  1 root  wheel  40 May 15 14:44 packages -> /tank0/packages/netbsd-11-x86_64-current

so I guess I have to add /links and those dirs to the mount list in
sandbox.  (On macOS it's /opt/links because you have to think different
gratuitously, but I'm not trying to macBob yet.)  I did, which was easy.

* random suggestions

One might want to write mk.conf code that's different under bob than not
(e.g. my WRKOBJDIR stuff), so there should be something defined you can
switch on.

* on to actually trying to set up bob

Adjusted config file based on reading above.

Ran "bob build".

I got errors because my zfs dataset was noexec, my fault, fixed.

I was confused by sandbox cleanup, did it manually once with
egrep/awk/while/read/unmount.  Then found 'bob sandbox'.  "bob sandbox
list" showed /tank0/bob/0 but "bob sandbox destroy" wants "0" not a
path.

Iterated a few times with my /links stuff and getting everything
mounted.

Ran again and succeeded (so my /etc/mk.conf symlink worries were unfounded):

  #> bob build
  Creating sandbox... done (0.4s)
  Scanning packages...
  Scanned 6 package paths in 1s (6 succeeded, 0 failed)
  Resolving dependencies... done (0.0s)
  Calculating package build status... done (0.1s)
  Loaded 5 cached build results
  Building packages...
         Built m4-1.4.21 (1m 11s)
  Built 6 in 1m 15s (1 succeeded, 5 cached, 0 failed, 0 skipped)
  Destroying sandbox... done (0.2s)
  Generating pkg_summary... done (0.0s)

and I see a new package and summary:

  -rw-r--r--  1 gdt   wheel     251513 Jun 16 14:46 bup-0.33.10.tgz
  -rw-r--r--  1 gdt   users      44167 Jun 21 19:50 pkg_summary.bz2
  -rw-r--r--  1 gdt   wheel    6882538 Jun 23 12:57 bob-1.0.0.tgz
  -rw-r--r--  1 gdt   wheel     278250 Jun 23 15:39 m4-1.4.21.tgz
  -rw-r--r--  1 root  users       1979 Jun 23 15:40 pkg_summary.gz

and the summary, unsurprisingly, only has m4 and the 5 infrastructure
deps.  But they weren't built, apparently reused.

Ran bob build, said nothing to do.

Ran "bob rebuild devel/m4", built again.   Seems not be using ccache,
as my mk.conf sets up.  Or maybe it's in the sandbox and destroyed.

Tried to ^Z bob.  Nothing happened.  Feels like a bug to me; I want to
be able to stop high-load processes when I want.  Maybe I have something
odd.

Changed my ccache setup to set CCACHE_DIR to /links/ccache and symlink
to /tank0/ccache/netbsd-11-amd64 and add that to the sandbox.

* mostly success

I haven't built everything I want, and I haven't decided whether/if to
flip my world to bob, but at this point things are working, and it's
under 2 hours for reading docs, setting things up, and writing this.

I realize that if I didn't have my own symlink/alt setup it would have
been easier.  But it wasn't hard.

I am still not set up for ccache, but not done debugging.

Greg