Re: trip report from a first-time bob user: zero to a built package in 2h

[Jonathan Perkin <jperkin%pkgsrc.org@localhost>]

Thanks Greg, I'll respond to the issues you've raised on GitHub in due 
course, and will just provide a little commentary to some of the general 
points here so that they're useful for others.

* On 2026-06-23 at 21:02 BST, Greg Troxel wrote:

> I noticed that the config file already existed, which seemed off from
> the usage output.  I diffed it to /usr/pkg/share/exampls/bob, and it
> matched (no surprise).  I ran bob init, and got an error that the file
> exists.  So I moved it aside to a backup name, and re-ran bob init, and
> the new filea matched the pre-installed file.  All of this is reasonable
> modulo the usage text not matching the package.

Yep, 'bob init' is a one-time command to generate an example 
configuration file for the target host.  As pkgsrc norms are that a 
program is installed with an example, I generate this in the package, so 
there's no need to run 'bob init' when installing via pkgtools/bob.

> Some of the config file contents are active, and some are commented out.
> I can't figure out why.  I can't figure out whether running bob with an
> empty config file assumes the contents that are in the config file, or
> runs with an empty config and thus mostly errors/does-nothing.

There are two extremes with default config files:

 - Minimal that requires reading lots of docs and adding necessary
   sections before you can run the command for the first time.  This can
   often put a lot of people off before they even start due to the
   cognitive overhead.

 - Overly verbose with many sections that may need to be trimmed or
   edited before you can run the command for the first time.

My goal with bob is a happy medium where should be able to run it out of 
the box with zero editing.  This is the case on NetBSD (on others you'd 
need to provide a bootstrap kit, I can't help that) - assuming a default 
NetBSD install, a 'bob init' followed by 'bob build' will immediately 
start building packages inside sandboxes with no other setup required.

The commented out sections are features that aren't required, but that I 
believe the vast majority of users will want, and so they are there to 
provide a quick starting point should people want them.  Not every 
feature is in the default configuration file.

> My preference is always for default behaviors (but not paths) to be baked
> in, and for configs to not set them unles they should be non-default,
> leading to small config files.  I realize that's at odds with how this
> probably needs to be, but it's perhaps part of why I felt unclear.

Very early versions of bob did actually try to go this way, where I 
wanted to see just how minimal a file I could get away with.  

The problem is that bob does a lot of dangerous things like managing 
mount points, running "rm -rf", killing processes, installing and 
deinstalling packages, etc, and I came to the conclusion that having 
anything hidden away was just going to be confusing at best and actively 
harmful at worst.  So now, other than some trivial things like log 
level which have defaults, everything must be defined in the config file.

> The wrkobjdir sub-table is confusing.  Must there be exactly 1, and if
> so, why is it sub?   What does 0 or 3 mean?

I'm not sure what you mean here.  wrkobjdir has its own section because 
there are multiple values to configure. jobs is just a single integer so 
doesn't.

> threshold appears to be an amount for each package build, so one could
> have build_threads * threshold usage.  I expected the same thing as
> threads, a max resource usage limit across all simultaneous jobs.

Remember that each build is performed inside its own sandbox, which will 
have its own /tmp etc, so there's no notion of allocation being required 
to support build_threads.

> I wonder why it doesn't default to 50% of the free space in the given
> filesystem, except that there would be complexities.

Yeh again trying to set defaults here would likely lead to more 
confusion than just setting the values manually.  Unfortunately these 
values can need tweaking because some packages misbehave, hence the 
always_disk setting and comment, for example.

As a real-world example, on my macOS daily bulk build host where I want 
as much on tmpfs as possible, I've had to tweak it so that on a 3GB ram 
disk, threshold is 1.3GB, because there are still some packages that 
will blow past 3GB during their build but only record a little over 
1.3GB by the time they have finished.

This ratio will change depending on what size tmpfs is set to and thus 
what types of packages are able to fit inside the target tmpfs.

> Might be later, but distfiles and binary packages aren't set in pkgsrc,
> and I wonder if they are read from mk.conf, or need to be set someplace
> else.

Yes, the design is that everything is read directly via (b)make.  Again, 
earlier versions of bob had settings for things like this, but the issue 
then is that you have the same variables defined in multiple places 
which can cause problems (which one has precedence for example) and also 
cause confusion.  The rare exceptions to this are things like MAKE_JOBS 
which if dynamic is set will be passed as environment variables to 
(b)make.

Set everything pkgsrc-related in mk.conf.  That way all builds are 
correct whether performed by bob or built manually.

> Arguably pkgsrc should be mounted ro by default.  Comments say about
> DISTDIR and PACKAGES but they don't address if just setting those in
> mk.conf is enough, and how that interacts with sandbox building.  I'm
> guessing I need to add mount lines for them.

Yes.  I think this is where adding some more example configuration files 
will help.

> I have distfiles and binary packages pointed elsewhere (one pkgsrc tree,
> NFS on multiple builders).  I don't see what happens; maybe there's auto
> null mounts.  Maybe not.

Nothing about sandbox actions is automatic.  Going back to the "mounting, 
killing, and rm -rfing", that's an absolute no-no.  If DISTDIR and 
PACKAGES are elsewhere, then add a new action mount line for them.

> I have /etc/mk.conf, a symlink into a file managed in CVS in my homedir.
> That does an include of /etc/mk.conf.local.    I don't feel confident
> that's going to be ok, and didn't find discussion about how system
> mk.conf* is carried into sandboxes but reading the setup seems like
>  - /etc is mapped, so mk.conf.local is ok
>  - symlink to home is a problem

Yes this likely won't work by default, the "copy" action will preserve 
symlinks not follow them, so you would need something like this added 
after the "/etc" copy:

  { action = "copy",
    src = "/home/gdt/cvs/mk.conf.local",
    dest = "/etc/mk.conf.local" },

I think this being missing might explain your ccache issue.

> One might want to write mk.conf code that's different under bob than not
> (e.g. my WRKOBJDIR stuff), so there should be something defined you can
> switch on.

You could use one of the existing environment variables for this, e.g. 
bob_make, though I think adding this feels like a code smell.  As long 
as you set e.g. WRKOBJDIR?= ... then it'll work fine outside bob, and 
bob's passing of WRKOBJDIR=/some/where/dynamic will take precedence, the 
same way it would if you did a manual (b)make WRKOBJDIR=/whatever.

> Tried to ^Z bob.  Nothing happened.  Feels like a bug to me; I want to
> be able to stop high-load processes when I want.  Maybe I have something
> odd.

No, this isn't supported yet, and would be a reasonable project I think 
given all of the threads and chroots and sandbox processes that bob has 
spawned.  It would be nice to have, but will require some thought and 
careful planning.

At the moment if you press ^C then bob will continue to run, but will 
not issue any new builds, so you will see each worker come to a stop 
once they are finished and bob then exits when all done.  A subsequent 
^C will forcibly kill everything and shutdown much more quickly.

It should always be safe to completely kill bob.  There are shutdown 
handlers that attempt to clean up all of the sandboxes, and you can 
simply run 'bob build' again to pick up from where it left off 
(obviously any in-transit package build progress is lost and they start 
from scratch).

--
Jonathan Perkin                    pkgsrc.smartos.org
Open Source Complete Cloud   www.tritondatacenter.com