tech-pkg archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: A new approach to package building

On 6/8/24 09:03, Taylor R Campbell wrote:
Date: Mon, 3 Jun 2024 15:17:54 -0500
From: Jason Bacon <>

Last time I used pbulk (which was a while ago), there was a tradeoff
between running multiple package builds in parallel vs using multiple
cores for a single build.  Things may have changed since then, but a
quick look at the pbulk docs didn't reveal anything.

There was an option for using multiple servers that would help, but
processors would still be left idle much of the time, while builds sat
in the queue.

I think most users have settled on using pbulk with MAKE_JOBS=1 or 2
and many parallel workers, plus tweaking with PBULK_WEIGHT to get
large packages like firefox and libreoffice started as early as
possible.  In my anecdotal experience that leaves all CPUs pretty well
utilized for bulk builds from scratch.

Incremental bulk builds don't do as well -- e.g., if a leaf package
like firefox is updated on a branch, then you only get MAKE_JOBS
parallelism, not parallelism across package builds.  But then if
someone updates a major dependency like cairo or something, the CPU
utilization is good again.

This makes sense.  I was thinking more of package testing, which is
basically the same as incremental bulk builds.  If we need to build 5
packages, and one of them is LLVM, then we want to maximize MAKE_JOBS
for LLVM, but increase the number of workers when LLVM is not building.
This is where LPJS comes in.

Note that LPJS could either be part of an alternative to pbulk, or a
back-end to pbulk to improve hardware utilization.  Both options should
be explored.

That sounds reasonable to explore, with the caveat that getting
MAKE_JOBS wired up to LPJS _inside a package build_ might be a lot of
work; you'll have to teach bmake, and gmake, and meson, and ninja, and
whatever else, to all talk to the job server the same way that pbulk

Some complexity, yes, but doable, and only has to be solved once +

All that said: the pbulk scan and resolve logic gives you the graph I
believe you're looking for (and pbulk scan itself is parallelized
across workers, rather effectively in my anecdotal experience).

As I recall, pbulk scan takes a loooooong time.  It might not be
necessary to actually build the entire graph ahead of time, though.  A
more dynamic approach, just descending the tree for one package at a
time might suffice.  Packages already present from a previous build can
be skipped.

I'll have to add job dependencies to LPJS before this is feasible.
That's a ways off, after the core functionality is well-debugged, but I
wanted to start chewing on this.

Thanks to everyone for the input...


Life is a game.  Play hard.  Play fair.  Have fun.

Home | Main Index | Thread Index | Old Index