Re: pbulk parallel builds

[Jonathan Perkin <jperkin%joyent.com@localhost>]

* On 2017-06-05 at 15:24 BST, Jason Bacon wrote:

> On 06/03/17 22:28, Johnny C. Lam wrote:
> > On Sat, Jun 03, 2017 at 07:20:45PM -0500, Jason Bacon wrote:
> > > Is it possible for pbulk to build multiple packages simultaneously on the
> > > same machine?
> > > 
> > > Right now I'm using MAKE_JOBS to utilize all available cores for individual
> > > builds where possible, but that still leaves a lot of resources idle where
> > > MAKE_JOBS cannot be used.
> > > 
> > > I've thought about using a trick such as
> > > 
> > > build_clients="localhost localhost localhost ..."
> > > 
> > > along with MAKE_JOBS=1
> > > 
> > > but as I'm building in a chroot env, I think I'd have to forward ssh
> > > connections into the chroot somehow.
> > > 
> > > Seems like there should be a more elegant way to utilize all the cores on a
> > > modern server.
> > This is a few years old, but the information is still relevant:
> > 
> > https://www.perkin.org.uk/posts/distributed-chrooted-pkgsrc-bulk-builds.html
> > 
> I had looked at this, but it seems to be focused on distributed builds
> (multiple machines or chroots).
> 
> I'm wondering if there's a simpler way to build multiple packages
> simultaneously within a single chroot, vm, or dedicated machine.
> 
> It seems to me that the hardware advancements are outpacing the ability to
> parallelize bulk builds.  Given the interdependencies among packages, there
> would be a point of diminishing returns that modern hardware may be
> approaching or have already reached.
> 
> My plan for CentOS builds is to use a pristine chroot on a single compute
> node in our newest cluster.  Each node has 24 cores, 64 GiB RAM, and access
> to NFS servers over Infiniband that I've benchmarked at 850 MB/sec sustained
> write speed.  I suspect that if a bulk build can utilize all this, I'm not
> going to gain much by distributing it to more nodes.
> 
> Is my gut feeling wrong?  Has anyone verified that a bulk build can scale
> well beyond 24 cores?

Yes, at peak I was able to perform a full bulk build in under 3 hours,
running across 6 zones each able to utilise up to 24 cores.

These days however, even on powerful hardware lang/rust can take up to
8 hours _just by itself_, so there is little point optimising the rest
of the build more than that.

Nowadays I try to optimise for concurrent builds rather than flat out
speed for single builds, so e.g. we now deliver daily 32-bit and
64-bit builds rather than just a fast single daily build, with the two
builds often overlapping on the same hardware.

-- 
Jonathan Perkin  -  Joyent, Inc.  -  www.joyent.com