Re: pbulk parallel builds

["Johnny C. Lam" <jlam%NetBSD.org@localhost>]

On Mon, Jun 05, 2017 at 09:36:02AM -0500, Jason Bacon wrote:
> On 06/04/17 02:20, Jonathan Perkin wrote:
> > * On 2017-06-04 at 01:20 BST, Jason Bacon wrote:
> > 
> > > Is it possible for pbulk to build multiple packages simultaneously
> > > on the same machine?
> > Yes!  I've been doing this for many years but it requires a few
> > patches that aren't yet in pkgsrc.  I proposed a change that will
> > hopefully be accepted upstream for people to test a while back:
> > 
> >    https://mail-index.netbsd.org/pkgsrc-users/2017/02/14/msg024455.html
> > 
> > I still use my previous patches for all my bulk builds, but would
> > appreciate testing of the above so we can make this easier for
> > everyone.
> > 
> Thanks, Jonathan!  Unfortunately, I've had to move away from mksandbox.  The
> problem is that on RHEL/CentOS, Yum installs things into /usr/bin, /usr/lib,
> etc., which has caused occasional leakage issues (pkgsrc picking up a tool
> or library installed by Yum instead of its own).  If I'm going to utilize
> our cluster nodes, which have a lot of Yum packages installed to support
> commercial software, I have to use a clean chroot image from a fresh install
> rather than mksandbox, which imports much of the chroot env from the host.

You can try using sysutils/sandboxctl to create the chroot sandboxes in
a consistent way.

> Also, I don't see a need to distribute my builds across multiple systems,
> given that our compute nodes have 24 cores, a lot of RAM, and fast RAIDs.
> 
> I'm hoping I can build multiple packages simultaneously in a single chroot,
> but if that's impossible for reasons I'm not yet aware of, I'll figure out
> an approach similar to what you've done with mksandbox, using multiple
> chroots.

As Jonathan pointed out, you don't need to distribute across multiple
systems.  His changes allow you to create multiple chroots on the same
system that independently use pbulk to build the packages in a
coordinated way.

Regards,
--
Johnny C. Lam
jlam%NetBSD.org@localhost