Re: pbulk parallel builds

[Jonathan Perkin <jperkin%joyent.com@localhost>]

* On 2017-06-04 at 15:41 BST, Johnny C. Lam wrote:

> On Sun, Jun 04, 2017 at 08:20:20AM +0100, Jonathan Perkin wrote:
> > * On 2017-06-04 at 01:20 BST, Jason Bacon wrote:
> > 
> > > Is it possible for pbulk to build multiple packages simultaneously
> > > on the same machine?
> > 
> > Yes!  I've been doing this for many years but it requires a few
> > patches that aren't yet in pkgsrc.  I proposed a change that will
> > hopefully be accepted upstream for people to test a while back:
> > 
> >   https://mail-index.netbsd.org/pkgsrc-users/2017/02/14/msg024455.html
> > 
> > I still use my previous patches for all my bulk builds, but would
> > appreciate testing of the above so we can make this easier for
> > everyone.
> 
> I looked at your diff.  They appear to strictly add functionality to
> pbulk and not change existing behavior, so they appear fine.  I have
> a few observations:
> 
> (1) There is a change to pkgsrc/mk/pbulk/pbulk.sh that changes the
>     default ${PKG_DBDIR} passed to the pkg_install tools from
>     ${PREFIX}/var/db/pkg to ${PREFIX}/pkgdb. That change seems
>     unnecessary, although I understand the convenience of it.  It
>     doesn't matter to me, but I'm just pointing it out as not needed
>     for the rest of your changes to work.

It is actually required, as it fixes the issue where pbulk.sh and the
bootstrap code have diverged.

> (2) There is an idiom used in the shell code that looks like:
> 
> 	case $list in
> 	"$word" | "$word "* | *" $word "* | *" $word" )
> 		: "word is in list" ;;
> 	esac
> 
>     That can be shortened a bit if you want to:
> 
> 	case " $list " in
> 	*" $word "*)
> 		: "word is in list" ;;
> 	esac

Joerg can comment on why he chose the previous style, as I was merely
copying the existing style in the code, but yes the second version is
cleaner.

> (3) I don't know how standard the seq(1) utility is but that's bridge
>     that can be crossed if it becomes a problem.

It's only an example so can be modified if necessary.  Users with
shells which support {N..M} can use that construct instead.

> You should commit your changes, especially as you've already been
> using them daily for the past several months.

Joerg had issues with it and I'd prefer we sorted out a design we're
all happy with first.  Also as I mentioned I don't actually use this
patch myself, and am still using my previous set of patches:

  https://github.com/joyent/pkgsrc/commits/joyent/feature/pbulk/trunk?author=jperkin

which is why I'm keen for others to test this new patch as much as
possible.  We need to be careful as automatic chroot creation and
deletion has lots of sharp edges and can easily result in loss of data
if things aren't configured correctly.

-- 
Jonathan Perkin  -  Joyent, Inc.  -  www.joyent.com