Subject: Re: pkgsrc on SMP machines
To: Manuel Bouyer <bouyer@antioche.eu.org>
From: Lars Nordlund <lars.nordlund@hem.utfors.se>
List: tech-pkg
Date: 12/22/2005 01:16:10
On Tue, 20 Dec 2005 19:22:30 +0100
Manuel Bouyer <bouyer@antioche.eu.org> wrote:
> > The sandbox must be cheap enough to create so the host system can
> > create 5000 of them rather fast.
> 
> Each require several null mounts, I'm not sure a system would support
> that much. You certainly don't want one sandbox per package, but
> one per jobs (if you can run 4 jobs in parallel, you want 4 sandboxes)

What kind of breakage do you foresee if builds are happening in more
than one place in the pkgsrc tree at the same time? Is this what you are
afraid of and therefore want one sandbox per job?

The following is probably already clear to you, but I'll say it anyway;

With the 'make parallel'-patch, and how make works and so on, only
"leaves" will be built at any given moment. When a package is built, it
will never have to discover that some other package is missing and go
and build it.

Yes I know about the pkgsrc locks and all that, and I have also seen
that it does not always work. But 'make parallel' shall never need to
rely on the locking. All builds will be in different packages all the
time.

> I'm not sure your system can be used for bulk builds. The scripts in
> pkgsrc/mk/bulk do much more than that (like making sure only the required
> packages are installed)

The make rule could be adjusted to this:

(example for package bar in category foo, which has no un-built
dependencies)

foo/bar: <some already built and packaged dependencies>
	<somehow pick a free sandbox and use it>
	<install required packages from package repository>
	make package clean
	<copy package to repository>
	<deinstall all packages in this sandbox>

I am at this point not sure how to 'pick a free sandbox' could be
implemented. Perhaps a wrapper script which does the rest of the above,
and chroot it? Hmm. It is late, I have probably missed something
obvious.. :-)

Never having used the bulk and tested mk/bulk/mksandbox.. Is it
possible to have two or more of these active on the same machine at the
same time?


Best regards,
	Lars Nordlund