Subject: Re: Fastest m68k box for bulk package builds
To: John Klos <firstname.lastname@example.org>
From: David Brownlee <email@example.com>
Date: 09/26/2001 12:39:22
On Mon, 24 Sep 2001, John Klos wrote:
> > > Perhaps a status page would be in order so that people can get an small
> > > occasional email telling them that it's been updated, so they can go
> > > look at the status page and see if what they're looking for has been
> > > built.
> > >
> > Sounds reasonable - it should probably be coordinated across
> > architectures so you could go to the same place for m68k, mips,
> > etc.
> > Maybe just something that generates a web page per category with
> > a table of package-name vs arch+osrev, with an indicator of built,
> > failed, not-for-this-arch, etc?
> Yes, definitely.
> An issue, though: if one person / machine is doing a bulk build, then this
> wouldn't be difficult, as the bulk-build scripts generate an index. But
> what about multiple people / machines? And merges / updates?
The table should probably be built by a cronjob on ftp.netbsd.org,
based on what packages have been uploaded.
> Would this be most appropriate for the bulk scripts (that is, adding
> something to output ewasily mergeable information), or for some code on
> the server?
It might make sense to add it to pkgsrc, then have the cron job on
ftp.netbsd.org run it - if you keep it with the bulk-packages then
people running independent builds can make more use of it.
> > That sounds very interesting, though pkgsrc version skews might
> > start to become an issue.
> I've been keeping the pkgsrc tree, distfiles, and finished binary tree on
> one disk and sharing via NFS. Compile temp directories are local.
> Is there a better way to do this? One thing that would be nice in the bulk
> package scripts would be if, when entering a directory, if a test can be
> done to see if another task is (trying to) build(ing) that package (and
> hasn't failed or succeeded yet), and if so, skip it.
> A few times when I was sharing via NFS, more than one computer would end
> up in the same package directory (usually because of a dependency), and
> things would get screwed up.
> If "package locking" existed, then I think building with multiple machines
> would be much closer to automatic.
I think package locking is useful in more than just bulk-build
situations, both in building, and in installation/removal.
It probably makes sense to keep a lockfile in the work directory,
and maybe try to adjust the clean logic so that everything except
the work directory itself and the lockfile would be deleted.
> > > I can use the already set up nodes of the Quadra cluster to start a bulk
> > > build on 1.4.3, if this would help.
> > >
> > More binary packages for releases are always a good thing :)
> I didn't realise that lots of people still run 1.4. But then again, I
> suppose lots of non-NetBSD people don't realise that we run VAX and m68k
:) - 1.5 is definitely more important, but there are still quite
a few people whe haven't moved up (I still have one machine on 1.3
for various reasons :)
David/absolute -- www.netbsd.org: No hype required --