Subject: Re: FreeBSD's package cluster
To: Tim Rightnour <root@garbled.net>
From: John Klos <john@ziaspace.com>
List: tech-pkg
Date: 05/27/2005 10:09:10
> I would like to comment on this really quickly. While I would be happy
> to see glunix get fixed in the pkg collection, the honest truth is its
> mostly useful as a reference tool, and a whole parallel system. It is
> *not* good for parallel compiling. Originally thats what I wanted to do
> with it when I ported it. After discovering that it wasn't what I
> needed at all, I ended up writing ClusterIt. ClusterIt was designed
> specifically for parallel compiling, and I think you will have more luck
> going that route. Specifically what you probably want is jsd/jsh, and
> barrier/barrierd.
I'd like to point out something that hasn't been discussed much in this
thread. FreeBSD supports systems which are generally fast. It makes sense
to cluster a few sparc64s or ia64s or whathaveyou. However, NetBSD
supports many archs, and clustering a gaggle of VAX is not exactly
practical (although it'd be unmatchable for bragging rights).
What'd be neat to see is a set of tools that could have one (fast) machine
preprocess, as much as possible, some or all of the packages, then have
the slower, real processors do the compiling. ClusterIt seems like it'd be
an ideal way to use one machine to coordinate all of them. It'd also be
neat to have a way to have multiple machines of the same arch work on
different, non-interdependent packages.
I started work on something like this a couple of years ago, but never got
too far. I wanted to have a database keep track of all of the packages in
the tree, all that had already been built for each arch / OS version, and
have it, when the tree changes, track the changes and tell each slave
machine to build whatever's new. It was going to use ssh in a chroot so
that machines which are not local to one another could still be used.
Has anyone worked on anything like this? Would there be interest?
John Klos