tech-cluster archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: FreeBSD's package cluster

I would like to comment on this really quickly. While I would be happy to see glunix get fixed in the pkg collection, the honest truth is its mostly useful as a reference tool, and a whole parallel system. It is *not* good for parallel compiling. Originally thats what I wanted to do with it when I ported it. After discovering that it wasn't what I needed at all, I ended up writing ClusterIt. ClusterIt was designed specifically for parallel compiling, and I think you will have more luck going that route. Specifically what you probably want is jsd/jsh, and barrier/barrierd.

I'd like to point out something that hasn't been discussed much in this thread. FreeBSD supports systems which are generally fast. It makes sense to cluster a few sparc64s or ia64s or whathaveyou. However, NetBSD supports many archs, and clustering a gaggle of VAX is not exactly practical (although it'd be unmatchable for bragging rights).

What'd be neat to see is a set of tools that could have one (fast) machine preprocess, as much as possible, some or all of the packages, then have the slower, real processors do the compiling. ClusterIt seems like it'd be an ideal way to use one machine to coordinate all of them. It'd also be neat to have a way to have multiple machines of the same arch work on different, non-interdependent packages.

I started work on something like this a couple of years ago, but never got too far. I wanted to have a database keep track of all of the packages in the tree, all that had already been built for each arch / OS version, and have it, when the tree changes, track the changes and tell each slave machine to build whatever's new. It was going to use ssh in a chroot so that machines which are not local to one another could still be used.

Has anyone worked on anything like this? Would there be interest?

John Klos

Home | Main Index | Thread Index | Old Index