Re: Distributed bulk building for slow machines

[Aleksey Cheusov <cheusov%tut.by@localhost>]

 >> 3) A method to use a fast machine to create a
 >> dependency tree which is used after the priority
 >> package list is finished

> Yep, this is really important, even if the dependency graph
> calculation is parallelized (ref. later entries in this thread).
> The time it takes to compute the dependency graph is significant,
I don't know about pbulk and old bulk build framework, but in distbb
dependency graph is built in two steps:
1) All packages are queried for their summary (PKGNAME, PKGPATH, DEPENDS
   and so on). These per-package summaries
   are collected in parallel on slave hosts,
   in our case running NetBSD on slow architectures on real iron or
   inside hardware emulators. These summaries are sent to a master host.
2) Building the dependency graph. It is built on master host that can
   run NetBSD on fast amd64/x86 machine. This step is very fast even on not
   so modern CPU. It takes 20 seconds(!) real time and 12 seconds user
   time on 3Ghz Pentium 4 CPU (GenuineIntel).

 >> Essentially, a sandbox would be created on worker
 >> machines where sshd is run inside of the sandbox with

> This one I'm not so sure of, as this would prevent users sitting
> behind NAT boxes from participating as clients or slaves.

For potential partificipants who are behind a firewall,
NAT can be organized over any open port, even 80 or 8080.
Anyway, I'd start with ssh as a default transport.
BTW transport can easily be changed at any time.
In case of distbb (and IIRC pbulk) this is just one variable in config
file.

> One thing to keep in mind, though, is this: all the machines
> participating in such a distributed effort need to run the same
> base OS release, and also run only the official release for the
> duration of its contribution.
Ideally only one person should be a sysadmin of all slave hosts per
architecture.  He should be a responsible person for the "equality" of
all slave hosts.

> Thought also needs to be given to how the (presumed local) pkgsrc
> repository is to be kept in sync,
It makes sense to sync pkgsrc tree between master and slaves hosts
automatically before starting a bulk build. This will avoid a lot of
potential problems.

> Then there's of course also the issue of ... um... security
> till now packages on ftp.netbsd.org have been built by
> developers, and a change to allow anonymous contributions (not a
As for me, this is the most serious (the only one?) problem in this
task.  Limiting contributors to a list of trusted NetBSD developers or
those who signed "papers" may significantly reduce amount of available
(potentially) resources. I have no idea how to solve this problem easily.

> Also... note that trying to build some packages are prone to
> cause panics on at least m68k hosts, the message seen on my hp300
> systems are (from memory) "out of address space", and I beleive
> this has to do with the pmap implementation.
Do you mean NetBSD itself crashes or gcc/clisp/whatever?

> So when doing distributed bulk builds for m68k, some packages need to
> be marked "no way"
In distbb there is a way this, all such packages can easily be marked
as "no way" in config file.

-- 
Best regards, Aleksey Cheusov.