pkgsrc-Users archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: Setting up bulkbuild


I'm interested in building stuff for VAX systems.


  It's now in the "Scanning..." phase and manages to "scan" about 250
packages per day, with a total of nearly 20k. So it'll be scanning for
about three months, give or take. Are there ways to speed this up? By
doing it parallel on several hosts, or create the scan result (I gues
it's inter-package dependencies?) on a different (faster) host and
then let the VAXen do actual build work (possibly proxying the
compilation work through distcc)?

Here's a general idea of how I've been doing VAX, m68k and other bulk (pbulk) builds. Note that there are lots of problems with trying to use pbulk as documented such as "/usr/bin/awk: no program filename" issues, issues (see PKG_DEVELOPER=no below), pbulk constantly wanting to rescan, "don't know how to make /nonexistent", and so on, so some of these steps might seem superfluous, but the process will fail if they're skipped. One day all of these should be documented in one place, along with their workarounds.

I run the scan part on a fast (amd64) machine. Note that I set up the chroot on the fast machine the same way as on the build machines, else we'll get lots of "don't know how to make /nonexistent" errors and wasted time. I then copy pbuild, presolve, pscan and friends, plus status, to a new pkgsrc/packages/bulklog/$architecture/meta/, then edit status to reflect unique times (so the reports of one build won't be in the same directory as another) and the correct architecture.

I then install pkgtools/libkver, then create a chroot (/usr/sandbox) on each bulk build machine, then, in screen, kver -r 9.0 (or 10.0) chroot /usr/sandbox

On my systems I tend to use /usr/local as LOCALBASE, which is advantageous because we don't want to install anything at all in to /usr/pkg/ in the chroot so it won't interfere with our builds:

mkdir -p /usr/sandbox/usr/local/lib /usr/sandbox/usr/local/sbin
cp /usr/local/lib/ /usr/sandbox/usr/local/lib/
cp /usr/local/sbin/kver /usr/sandbox/usr/local/sbin/

Next, echo "PKG_DEFAULT_OPTIONS=-pbulk-rsync" > /etc/mk.conf (in the sandbox) because I don't need / want to build perl and rsync when I'm using an NFS server that already has them.

I edit pkgsrc/mk/pbulk/ to change PKG_DEVELOPER to no and add this right after the "(cd ${PKGSRCDIR}/pkgtools/pbulk" line:

(cd ${PKGSRCDIR}/devel/distcc && PACKAGES=${TMPDIR}/packages-pbulk WRKOBJDIR=${TMPDIR}/obj-pbulk ${PBULKPREFIX}/bin/bmake install)

(glad to see you've found :)

Just for consistency between all of my bulk build machines, I also:

groupadd -g 70 distcc
useradd -u 70 -g 70 -d /nonexistent -s /sbin/nologin distcc
useradd -u 999 -g 100 -d /home/pbulk -s /bin/sh -m pbulk

Then I run:

sh /usr/pkgsrc/mk/pbulk/ -n -c /etc/mk.conf

After that's done, I create a new /etc/mk.conf:

.for DISTCCDEPS in sysutils/checkperms pkgtools/digest devel/distcc devel/popt devel/libtool-base devel/gmake devel/m4 pkgtools/cwrappers pkgtools/mktools
.if !defined(IGNORE_DISTCC)
  PKGSRC_COMPILER=distcc gcc

WRKOBJDIR is in /tmp because local disk is faster than NFS, and /tmp/pkgbuild needs to be 1777 permissions. DISTCC_HOSTS is, of course, the machine you're going to use to run distcc to help speed things up.

I then edit /usr/pbulk/bin/bulkbuild-restart and add "restart_build=yes" so that script_phase_pre_build and script_phase_scan are skipped.

If I'm just going to run a pbulk on this machine, I u(n)limit, then run /usr/pbulk/bin/bulkbuild-restart. If I'm going to run this machine as part of a clustered build, there are more steps, which I'll summarize here (but this part isn't complete).

Since we want the chroots to think they're running a very specific version of NetBSD, we edit /usr/sandbox/usr/pbulk/etc/pbulk.conf and change:

pbuild="/usr/local/sbin/kver -r 9.0 /usr/pbulk/bin/pbulk-build"

Plus the same for presolve and pscan, too.

I give each build machine a separate local address (either IPv4 or IPv6), configure the main system's ListenAddresses in /etc/ssh/sshd_config to not use the separate local address and configure the chroot's ListenAddress to listen. On boot I have "chroot /usr/sandbox /etc/rc.d/sshd onestart" in /etc/rc.local so that the chroot's sshd is always running.

I then configure and run pbulk on a fast(er) machine with a list of build_clients. This process has all sorts of caveats, too, which I'll have to document the next time I run through them. Also note that this machine needs to be reachable from any build machine, just as the build machines obviously need to be reachable from this machine (I forget why contact needs to be bidirectional, but with IPv6 this is easy, even over the Internet).

This should be a good start, I think, and I'll update with more details when I have a chance.

I didn't get to move from 2023Q1 to 2023Q2 for m68k, sparc, sparc64, vax, sh3el, mipsel, mips64eb, earmv4, or aarch64eb because my colo provider went out of business, but I hope to have many set back up for 2023Q3 :)

(As an aside, if anyone knows of anyone who has space, power and a little bandwidth for build machines, please contact me)

Also, I have a script for going through a previous quarter's binary packages and keeping / copying packages that haven't been updated and don't have prerequisites that have been updated, plus a simple edit to make sure that "make bin-install" doesn't pull older binary packages from the NetBSD servers.

When it comes to doing a cvs update and worrying about rescanning, I simply don't worry about rescanning. If a popular package is updated, and/or if there's an egregious security issue, I just manually kver / chroot, then "make bin-install" for that package. Perhaps this could be handled better in the future.

I hope this helps!


Home | Main Index | Thread Index | Old Index