Subject: pkg_comp + SMP (tested on real hardware)
To: None <firstname.lastname@example.org>
From: Lars Nordlund <email@example.com>
Date: 07/06/2006 23:56:02
I have finally had the opportunity to do some testing of pkg_comp on
SMP gear. The machine is equipped with 2.5GiB RAM and two "Intel(R) Xeon
(TM) CPU 3.20GHz" running with hyper threading enabled. It was running
NetBSD 3.0 release and I also used the 3.0 release sets in pkg_comp.
All builds has started with empty /usr/pkgsrc/packages/All directories
to have all dependencies built in each test run. I also made sure that
all distfiles were already downloaded.
I started out by building meta-pkgs/kde3 with 4 parallel build
processes. Due to various issues, it became a longer test then I had
expected. So I only did one run. (I also wanted to test that my
mkdir-based locking scheme held up. It did.)
meta-pkgs/kde3, 4 jobs
Total time: 5:36:00
ksh time: 20170.81 real 43677.84 user 17988.42 sys
meta-pkgs/kde3 has an unfortunate "waist" in the dependency graph.
qt3-libs -> qt3-tools -> kdelibs3 -> kdebase3. These packages were
mainly built alone. Nearly half the time was spent building only one
Next I wanted to compare between different number of parallel jobs. To
save time I needed a smaller work set. I listed the following packages
in the pkg_comp config file:
devel/subversion editors/emacs devel/check devel/cmake shells/zsh
misc pkgs, 1 job
Total time: 50:53
perl build time: 4:44
apache2 build time: 2:11
ksh time: 3054.04 real 3603.45 user 1882.48 sys
misc pkgs, 2 jobs
Total time: 25:15
perl build time: 5:18
apache2 build time: 2:35
ksh time: 2115.49 real 3693.54 user 1929.13 sys
misc pkgs, 4 jobs
Total time, 31:15
perl build time: 8:13
apache2 build time: 4:43
ksh time: 1875.15 real 4729.19 user 2804.02 sys
The machine is using IDE which may explain why individual packages took
so much longer when going from 1 to 2 parallel jobs.
100% improvement on the total time when going from -j 1 to -j 2 is
probably thanks to hyper threading making process switching faster. And
in pkgsrc there are a lot of processes to switch between (buildlink3).
Taking advantage of SMP in pkgsrc works out quite nicely. In
combination with pkg_comp it is very convenient to start larger builds
without having to pkg_delete half the running system. I also welcome
the changes to make pkg_comp portable between operating systems, as has
been discussed in pkgsrc forums not long ago.
Many thanks to GMQ Consulting (www.gmq.se) for providing the hardware!