pkgsrc scanning performance benchmarks

[John Marino <netbsd%marino.st@localhost>]

As a natural part of my effort to port the Synth repository builder to 
the pkgsrc platforms that can support it, I've incidentally acquired 
directly comparable benchmarks over full tree scanning between FreeBSD's 
Ports Collection and pkgsrc.  I did it over two different boxes and I 
think at least some people will be interested in the numbers.  (Please 
don't get hung up that I call the pkgsrc analog to ports "ports" as well 
because I don't have a better word to describe the pkgsrc makefiles that 
build pkgsrc packages.)

In each run, Synth is scanning the entire tree systematically in 
parallel.   For example, it splits the tree into 12 equal batches.  In 
sequential order, it uses bmake to return the value of 10 variables.  It 
is not recursive since the entire tree is covered when all 12 
independent builders finish.

The first box is a home home machine (DragonFly 4.7):

hw.machine: x86_64
hw.model: Intel(R) Core(TM) i5-2500 CPU @ 3.30GHz
UFS / 8G PhysMem

Ports Collection:
------------------------------
--  Statistics
------------------------------
 Ports scanned : 26489
  Elapsed time : 00:01:59
   Parallelism : 12 scanners
          ncpu : 4

pkgsrc:
------------------------------
--  Statistics
------------------------------
 Ports scanned : 15171
  Elapsed time : 00:32:10
   Parallelism : 12 scanners
          ncpu : 4

P.Coll 222.6 ports/second
pkgsrc   7.9 ports/second
28.2x faster

Normalized, it takes pkgsrc 28 times longer to complete an analogous 
task.  The second box is beefier, it's the DF package building machine 
known as 'muscles'.

hw.machine: x86_64
hw.model: Intel(R) Xeon(R) CPU E3-1240 v3 @ 3.40GHz
HAMMER FS / 32G PhysMem

Ports Collection:
------------------------------
--  Statistics
------------------------------
 Ports scanned : 26540
  Elapsed time : 00:01:14
   Parallelism : 24 scanners
          ncpu : 8

pksrc:
------------------------------
--  Statistics
------------------------------
 Ports scanned : 15171
  Elapsed time : 00:19:30
   Parallelism : 24 scanners
          ncpu : 8

P.Coll 358.6 ports/second
pkgsrc  13.0 ports/second
27.6x faster

While the better machine can shave 13 minutes off the scan time, the 
performance ratio of 28:1 is maintained.

I have not looked at the cause at all, but I have some experience with 
identifying usual suspects, which are:
- unnecessary spawning of shell commands (e.g. better alternatives exist)
- chaining commands together with pipes  (ditto)
- a killer: using ${MAKE} to call itself to get the value of a makefile 
variable.  As an example, even grepping for a constant is a couple of 
magnitudes faster than ${MAKE} -C ${dir} -VMYVAR

Individually it may not seem like one approach is a performance hit, but 
compounded 15,000 times it becomes a huge one.  Based on experience, I 
suspect that finding low-hanging fruit and providing more efficient 
methods could cut the scan time by two-thirds pretty easily.

It's not just people that do full tree scans (how long does that take on 
a core2duo, like 3 hours if done serially?), but I was very surprised on 
long it took to scan the dependency tree of wm/xfce4-wm.  That was only 
about 150 "ports" but took a noticeably long time.

I do not believe the pkgsrc framework is 28 times more complex than the 
Ports Collection framework.  It's just much more inefficient.  I know 
such statements rankle some pkgsrc devs, but numbers don't lie.  I tried 
to be very objective here and I sort of hope that spurs an interest to 
greatly improve efficiency (or at least not block proposals with the 
goal of improving efficiency).

John

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus