Doubling performance of pkgsrc/mk

[Jonathan Perkin <jperkin%mnx.io@localhost>]

Hi,

I got fed up with pkgsrc being slow again, so have spent a bit of time 
recently improving performance.

Here are some benchmarks for the build of pkgtools/url2pkg (chosen as 
its "build" time is very short so exposes any delays with the 
infrastructure):

  +----------+---------+--------+-------+
  | OS       | SmartOS | NetBSD | macOS |
  +----------+---------+--------+-------+
  | Baseline |   12.5s |  10.3s |  3.0s |
  | Patched  |    5.7s |   5.3s |  1.4s |
  +----------+---------+--------+-------+

All of the builds were done on very different hardware so the OPSYS 
differences aren't interesting, but the patched times show a consistent 
2x speedup regardless.

I'm obviously testing this work a lot as it touches a huge amount of 
very important code, but I'd be intested in feedback early on, as well 
as reports of testing from anyone who wants to try it out.

The branch is here:

  https://github.com/TritonDataCenter/pkgsrc/tree/feature/performance/trunk

I may occasionally rebase against trunk as well as squashing/improving 
patches, so expect to sometimes see a forced update.

As with previous performance work the bulk of this is simply thanks to 
reducing the number of fork+exec, using DTrace to identify areas that 
can be optimised.  On SmartOS the baseline build invoked 2391 processes, 
the patched build only 877.

The good news is there's still a bunch of things that can be improved, 
most notably around tools creation which is still very expensive.

Cheers,

--
Jonathan Perkin   -   mnx.io   -   pkgsrc.smartos.org
Open Source Complete Cloud   www.tritondatacenter.com