Re: Doubling performance of pkgsrc/mk

[Adam <adam%netbsd.org@localhost>]

> Hi,
> 
> I got fed up with pkgsrc being slow again, so have spent a bit of time recently improving performance.
> 
> Here are some benchmarks for the build of pkgtools/url2pkg (chosen as its "build" time is very short so exposes any delays with the infrastructure):
> 
>  +----------+---------+--------+-------+
>  | OS       | SmartOS | NetBSD | macOS |
>  +----------+---------+--------+-------+
>  | Baseline |   12.5s |  10.3s |  3.0s |
>  | Patched  |    5.7s |   5.3s |  1.4s |
>  +----------+---------+--------+-------+
> 
> All of the builds were done on very different hardware so the OPSYS differences aren't interesting, but the patched times show a consistent 2x speedup regardless.
> 
> I'm obviously testing this work a lot as it touches a huge amount of very important code, but I'd be intested in feedback early on, as well as reports of testing from anyone who wants to try it out.
> 
> The branch is here:
> 
>  https://github.com/TritonDataCenter/pkgsrc/tree/feature/performance/trunk
> 
> I may occasionally rebase against trunk as well as squashing/improving patches, so expect to sometimes see a forced update.
> 
> As with previous performance work the bulk of this is simply thanks to reducing the number of fork+exec, using DTrace to identify areas that can be optimised.  On SmartOS the baseline build invoked 2391 processes, the patched build only 877.
> 
> The good news is there's still a bunch of things that can be improved, most notably around tools creation which is still very expensive.
> 
> Cheers,
> 
> -- 
> Jonathan Perkin   -   mnx.io   -   pkgsrc.smartos.org
> Open Source Complete Cloud   www.tritondatacenter.com

Very nice :)

I wonder, is "${TEST} -d ${.TARGET:H} || ${MKDIR} ${.TARGET:H}" really faster than just "${MKDIR} ${.TARGET:H}"?

Kind regards,
Adam