Increases in build system time

[Andreas Gustafsson <gson%gson.org@localhost>]

Hi all,

Back in September, I wrote:
> I'm trying to run a bisection to determine why builds hosted on recent
> versions of NetBSD seem to be taking significantly more system time
> than they used to, building the same thing.

I finally have some results to report.  These are from builds of the
NetBSD-8/amd64 release hosted on various versions of -current/amd64,
on a HP DL360 G7 with dual Xeon L5630 CPUs (8 cores in all).  The
amount of system time taken by each build was measured using time(1).

Between a -current from September 2016 and one from October 2019, the
system time more than doubled, from 4245 seconds to 9344 seconds.
The time(1) output from the oldest and newest version was:

    3930.86 real     15737.04 user      4245.26 sys
    4461.47 real     16687.37 user      9344.68 sys

This means that on the recent -current, on average, roughly four of
the eight cores were executing the build tools (compilers, etc),
roughly two were executing the kernel, and the remaining two were
presumably idle.

The increase did not happen all at once but in several smaller steps
as shown in this graph:

  http://www.gson.org/netbsd/bugs/system-time/graph.png

For each step, finding the commits that caused it required a separate
bisection.  Each bisection took 1-2 days to run, so I have only
bisected the largest steps, those of 5 percent or more.  They are
listed below in order from largest to smallest, with CVS revisions
and commit messages.

  38% increase:

    2018.04.04.12.59.49 maxv src/sys/arch/amd64/amd64/machdep.c 1.303
    2018.04.04.12.59.49 maxv src/sys/arch/x86/include/cpu.h 1.91
    2018.04.04.12.59.49 maxv src/sys/arch/x86/x86/cpu.c 1.154
    2018.04.04.12.59.49 maxv src/sys/arch/x86/x86/spectre.c 1.8

    Enable the SpectreV2 mitigation by default at boot time.

  12% increase:

    2019.03.08.20.35.10 christos src/share/mk/bsd.own.mk 1.1108

    Back to using jemalloc for x86_64; all problems have been resolved.

  9% increase:

    2018.02.26.05.52.50 maxv src/sys/arch/amd64/conf/GENERIC 1.485

    Enable SVS by default.

  7% increase:

    2016.12.14.15.49.35 hannken src/sys/kern/vfs_vnode.c 1.63

    Change the freelists to lrulists, all vnodes are always on one
    of the lists.  Speeds up namei on cached vnodes by ~3 percent.

    Merge "vrele_thread" into "vdrain_thread" so we have one thread
    working on the lrulists.  Adapt vfs_drainvnodes() to always wait
    for a complete cycle of vdrain_thread().

  5% increase:

    2018.04.07.22.39.31 christos src/external/Makefile 1.21
    2018.04.07.22.39.31 christos src/external/README 1.16
    [302 more revisions by christos elided]
    2018.04.07.22.39.53 christos src/external/bsd/Makefile 1.59
    2018.04.07.22.41.55 christos src/doc/3RDPARTY 1.1515
    2018.04.07.22.41.55 christos src/doc/CHANGES 1.2376
    2018.04.08.00.52.38 mrg src/sys/arch/amd64/conf/ALL 1.85
    2018.04.08.00.52.38 mrg src/sys/arch/amd64/conf/GENERIC 1.489
    2018.04.08.00.52.38 mrg src/sys/arch/i386/conf/ALL 1.437
    2018.04.08.00.52.38 mrg src/sys/arch/i386/conf/GENERIC 1.1177
    2018.04.08.01.30.01 christos src/external/mpl/Makefile 1.1

    [Too many commit messages to list here, but the following from
    mrg's commit of src/sys/arch/amd64/conf/GENERIC 1.489 may
    be relevant]

    turn on GCC spectre v2 mitigation options.

  5% increase:

    2019.03.10.15.32.42 christos src/external/bsd/jemalloc/lib/Makefile.inc 1.5

    turn on debugging to help find problems

  5% decrease:

    2019.07.23.06.31.20 martin src/external/bsd/jemalloc/lib/Makefile.inc 1.10

    Disable JEMALLOC_DEBUG, it served us well, but now we want performance
    back. Discussed with christos.

To summarize, most of the increase was due to Spectre and Meltdown
mitigations, which I guess is not really surprising.  But the 12% net
increase from jemalloc and the 7% increase from vfs_vnode.c 1.63 seem
to call for closer investigation.
-- 
Andreas Gustafsson, gson%gson.org@localhost