Re: 8.0 performance issue when running build.sh?

[Maxime Villard <max%m00nbsd.net@localhost>]

Le 10/07/2018 à 11:01, Martin Husemann a écrit :
> On Fri, Jul 06, 2018 at 04:04:50PM +0200, Martin Husemann wrote:
> > I have no scientific data yet, but just noticed that build times on the
> > auto build cluster did rise very dramatically since it has been updated
> > to run NetBSD 8.0 RC2.
> >
> > Since builds move around build slaves sometimes (not exactly randomly,
> > but anyway) I picked the alpha port as an example (the first few
> > architectures in the alphabetical list get build slaves assigned pretty
> > consistently).
>
> Here is an intermediate result from further experiments and statistics:
>
>   - fpu_eager (as it is on NetBSD 8.0 RC2, which is not what is in -current
>     and not what will be in the final 8.0 release) has a measurable performance
>     impact - but it is not the big issue here.

For the record: EagerFPU has a fixed performance cost, which is,
saving+restoring the FPU context during each context switch. LazyFPU, however,
had a variable performance cost: during context switches the FPU state was
kept on the CPU, in the hope that if we switched back to the owner lwp we may
not have had to do a save+restore. If we did have to, however, we needed to
send an IPI, and cost(IPI+save+restore) > cost(save+restore).

So LazyFPU may have been less/more expensive than EagerFPU, depending on the
workload/scheduling.

The reason it is more expensive for you is maybe because on your machine each
lwp ("make" thread) stays on the same CPU, and the kpreemtions cause a
save+restore that is not actually necessary since each CPU always comes back
to the owner lwp. (As you said, also, you have the old version of EagerFPU in
RC2, which is more expensive than that of the current -current, so it's part
of the problem too.)

I've already said it, but XSAVEOPT actually eliminates this problem, since it
performs the equivalent of LazyFPU (not saving+restoring when not needed)
without requiring an IPI.

>   - if we ignore netbsd-7* branches, the performance loss is reasonable
>     explainable by the SVS penalty - we are going to check that theory soon.
>
>   - maybe the netbsd-7 /bin/sh and/or /usr/bin/make cause some very bad
>     interaction with SVS, making those build times sky rocket - if turning
>     off SVS does not solve this, we will need to dig deeper.
>
> So stay tuned, maybe only Intel to blame ;-)
>
> If anyone has concrete pointers for the last issue (or ideas what to change/
> measure) please speak up.

Not sure this is related, but it seems that the performance of build.sh on
netbsd-current is oscillating as fuck. Maybe I just hadn't noticed that it
was oscillating this much before, but right now a

	build.sh -j 4 kernel=GENERIC

is in [5min; 5min35s]. Even with SpectreV2/SpectreV4/SVS/EagerFPU all
disabled. It seems to me that at a time doing two or three builds was enough
and you would get an oscillation that was <5s. Now it looks like I have to
do more than 5 builds to get a relevant average.

Maxime