Re: xennet performance collapses with multiple vCPU

[Andrew Doran <ad%netbsd.org@localhost>]

On Wed, Mar 25, 2020 at 07:06:58PM +0100, Jarom?r Dole?ek wrote:
> Le mer. 25 mars 2020 ? 17:16, Jarom?r Dole?ek <jaromir.dolecek%gmail.com@localhost> a
> ?crit :
> 
> > Le mar. 24 mars 2020 ? 15:13, Stephen Borrill <netbsd%precedence.co.uk@localhost> a
> > ?crit :
> >
> >> All using NetBSD 9.99.51 (XEN3_DOMU) #0: Sun Mar 22 17:35:29 UTC 2020
> >>
> >> Note that increasing the vCPUs from 1 to 4 (2 gives same result as 4)
> >> drops the throughput to about 10%. No offloads configured in domU:
> >>
> >>
> > I found out the problem appeared between 2020-01-08 00:00 (that was still
> > good) and 2020-01-14 00:00 (that is already bad).
> >
> > There were some CPU scheduling changes by Andrew Doran around that time,
> > so it's likely the culprit.
> >
> > I'm working on bisecting this further. It's complicated, between 01-08 and
> > 01-14 there were some bugs which prevent the Xen DomU MP kernel from
> > booting.
> >
> 
> I've now tracked it down to this change:
> 
> Module Name:    src
> Committed By:   ad
> Date:           Mon Jan 13 20:30:08 UTC 2020
> 
> Modified Files:
>         src/sys/kern: subr_cpu.c
> 
> Log Message:
> Fix some more bugs in the topo stuff, that prevented it from working
> properly with fake topo info + MP.
> 
> 
> To generate a diff of this commit:
> cvs rdiff -u -r1.10 -r1.11 src/sys/kern/subr_cpu.c
> 
> After this change the DomU even boots visibly slower. Maybe this change
> makes MP system scheduler use all CPUs, but introduces too much switching
> between them? Andy, can you have a look?

Before I get to try this at the weekend, I simulated this on native x86 by
disabling topology, preemption and fast softints, and it all seems fine.

Andrew