xennet performance collapses with multiple vCPU

[Stephen Borrill <netbsd%precedence.co.uk@localhost>]

All using NetBSD 9.99.51 (XEN3_DOMU) #0: Sun Mar 22 17:35:29 UTC 2020

Note that increasing the vCPUs from 1 to 4 (2 gives same result as 4) 
drops the throughput to about 10%. No offloads configured in domU:

xennet0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	capabilities=3ff00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
	capabilities=3ff00<UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx,TCP6CSUM_Tx>
	capabilities=3ff00<UDP6CSUM_Rx,UDP6CSUM_Tx>
	enabled=0
	ec_capabilities=1<VLAN_MTU>
	ec_enabled=0
	address: aa:b8:ba:d1:cc:d5
	inet 192.168.10.123/24 broadcast 192.168.10.255 flags 0x0
	inet6 fe80::7edc:79d:80f5:86fd%xennet0/64 flags 0x0 scopeid 0x1

netio same host (1 vCPU both ends):
Packet size  1k bytes:  96509 KByte/s Tx,  92083 KByte/s Rx.
Packet size  2k bytes:  124560 KByte/s Tx,  125777 KByte/s Rx.
Packet size  4k bytes:  117528 KByte/s Tx,  124900 KByte/s Rx.
Packet size  8k bytes:  123113 KByte/s Tx,  118255 KByte/s Rx.
Packet size 16k bytes:  120673 KByte/s Tx,  126248 KByte/s Rx.
Packet size 32k bytes:  127362 KByte/s Tx,  123743 KByte/s Rx.

iperf3 same host (1 vCPU both ends):
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.05 GBytes   902 Mbits/sec

netio same host (4 vCPU both ends):
Packet size  1k bytes:  9814 KByte/s Tx,  9443 KByte/s Rx.
Packet size  2k bytes:  11455 KByte/s Tx,  12041 KByte/s Rx.
Packet size  4k bytes:  11425 KByte/s Tx,  11489 KByte/s Rx.
Packet size  8k bytes:  12266 KByte/s Tx,  12119 KByte/s Rx.
Packet size 16k bytes:  12385 KByte/s Tx,  12269 KByte/s Rx.
Packet size 32k bytes:  12310 KByte/s Tx,  12277 KByte/s Rx.

iperf3 same host (4 vCPU both ends):
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   123 MBytes   103 Mbits/sec

netio different host (1 vCPU both ends):
Packet size  1k bytes:  42956 KByte/s Tx,  43926 KByte/s Rx.
Packet size  2k bytes:  63425 KByte/s Tx,  64284 KByte/s Rx.
Packet size  4k bytes:  76534 KByte/s Tx,  45076 KByte/s Rx.
Packet size  8k bytes:  116432 KByte/s Tx,  44534 KByte/s Rx.
Packet size 16k bytes:  134190 KByte/s Tx,  44474 KByte/s Rx.
Packet size 32k bytes:  135365 KByte/s Tx,  45204 KByte/s Rx.

[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.21 GBytes  1.04 Gbits/sec

netio different host (4 vCPU both ends):
Packet size  1k bytes:  10133 KByte/s Tx,  9498 KByte/s Rx.
Packet size  2k bytes:  12198 KByte/s Tx,  12218 KByte/s Rx.
Packet size  4k bytes:  12491 KByte/s Tx,  12224 KByte/s Rx.
Packet size  8k bytes:  12644 KByte/s Tx,  12283 KByte/s Rx.
Packet size 16k bytes:  12643 KByte/s Tx,  12339 KByte/s Rx.
Packet size 32k bytes:  12624 KByte/s Tx,  12252 KByte/s Rx.
Done.

iperf3 different host (4 vCPU both ends):
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   151 MBytes   127 Mbits/sec

By comparison:

Windows to Windows same host gives:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  4.54 GBytes  3.90 Gbits/sec

Windows to Windows different host gives:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  2.76 GBytes  2.37 Gbits/sec

--
Stephen