Dom0 bad network performance (was: NetBSD/Xen samba performance low (compared to NetBSD/amd64))

[Matthias Petermann <mp%petermann-it.de@localhost>]

Hello all,

last summer I already had a request here, which I had classified as a 
Samba problem at that time due to insufficient preliminary 
investigations. You guys helped me a lot to investigate the problem 
further, but unfortunately I didn't have the chance to follow up on it.

Now I have made a new attempt on the same hardware (Intel NUC5) and have 
gone a little deeper into the matter. My new findings are:

 - the raw read/write performance in the filesystem is hardly 
different, no matter if I use NetBSD as Dom0 or on the same system a 
"pure" NetBSD kernel without Xen. In both cases it is about 60 MByte/s 
which appears ok for this type of hardware.

 - when transferring files of this file system over the network the 
performance is lost - with NetBSD as Dom0 the transfer rate is max. 12 
MByte/s, while it is about 60 MByte/s when using a "pure" NetBSD kernel 
without Xen. It does not matter whether the transfer is done with Samba 
or another protocol (sftp, NFS).

I just measured the bandwidth with iperf for both cases:

1) NetBSD/Xen Dom0

```
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  21.0 MBytes   176 Mbits/sec    0   36.8 KBytes 

[  5]   1.00-2.00   sec  20.9 MBytes   175 Mbits/sec    0   36.8 KBytes 

[  5]   2.00-3.00   sec  21.0 MBytes   176 Mbits/sec    0   36.8 KBytes 

[  5]   3.00-4.00   sec  21.5 MBytes   180 Mbits/sec    0   36.8 KBytes 

[  5]   4.00-5.00   sec  21.0 MBytes   176 Mbits/sec    0   36.8 KBytes 

[  5]   5.00-6.00   sec  21.1 MBytes   177 Mbits/sec    0   36.8 KBytes 

[  5]   6.00-7.00   sec  20.8 MBytes   174 Mbits/sec    0   36.8 KBytes 

[  5]   7.00-8.00   sec  20.7 MBytes   174 Mbits/sec    0   36.8 KBytes 

[  5]   8.00-9.00   sec  20.7 MBytes   174 Mbits/sec    0   36.8 KBytes 

[  5]   9.00-10.00  sec  20.3 MBytes   171 Mbits/sec    0   36.8 KBytes
```

2) NetBSD/amd64 (pure Kernel without Xen)

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  54.8 MBytes   460 Mbits/sec    0   35.4 KBytes 

[  5]   1.00-2.00   sec  54.9 MBytes   460 Mbits/sec    0   35.4 KBytes 

[  5]   2.00-3.00   sec  55.5 MBytes   465 Mbits/sec    0   35.4 KBytes 

[  5]   3.00-4.00   sec  54.7 MBytes   459 Mbits/sec    0   35.4 KBytes 

[  5]   4.00-5.00   sec  54.7 MBytes   459 Mbits/sec    0   35.4 KBytes 

[  5]   5.00-6.00   sec  55.6 MBytes   467 Mbits/sec    0   35.4 KBytes 

[  5]   6.00-7.00   sec  55.2 MBytes   463 Mbits/sec    0   35.4 KBytes 

[  5]   7.00-8.00   sec  55.2 MBytes   463 Mbits/sec    0   35.4 KBytes 

[  5]   8.00-9.00   sec  55.6 MBytes   467 Mbits/sec    0   35.4 KBytes 

[  5]   9.00-10.00  sec  55.5 MBytes   465 Mbits/sec    0   35.4 KBytes

The results are not quite linear with the observed "gross" values for 
file transfer, but still clearly show that there is a measurable drop in 
network speed.

Are these values common for a Xen Dom0, or would you also consider this 
a problem? If it looks like a problem (which I think it is at the 
moment), what can I do to get further?

Kind regards
Matthias


[1] http://mail-index.netbsd.org/port-xen/2020/08/03/msg009779.html