Re: Low capped TCP network speeds over the Internet

[z411 <z411%omaera.org@localhost>]

Hi Greg, I'm really thankful for your analysis.

By now out of the desperation I decided to switch the network interface
from VirtIO (vioif(4)) to Intel E1000 (wm(4)). This actually fixed the
speeds:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   256 KBytes  2.09 Mbits/sec    0   41.0 KBytes
[  5]   1.00-2.01   sec  1.38 MBytes  11.4 Mbits/sec    0    440 KBytes
[  5]   2.01-3.01   sec  7.62 MBytes  64.0 Mbits/sec    0   2.32 MBytes
[  5]   3.01-4.01   sec  17.4 MBytes   146 Mbits/sec    0   9.33 MBytes
[  5]   4.01-5.01   sec  19.0 MBytes   159 Mbits/sec    0   14.4 MBytes
[  5]   5.01-6.01   sec  19.1 MBytes   160 Mbits/sec    0   15.5 MBytes
[  5]   6.01-7.01   sec  19.0 MBytes   159 Mbits/sec    0   16.6 MBytes
[  5]   7.01-8.01   sec  19.1 MBytes   160 Mbits/sec    0   17.7 MBytes
[  5]   8.01-9.01   sec  19.0 MBytes   159 Mbits/sec    0   18.8 MBytes
[  5]   9.01-10.01  sec  17.9 MBytes   150 Mbits/sec    0   19.7 MBytes
[  5]  10.01-10.21  sec  3.88 MBytes   166 Mbits/sec    0   20.0 MBytes

I'm not sure if this has to do with the vioif(4) driver not supporting
offloading or something else, but it really went from 8.39 Mbps to 160
Mbps (with tunables).

That said, while packet loss isn't as common as before, it still
happens; I recorded a bad case of it:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.01   sec   128 KBytes  1.04 Mbits/sec    0   48.1 KBytes

[  5]   1.01-2.02   sec  1.75 MBytes  14.5 Mbits/sec    0    492 KBytes

[  5]   2.02-3.02   sec  5.75 MBytes  48.2 Mbits/sec    0   3.45 MBytes

[  5]   3.02-4.02   sec  7.75 MBytes  65.0 Mbits/sec    0   5.69 MBytes

[  5]   4.02-5.02   sec  1.88 MBytes  15.7 Mbits/sec    0   4.24 KBytes

[  5]   5.02-6.02   sec   128 KBytes  1.05 Mbits/sec    0   9.90 KBytes

[  5]   6.02-7.02   sec   128 KBytes  1.05 Mbits/sec    0   12.7 KBytes

[  5]   7.02-8.02   sec  0.00 Bytes  0.00 bits/sec    0   11.3 KBytes
[  5]   8.02-9.02   sec   128 KBytes  1.05 Mbits/sec    0   9.90 KBytes

[  5]   9.02-10.02  sec  0.00 Bytes  0.00 bits/sec    0   9.90 KBytes
[  5]  10.02-10.21  sec  0.00 Bytes  0.00 bits/sec    0   11.3 KBytes

I saved a capture of this as well: http://u.omaera.org/nbsd2.pcapng

I'll try to install NetBSD in a different datacenter to see if there's
anything wrong with this particular network.