Re: Low capped TCP network speeds over the Internet

[z411 <z411%omaera.org@localhost>]

Hi Michael.

I have some new findings. I installed NetBSD in a KVM hypervisor in a
Japanese datacenter this time. NetBSD can reach very high speeds
compared to previous attempts (40 MB/s), and even though it still
collapses here and there, it is much more stable. It makes me think that
the US datacenter has a "dirtier" network path, or is applying some kind
of traffic shaping, or both. Linux has no issues on the same node, and
FreeBSD/OpenBSD have some trouble but not as much as NetBSD. I can only
imagine they just have TCP stacks that are less sensitive to packet loss.

It's worth noting that UDP transfers show 0% loss every time. So maybe
it's not loss specifically but something else that affects TCP transfers.

Another interesting finding is that, at least for me, all operating
systems suffer negatively from VirtIO NIC. They all improve their speeds
when switching to E1000, which goes against what I often hear about
VirtIO being faster due to paravirtualization.

I made a general overview here:

https://u.omaera.org/e1000.png

All operating systems simply do better with E1000, but VirtIO caps
NetBSD in particular to 1 MB/s even when tuned, so while other OSes do
"okay" with VirtIO, NetBSD has serious trouble with it.

On 3/7/26 03:41, Michael van Elst wrote:
> It would be interesting to see how that affects your setup.

I tried your patchset. Sadly I don't think I saw much improvement.
Here's a worst case measurement (xpl files inside):

https://u.omaera.org/xpl_us2.tar.gz

The Japanese server does much better: almost no drops, often stable 40
MB/s (!). I was planning to move to that server anyway so this works for
me. I will publish these results to my blog soon.

===

I found a new issue though: receiving files with SCP from the
server->client is fast, but when sending files to the server it only
sends 255KB and then it stalls for about half a minute, then sends the
rest at a slow pace (around 200-300 KB/s). If I send it using legacy SCP
(-O flag) the initial stall doesn't happen, but it's still slow.
The server has big recv buffers and it has no trouble downloading
(downloads using ftp(1) are very fast), it's just SCP.

These are my tunables:
net.inet6.tcp6.recvbuf_inc=131052
net.inet6.tcp6.sendbuf_inc=131052
net.inet6.tcp6.recvbuf_max=5042880
net.inet6.tcp6.sendbuf_max=5042880
net.inet6.tcp6.congctl.selected=cubic
net.inet.tcp.recvbuf_inc=131052
net.inet.tcp.sendbuf_inc=131052
net.inet.tcp.recvbuf_max=5042880
net.inet.tcp.sendbuf_max=5042880
net.inet.tcp.congctl.selected=cubic
kern.sbmax=5042880

Do you have an idea of what could be happening?

Thanks for all the help.
- z411