Re: Low capped TCP network speeds over the Internet

[z411 <z411%omaera.org@localhost>]

Hi Michael, I've got some good news.

I switched the network interface from VirtIO to an emulated Intel E1000,
that is from vioif(4) to wm(4), and the speed cap disappears. With the
proper buffer sizes I can reach very good speeds:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   256 KBytes  2.09 Mbits/sec    0   41.0 KBytes
[  5]   1.00-2.01   sec  1.38 MBytes  11.4 Mbits/sec    0    440 KBytes
[  5]   2.01-3.01   sec  7.62 MBytes  64.0 Mbits/sec    0   2.32 MBytes
[  5]   3.01-4.01   sec  17.4 MBytes   146 Mbits/sec    0   9.33 MBytes
[  5]   4.01-5.01   sec  19.0 MBytes   159 Mbits/sec    0   14.4 MBytes
[  5]   5.01-6.01   sec  19.1 MBytes   160 Mbits/sec    0   15.5 MBytes
[  5]   6.01-7.01   sec  19.0 MBytes   159 Mbits/sec    0   16.6 MBytes
[  5]   7.01-8.01   sec  19.1 MBytes   160 Mbits/sec    0   17.7 MBytes
[  5]   8.01-9.01   sec  19.0 MBytes   159 Mbits/sec    0   18.8 MBytes
[  5]   9.01-10.01  sec  17.9 MBytes   150 Mbits/sec    0   19.7 MBytes
[  5]  10.01-10.21  sec  3.88 MBytes   166 Mbits/sec    0   20.0 MBytes

So either the vioif(4) driver has some nasty bug, or maybe it has to do
with the note in the man page about it not supporting offloading.

That said, even though the packet loss got much better, with wm(4) it
still enters retransmission loops fairly often before it is able to
recover again. I recorded the worst case:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   128 KBytes  1.05 Mbits/sec    0   46.7 KBytes

[  5]   1.00-2.01   sec  1.88 MBytes  15.6 Mbits/sec    0    516 KBytes

[  5]   2.01-3.01   sec  2.00 MBytes  16.8 Mbits/sec    0   1.41 KBytes

[  5]   3.01-4.01   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes
[  5]   4.01-5.01   sec  0.00 Bytes  0.00 bits/sec    0   12.7 KBytes
[  5]   5.01-6.01   sec   128 KBytes  1.05 Mbits/sec    0   9.90 KBytes

[  5]   6.01-7.01   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes
[  5]   7.01-8.01   sec   128 KBytes  1.05 Mbits/sec    0   2.83 KBytes

[  5]   8.01-9.01   sec  0.00 Bytes  0.00 bits/sec    0   2.83 KBytes
[  5]   9.01-10.01  sec  0.00 Bytes  0.00 bits/sec    0   2.83 KBytes
[  5]  10.01-10.20  sec  0.00 Bytes  0.00 bits/sec    0   2.83 KBytes

It does seem to me that it starts sending out of order packets, although
I'm not sure if it's just retransmission.

On 3/5/26 20:21, Michael van Elst wrote:
> The slow recovery comes from limiting packet bursts as requested
> in early RFCs. Other BSDs have relaxed that a long time ago,
> and I'm currently running NetBSD with similar modifications in
> the TCP stack.

I'd be interested in hearing about the modifications in the TCP stack,
if you don't mind.

> What I do not yet understand is what causes the packet loss.

Most of it, or at least the low speeds, seemed to be caused by vioif(4).

> Another issue is e.g. the wm(4) driver that easily sends
> packets out of order (and that I work around with a small
> patch).

Now that I switched to wm(4) that might be my issue right now. I'd be
also grateful if you could point me to said patch.

I'm happy that I'm getting much better speeds now, but I'm still
interested in hearing about your modifications, in case they have to do
with the drops I'm experiencing now.

Thank you for your time.
- z411