Re: Low capped TCP network speeds over the Internet

[z411 <z411%omaera.org@localhost>]

On 3/5/26 02:54, Michael van Elst wrote:
> You need to increase kern.sbmax too, this is a global limit on
> the socket buffer size. The default is 256kB, so at 200ms RTT
> you are limited to about 256kB/200ms ~ 10Mbit/s.

Thank you very much. This makes a difference - although at higher values
the speed doesn't increase by much, but instead the transfer becomes
visibly more unstable.

kern.sbmax=262144 (Default value): Very stable at 8.39 Mbit/s.
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   1.00-2.00   sec   768 KBytes  6.29 Mbits/sec
[  5]   2.00-3.00   sec  1.00 MBytes  8.39 Mbits/sec
[  5]   3.00-4.00   sec  1.00 MBytes  8.39 Mbits/sec
[  5]   4.00-5.00   sec  1.00 MBytes  8.39 Mbits/sec
[  5]   5.00-6.00   sec  1.00 MBytes  8.39 Mbits/sec
[  5]   6.00-7.00   sec   896 KBytes  7.34 Mbits/sec
[  5]   7.00-8.00   sec  1.00 MBytes  8.39 Mbits/sec
[  5]   8.00-9.00   sec  1.00 MBytes  8.39 Mbits/sec
[  5]   9.00-10.00  sec  1.00 MBytes  8.39 Mbits/sec

kern.sbmax=524288 or 16777216: Increases packet loss dramatically, speed
becomes immediately unstable:

Worst case:
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   1.00-2.00   sec  1.25 MBytes  10.5 Mbits/sec
[  5]   2.00-3.00   sec   256 KBytes  2.10 Mbits/sec
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec

Best case:
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   1.00-2.00   sec  1.00 MBytes  8.39 Mbits/sec
[  5]   2.00-3.00   sec  1.12 MBytes  9.44 Mbits/sec
[  5]   3.00-4.00   sec   256 KBytes  2.10 Mbits/sec
[  5]   4.00-5.00   sec  1.88 MBytes  15.7 Mbits/sec
[  5]   5.00-6.00   sec  1.62 MBytes  13.6 Mbits/sec
[  5]   6.00-7.00   sec   512 KBytes  4.19 Mbits/sec
[  5]   7.00-8.00   sec   256 KBytes  2.10 Mbits/sec
[  5]   8.00-9.00   sec   640 KBytes  5.24 Mbits/sec
[  5]   9.00-10.00  sec  1.12 MBytes  9.43 Mbits/sec

So with a increased the value it can occasionally hit higher speeds,
it's not much. And what's worse, it makes it very unstable.

In Wireshark, with kern.sbmax=262144 I very rarely see any packet loss
or retransmissions. Any higher than that and packet loss and slow
retransmissions become very frequent. Switching CC to CUBIC improves
speeds but not the heavy packet loss and slow recovery.

I also often see duplicate ACKs by the client for a good while before
slow retransmissions start (RTO?) with no fast retransmit happening
before that. But again, no loss happens in the first place when the
buffer is low.

I uploaded a sample pcap:
http://u.omaera.org/nbsd.pcapng

- Why could it be that packet loss almost doesn't happen with the
default buffer size, but becomes very frequent when it's increased?
- Is fast retransmit not being triggered?
- Would it be appropriate to take this to tech-net@ as well?

Thank you in advance.

> If you have many connections you may also need to increase
> kern.somaxkva that limits how much memory can be allocated
> to the network in total.
> 
> 
>> Since post-tuning it's very stable I don't think it's about congestion
>> anymore - but I'm not sure why I'm hitting this hard cap, even after
>> increasing the buffer sizes. Could it be that NetBSD is not increasing
>> the window size?
> 
> NetBSD will grow the window and has window scaling enabled by default.
> However, it is very slow growing the send window after a congestion event.
>