Re: network outages

[Greg Troxel <gdt%lexort.com@localhost>]

I have reproduced trouble on 11-rc2, in a domU with 6000 MB.

My machine has had in sysctl.conf:

  net.inet.tcp.sendspace=131072
  net.inet.tcp.recvspace=131072
  net.inet6.tcp6.sendspace=131072
  net.inet6.tcp6.recvspace=131072

  net.inet.tcp.recvbuf_auto=0
  net.inet.tcp.sendbuf_auto=0
  net.inet6.tcp6.recvbuf_auto=0
  net.inet6.tcp6.sendbuf_auto=0


First, I applied all of your sysctls (with a shell script to set them,
not sysctl.conf) and noticed that iperf3 failed.  My existing ssh
connection remained ok.

I then backed off sendspace/recvspace to my values and things were back
to normal.

I then did binary search on just changing sendspace.  The highest value
that works exactly is 209715 which is 0x33333.

It would be interesting to see if others see the same value, or
different.

n11-amd64 root 79 ~ #> sysctl -w net.inet.tcp.sendspace=209715; iperf3 -4 -c foo.example.com;   sysctl -w net.inet.tcp.sendspace=209716; iperf3 -4 -c foo.example.com
net.inet.tcp.sendspace: 209716 -> 209715
Connecting to host foo.example.com, port 5201
[  7] local 172.16.32.56 port 65498 connected to 71.19.148.97 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  7]   0.00-1.01   sec  1.38 MBytes  11.5 Mbits/sec    0    512 KBytes       
[  7]   1.01-2.02   sec  2.25 MBytes  18.7 Mbits/sec    0    512 KBytes       
^C[  7]   2.02-2.56   sec  1.50 MBytes  23.0 Mbits/sec    0    512 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  7]   0.00-2.56   sec  5.12 MBytes  16.8 Mbits/sec    0            sender
[  7]   0.00-2.56   sec  0.00 Bytes  0.00 bits/sec                  receiver
iperf3: interrupt - the client has terminated by signal Interrupt(2)

n11-amd64 root 80 ~ #> !77
  sysctl -w net.inet.tcp.sendspace=209716; iperf3 -4 -c foo.example.com
net.inet.tcp.sendspace: 209716 -> 209716
iperf3: error - unable to connect to server - server may have stopped running or use a different port, firewall issue, etc.: No buffer space available

diffing 'netstat -s' was not illuminating.

Speculation: sendspace is reserving memory somehow, and depending on
[random stuff] there is sometimes not enough.

On this machine a few minutes later:

  Memory: 2639M Act, 1254M Inact, 22M Wired, 28M Exec, 3646M File, 254M Free

I freed up RAM by running a program that touches memory and exits:

  Memory: 12M Act, 1092M Inact, 22M Wired, 28M Exec, 862M File, 3007M Free

but still 209716 causes ENOBUFS.