Re: NPF/interface tuning? shell unusable on gateway

["Jeremy C. Reed" <reed%reedmedia.net@localhost>]

On Sun, 27 Mar 2022, David Young wrote:

> Are there any packet drops or other errors? `sysctl net.interfaces`,
> `sysctl net.inet6.ip6.ifq`, `sysctl net.inet.ip.ifq`, and `netstat -dvI
> re0; netstat -dvI re1` may be revealing.

David, thank you for the feedback and hints. I switched re1 to outside 
and re0 to my WAN after my previous email just to see if there was any 
change. It still had same problem. But you helped me track down to what 
it appears to be one link and one service with a problem.  I will 
provide answers below first:

$ sysctl net.interfaces 
net.interfaces.athn0.rcvq.drops = 0
net.interfaces.athn0.sndq.len = 0
net.interfaces.athn0.sndq.maxlen = 256
net.interfaces.athn0.sndq.drops = 0
net.interfaces.re0.rcvq.drops = 0
net.interfaces.re0.sndq.len = 0
net.interfaces.re0.sndq.maxlen = 512
net.interfaces.re0.sndq.drops = 0
net.interfaces.re1.rcvq.drops = 0
net.interfaces.re1.sndq.len = 0
net.interfaces.re1.sndq.maxlen = 512
net.interfaces.re1.sndq.drops = 0
net.interfaces.lo0.rcvq.drops = 0
net.interfaces.lo0.sndq.len = 0
net.interfaces.lo0.sndq.maxlen = 256
net.interfaces.lo0.sndq.drops = 0

$ sysctl net.inet6.ip6.ifq
net.inet6.ip6.ifq.len = 0
net.inet6.ip6.ifq.maxlen = 256
net.inet6.ip6.ifq.drops = 0

(I am not purposely using IPv6.)

$ sysctl net.inet.ip.ifq
net.inet.ip.ifq.len = 0
net.inet.ip.ifq.maxlen = 256
net.inet.ip.ifq.drops = 0

$ netstat -dvI re0; netstat -dvI re1
Name  Mtu   Network       Address              Ipkts Ierrs Idrops    Opkts Oerrs Colls Odrops
re0   1500  <Link>        b8:ac:6f:df:49:9d 19529216     0      0 31150053     0     0      0
re0   1500  172.16/16     172.16.1.1        19529216     0      0 31150053     0     0      0
re0   1500  fe80::%re0/64 fe80::baac:6fff:fedf:499d%re0 19529216     0      0 31150053     0     0      0
Name  Mtu   Network       Address              Ipkts Ierrs Idrops    Opkts Oerrs Colls Odrops
re1   1500  <Link>        f4:f2:6d:00:b7:57 30856346     0      0 19509672     0     0      0
re1   1500  fe80::%re1/64 fe80::200e:d2e4:6900:afc6%re1 30856346     0      0 19509672     0     0      0
re1   1500  47.185.18/24  47.185.18.26        30856346     0      0 19509672     0     0      0

> What link speed is negotiated on WAN and LAN ports?
> Is any flow-control negotiated?

My LAN interface:
$ ifconfig re0                                                          
re0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        
capabilities=3f80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
        capabilities=3f80<UDP4CSUM_Rx,UDP4CSUM_Tx>
        enabled=0
        ec_capabilities=3<VLAN_MTU,VLAN_HWTAGGING>
        ec_enabled=0
        address: b8:ac:6f:df:49:9d
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active
        inet 172.16.1.1/16 broadcast 172.16.255.255 flags 0x0
        inet6 fe80::baac:6fff:fedf:499d%re0/64 flags 0x0 scopeid 0x2

My interface to outside:
$ ifconfig re1 
re1: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        
capabilities=3f80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
        capabilities=3f80<UDP4CSUM_Rx,UDP4CSUM_Tx>
        enabled=0
        ec_capabilities=3<VLAN_MTU,VLAN_HWTAGGING>
        ec_enabled=0
        address: f4:f2:6d:00:b7:57
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet 47.185.18.26/24 broadcast 47.185.18.255 flags 0x0

> It sounds like the LAN is quite slow?  I may have misunderstood.  Is the
> LAN all wired or is there any wireless involved?

Currently the LAN goes to a wireless router (then is double NAT) and it 
is primarily used with wifi. It also has a few ethernet from it 
including to my main workstation (which happens to be over 
ethernet-over-power).

So tried to reproduce problem on a laptop over wifi (second LAN router) 
and no problem.

I bypassed both wifi and ethernet-over-power and had no problem.

I bypassed second LAN router and did have problem over 
ethernet-over-power.

I used a different computer over the second router and over 
ethernet-over-power and did have the problem.

So the problem is over the ethernet-over-power (regardless if it goes 
through second router or not).

The problem is: using shell on the netbsd router is basically locked up, 
not just slow but entirely unusable, for the same client over 
ethernet-over-power that is also doing some downloads/uploads.

I have had a ethernet-over-power fail before and I replaced 
it.

I am confused by some things:

- I didn't see the problem before my change to NetBSD as the router. 
Maybe I didn't use it enough before to notice it (but had used it over 
5+ years).  Maybe ethernet-over-power just started failing recent so was 
a coincidence.

- While the ssh hangs (even on different port), I can use echo (7/tcp), 
chargen (19/tcp), and telnet services fine from the same client to 
the same system.  (sshd on different port and via inetd does not work.)

- I don't understand why even though my same client cannot use the 
NetBSD router's shell, I can route through it fine and use outside shell 
fine at same time.

I will replace the ethernet-over-power, but I wonder still how I can 
tune my NetBSD router so I can use ssh to it. Maybe some quality of 
service configuration. Meanwhile I can use telnet to use my router :(

When unusable, pings to it from same client are like
64 bytes from 172.16.1.1: icmp_seq=32 ttl=254 time=426 ms
64 bytes from 172.16.1.1: icmp_seq=33 ttl=254 time=401 ms
64 bytes from 172.16.1.1: icmp_seq=34 ttl=254 time=329 ms
64 bytes from 172.16.1.1: icmp_seq=35 ttl=254 time=386 ms

While normal is:
64 bytes from 172.16.1.1: icmp_seq=36 ttl=254 time=4.08 ms
64 bytes from 172.16.1.1: icmp_seq=37 ttl=254 time=3.37 ms

Any ideas why telnet works slowly but ssh does not at all in these 
cases? telnet is usable but cannot even see one character sent over ssh 
when ssh locks up (again it restores about 5 to 10 seconds after I stop 
or suspend a speedtest or rsync job).