Re: ixg wierdness

[SAITOH Masanobu <msaitoh%execsw.org@localhost>]

Hi.

On 2021/12/21 19:53, Patrick Welche wrote:
> On a box with 4 bnx and 4 ixg interfaces, I just hit PR kern/53155
> when trying to use bnx1. (Built LOCKDEBUG etc kernel, with serial
> console. Hang such that ~# doesn't drop into ddb) No problems
> running as an NFS server for a year or two just using bnx0.
> (I didn't try "up"ing bnx2)
>
> So I tried swapping to use ixg0 and ixg1 instead.
>
> I see a strange bursty pattern with what looks like a 1s count down, e.g.:
>
> 64 bytes from 10.0.0.236: icmp_seq=642 ttl=255 time=37004.721972 ms
> 64 bytes from 10.0.0.236: icmp_seq=643 ttl=255 time=36004.533428 ms
> 64 bytes from 10.0.0.236: icmp_seq=644 ttl=255 time=35004.224479 ms
> 64 bytes from 10.0.0.236: icmp_seq=645 ttl=255 time=34003.925027 ms
> 64 bytes from 10.0.0.236: icmp_seq=646 ttl=255 time=33003.615239 ms
> 64 bytes from 10.0.0.236: icmp_seq=647 ttl=255 time=32003.313832 ms
> 64 bytes from 10.0.0.236: icmp_seq=648 ttl=255 time=31003.008233 ms
> 64 bytes from 10.0.0.236: icmp_seq=649 ttl=255 time=30002.702356 ms
> 64 bytes from 10.0.0.236: icmp_seq=650 ttl=255 time=29002.396480 ms
> 64 bytes from 10.0.0.236: icmp_seq=651 ttl=255 time=28002.090882 ms
> 64 bytes from 10.0.0.236: icmp_seq=652 ttl=255 time=27001.772992 ms
> 64 bytes from 10.0.0.236: icmp_seq=653 ttl=255 time=26001.477731 ms
> 64 bytes from 10.0.0.236: icmp_seq=654 ttl=255 time=25001.291421 ms
> 64 bytes from 10.0.0.236: icmp_seq=655 ttl=255 time=24000.965150 ms
> 64 bytes from 10.0.0.236: icmp_seq=656 ttl=255 time=23000.622398 ms
> 64 bytes from 10.0.0.236: icmp_seq=657 ttl=255 time=22000.278807 ms
> 64 bytes from 10.0.0.236: icmp_seq=658 ttl=255 time=20999.931305 ms
> 64 bytes from 10.0.0.236: icmp_seq=659 ttl=255 time=19999.592463 ms
> 64 bytes from 10.0.0.236: icmp_seq=660 ttl=255 time=19009.253137 ms
> 64 bytes from 10.0.0.236: icmp_seq=661 ttl=255 time=18008.910105 ms
> 64 bytes from 10.0.0.236: icmp_seq=662 ttl=255 time=17008.551987 ms
> 64 bytes from 10.0.0.236: icmp_seq=663 ttl=255 time=16008.224040 ms
> 64 bytes from 10.0.0.236: icmp_seq=664 ttl=255 time=15007.874862 ms
> 64 bytes from 10.0.0.236: icmp_seq=665 ttl=255 time=14007.533506 ms
> 64 bytes from 10.0.0.236: icmp_seq=666 ttl=255 time=13007.194943 ms
> 64 bytes from 10.0.0.236: icmp_seq=667 ttl=255 time=12006.852469 ms
> 64 bytes from 10.0.0.236: icmp_seq=668 ttl=255 time=11006.509437 ms
> 64 bytes from 10.0.0.236: icmp_seq=669 ttl=255 time=10006.193223 ms
> 64 bytes from 10.0.0.236: icmp_seq=670 ttl=255 time=9005.846559 ms
> 64 bytes from 10.0.0.236: icmp_seq=671 ttl=255 time=8005.508556 ms
> 64 bytes from 10.0.0.236: icmp_seq=672 ttl=255 time=7005.165803 ms
> 64 bytes from 10.0.0.236: icmp_seq=673 ttl=255 time=6004.818579 ms
> 64 bytes from 10.0.0.236: icmp_seq=674 ttl=255 time=5004.479458 ms
> 64 bytes from 10.0.0.236: icmp_seq=675 ttl=255 time=4004.132514 ms
> 64 bytes from 10.0.0.236: icmp_seq=676 ttl=255 time=3003.794232 ms
> 64 bytes from 10.0.0.236: icmp_seq=677 ttl=255 time=2003.431084 ms
> 64 bytes from 10.0.0.236: icmp_seq=678 ttl=255 time=1003.103697 ms
> 64 bytes from 10.0.0.236: icmp_seq=679 ttl=255 time=2.761223 ms
> 64 bytes from 10.0.0.236: icmp_seq=717 ttl=255 time=6373.442427 ms
> 64 bytes from 10.0.0.236: icmp_seq=718 ttl=255 time=5373.238237 ms
> 64 bytes from 10.0.0.236: icmp_seq=719 ttl=255 time=4372.937388 ms
> 64 bytes from 10.0.0.236: icmp_seq=720 ttl=255 time=3372.631791 ms
> 64 bytes from 10.0.0.236: icmp_seq=721 ttl=255 time=2372.325913 ms
> 64 bytes from 10.0.0.236: icmp_seq=722 ttl=255 time=1372.006627 ms
> 64 bytes from 10.0.0.236: icmp_seq=723 ttl=255 time=371.714159 ms
> ping: sendto: Host is down
> ping: sendto: Host is down
> ping: sendto: Host is down
> ping: sendto: Host is down
> ping: sendto: Host is down
> ...
> then eventually wakes up again
>
> when pinging to its ixg0 interface.
>
> You see the bursts while running tcpdump -ni ixg0.
>
> ixg0 at pci8 dev 0 function 0: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 4.0.1-k
> ixg0: device 82599EB
> ixg0: ETrackID 800001a5
> ixg0: autoconfiguration error: failed to allocate MSI-X interrupt
> ixg0: interrupting at ioapic1 pin 22
> ixg0: Ethernet address 00:1b:21:9a:d4:84
> ixg0: PHY: OUI 0x0014a6 model 0x0001, rev. 0
> ixg0: PCI Express Bus: Speed 5.0GT/s Width x8
> ixg0: feature cap 0x1780<LEGACY_TX,FDIR,MSI,MSIX,LEGACY_IRQ>
> ixg0: feature ena 0x1000<LEGACY_IRQ>
>
> ixg0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>          capabilities=0x7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>
>          capabilities=0x7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
>          capabilities=0x7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
>          enabled=0
>          ec_capabilities=0xf<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWFILTER>
>          ec_enabled=0x7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>
>          address: 00:1b:21:9a:d4:84
>          media: Ethernet autoselect (1000baseT full-duplex)
>          status: active
>          inet6 fe80::21b:21ff:fe9a:d484%ixg0/64 flags 0 scopeid 0x5
>          inet 10.0.0.236/24 broadcast 10.0.0.255 flags 0
>
> This is with 15 December 2021 -current/amd64.
>
> Any ideas on what might be going on?
>
>
> Cheers,
>
> Patrick

One of the possibility of the problem is the shortage
of mbuf cluster. Could you show me the output of netstat -m?

> % netstat -m
> 9222 mbufs in use:
>         9217 mbufs allocated to data
>         3 mbufs allocated to packet headers
>         2 mbufs allocated to socket names and addresses
> 0 calls to protocol drain routines

If the last line's number is not 0, increase kern.mbuf.nmbclusters.

> % vmstat -ev | grep "Rx no mbuf"
> ixg0 q0 Rx no mbuf                                            0    0 misc
> ixg0 q1 Rx no mbuf                                            0    0 misc
> ixg0 q2 Rx no mbuf                                            0    0 misc
> ixg0 q3 Rx no mbuf                                            0    0 misc

When MCLGET() failed, the above evcnt also increased.

> > ixg0: autoconfiguration error: failed to allocate MSI-X interrupt
> > ixg0: interrupting at ioapic1 pin 22
> > ixg0: Ethernet address 00:1b:21:9a:d4:84
> > ixg0: PHY: OUI 0x0014a6 model 0x0001, rev. 0
> > ixg0: PCI Express Bus: Speed 5.0GT/s Width x8
> > ixg0: feature cap 0x1780<LEGACY_TX,FDIR,MSI,MSIX,LEGACY_IRQ>
> > ixg0: feature ena 0x1000<LEGACY_IRQ>

The allocation of an MSI-X vector failed and it uses INTx.
Could you show me the full dmesg?

And, please show me the output of "vmstat -ev | grep ixg".
I will take a look of it.


--
-----------------------------------------------
                SAITOH Masanobu (msaitoh%execsw.org@localhost
                                 msaitoh%netbsd.org@localhost)