Current-Users archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: ixg wierdness

On 2021/12/22 9:38, SAITOH Masanobu wrote:

Hi.

On 2021/12/21 19:53, Patrick Welche wrote:

On a box with 4 bnx and 4 ixg interfaces, I just hit PR kern/53155
when trying to use bnx1. (Built LOCKDEBUG etc kernel, with serial
console. Hang such that ~# doesn't drop into ddb) No problems
running as an NFS server for a year or two just using bnx0.
(I didn't try "up"ing bnx2)

So I tried swapping to use ixg0 and ixg1 instead.

I see a strange bursty pattern with what looks like a 1s count down, e.g.:

64 bytes from 10.0.0.236: icmp_seq=642 ttl=255 time=37004.721972 ms
64 bytes from 10.0.0.236: icmp_seq=643 ttl=255 time=36004.533428 ms
64 bytes from 10.0.0.236: icmp_seq=644 ttl=255 time=35004.224479 ms
64 bytes from 10.0.0.236: icmp_seq=645 ttl=255 time=34003.925027 ms
64 bytes from 10.0.0.236: icmp_seq=646 ttl=255 time=33003.615239 ms
64 bytes from 10.0.0.236: icmp_seq=647 ttl=255 time=32003.313832 ms
64 bytes from 10.0.0.236: icmp_seq=648 ttl=255 time=31003.008233 ms
64 bytes from 10.0.0.236: icmp_seq=649 ttl=255 time=30002.702356 ms
64 bytes from 10.0.0.236: icmp_seq=650 ttl=255 time=29002.396480 ms
64 bytes from 10.0.0.236: icmp_seq=651 ttl=255 time=28002.090882 ms
64 bytes from 10.0.0.236: icmp_seq=652 ttl=255 time=27001.772992 ms
64 bytes from 10.0.0.236: icmp_seq=653 ttl=255 time=26001.477731 ms
64 bytes from 10.0.0.236: icmp_seq=654 ttl=255 time=25001.291421 ms
64 bytes from 10.0.0.236: icmp_seq=655 ttl=255 time=24000.965150 ms
64 bytes from 10.0.0.236: icmp_seq=656 ttl=255 time=23000.622398 ms
64 bytes from 10.0.0.236: icmp_seq=657 ttl=255 time=22000.278807 ms
64 bytes from 10.0.0.236: icmp_seq=658 ttl=255 time=20999.931305 ms
64 bytes from 10.0.0.236: icmp_seq=659 ttl=255 time=19999.592463 ms
64 bytes from 10.0.0.236: icmp_seq=660 ttl=255 time=19009.253137 ms
64 bytes from 10.0.0.236: icmp_seq=661 ttl=255 time=18008.910105 ms
64 bytes from 10.0.0.236: icmp_seq=662 ttl=255 time=17008.551987 ms
64 bytes from 10.0.0.236: icmp_seq=663 ttl=255 time=16008.224040 ms
64 bytes from 10.0.0.236: icmp_seq=664 ttl=255 time=15007.874862 ms
64 bytes from 10.0.0.236: icmp_seq=665 ttl=255 time=14007.533506 ms
64 bytes from 10.0.0.236: icmp_seq=666 ttl=255 time=13007.194943 ms
64 bytes from 10.0.0.236: icmp_seq=667 ttl=255 time=12006.852469 ms
64 bytes from 10.0.0.236: icmp_seq=668 ttl=255 time=11006.509437 ms
64 bytes from 10.0.0.236: icmp_seq=669 ttl=255 time=10006.193223 ms
64 bytes from 10.0.0.236: icmp_seq=670 ttl=255 time=9005.846559 ms
64 bytes from 10.0.0.236: icmp_seq=671 ttl=255 time=8005.508556 ms
64 bytes from 10.0.0.236: icmp_seq=672 ttl=255 time=7005.165803 ms
64 bytes from 10.0.0.236: icmp_seq=673 ttl=255 time=6004.818579 ms
64 bytes from 10.0.0.236: icmp_seq=674 ttl=255 time=5004.479458 ms
64 bytes from 10.0.0.236: icmp_seq=675 ttl=255 time=4004.132514 ms
64 bytes from 10.0.0.236: icmp_seq=676 ttl=255 time=3003.794232 ms
64 bytes from 10.0.0.236: icmp_seq=677 ttl=255 time=2003.431084 ms
64 bytes from 10.0.0.236: icmp_seq=678 ttl=255 time=1003.103697 ms
64 bytes from 10.0.0.236: icmp_seq=679 ttl=255 time=2.761223 ms
64 bytes from 10.0.0.236: icmp_seq=717 ttl=255 time=6373.442427 ms
64 bytes from 10.0.0.236: icmp_seq=718 ttl=255 time=5373.238237 ms
64 bytes from 10.0.0.236: icmp_seq=719 ttl=255 time=4372.937388 ms
64 bytes from 10.0.0.236: icmp_seq=720 ttl=255 time=3372.631791 ms
64 bytes from 10.0.0.236: icmp_seq=721 ttl=255 time=2372.325913 ms
64 bytes from 10.0.0.236: icmp_seq=722 ttl=255 time=1372.006627 ms
64 bytes from 10.0.0.236: icmp_seq=723 ttl=255 time=371.714159 ms
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
...
then eventually wakes up again

when pinging to its ixg0 interface.

You see the bursts while running tcpdump -ni ixg0.

ixg0 at pci8 dev 0 function 0: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 4.0.1-k
ixg0: device 82599EB
ixg0: ETrackID 800001a5
ixg0: autoconfiguration error: failed to allocate MSI-X interrupt
ixg0: interrupting at ioapic1 pin 22
ixg0: Ethernet address 00:1b:21:9a:d4:84
ixg0: PHY: OUI 0x0014a6 model 0x0001, rev. 0
ixg0: PCI Express Bus: Speed 5.0GT/s Width x8
ixg0: feature cap 0x1780<LEGACY_TX,FDIR,MSI,MSIX,LEGACY_IRQ>
ixg0: feature ena 0x1000<LEGACY_IRQ>

ixg0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
         capabilities=0x7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>
         capabilities=0x7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
         capabilities=0x7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
         enabled=0
         ec_capabilities=0xf<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWFILTER>
         ec_enabled=0x7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>
         address: 00:1b:21:9a:d4:84
         media: Ethernet autoselect (1000baseT full-duplex)
         status: active
         inet6 fe80::21b:21ff:fe9a:d484%ixg0/64 flags 0 scopeid 0x5
         inet 10.0.0.236/24 broadcast 10.0.0.255 flags 0

This is with 15 December 2021 -current/amd64.

Any ideas on what might be going on?


Cheers,

Patrick


One of the possibility of the problem is the shortage
of mbuf cluster. Could you show me the output of netstat -m?


% netstat -m
9222 mbufs in use:
        9217 mbufs allocated to data
        3 mbufs allocated to packet headers
        2 mbufs allocated to socket names and addresses
0 calls to protocol drain routines


If the last line's number is not 0, increase kern.mbuf.nmbclusters.


% vmstat -ev | grep "Rx no mbuf"
ixg0 q0 Rx no mbuf                                            0    0 misc
ixg0 q1 Rx no mbuf                                            0    0 misc
ixg0 q2 Rx no mbuf                                            0    0 misc
ixg0 q3 Rx no mbuf                                            0    0 misc


When MCLGET() failed, the above evcnt also increased.


ixg0: autoconfiguration error: failed to allocate MSI-X interrupt
ixg0: interrupting at ioapic1 pin 22
ixg0: Ethernet address 00:1b:21:9a:d4:84
ixg0: PHY: OUI 0x0014a6 model 0x0001, rev. 0
ixg0: PCI Express Bus: Speed 5.0GT/s Width x8
ixg0: feature cap 0x1780<LEGACY_TX,FDIR,MSI,MSIX,LEGACY_IRQ>
ixg0: feature ena 0x1000<LEGACY_IRQ>


The allocation of an MSI-X vector failed and it uses INTx.
Could you show me the full dmesg?

And, please show me the output of "vmstat -ev | grep ixg".
I will take a look of it.



Plus,
	cpuctl list
	intrctl list
	diff if you modified GENERIC kernel

If the machine is the same as kern/53155 I don't know why
the MSI-X allocation fails. If the fallback code in ixgbe.c
has a bug, it would worth to try the following diff:

Index: ixgbe.c
===================================================================
RCS file: /cvsroot/src/sys/dev/pci/ixgbe/ixgbe.c,v
retrieving revision 1.300
diff -u -p -r1.300 ixgbe.c
--- ixgbe.c     16 Dec 2021 10:48:49 -0000      1.300
+++ ixgbe.c     22 Dec 2021 03:03:54 -0000
@@ -350,7 +350,7 @@ static int ixgbe_smart_speed = ixgbe_sma
  * MSI-X should be the default for best performance,
  * but this allows it to be forced off for testing.
  */
-static int ixgbe_enable_msix = 1;
+static int ixgbe_enable_msix = 0;
 SYSCTL_INT(_hw_ix, OID_AUTO, enable_msix, CTLFLAG_RDTUN, &ixgbe_enable_msix, 0,
     "Enable MSI-X interrupts");


Another possibility is that someone doing splhigh() for long time.

--
-----------------------------------------------
                SAITOH Masanobu (msaitoh%execsw.org@localhost
                                 msaitoh%netbsd.org@localhost)
Home | Main Index | Thread Index | Old Index