ixg wierdness

[Patrick Welche <prlw1%cam.ac.uk@localhost>]

On a box with 4 bnx and 4 ixg interfaces, I just hit PR kern/53155
when trying to use bnx1. (Built LOCKDEBUG etc kernel, with serial
console. Hang such that ~# doesn't drop into ddb) No problems
running as an NFS server for a year or two just using bnx0.
(I didn't try "up"ing bnx2)

So I tried swapping to use ixg0 and ixg1 instead.

I see a strange bursty pattern with what looks like a 1s count down, e.g.:

64 bytes from 10.0.0.236: icmp_seq=642 ttl=255 time=37004.721972 ms
64 bytes from 10.0.0.236: icmp_seq=643 ttl=255 time=36004.533428 ms
64 bytes from 10.0.0.236: icmp_seq=644 ttl=255 time=35004.224479 ms
64 bytes from 10.0.0.236: icmp_seq=645 ttl=255 time=34003.925027 ms
64 bytes from 10.0.0.236: icmp_seq=646 ttl=255 time=33003.615239 ms
64 bytes from 10.0.0.236: icmp_seq=647 ttl=255 time=32003.313832 ms
64 bytes from 10.0.0.236: icmp_seq=648 ttl=255 time=31003.008233 ms
64 bytes from 10.0.0.236: icmp_seq=649 ttl=255 time=30002.702356 ms
64 bytes from 10.0.0.236: icmp_seq=650 ttl=255 time=29002.396480 ms
64 bytes from 10.0.0.236: icmp_seq=651 ttl=255 time=28002.090882 ms
64 bytes from 10.0.0.236: icmp_seq=652 ttl=255 time=27001.772992 ms
64 bytes from 10.0.0.236: icmp_seq=653 ttl=255 time=26001.477731 ms
64 bytes from 10.0.0.236: icmp_seq=654 ttl=255 time=25001.291421 ms
64 bytes from 10.0.0.236: icmp_seq=655 ttl=255 time=24000.965150 ms
64 bytes from 10.0.0.236: icmp_seq=656 ttl=255 time=23000.622398 ms
64 bytes from 10.0.0.236: icmp_seq=657 ttl=255 time=22000.278807 ms
64 bytes from 10.0.0.236: icmp_seq=658 ttl=255 time=20999.931305 ms
64 bytes from 10.0.0.236: icmp_seq=659 ttl=255 time=19999.592463 ms
64 bytes from 10.0.0.236: icmp_seq=660 ttl=255 time=19009.253137 ms
64 bytes from 10.0.0.236: icmp_seq=661 ttl=255 time=18008.910105 ms
64 bytes from 10.0.0.236: icmp_seq=662 ttl=255 time=17008.551987 ms
64 bytes from 10.0.0.236: icmp_seq=663 ttl=255 time=16008.224040 ms
64 bytes from 10.0.0.236: icmp_seq=664 ttl=255 time=15007.874862 ms
64 bytes from 10.0.0.236: icmp_seq=665 ttl=255 time=14007.533506 ms
64 bytes from 10.0.0.236: icmp_seq=666 ttl=255 time=13007.194943 ms
64 bytes from 10.0.0.236: icmp_seq=667 ttl=255 time=12006.852469 ms
64 bytes from 10.0.0.236: icmp_seq=668 ttl=255 time=11006.509437 ms
64 bytes from 10.0.0.236: icmp_seq=669 ttl=255 time=10006.193223 ms
64 bytes from 10.0.0.236: icmp_seq=670 ttl=255 time=9005.846559 ms
64 bytes from 10.0.0.236: icmp_seq=671 ttl=255 time=8005.508556 ms
64 bytes from 10.0.0.236: icmp_seq=672 ttl=255 time=7005.165803 ms
64 bytes from 10.0.0.236: icmp_seq=673 ttl=255 time=6004.818579 ms
64 bytes from 10.0.0.236: icmp_seq=674 ttl=255 time=5004.479458 ms
64 bytes from 10.0.0.236: icmp_seq=675 ttl=255 time=4004.132514 ms
64 bytes from 10.0.0.236: icmp_seq=676 ttl=255 time=3003.794232 ms
64 bytes from 10.0.0.236: icmp_seq=677 ttl=255 time=2003.431084 ms
64 bytes from 10.0.0.236: icmp_seq=678 ttl=255 time=1003.103697 ms
64 bytes from 10.0.0.236: icmp_seq=679 ttl=255 time=2.761223 ms
64 bytes from 10.0.0.236: icmp_seq=717 ttl=255 time=6373.442427 ms
64 bytes from 10.0.0.236: icmp_seq=718 ttl=255 time=5373.238237 ms
64 bytes from 10.0.0.236: icmp_seq=719 ttl=255 time=4372.937388 ms
64 bytes from 10.0.0.236: icmp_seq=720 ttl=255 time=3372.631791 ms
64 bytes from 10.0.0.236: icmp_seq=721 ttl=255 time=2372.325913 ms
64 bytes from 10.0.0.236: icmp_seq=722 ttl=255 time=1372.006627 ms
64 bytes from 10.0.0.236: icmp_seq=723 ttl=255 time=371.714159 ms
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
...
then eventually wakes up again

when pinging to its ixg0 interface.

You see the bursts while running tcpdump -ni ixg0.

ixg0 at pci8 dev 0 function 0: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 4.0.1-k
ixg0: device 82599EB
ixg0: ETrackID 800001a5
ixg0: autoconfiguration error: failed to allocate MSI-X interrupt
ixg0: interrupting at ioapic1 pin 22
ixg0: Ethernet address 00:1b:21:9a:d4:84
ixg0: PHY: OUI 0x0014a6 model 0x0001, rev. 0
ixg0: PCI Express Bus: Speed 5.0GT/s Width x8
ixg0: feature cap 0x1780<LEGACY_TX,FDIR,MSI,MSIX,LEGACY_IRQ>
ixg0: feature ena 0x1000<LEGACY_IRQ>

ixg0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        capabilities=0x7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>
        capabilities=0x7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
        capabilities=0x7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
        enabled=0
        ec_capabilities=0xf<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWFILTER>
        ec_enabled=0x7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>
        address: 00:1b:21:9a:d4:84
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active
        inet6 fe80::21b:21ff:fe9a:d484%ixg0/64 flags 0 scopeid 0x5
        inet 10.0.0.236/24 broadcast 10.0.0.255 flags 0

This is with 15 December 2021 -current/amd64.

Any ideas on what might be going on?


Cheers,

Patrick