kern/55534: ixg(4) does not recognize link-up state at boot time

[clare%csel.org@localhost]

>Number:         55534
>Category:       kern
>Synopsis:       ixg(4) does not recognize link-up state at boot time
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Aug 03 12:00:00 +0000 2020
>Originator:     Shinichi Doyashiki
>Release:        NetBSD 9.99.69
>Organization:
	at home
>Environment:
System: NetBSD esperia.nas.csel.org 9.99.69 NetBSD 9.99.69 (ESPERIA) #1: Mon Aug  3 19:06:56 JST 2020  clare%esperia.nas.csel.org@localhost:/export/tmp/current/src/sys/arch/amd64/compile/ESPERIA amd64
Machine: amd64
>Description:
  the kernel as of version 9.99.69 with ixg(4) driver does not recognize link-up
  state of the SFP+ module attached to the Intel 82599 at boot time, in my hardware
  environment.

[     1.034623] ppb0: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x8 @ 8.0GT/s
[     1.034623] ppb0: link is x8 @ 5.0GT/s
[     1.034623] pci1 at ppb0 bus 1
[     1.034623] pci1: i/o space, memory space enabled, rd/line, wr/inv ok
[     1.034623] ixg0 at pci1 dev 0 function 0: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 4.0.1-k
[     1.034623] ixg0: clearing prefetchable bit
[     1.034623] ixg0: device 82599EB
[     1.034623] ixg0: ETrackID 800003df
[     1.034623] ixg0: for TX/RX, interrupting at msix0 vec 0, bound queue 0 to cpu 0
[     1.034623] ixg0: for TX/RX, interrupting at msix0 vec 1, bound queue 1 to cpu 1
[     1.034623] ixg0: for TX/RX, interrupting at msix0 vec 2, bound queue 2 to cpu 2
[     1.034623] ixg0: for TX/RX, interrupting at msix0 vec 3, bound queue 3 to cpu 3
[     1.034623] ixg0: for TX/RX, interrupting at msix0 vec 4, bound queue 4 to cpu 4
[     1.034623] ixg0: for TX/RX, interrupting at msix0 vec 5, bound queue 5 to cpu 5
[     1.034623] ixg0: for TX/RX, interrupting at msix0 vec 6, bound queue 6 to cpu 6
[     1.034623] ixg0: for TX/RX, interrupting at msix0 vec 7, bound queue 7 to cpu 7
[     1.034623] ixg0: for link, interrupting at msix0 vec 8, affinity to cpu 0
[     1.034623] ixg0: Using MSI-X interrupts with 9 vectors
[     1.034623] ixg0: Ethernet address 80:61:5f:**:**:**
[     1.034623] ixg0: PCI Express Bus: Speed 5.0GT/s Width x8
[     1.034623] ixg0: feature cap 0x1780<LEGACY_TX,FDIR,MSI,MSIX,LEGACY_IRQ>
[     1.034623] ixg0: feature ena 0x400<MSIX>
[     1.034623] ixg1 at pci1 dev 0 function 1: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 4.0.1-k
[     1.034623] ixg1: clearing prefetchable bit
[     1.034623] ixg1: device 82599EB
[     1.034623] ixg1: ETrackID 800003df
[     1.034623] ixg1: for TX/RX, interrupting at msix1 vec 0, bound queue 0 to cpu 0
[     1.034623] ixg1: for TX/RX, interrupting at msix1 vec 1, bound queue 1 to cpu 1
[     1.034623] ixg1: for TX/RX, interrupting at msix1 vec 2, bound queue 2 to cpu 2
[     1.034623] ixg1: for TX/RX, interrupting at msix1 vec 3, bound queue 3 to cpu 3
[     1.034623] ixg1: for TX/RX, interrupting at msix1 vec 4, bound queue 4 to cpu 4
[     1.034623] ixg1: for TX/RX, interrupting at msix1 vec 5, bound queue 5 to cpu 5
[     1.034623] ixg1: for TX/RX, interrupting at msix1 vec 6, bound queue 6 to cpu 6
[     1.034623] ixg1: for TX/RX, interrupting at msix1 vec 7, bound queue 7 to cpu 7
[     1.034623] ixg1: for link, interrupting at msix1 vec 8, affinity to cpu 0
[     1.034623] ixg1: Using MSI-X interrupts with 9 vectors
[     1.034623] ixg1: Ethernet address 80:61:5f:**:**:**
[     1.034623] ixg1: PCI Express Bus: Speed 5.0GT/s Width x8
[     1.034623] ixg1: feature cap 0x1780<LEGACY_TX,FDIR,MSI,MSIX,LEGACY_IRQ>
[     1.034623] ixg1: feature ena 0x400<MSIX>


  hot-plugging of the SFP+ module after boot was correctly recognized by the
  new kernel, and it was okay.

  plugged in the SFP+ module before boot (bad case):

ixg0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 9000
	capabilities=7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>
	capabilities=7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
	capabilities=7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
	enabled=0
	ec_capabilities=f<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWFILTER>
	ec_enabled=7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>
	address: 80:61:5f:**:**:**
	media: Ethernet autoselect (none)
	status: no carrier
	inet6 fe80::****:**ff:fe**:****%ixg0/64 flags 0x8<DETACHED> scopeid 0x1


  plugged in the SFP+ module after boot (good case):

ixg0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 9000
        capabilities=7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>
        capabilities=7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
        capabilities=7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
        enabled=0
        ec_capabilities=f<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWFILTER>
        ec_enabled=7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>
        address: 80:61:5f:**:**:**
        media: Ethernet autoselect (10Gbase-Twinax full-duplex,rxpause,txpause)
        status: active
        inet6 fe80::****:**ff:fe**:****%ixg0/64 flags 0x0 scopeid 0x1


>How-To-Repeat:
1) install kernel with ixg(4) and the Intel 82599 NIC with
   SFP+ module(s).

2) configure /etc/ifconfig.ixg0 as usual (for boot time):
        mtu 9000
	up

3) re-boot the kernel.

4) run ifconfig(8) to review the link status.
	# ifconfig ixg0


>Fix:
  unknown yet.