Re: kern/54837: ixl* + wm* interrupt allocation fails

[Masanobu SAITOH <msaitoh%execsw.org@localhost>]

The following reply was made to PR kern/54837; it has been noted by GNATS.

From: Masanobu SAITOH <msaitoh%execsw.org@localhost>
To: gnats-bugs%netbsd.org@localhost, kern-bug-people%netbsd.org@localhost,
 gnats-admin%netbsd.org@localhost, netbsd-bugs%netbsd.org@localhost
Cc: msaitoh%execsw.org@localhost
Subject: Re: kern/54837: ixl* + wm* interrupt allocation fails
Date: Mon, 6 Jan 2020 18:23:55 +0900

 On 2020/01/06 17:45, kardel%netbsd.org@localhost wrote:
 >> Number:         54837
 >> Category:       kern
 >> Synopsis:       ixl* + wm* interrupt allocation fails
 >> Confidential:   no
 >> Severity:       serious
 >> Priority:       high
 >> Responsible:    kern-bug-people
 >> State:          open
 >> Class:          sw-bug
 >> Submitter-Id:   net
 >> Arrival-Date:   Mon Jan 06 08:45:00 +0000 2020
 >> Originator:     Frank Kardel
 >> Release:        NetBSD 9.99.33
 >> Organization:
 > 	
 >> Environment:
 > 	
 > 	
 > System: NetBSD abstest2 9.99.33 NetBSD 9.99.33 (ABST2GEN) #0: Fri Jan 3 16:14:02 CET 2020 kardel@:/src/NetBSD/cur/src/obj.amd64/sys/arch/amd64/compile/ABST2GEN amd64
 > Architecture: x86_64
 > Machine: amd64
 >> Description:
 > 	On a system with Intel X722 10GbaseT and Intel I350 Gigabit interfaces the wm* interfaces
 > 	fail allocate interrupts after the X722 interfaces where successfully initialized.
 > 
 > dmesg:
 > [...]
 > pci5 at mainbus0 bus 58
 > pci5: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
 > ppb3 at pci5 dev 0 function 0: Intel product 2030 (rev. 0x04)
 > ppb3: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x16 @ 8.0GT/s
 > pci6 at ppb3 bus 59
 > pci6: i/o space, memory space enabled
 > ppb4 at pci6 dev 0 function 0: Intel C620 PCIe x16 Uplink (NPX16) (rev. 0x09)
 > ppb4: PCI Express capability version 2 <Upstream Port of PCI-E Switch>
 > pci7 at ppb4 bus 60
 > pci7: i/o space, memory space enabled
 > ppb5 at pci7 dev 3 function 0: Intel C620 Virtual Switch Port (for 10GbE LAN) (rev. 0x09)
 > ppb5: PCI Express capability version 2 <Downstream Port of PCI-E Switch> x1 @ 2.5GT/s
 > pci8 at ppb5 bus 61
 > pci8: i/o space, memory space enabled
 > ixl0 at pci8 dev 0 function 0: port 1, FW 3.1.55727 API 1.5
 > ixl0: Ethernet address ...
 > ixl0: interrupt at msix4 vec 0
 > ixl0: interrupt at msix4 vec 1
 > ixl0: interrupt at msix4 vec 2
 > ixl0: interrupt at msix4 vec 3
 > ixl0: interrupt at msix4 vec 4
 > allocated pic msix4 type edge pin 5 level 6 to cpu1 slot 0 idt entry 107
 > ixl0: interrupt at msix4 vec 5
 > allocated pic msix4 type edge pin 6 level 6 to cpu1 slot 1 idt entry 108
 > ixl0: interrupt at msix4 vec 6
 > allocated pic msix4 type edge pin 7 level 6 to cpu1 slot 2 idt entry 109
 > ixl0: interrupt at msix4 vec 7
 > allocated pic msix4 type edge pin 8 level 6 to cpu1 slot 3 idt entry 110
 > ixl0: interrupt at msix4 vec 8
 > allocated pic msix4 type edge pin 9 level 6 to cpu1 slot 4 idt entry 111
 > ixl0: interrupt at msix4 vec 9
 > allocated pic msix4 type edge pin 10 level 6 to cpu1 slot 5 idt entry 112
 > ixl0: interrupt at msix4 vec 10
 > allocated pic msix4 type edge pin 11 level 6 to cpu1 slot 6 idt entry 113
 > ixl0: interrupt at msix4 vec 11
 > allocated pic msix4 type edge pin 12 level 6 to cpu1 slot 7 idt entry 114
 > ixl0: interrupt at msix4 vec 12
 > allocated pic msix4 type edge pin 13 level 6 to cpu1 slot 8 idt entry 115
 > ixl0: interrupt at msix4 vec 13
 > allocated pic msix4 type edge pin 14 level 6 to cpu1 slot 9 idt entry 116
 > ixl0: interrupt at msix4 vec 14
 > allocated pic msix4 type edge pin 15 level 6 to cpu1 slot 10 idt entry 117
 > ixl0: interrupt at msix4 vec 15
 > allocated pic msix4 type edge pin 16 level 6 to cpu1 slot 11 idt entry 118
 > ixl0: interrupt at msix4 vec 16
 > allocated pic msix4 type edge pin 17 level 6 to cpu1 slot 12 idt entry 119
 > ixl0: interrupt at msix4 vec 17
 > allocated pic msix4 type edge pin 18 level 6 to cpu1 slot 13 idt entry 120
 > ixl0: interrupt at msix4 vec 18
 > allocated pic msix4 type edge pin 19 level 6 to cpu1 slot 14 idt entry 121
 > ixl0: interrupt at msix4 vec 19
 > allocated pic msix4 type edge pin 20 level 6 to cpu1 slot 15 idt entry 122
 > ixl0: interrupt at msix4 vec 20
 > allocated pic msix4 type edge pin 21 level 6 to cpu1 slot 16 idt entry 123
 > ixl0: interrupt at msix4 vec 21
 > allocated pic msix4 type edge pin 22 level 6 to cpu1 slot 17 idt entry 124
 > ixl0: interrupt at msix4 vec 22
 > allocated pic msix4 type edge pin 23 level 6 to cpu1 slot 18 idt entry 125
 > ixl0: interrupt at msix4 vec 23
 > allocated pic msix4 type edge pin 24 level 6 to cpu1 slot 19 idt entry 126
 > ixl0: interrupt at msix4 vec 24
 > allocated pic msix4 type edge pin 25 level 6 to cpu1 slot 20 idt entry 127
 > ixl0: interrupt at msix4 vec 25
 > allocated pic msix4 type edge pin 26 level 6 to cpu1 slot 21 idt entry 130
 > ixl0: interrupt at msix4 vec 26
 > allocated pic msix4 type edge pin 27 level 6 to cpu1 slot 22 idt entry 131
 > ixl0: interrupt at msix4 vec 27
 > allocated pic msix4 type edge pin 28 level 6 to cpu1 slot 23 idt entry 132
 > ixl0: interrupt at msix4 vec 28
 > allocated pic msix4 type edge pin 29 level 6 to cpu1 slot 29 idt entry 133
 > ixl0: interrupt at msix4 vec 29
 > allocated pic msix4 type edge pin 30 level 6 to cpu2 slot 0 idt entry 134
 > ixl0: interrupt at msix4 vec 30
 > allocated pic msix4 type edge pin 31 level 6 to cpu2 slot 1 idt entry 135
 > ixl0: interrupt at msix4 vec 31
 > allocated pic msix4 type edge pin 32 level 6 to cpu2 slot 2 idt entry 136
 > ixl0: interrupt at msix4 vec 32
 > ixl0: for TXRX0 interrupting at msix4 vec 1 affinity to 0
 > ixl0: for TXRX1 interrupting at msix4 vec 2
 > ixl0: for TXRX2 interrupting at msix4 vec 3 affinity to 2
 > ixl0: for TXRX3 interrupting at msix4 vec 4 affinity to 3
 > ixl0: for TXRX4 interrupting at msix4 vec 5 affinity to 4
 > ixl0: for TXRX5 interrupting at msix4 vec 6 affinity to 5
 > ixl0: for TXRX6 interrupting at msix4 vec 7 affinity to 6
 > ixl0: for TXRX7 interrupting at msix4 vec 8 affinity to 7
 > ixl0: for TXRX8 interrupting at msix4 vec 9 affinity to 8
 > ixl0: for TXRX9 interrupting at msix4 vec 10 affinity to 9
 > ixl0: for TXRX10 interrupting at msix4 vec 11 affinity to 10
 > ixl0: for TXRX11 interrupting at msix4 vec 12 affinity to 11
 > ixl0: for TXRX12 interrupting at msix4 vec 13 affinity to 12
 > ixl0: for TXRX13 interrupting at msix4 vec 14 affinity to 13
 > ixl0: for TXRX14 interrupting at msix4 vec 15 affinity to 14
 > ixl0: for TXRX15 interrupting at msix4 vec 16 affinity to 15
 > ixl0: for TXRX16 interrupting at msix4 vec 17 affinity to 16
 > ixl0: for TXRX17 interrupting at msix4 vec 18 affinity to 17
 > ixl0: for TXRX18 interrupting at msix4 vec 19 affinity to 18
 > ixl0: for TXRX19 interrupting at msix4 vec 20 affinity to 19
 > ixl0: for TXRX20 interrupting at msix4 vec 21 affinity to 20
 > ixl0: for TXRX21 interrupting at msix4 vec 22 affinity to 21
 > ixl0: for TXRX22 interrupting at msix4 vec 23 affinity to 22
 > ixl0: for TXRX23 interrupting at msix4 vec 24 affinity to 23
 > ixl0: for TXRX24 interrupting at msix4 vec 25 affinity to 24
 > ixl0: for TXRX25 interrupting at msix4 vec 26 affinity to 25
 > ixl0: for TXRX26 interrupting at msix4 vec 27 affinity to 26
 > ixl0: for TXRX27 interrupting at msix4 vec 28 affinity to 27
 > ixl0: for TXRX28 interrupting at msix4 vec 29 affinity to 28
 > ixl0: for TXRX29 interrupting at msix4 vec 30 affinity to 29
 > ixl0: for TXRX30 interrupting at msix4 vec 31 affinity to 30
 > ixl0: for TXRX31 interrupting at msix4 vec 32 affinity to 31
 > ixl0: for other interrupting at msix4 vec 0 affinity to 0
 > ixl1 at pci8 dev 0 function 1: port 0, FW 3.1.55727 API 1.5
 > ixl1: Ethernet address ...
 > ixl1: interrupt at msix5 vec 0
 > ixl1: interrupt at msix5 vec 1
 > allocated pic msix5 type edge pin 2 level 6 to cpu1 slot 0 idt entry 139
 > ixl1: interrupt at msix5 vec 2
 > allocated pic msix5 type edge pin 3 level 6 to cpu1 slot 1 idt entry 140
 > ixl1: interrupt at msix5 vec 3
 > allocated pic msix5 type edge pin 4 level 6 to cpu1 slot 2 idt entry 141
 > ixl1: interrupt at msix5 vec 4
 > allocated pic msix5 type edge pin 5 level 6 to cpu1 slot 3 idt entry 142
 > ixl1: interrupt at msix5 vec 5
 > allocated pic msix5 type edge pin 6 level 6 to cpu1 slot 4 idt entry 143
 > ixl1: interrupt at msix5 vec 6
 > allocated pic msix5 type edge pin 7 level 6 to cpu1 slot 5 idt entry 144
 > ixl1: interrupt at msix5 vec 7
 > allocated pic msix5 type edge pin 8 level 6 to cpu1 slot 6 idt entry 145
 > ixl1: interrupt at msix5 vec 8
 > allocated pic msix5 type edge pin 9 level 6 to cpu1 slot 7 idt entry 146
 > ixl1: interrupt at msix5 vec 9
 > allocated pic msix5 type edge pin 10 level 6 to cpu1 slot 8 idt entry 147
 > ixl1: interrupt at msix5 vec 10
 > allocated pic msix5 type edge pin 11 level 6 to cpu1 slot 9 idt entry 148
 > ixl1: interrupt at msix5 vec 11
 > allocated pic msix5 type edge pin 12 level 6 to cpu1 slot 10 idt entry 149
 > ixl1: interrupt at msix5 vec 12
 > allocated pic msix5 type edge pin 13 level 6 to cpu1 slot 11 idt entry 150
 > ixl1: interrupt at msix5 vec 13
 > allocated pic msix5 type edge pin 14 level 6 to cpu1 slot 12 idt entry 151
 > ixl1: interrupt at msix5 vec 14
 > allocated pic msix5 type edge pin 15 level 6 to cpu1 slot 13 idt entry 152
 > ixl1: interrupt at msix5 vec 15
 > allocated pic msix5 type edge pin 16 level 6 to cpu1 slot 14 idt entry 153
 > ixl1: interrupt at msix5 vec 16
 > allocated pic msix5 type edge pin 17 level 6 to cpu1 slot 15 idt entry 154
 > ixl1: interrupt at msix5 vec 17
 > allocated pic msix5 type edge pin 18 level 6 to cpu1 slot 16 idt entry 155
 > ixl1: interrupt at msix5 vec 18
 > allocated pic msix5 type edge pin 19 level 6 to cpu1 slot 17 idt entry 156
 > ixl1: interrupt at msix5 vec 19
 > allocated pic msix5 type edge pin 20 level 6 to cpu1 slot 18 idt entry 157
 > ixl1: interrupt at msix5 vec 20
 > allocated pic msix5 type edge pin 21 level 6 to cpu1 slot 19 idt entry 158
 > ixl1: interrupt at msix5 vec 21
 > allocated pic msix5 type edge pin 22 level 6 to cpu1 slot 20 idt entry 159
 > ixl1: interrupt at msix5 vec 22
 > allocated pic msix5 type edge pin 23 level 6 to cpu1 slot 21 idt entry 160
 > ixl1: interrupt at msix5 vec 23
 > allocated pic msix5 type edge pin 24 level 6 to cpu1 slot 22 idt entry 161
 > ixl1: interrupt at msix5 vec 24
 > allocated pic msix5 type edge pin 25 level 6 to cpu1 slot 23 idt entry 162
 > ixl1: interrupt at msix5 vec 25
 > allocated pic msix5 type edge pin 26 level 6 to cpu1 slot 29 idt entry 163
 > ixl1: interrupt at msix5 vec 26
 > allocated pic msix5 type edge pin 27 level 6 to cpu2 slot 0 idt entry 164
 > ixl1: interrupt at msix5 vec 27
 > allocated pic msix5 type edge pin 28 level 6 to cpu2 slot 1 idt entry 165
 > ixl1: interrupt at msix5 vec 28
 > allocated pic msix5 type edge pin 29 level 6 to cpu2 slot 2 idt entry 166
 > ixl1: interrupt at msix5 vec 29
 > allocated pic msix5 type edge pin 30 level 6 to cpu2 slot 4 idt entry 167
 > ixl1: interrupt at msix5 vec 30
 > allocated pic msix5 type edge pin 31 level 6 to cpu2 slot 5 idt entry 168
 > ixl1: interrupt at msix5 vec 31
 > allocated pic msix5 type edge pin 32 level 6 to cpu2 slot 6 idt entry 169
 > ixl1: interrupt at msix5 vec 32
 > ixl1: for TXRX0 interrupting at msix5 vec 1 affinity to 0
 > ixl1: for TXRX1 interrupting at msix5 vec 2 affinity to 1
 > ixl1: for TXRX2 interrupting at msix5 vec 3 affinity to 2
 > ixl1: for TXRX3 interrupting at msix5 vec 4 affinity to 3
 > ixl1: for TXRX4 interrupting at msix5 vec 5 affinity to 4
 > ixl1: for TXRX5 interrupting at msix5 vec 6 affinity to 5
 > ixl1: for TXRX6 interrupting at msix5 vec 7 affinity to 6
 > ixl1: for TXRX7 interrupting at msix5 vec 8 affinity to 7
 > ixl1: for TXRX8 interrupting at msix5 vec 9 affinity to 8
 > ixl1: for TXRX9 interrupting at msix5 vec 10 affinity to 9
 > ixl1: for TXRX10 interrupting at msix5 vec 11 affinity to 10
 > ixl1: for TXRX11 interrupting at msix5 vec 12 affinity to 11
 > ixl1: for TXRX12 interrupting at msix5 vec 13 affinity to 12
 > ixl1: for TXRX13 interrupting at msix5 vec 14 affinity to 13
 > ixl1: for TXRX14 interrupting at msix5 vec 15 affinity to 14
 > ixl1: for TXRX15 interrupting at msix5 vec 16 affinity to 15
 > ixl1: for TXRX16 interrupting at msix5 vec 17 affinity to 16
 > ixl1: for TXRX17 interrupting at msix5 vec 18 affinity to 17
 > ixl1: for TXRX18 interrupting at msix5 vec 19 affinity to 18
 > ixl1: for TXRX19 interrupting at msix5 vec 20 affinity to 19
 > ixl1: for TXRX20 interrupting at msix5 vec 21 affinity to 20
 > ixl1: for TXRX21 interrupting at msix5 vec 22 affinity to 21
 > ixl1: for TXRX22 interrupting at msix5 vec 23 affinity to 22
 > ixl1: for TXRX23 interrupting at msix5 vec 24 affinity to 23
 > ixl1: for TXRX24 interrupting at msix5 vec 25 affinity to 24
 > ixl1: for TXRX25 interrupting at msix5 vec 26 affinity to 25
 > ixl1: for TXRX26 interrupting at msix5 vec 27 affinity to 26
 > ixl1: for TXRX27 interrupting at msix5 vec 28 affinity to 27
 > ixl1: for TXRX28 interrupting at msix5 vec 29 affinity to 28
 > ixl1: for TXRX29 interrupting at msix5 vec 30 affinity to 29
 > ixl1: for TXRX30 interrupting at msix5 vec 31 affinity to 30
 > ixl1: for TXRX31 interrupting at msix5 vec 32 affinity to 31
 > ixl1: for other interrupting at msix5 vec 0 affinity to 0
 > [...]
 > pci9 at mainbus0 bus 93
 > pci9: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
 > ppb6 at pci9 dev 2 function 0: Intel product 2032 (rev. 0x04)
 > ppb6: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x4 @ 8.0GT/s
 > pci10 at ppb6 bus 94
 > pci10: i/o space, memory space enabled
 > nvme0 at pci10 dev 0 function 0: Intel SSD DC P4500 (rev. 0x00)
 > nvme0: NVMe 1.2
 > allocated pic msix6 type edge pin 0 level 6 to cpu1 slot 1 idt entry 170
 > nvme0: for admin queue interrupting at msix6 vec 0
 > nvme0: INTEL SSDPE2KX040T8, firmware VDV10131, serial ...
 > allocated pic msix6 type edge pin 1 level 6 to cpu1 slot 2 idt entry 171
 > nvme0: for io queue 1 interrupting at msix6 vec 1 affinity to cpu0
 > allocated pic msix6 type edge pin 2 level 6 to cpu1 slot 2 idt entry 172
 > nvme0: for io queue 2 interrupting at msix6 vec 2 affinity to cpu1
 > allocated pic msix6 type edge pin 3 level 6 to cpu1 slot 3 idt entry 173
 > nvme0: for io queue 3 interrupting at msix6 vec 3 affinity to cpu2
 > allocated pic msix6 type edge pin 4 level 6 to cpu1 slot 3 idt entry 174
 > nvme0: for io queue 4 interrupting at msix6 vec 4 affinity to cpu3
 > allocated pic msix6 type edge pin 5 level 6 to cpu1 slot 3 idt entry 175
 > nvme0: for io queue 5 interrupting at msix6 vec 5 affinity to cpu4
 > allocated pic msix6 type edge pin 6 level 6 to cpu1 slot 3 idt entry 176
 > nvme0: for io queue 6 interrupting at msix6 vec 6 affinity to cpu5
 > allocated pic msix6 type edge pin 7 level 6 to cpu1 slot 3 idt entry 177
 > nvme0: for io queue 7 interrupting at msix6 vec 7 affinity to cpu6
 > allocated pic msix6 type edge pin 8 level 6 to cpu1 slot 3 idt entry 178
 > nvme0: for io queue 8 interrupting at msix6 vec 8 affinity to cpu7
 > allocated pic msix6 type edge pin 9 level 6 to cpu1 slot 3 idt entry 179
 > nvme0: for io queue 9 interrupting at msix6 vec 9 affinity to cpu8
 > allocated pic msix6 type edge pin 10 level 6 to cpu1 slot 3 idt entry 180
 > nvme0: for io queue 10 interrupting at msix6 vec 10 affinity to cpu9
 > allocated pic msix6 type edge pin 11 level 6 to cpu1 slot 3 idt entry 181
 > nvme0: for io queue 11 interrupting at msix6 vec 11 affinity to cpu10
 > allocated pic msix6 type edge pin 12 level 6 to cpu1 slot 3 idt entry 182
 > nvme0: for io queue 12 interrupting at msix6 vec 12 affinity to cpu11
 > allocated pic msix6 type edge pin 13 level 6 to cpu1 slot 3 idt entry 183
 > nvme0: for io queue 13 interrupting at msix6 vec 13 affinity to cpu12
 > allocated pic msix6 type edge pin 14 level 6 to cpu1 slot 3 idt entry 184
 > nvme0: for io queue 14 interrupting at msix6 vec 14 affinity to cpu13
 > allocated pic msix6 type edge pin 15 level 6 to cpu1 slot 3 idt entry 185
 > nvme0: for io queue 15 interrupting at msix6 vec 15 affinity to cpu14
 > allocated pic msix6 type edge pin 16 level 6 to cpu1 slot 3 idt entry 186
 > nvme0: for io queue 16 interrupting at msix6 vec 16 affinity to cpu15
 > allocated pic msix6 type edge pin 17 level 6 to cpu1 slot 3 idt entry 187
 > nvme0: for io queue 17 interrupting at msix6 vec 17 affinity to cpu16
 > allocated pic msix6 type edge pin 18 level 6 to cpu1 slot 3 idt entry 188
 > nvme0: for io queue 18 interrupting at msix6 vec 18 affinity to cpu17
 > allocated pic msix6 type edge pin 19 level 6 to cpu1 slot 3 idt entry 189
 > nvme0: for io queue 19 interrupting at msix6 vec 19 affinity to cpu18
 > allocated pic msix6 type edge pin 20 level 6 to cpu1 slot 3 idt entry 190
 > nvme0: for io queue 20 interrupting at msix6 vec 20 affinity to cpu19
 > allocated pic msix6 type edge pin 21 level 6 to cpu1 slot 3 idt entry 191
 > nvme0: for io queue 21 interrupting at msix6 vec 21 affinity to cpu20
 > allocated pic msix6 type edge pin 22 level 6 to cpu1 slot 3 idt entry 193
 > nvme0: for io queue 22 interrupting at msix6 vec 22 affinity to cpu21
 > allocated pic msix6 type edge pin 23 level 6 to cpu1 slot 3 idt entry 194
 > nvme0: for io queue 23 interrupting at msix6 vec 23 affinity to cpu22
 > allocated pic msix6 type edge pin 24 level 6 to cpu1 slot 3 idt entry 195
 > nvme0: for io queue 24 interrupting at msix6 vec 24 affinity to cpu23
 > allocated pic msix6 type edge pin 25 level 6 to cpu1 slot 3 idt entry 196
 > nvme0: for io queue 25 interrupting at msix6 vec 25 affinity to cpu24
 > allocated pic msix6 type edge pin 26 level 6 to cpu1 slot 3 idt entry 197
 > nvme0: for io queue 26 interrupting at msix6 vec 26 affinity to cpu25
 > allocated pic msix6 type edge pin 27 level 6 to cpu1 slot 3 idt entry 198
 > nvme0: for io queue 27 interrupting at msix6 vec 27 affinity to cpu26
 > allocated pic msix6 type edge pin 28 level 6 to cpu1 slot 3 idt entry 199
 > nvme0: for io queue 28 interrupting at msix6 vec 28 affinity to cpu27
 > allocated pic msix6 type edge pin 29 level 6 to cpu1 slot 3 idt entry 200
 > nvme0: for io queue 29 interrupting at msix6 vec 29 affinity to cpu28
 > allocated pic msix6 type edge pin 30 level 6 to cpu1 slot 3 idt entry 201
 > nvme0: for io queue 30 interrupting at msix6 vec 30 affinity to cpu29
 > allocated pic msix6 type edge pin 31 level 6 to cpu1 slot 3 idt entry 202
 > nvme0: for io queue 31 interrupting at msix6 vec 31 affinity to cpu30
 > allocated pic msix6 type edge pin 32 level 6 to cpu1 slot 3 idt entry 203
 > nvme0: for io queue 32 interrupting at msix6 vec 32 affinity to cpu31
 > ld0 at nvme0 nsid 1
 > ld0: 3726 GB, 486401 cyl, 255 head, 63 sec, 512 bytes/sect x 7814037168 sectors
 > ppb7 at pci9 dev 3 function 0: Intel product 2033 (rev. 0x04)
 > ppb7: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x4 @ 8.0GT/s
 > pci11 at ppb7 bus 95
 > pci11: i/o space, memory space enabled
 > nvme1 at pci11 dev 0 function 0: Intel SSD DC P4500 (rev. 0x00)
 > nvme1: NVMe 1.2
 > allocated pic msix7 type edge pin 0 level 6 to cpu1 slot 3 idt entry 204
 > nvme1: for admin queue interrupting at msix7 vec 0
 > nvme1: INTEL SSDPE2KX040T8, firmware VDV10131, serial ...
 > allocated pic msix7 type edge pin 1 level 6 to cpu1 slot 4 idt entry 205
 > nvme1: for io queue 1 interrupting at msix7 vec 1 affinity to cpu0
 > allocated pic msix7 type edge pin 2 level 6 to cpu1 slot 4 idt entry 206
 > nvme1: for io queue 2 interrupting at msix7 vec 2 affinity to cpu1
 > allocated pic msix7 type edge pin 3 level 6 to cpu1 slot 5 idt entry 207
 > nvme1: for io queue 3 interrupting at msix7 vec 3 affinity to cpu2
 > allocated pic msix7 type edge pin 4 level 6 to cpu1 slot 5 idt entry 208
 > nvme1: for io queue 4 interrupting at msix7 vec 4 affinity to cpu3
 > allocated pic msix7 type edge pin 5 level 6 to cpu1 slot 5 idt entry 209
 > nvme1: for io queue 5 interrupting at msix7 vec 5 affinity to cpu4
 > allocated pic msix7 type edge pin 6 level 6 to cpu1 slot 5 idt entry 210
 > nvme1: for io queue 6 interrupting at msix7 vec 6 affinity to cpu5
 > allocated pic msix7 type edge pin 7 level 6 to cpu1 slot 5 idt entry 211
 > nvme1: for io queue 7 interrupting at msix7 vec 7 affinity to cpu6
 > allocated pic msix7 type edge pin 8 level 6 to cpu1 slot 5 idt entry 212
 > nvme1: for io queue 8 interrupting at msix7 vec 8 affinity to cpu7
 > allocated pic msix7 type edge pin 9 level 6 to cpu1 slot 5 idt entry 213
 > nvme1: for io queue 9 interrupting at msix7 vec 9 affinity to cpu8
 > allocated pic msix7 type edge pin 10 level 6 to cpu1 slot 5 idt entry 214
 > nvme1: for io queue 10 interrupting at msix7 vec 10 affinity to cpu9
 > allocated pic msix7 type edge pin 11 level 6 to cpu1 slot 5 idt entry 215
 > nvme1: for io queue 11 interrupting at msix7 vec 11 affinity to cpu10
 > allocated pic msix7 type edge pin 12 level 6 to cpu1 slot 5 idt entry 216
 > nvme1: for io queue 12 interrupting at msix7 vec 12 affinity to cpu11
 > allocated pic msix7 type edge pin 13 level 6 to cpu1 slot 5 idt entry 217
 > nvme1: for io queue 13 interrupting at msix7 vec 13 affinity to cpu12
 > allocated pic msix7 type edge pin 14 level 6 to cpu1 slot 5 idt entry 218
 > nvme1: for io queue 14 interrupting at msix7 vec 14 affinity to cpu13
 > allocated pic msix7 type edge pin 15 level 6 to cpu1 slot 5 idt entry 219
 > nvme1: for io queue 15 interrupting at msix7 vec 15 affinity to cpu14
 > allocated pic msix7 type edge pin 16 level 6 to cpu1 slot 5 idt entry 220
 > nvme1: for io queue 16 interrupting at msix7 vec 16 affinity to cpu15
 > allocated pic msix7 type edge pin 17 level 6 to cpu1 slot 5 idt entry 221
 > nvme1: for io queue 17 interrupting at msix7 vec 17 affinity to cpu16
 > allocated pic msix7 type edge pin 18 level 6 to cpu1 slot 5 idt entry 222
 > nvme1: for io queue 18 interrupting at msix7 vec 18 affinity to cpu17
 > allocated pic msix7 type edge pin 19 level 6 to cpu1 slot 5 idt entry 223
 > nvme1: for io queue 19 interrupting at msix7 vec 19 affinity to cpu18
 > allocated pic msix7 type edge pin 20 level 6 to cpu1 slot 5 idt entry 226
 > nvme1: for io queue 20 interrupting at msix7 vec 20 affinity to cpu19
 > allocated pic msix7 type edge pin 21 level 6 to cpu1 slot 5 idt entry 227
 > nvme1: for io queue 21 interrupting at msix7 vec 21 affinity to cpu20
 > allocated pic msix7 type edge pin 22 level 6 to cpu1 slot 5 idt entry 228
 > nvme1: for io queue 22 interrupting at msix7 vec 22 affinity to cpu21
 > allocated pic msix7 type edge pin 23 level 6 to cpu1 slot 5 idt entry 229
 > nvme1: for io queue 23 interrupting at msix7 vec 23 affinity to cpu22
 > allocated pic msix7 type edge pin 24 level 6 to cpu1 slot 5 idt entry 230
 > nvme1: for io queue 24 interrupting at msix7 vec 24 affinity to cpu23
 > allocated pic msix7 type edge pin 25 level 6 to cpu1 slot 5 idt entry 231
 > nvme1: for io queue 25 interrupting at msix7 vec 25 affinity to cpu24
 > allocated pic msix7 type edge pin 26 level 6 to cpu1 slot 5 idt entry 232
 > nvme1: for io queue 26 interrupting at msix7 vec 26 affinity to cpu25
 > allocated pic msix7 type edge pin 27 level 6 to cpu1 slot 5 idt entry 233
 > nvme1: for io queue 27 interrupting at msix7 vec 27 affinity to cpu26
 > allocated pic msix7 type edge pin 28 level 6 to cpu1 slot 5 idt entry 234
 > nvme1: for io queue 28 interrupting at msix7 vec 28 affinity to cpu27
 > allocated pic msix7 type edge pin 29 level 6 to cpu1 slot 5 idt entry 235
 > nvme1: for io queue 29 interrupting at msix7 vec 29 affinity to cpu28
 > allocated pic msix7 type edge pin 30 level 6 to cpu1 slot 5 idt entry 236
 > nvme1: for io queue 30 interrupting at msix7 vec 30 affinity to cpu29
 > allocated pic msix7 type edge pin 31 level 6 to cpu1 slot 5 idt entry 237
 > nvme1: for io queue 31 interrupting at msix7 vec 31 affinity to cpu30
 > allocated pic msix7 type edge pin 32 level 6 to cpu1 slot 5 idt entry 238
 > nvme1: for io queue 32 interrupting at msix7 vec 32 affinity to cpu31
 > ld1 at nvme1 nsid 1
 > ld1: 3726 GB, 486401 cyl, 255 head, 63 sec, 512 bytes/sect x 7814037168 sectors
 > [...]
 > PCI12 AT mainbus0 bus 128
 > pci12: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
 > [...]
 > pci13 at mainbus0 bus 133
 > pci13: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
 > [...]
 > pci14 at mainbus0 bus 174
 > pci14: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
 > ppb8 at pci14 dev 2 function 0: Intel product 2032 (rev. 0x04)
 > ppb8: PCI Express capability version 2 <Root Port of PCI-E Root Complex> x8 @ 8.0GT/s
 > ppb8: link is x4 @ 5.0GT/s
 > pci15 at ppb8 bus 175
 > pci15: i/o space, memory space enabled
 > wm0 at pci15 dev 0 function 0: I350 Gigabit Network Connection (rev. 0x01)
 > failed to allocate interrupt slot for PIC msix8 pin 0
 > wm0: autoconfiguration error: unable to establish MSI-X(for TX and RX) at msix8 vec 0
 > failed to allocate interrupt slot for PIC msi8 pin 0
 > wm0: autoconfiguration error: unable to establish MSI
 > failed to allocate interrupt slot for PIC ioapic7 pin 2
 > wm0: autoconfiguration error: unable to establish INTx
 > wm1 at pci15 dev 0 function 1: I350 Gigabit Network Connection (rev. 0x01)
 > failed to allocate interrupt slot for PIC msix8 pin 0
 > wm1: autoconfiguration error: unable to establish MSI-X(for TX and RX) at msix8 vec 0
 > failed to allocate interrupt slot for PIC msi8 pin 0
 > wm1: autoconfiguration error: unable to establish MSI
 > failed to allocate interrupt slot for PIC ioapic7 pin 5
 > wm1: autoconfiguration error: unable to establish INTx
 > [...]
 > 	unconfiguring ixl* interface bring the wm* interfaces back to life
 > 
 > 	Full dmesg output avaliable on request.
 >> How-To-Repeat:
 > 	attempt to boot 9.99.33 on a larger system with X722 and I350 ethernet interfaces.
 > 
 >> Fix:
 > 	disable X722 interface, but that defeats the purpose.
 > 
 >> Unformatted:
 
 To mitigate this problem, x86's softint bitmask should be
 increased from 32bit to 64bit. Even if it's done, the problem
 won't be solved on many core systems. To resolve the problem,
 IDTVEC should be per-CPU.
 
 -- 
 -----------------------------------------------
                 SAITOH Masanobu (msaitoh%execsw.org@localhost
                                  msaitoh%netbsd.org@localhost)