Re: pci_intr_alloc() vs pci_intr_establish() - retry type?

[Masanobu SAITOH <msaitoh%execsw.org@localhost>]

On 2018/12/04 16:44, Masanobu SAITOH wrote:
> On 2018/12/04 4:55, Jaromír Doleček wrote:
> > Le lun. 3 déc. 2018 à 12:09, Masanobu SAITOH <msaitoh%execsw.org@localhost> a écrit :
> > > C3000's AHCI has multi-vector MSI-X table and it doesn't work since
> > > Nobember 20th...
> >
> > Can you try if by chance this code adapted nvme(4) changes anything on
> > your system?
> >
> > http://www.netbsd.org/~jdolecek/ahcisata_msixoff.diff
>
> It panics:
> > [   1.0427481] panic: extent_alloc_region: bad size
> > [   1.0427481] cpu0: Begin traceback...
> > [   1.0427481] vpanic() at netbsd:vpanic+0x16f
> > [   1.0427481] snprintf() at netbsd:snprintf
> > [   1.0427481] extent_alloc_region() at netbsd:extent_alloc_region+0x29e
> > [   1.0427481] bus_space_reserve() at netbsd:bus_space_reserve+0x95
> > [   1.0427481] bus_space_map() at netbsd:bus_space_map+0x85
> > [   1.0427481] ahci_pci_attach() at netbsd:ahci_pci_attach+0xde
> > [   1.0427481] config_attach_loc() at netbsd:config_attach_loc+0x1a8
> > [   1.0427481] config_found_sm_loc() at netbsd:config_found_sm_loc+0x48
> > [   1.0427481] pci_probe_device() at netbsd:pci_probe_device+0x582
> > [   1.0427481] pci_enumerate_bus() at netbsd:pci_enumerate_bus+0x197
> > [   1.0427481] pcirescan() at netbsd:pcirescan+0x47
> > [   1.0427481] pciattach() at netbsd:pciattach+0x193
> > [   1.0427481] config_attach_loc() at netbsd:config_attach_loc+0x1a8
> > [   1.0427481] config_found_sm_loc() at netbsd:config_found_sm_loc+0x48
> > [   1.0427481] mp_pci_scan() at netbsd:mp_pci_scan+0xa3
> > [   1.0427481] mainbus_attach() at netbsd:mainbus_attach+0x332
> > [   1.0427481] config_attach_loc() at netbsd:config_attach_loc+0x1a8
> > [   1.0427481] cpu_configure() at netbsd:cpu_configure+0x2b
> > [   1.0427481] main() at netbsd:main+0x331
> > [   1.0427481] cpu0: End traceback...
> > [   1.0427481] fatal breakpoint trap in supervisor mode
> > [   1.0427481] trap type 1 code 0 rip 0xffffffff8021de45 cs 0x8 rflags 0x202 cr2 0 ilevel 0x8 rsp 0xffffffff81d027f0
> > [   1.0427481] curlwp 0xffffffff81858900 pid 0.1 lowest kstack 0xffffffff81cfe2c0
> > Stopped in pid 0.1 (system) at  netbsd:breakpoint+0x5:  leave
>
> pcictl dump shows:
> > PCI configuration registers:
> >   Common header:
> >     0x00: 0x19c28086 0x02b00147 0x01060110 0x00000000
> >
> >     Vendor Name: Intel (0x8086)
> >     Device Name: C3000 SATA Controller 1 (0x19c2)
> >     Command register: 0x0147
> >       I/O space accesses: on
> >       Memory space accesses: on
> >       Bus mastering: on
> >       Special cycles: off
> >       MWI transactions: off
> >       Palette snooping: off
> >       Parity error checking: on
> >       Address/data stepping: off
> >       System error (SERR): on
> >       Fast back-to-back transactions: off
> >       Interrupt disable: off
> >     Status register: 0x02b0
> >       Immediate Readiness: off
> >       Interrupt status: inactive
> >       Capability List support: on
> >       66 MHz capable: on
> >       User Definable Features (UDF) support: off
> >       Fast back-to-back capable: on
> >       Data parity error detected: off
> >       DEVSEL timing: medium (0x1)
> >       Slave signaled Target Abort: off
> >       Master received Target Abort: off
> >       Master received Master Abort: off
> >       Asserted System Error (SERR): off
> >       Parity error detected: off
> >     Class Name: mass storage (0x01)
> >     Subclass Name: SATA (0x06)
> >     Interface Name: AHCI 1.0 (0x01)
> >     Revision ID: 0x10
> >     BIST: 0x00
> >     Header Type: 0x00 (0x00)
> >     Latency Timer: 0x00
> >     Cache Line Size: 0bytes (0x00)
> >
> >   Type 0 ("normal" device) header:
> >     0x10: 0xdffb4000 0xdffbd000 0x0000e071 0x0000e061
> >     0x20: 0x0000e021 0xdffbc000 0x00000000 0x72708086
> >     0x30: 0x00000000 0x00000080 0x00000000 0x0000010e
> >
> >     Base address register at 0x10
> >       type: 32-bit nonprefetchable memory
> >       base: 0xdffb4000
> >     Base address register at 0x14
> >       type: 32-bit nonprefetchable memory
> >       base: 0xdffbd000
> >     Base address register at 0x18
> >       type: I/O
> >       base: 0x0000e070
> >     Base address register at 0x1c
> >       type: I/O
> >       base: 0x0000e060
> >     Base address register at 0x20
> >       type: I/O
> >       base: 0x0000e020
> >     Base address register at 0x24
> >       type: 32-bit nonprefetchable memory
> >       base: 0xdffbc000
> >     Cardbus CIS Pointer: 0x00000000
> >     Subsystem vendor ID: 0x8086
> >     Subsystem ID: 0x7270
> >     Expansion ROM Base Address Register: 0x00000000
> >       base: 0x00000000
> >       Expansion ROM Enable: off
> >       Validation Status: Validation not supported
> >       Validation Details: 0x0
> >     Capability list pointer: 0x80
> >     Reserved @ 0x38: 0x00000000
> >     Maximum Latency: 0x00
> >     Minimum Grant: 0x00
> >     Interrupt pin: 0x01 (pin A)
> >     Interrupt line: 0x0e
> >
> >   Capability register at 0x80
> >     type: 0x05 (MSI)
> >   Capability register at 0x70
> >     type: 0x01 (Power Management)
> >   Capability register at 0xa8
> >     type: 0x12 (SATA)
> >   Capability register at 0xd0
> >     type: 0x11 (MSI-X)
> >
> >   PCI Power Management Capabilities Register
> >     Capabilities register: 0x4003
> >       Version: 1.2
> >       PME# clock: off
> >       Device specific initialization: off
> >       3.3V auxiliary current: self-powered
> >       D1 power management state support: off
> >       D2 power management state support: off
> >       PME# support D0: off
> >       PME# support D1: off
> >       PME# support D2: off
> >       PME# support D3 hot: on
> >       PME# support D3 cold: off
> >     Control/status register: 0x00000008
> >       Power state: D0
> >       PCI Express reserved: off
> >       No soft reset: on
> >       PME# assertion: disabled
> >       Data Select: 0
> >       Data Scale: 0
> >       PME# status: off
> >     Bridge Support Extensions register: 0x00
> >       B2/B3 support: off
> >       Bus Power/Clock Control Enable: off
> >     Data register: 0x00
> >
> >   PCI Message Signaled Interrupt
> >     Message Control register: 0x0001
> >       MSI Enabled: on
> >       Multiple Message Capable: no (1 vector)
> >       Multiple Message Enabled: off (1 vector)
> >       64 Bit Address Capable: off
> >       Per-Vector Masking Capable: off
> >       Extended Message Data Capable: off
> >       Extended Message Data Enable: off
> >     Message Address register: 0xfee04000
> >     Message Data register: 0x006f
> >
> >   MSI-X Capability Register
> >     Message Control register: 0x0007
> >       Table Size: 8
> >       Function Mask: off
> >       MSI-X Enable: off
> >     Table offset register: 0x00000000
> >       Table offset: 0x00000000
> >       BIR: 0x0        <========================
> >     Pending bit array register: 0x00000001
> >       Pending bit array offset: 0x00000000
> >       BIR: 0x1
> >
> >   Serial ATA Capability Register
> >     Revision register: 0x0010
> >       Revision: 1.0
> >     BAR Register: 0x00000048
> >       Register location: BAR 4  <======================
> >       BAR offset: 0x00000010
> >
> >   Device-dependent header:
> >     0x40: 0x00000000 0x00000000 0x00000000 0x00000000
> >     0x50: 0x00000000 0x00000000 0x00000000 0x00000000
> >     0x60: 0x00000000 0x00000000 0x00000000 0x00000000
> >     0x70: 0x4003a801 0x00000008 0x00000000 0x00000000
> >     0x80: 0x00017005 0xfee04000 0x0000006f 0x00000000
> >     0x90: 0x00cf00cf 0x00200030 0x00000000 0x8000fa30
> >     0xa0: 0x000000d4 0x2c1e1108 0x0010d012 0x00000048
> >     0xb0: 0x00000000 0x00000000 0x00000000 0x00000000
> >     0xc0: 0x00000000 0x00000000 0x00000000 0x00000000
> >     0xd0: 0x00070011 0x00000000 0x00000001 0x00000000
> >     0xe0: 0x00000000 0x00000000 0x00000000 0x00000000
> >     0xf0: 0x00000000 0x00000000 0x01100f1c 0x00000000
>
>
>
> > ahcisata(4) works for me on two different computers in MSI mode. I
> > don't have any ahcisata device which supports MSI-X, but it is working
> > on Cavium which has MSI-X. Perhaps some different BAR wierdness than
> > Cavium, or simply the boards using MSI-X should use the "cavium" BAR?
> >
> > So another thing to try would be to make ahci_pci_abar() return
> > AHCI_PCI_ABAR_CAVIUM always and see if it changes things.
> >
> > Also you can try rev 1.44 of ahcisata_pci.c with AHCISATA_DISABLE_MSIX
> > to see if it works if forced to MSI only.
>
> I've tested rev. 1.44 now and it worked on this machine.
ofcourse it's with AHCISATA_DISABLE_MSIX.

> [   1.0426481] ahcisata0 at pci0 dev 19 function 0: vendor 8086 product 19b2 (rev. 0x10)
> [   1.0426481] allocated pic msi5 type edge pin 0 level 6 to cpu0 slot 21 idt entry 110
> [   1.0426481] ahcisata0: interrupting at msi5 vec 0
> [   1.0426481] ahcisata0: 64-bit DMA
> [   1.0426481] ahcisata0: AHCI revision 1.31, 1 port, 32 slots, CAP 0xc3369f40<EMS,PMD,SPM,SAM,ISS=0x3=Gen3,SCLO,SAL,SNCQ,S64A>
> [   1.0426481] atabus0 at ahcisata0 channel 3
> [   1.0426481] ahcisata1 at pci0 dev 20 function 0: vendor 8086 product 19c2 (rev. 0x10)
> [   1.0426481] allocated pic msi6 type edge pin 0 level 6 to cpu0 slot 22 idt entry 111
> [   1.0426481] ahcisata1: interrupting at msi6 vec 0
> [   1.0426481] ahcisata1: 64-bit DMA
> [   1.0426481] ahcisata1: AHCI revision 1.31, 2 ports, 32 slots, CAP 0xc3369f41<EMS,PMD,SPM,SAM,ISS=0x3=Gen3,SCLO,SAL,SNCQ,S64A>
> [   1.0426481] atabus1 at ahcisata1 channel 4
> [   1.0426481] atabus2 at ahcisata1 channel 5
(snip)
> [   1.5749882] ahcisata1 port 5: device present, speed: 6.0Gb/s
> [   3.1463325] wd0 at atabus2 drive 0
> [   3.1563421] wd0: <INTEL SSDSC2CW240A3>
> [   3.1563421] wd0: drive supports 16-sector PIO transfers, LBA48 addressing
> [   3.1695774] wd0: 223 GB, 465141 cyl, 16 head, 63 sec, 512 bytes/sect x 468862128 sectors
> [   3.1863674] wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133), WRITE DMA FUA, NCQ (32 tags)
> [   3.1975821] wd0(ahcisata1:5:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA), NCQ (31 tags)


> > I wonder whether there is something we still need to adjust during
> > general initialization, I see this on my hardware:
> >
> > re(4) - on my system works when attached either MSI or MSI-X
> > nvme(4) - on my system works only when attached via MSI-X (regardless
> > of the PCI_CAP_MSIX), doesn't work when forced to MSI
> > ahcisata(4) works in MSI on my system, as MSI-X for skrll@, MSI-X
> > doesn't work for you
> >
> > Jaromir


--
-----------------------------------------------
                SAITOH Masanobu (msaitoh%execsw.org@localhost
                                 msaitoh%netbsd.org@localhost)