RE: Support for Intel P3xxx(3500) series PCIe SSD

[Derrick Lobo <derrick.lobo%givex.com@localhost>]

Thanks @Saitoh

Derrick Lobo

-----Original Message-----
From: Masanobu SAITOH [mailto:msaitoh%execsw.org@localhost] 
Sent: Friday, May 20, 2016 2:35 AM
To: Derrick Lobo; Kimihiro Nonaka
Cc: msaitoh%execsw.org@localhost; Joerg Sonnenberger; port-amd64%netbsd.org@localhost;
port-i386%netbsd.org@localhost; NONAKA Kimihiro
Subject: Re: Support for Intel P3xxx(3500) series PCIe SSD

Hi.

On 2016/05/20 2:02, Derrick Lobo wrote:
>
> File is attached here.
>
> https://dev-wwws.givex.com/public/0ccfc54fa55b8f957e062eec8b925831.bin

  Thanks.

The all setting of your machine's MSI mappings are fixed.

>   HyperTransport Capability Register at 0xf0
>     Command register: 0xa803
>       Capability Type: 0x15 (MSI Mapping)
>       Enable: on
>       Fixed: on                        <=== this
>     Address Low register: 0x00808000
>     Address high register: 0x00000000
>       Address: 0x0000000000800000

But, x86/pci/pci_attach_hook() excludes MSI and MSI-X on your machine.
See:

	http://nxr.netbsd.org/xref/src/sys/arch/x86/pci/pci_machdep.c#507

If 0:24:0 has HyperTransport capability, MSI/MSI-X is disabled.
Your log shows:

> pcictl pci0 dump -b 0 -d 24 -f 0
> PCI configuration registers:
>   Common header:
>     0x00: 0x16001022 0x00100000 0x06000000 0x00800000
>
>     Vendor Name: AMD (0x1022)
>     Device Name: Family15h HyperTransport Configuration (0x1600)

	(snip)

>   Capability register at 0x80
>     type: 0x08 (HyperTransport)
>   Capability register at 0xa0
>     type: 0x08 (HyperTransport)
>   Capability register at 0xc0
>     type: 0x08 (HyperTransport)
>   Capability register at 0xe0
>     type: 0x08 (HyperTransport)
>

The code was taken from OpenBSD.

(We don't know whether this code is really required or not now...)

  Anyway, thank you for your cooperation!


> Derrick Lobo
> -----Original Message-----
> From: Masanobu SAITOH [mailto:msaitoh%execsw.org@localhost]
> Sent: Thursday, May 19, 2016 1:07 AM
> To: Derrick Lobo; Kimihiro Nonaka
> Cc: msaitoh%execsw.org@localhost; Joerg Sonnenberger; port-amd64%netbsd.org@localhost; 
> port-i386%netbsd.org@localhost; NONAKA Kimihiro
> Subject: Re: Support for Intel P3xxx(3500) series PCIe SSD
>
> Hi.
>
> On 2016/05/13 23:06, Derrick Lobo wrote:
>> Sorry Im checking on the details.. Heres the output
>>
>> 000:00:0: ATI Technologies RD890 PCI Bridge GFX0 Port A (host bridge, 
>> revision 0x02)
>> 000:09:0: ATI Technologies RD890 PCI Express Bridge GPP Port H (PCI
>> bridge)
>> 000:10:0: ATI Technologies RD890 PCI Bridge GFX1 Port A (PCI bridge)
>> 000:17:0: ATI Technologies SB700-SB900 SATA Controller (IDE mode) 
>> (IDE mass storage, interface 0x8f)
>> 000:18:0: ATI Technologies SB700-SB900 USB OHCI Controller (USB 
>> serial bus, OHCI)
>> 000:18:1: ATI Technologies SB700-SB900 USB OHCI Controller (USB 
>> serial bus, OHCI)
>> 000:18:2: ATI Technologies SB700-SB900 USB EHCI Controller (USB 
>> serial bus, EHCI)
>> 000:19:0: ATI Technologies SB700-SB900 USB OHCI Controller (USB 
>> serial bus, OHCI)
>> 000:19:1: ATI Technologies SB700-SB900 USB OHCI Controller (USB 
>> serial bus, OHCI)
>> 000:19:2: ATI Technologies SB700-SB900 USB EHCI Controller (USB 
>> serial bus, EHCI)
>> 000:20:0: ATI Technologies SBx00 SMBus Controller (SMBus serial bus, 
>> revision 0x3d)
>> 000:20:1: ATI Technologies SB700-SB900 IDE Controller (IDE mass 
>> storage, interface 0x8a)
>> 000:20:3: ATI Technologies SB700-SB900 LPC Host Controller (ISA
>> bridge)
>> 000:20:4: ATI Technologies SBx00 PCI to PCI Bridge (PCI bridge, 
>> subtractive decode)
>> 000:20:5: ATI Technologies SB700-SB900 USB OHCI Controller (USB 
>> serial bus, OHCI)
>> 000:24:0: AMD Family15h HyperTransport Configuration (host bridge)
>> 000:24:1: AMD Family15h Address Map Configuration (host bridge)
>> 000:24:2: AMD Family15h DRAM Configuration (host bridge)
>> 000:24:3: AMD Family15h Miscellaneous Configuration (host bridge)
>> 000:24:4: AMD Family15h Link Configuration (host bridge)
>> 000:24:5: AMD Family15h North Bridge Configuration (host bridge)
>> 000:25:0: AMD Family15h HyperTransport Configuration (host bridge)
>> 000:25:1: AMD Family15h Address Map Configuration (host bridge)
>> 000:25:2: AMD Family15h DRAM Configuration (host bridge)
>> 000:25:3: AMD Family15h Miscellaneous Configuration (host bridge)
>> 000:25:4: AMD Family15h Link Configuration (host bridge)
>> 000:25:5: AMD Family15h North Bridge Configuration (host bridge)
>> 001:00:0: Intel i82574L 1000baseT Ethernet (ethernet network)
>> 002:00:0: Intel i82574L 1000baseT Ethernet (ethernet network)
>> 003:04:0: Matrox MGA G200eW (VGA display, revision 0x0a)
>
> Thanks. Perhaps the device which has some information I want is 0:0:0, 
> but it might be not. If you are ok, could you get the following shell 
> script, execute it and put the whole output to somewhere? The output 
> helps not only MSI/MSI-X stuff but also help to add some code to 
> decode PCI(and/or
> HyperTransport) capabilities used by "pcictl <pciN> dump"
>
> 	http://www.netbsd.org/~msaitoh/pcidump
>
> Regards.
>
>
>
>>
>> -----Original Message-----
>> From: Masanobu SAITOH [mailto:msaitoh%execsw.org@localhost]
>> Sent: Thursday, May 12, 2016 10:56 PM
>> To: Derrick Lobo; Kimihiro Nonaka
>> Cc: msaitoh%execsw.org@localhost; Joerg Sonnenberger; port-amd64%netbsd.org@localhost; 
>> port-i386%netbsd.org@localhost; NONAKA Kimihiro
>> Subject: Re: Support for Intel P3xxx(3500) series PCIe SSD
>>
>> Hello.
>>
>> On 2016/05/12 23:16, Derrick Lobo wrote:
>>> Used nvme_pci.c r1.2 and it works.. I am able to mount the drive.
>> Excellent.
>>> Thank you nonakap@
>>
>>   Thank you for your report.
>>
>>   As I wrote before:
>>
>>>>> me show the full dmesg and "pcictl pci0 list" on your machine 
>>>>> without the SSD.
>>
>> Could you show the output of "pcictl pci0 list"?
>>
>> The reason why is that your machine currently can't use MSI-X because 
>> of the HyperTransport setting. See:
>>
>> 	http://nxr.netbsd.org/xref/src/sys/dev/pci/pci.c#416
>>
>> This code was taken from FreeBSD. This is the first time for us 
>> really disabled by this code. Is it possible for you to check the 
>> HyperTransport capability's setting? If not, please show me the 
>> output of "pcictl pci0 list" first.
>>
>>   Regards.
>>
>>
>>
>>> -----Original Message-----
>>> From: Kimihiro Nonaka [mailto:nonakap%gmail.com@localhost]
>>> Sent: Wednesday, May 11, 2016 9:59 AM
>>> To: Derrick Lobo
>>> Cc: Masanobu SAITOH; Joerg Sonnenberger; port-amd64%netbsd.org@localhost; 
>>> port-i386%netbsd.org@localhost; NONAKA Kimihiro
>>> Subject: Re: Support for Intel P3xxx(3500) series PCIe SSD
>>>
>>> Hi,
>>>
>>> Please try src/sys/dev/pci/nvme_pci.c r1.2.
>>>
>>> 2016-05-11 21:57 GMT+09:00 Derrick Lobo <derrick.lobo%givex.com@localhost>:
>>>> Im using amd64
>>>>
>>>> Dmesg -a
>>>> NetBSD 7.99.29 (GENERIC.201605101220Z) amd64
>>>>
>>>> The dmesg without the card installed is attached.
>>>>
>>>> Thanks for your help
>>>>
>>>> Derrick Lobo
>>>> -----Original Message-----
>>>> From: Masanobu SAITOH [mailto:msaitoh%execsw.org@localhost]
>>>> Sent: Wednesday, May 11, 2016 6:29 AM
>>>> To: Derrick Lobo; Joerg Sonnenberger; port-amd64%netbsd.org@localhost; 
>>>> port-i386%NetBSD.org@localhost
>>>> Cc: msaitoh%execsw.org@localhost; nonaka%netbsd.org@localhost
>>>> Subject: Re: Support for Intel P3xxx(3500) series PCIe SSD
>>>>
>>>> On 2016/05/11 19:17, Masanobu SAITOH wrote:
>>>>> Hi
>>>>>
>>>>> On 2016/05/11 6:34, Derrick Lobo wrote:
>>>>>> I used the netbsd current build from today and rcvd the following
>>>> error..
>>>>>> see pic attached. The drive is being recognized as nvme0.
>>>>>> This is the first build I tried Since Jan 29th.
>>>>>
>>>>>  In your pic:
>>>>>
>>>>>  > nvme0: interrupting at ioapic1 pin 4
>>>>
>>>>   Perhaps the driver have a bug when INTx is used. nonaka@ will 
>>>> test and fix the problem.
>>>>
>>>>> Are you using XEN_DOM0?
>>>>
>>>>   NetBSD/xen doesn't support MSI-X. If it's not XEN_DOM0, please 
>>>> let me show the full dmesg and "pcictl pci0 list" on your machine 
>>>> without the SSD.
>>>>
>>>>   Regards.
>>>>
>>>>
>>>>>
>>>>>> Derrick Lobo
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: port-amd64-owner%NetBSD.org@localhost 
>>>>>> [mailto:port-amd64-owner%NetBSD.org@localhost] On Behalf Of Joerg 
>>>>>> Sonnenberger
>>>>>> Sent: Friday, January 29, 2016 4:50 AM
>>>>>> To: port-amd64%netbsd.org@localhost; port-i386%netbsd.org@localhost
>>>>>> Subject: Re: Support for Intel P3xxx(3500) series PCIe SSD
>>>>>>
>>>>>> On Wed, Jan 27, 2016 at 12:00:06PM -0500, Derrick Lobo wrote:
>>>>>>> Seeing the following error after I added th Intel P3500  PCie 
>>>>>>> SSD 2Tb on the server.. upgraded to netbsd 7 current  dated 26th 
>>>>>>> Jan and got the same error
>>>>>>
>>>>>> That's NVME, isn't it?
>>>>>>
>>>>>> Joerg
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> -----------------------------------------------
>>>>                  SAITOH Masanobu (msaitoh%execsw.org@localhost
>>>>                                   msaitoh%netbsd.org@localhost)
>>
>>
>> --
>> -----------------------------------------------
>>                  SAITOH Masanobu (msaitoh%execsw.org@localhost
>>                                   msaitoh%netbsd.org@localhost)
>>
>
>
> --
> -----------------------------------------------
>                  SAITOH Masanobu (msaitoh%execsw.org@localhost
>                                   msaitoh%netbsd.org@localhost)
>


--
-----------------------------------------------
                 SAITOH Masanobu (msaitoh%execsw.org@localhost
                                  msaitoh%netbsd.org@localhost)