FW: ixg(4) performances

["Terry Moore" <tmm%mcci.com@localhost>]

Forgot to cc the list.

-----Original Message-----
From: Terry Moore [mailto:tmm%mcci.com@localhost] 
Sent: Friday, August 29, 2014 15:13
To: 'Emmanuel Dreyfus'
Subject: RE: ixg(4) performances

> > But it's running at gen1. I strongly suspect that the benchmark case 
> > was
> > gen2 (since the ixg is capable of it).
> 
> gen1 vs gen2 is 2.5 Gb.s vs 5 Gb/s?

Yes. Actually, 2.5Gbps is symbol rate -- it's 8/10 encoded, so one lane is
really 2Gbps.  So 8 lanes is 16Gbps which *should* be enough, but... there's
overhead and a variety of sources of wastage. 

I just saw today a slide that says that 8 lanes gen 1 is just barely enough
for 10Gb Eth
(http://www.eetimes.com/document.asp?doc_id=1323695&page_number=4). 

It's possible that the benchmark system was using 8 lanes of Gen2. 

Another possibility: earlier you wrote:

> No reference to MMRBC in this document, but I understand "Max Read Request
Size" 
> is the same thing. Page 765 tells us about register A8, bits 12-14 
> that should be set to 100.
> pcictl /dev/pci5 read -d 0 -f 1 0x18 tells me the value 0x00092810
> 
> I tried this command:
> pcictl /dev/pci5 write -d 0 -f 1 0x18 0x00094810

In the PCIe spec, this is controlled via the Device Control register which
is a 16-bit value.  You want to set *two* fields, to different values.  0x18
looks like the wrong offset. The PCIe spec says offset 0x08, but that's
relative to the base of the capability structure, the offset of which is in
the low byte of the dword at 0x34.  I am running NetBSD 5; my pcictl doesn't
support write as one of its options, but I'd expect that to be relative to
the base of the function config space, and *that's not the device
capabilities register*. It's a read/write register, which is one of the Base
Address Registers.  

In any case, in the Device Control Register, bits 7:5 are the max payload
size *for writes* by the igx to the system.  These must be set to 101b for
4K max payload size.

Similarly, bits 14:12 are the max read request size.  These must also be set
to 101b for 4K max read request. 

Since you did a dword read, the extra 0x9.... is the device status register.
This makes me suspicious as the device status register is claiming that you
have "unsupported request detected)" [bit 3] and "correctable error
detected" [bit 0].  Further, this register is RW1C for all these bits -- so
when you write 94810, it should have cleared the 9 (so a subsequent read
should have returned 4810).

Please check.

Might be good to post a "pcictl dump" of your device, just to expose all the
details.

> > Is the ixg in an expansion slot or integrated onto the main board?
> 
> In a slot.

Check the manual on the main board and find out whether other slots have 8
lanes of Gen2. 

If so, move the board.

Best regards,
--Terry