Even with 66/64, there are some other issues that showed up in testing
at solidum.com when using the SysKonnect boards and DMA'ing to memory
on another PCI board (where a co-processor does the packet processing)

  1) the stupid host bridge was breaking the 66/64s writes into 66/32s!!!
     We wound up putting the device into a 32/64 slot adjacent to our
     co-processor. 

  2) if the SysKonnect got a PCI retry while transfering data, it backed
     off for essentially four or five packet times. This instantly meant
     that we got less than 200Mb/s.

     Curiously, we could run two SysKonnect cards and double that number :-)

     Alas, we could not get rid of the retry on the PCI interface to our
     card. Lucent ORCA FPGA was just too slow.

  3) when testing to memory, we never quite figured out some of the host
     bridge issues - the writes out to have been pipelineable. PCI doesn't
     say anything about that though - it just says if memory can be
     prefetched. Most things assume that if you prefetch that means that you
     can cache on writes as well. That isn't entirely certain, and host
     bridges (memory interfaces) seem to have various ideas on this.

  4) note that we weren't even trying to look at the packets with the
     host CPU. (A P-III 650 at the time)
	
  So, I wouldn't try building anything above 500Mb/s (full duplex rate) even
with 64/66. SPI4.2/CSIX is your friend :-)

]       ON HUMILITY: to err is human. To moo, bovine.           |  firewalls  [
]   Michael Richardson, Sandelman Software Works, Ottawa, ON    |net architect[
] mcr@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[
] panic("Just another NetBSD/notebook using, kernel hacking, security guy");  [