Subject: Re: hardware compatability Re: Linux emulation, binary software, cluster/grid and SMP
To: George Georgalis <george@galis.org>
From: None <jonathan@dsg.stanford.edu>
List: tech-kern
Date: 03/26/2006 12:44:41
In message <20060326064745.GB7637@sta.duo>"George Georgalis" writes

>>> vendor 0x14e4 product 0x166b (ethernet network, revision 0x03) at pci4 dev 
>4 function 0 not configured
>>> vendor 0x14e4 product 0x166b (ethernet network, revision 0x03) at pci4 dev 
>4 function 1 not configured
>>> 
>>> we've had some difficulty communicating with penguincomputing
>>> but today I was able to obtain an 'open source network driver'
>>
>>That's a variant of the Tigon III that I don't think we support right now. 
>>Check out sys/dev/if_bge.c.

George,

I beleive the variant you have is a bcm5780S.  From painstakingly 
perusing the LInux drivers, the following chips:

	bcm5714: PCI-e x4, dual-port gig-e with PCIe-to-external-PCI-X
	bcm5715: as 5714, but without PCI-X bus
	HT-2000:  Serverworks Hypertransport-to-PCI-e tunnel
		(sometimes called a "Northbridge", but that terminology
		carries inappropriate Intel CPU/memory-controller baggage)
		with embedded dual gigabit ethernet.
		The gig-e parts show up as "bcm5780".

all appear to use, in essesce, the same acutal gig-e core: at least,
all three require approximately the same set of software workarounds
for hardware bugs.


However, the issue with the chips on your blade systems is (as I
beleive I tried to explain a couple of weeks back) they're not the
common-or-garden versions with integral 1000base-T
transceiver. Instead, your blades appear to have the "bcm5780S"
version, which has builtin interfaces with 8-bit-to-10-bit encoded
signalling) (aka "TBI" or sometimes "SERDES"), suitable for wiring
direct to a 1000base-X optical GBIC.

The problem for the *BSD driver is that no-one besides me is
contributing if_bge support for these new Broadcom parts. (FreeBSD,
for instance, tries to treat the revisions above like a 5750, but that
misses a cupole of essential bug-workarounds in DMA setup).
And when I extracted the necessary hardware bug-workarounds from the
Linux driver, I never paid attention to code specific to the
TBI/Serdes parts.

Thus, if additional reverse engineering is required, it's hours to
paste in the relevant portoins, but days-to-weeks or more before I
acutally find time to grovel through the Linux code finding all the
requsisite parts.

There's a minor NetBSD-specific issue, that our if_bge matches and
attaches chips using either an exact chip-id match, or a "default"
keyed off the ASIC family; and right now we have no 5780/5780S exact
device-id matches. OTOH, we do have a "default" entry for the ASIC
revision used in 5714/5715/5780 cores, but (AFAIK) the "core revision
in the 5780 and 5780S are identical, so our "default" entry might
match both, but that entry has hardcoded "quirks" designed to match
the chips with integral 1000baseT phy; which is wrong for your 5780S
(e.g., ignores all phys except the expected integraly PHY at
PHY-address 1), and probably won't work with your 5780S.


>I was able to share that file with a penguincomputing engineer Friday.
>He was concerned that this is an issue and would escalate to broadcom
>hw engineering if necessary. unfortunately I think there has been a
>requirements/communication problem in the design spec.

Just personally, I wouldn't hold out much hope for *BSD driver support
from broadcom HW engineering. OTOH, feel free to suggest my name as a
possible recipient for an up-to-date BCM57xx programmer's manual :).