Subject: Still occasional PCIe bge0 problems
To: NetBSD current-users <current-users@netbsd.org>
From: Chris Tribo <ctribo@dtcc.edu>
List: current-users
Date: 03/08/2006 11:53:41
Typically when rebooting from Windows - perhaps the windows Broadcom  
driver puts the chip into powersave?

This looks weird too, in the pci dump, it says
     Interrupt pin: 0x01 (pin A)
     Interrupt line: 0x0b

but the dmesg says ioapic0 pin 16 (irq 11). So is the interrupt pin  
being read in reverse and it should be 0x10 instead of 0x01? Or does  
this information not reflect the apic pins? ioapic pin 16 is not  
listed in vmstat -i when the chip isn't working, and it is when it  
does work.

The pci dump is different between the working and non-working dump as  
well. Here's a diff:

$ diff -bu dump.n dump.w
--- dump.n      2006-03-08 11:46:00.000000000 -0500
+++ dump.w      2006-03-08 11:43:11.000000000 -0500
@@ -1,6 +1,6 @@
PCI configuration registers:
    Common header:
-    0x00: 0x167714e4 0x00180006 0x02000001 0x00000010
+    0x00: 0x167714e4 0x00100006 0x02000001 0x00000010
     Vendor Name: Broadcom (0x14e4)
     Device Name: BCM5751 10/100/1000 Ethernet (0x1677)
@@ -15,7 +15,7 @@
      Address/data stepping: off
      System error (SERR): off
      Fast back-to-back transactions: off
-    Status register: 0x0018
+    Status register: 0x0010
      Capability List support: on
      66 MHz capable: off
      User Definable Features (UDF) support: off
@@ -67,22 +67,21 @@
       type: 0x01 (Power Management, rev. 1.0)
     Capability register at 0x50
       type: 0x03 (VPD)
     Capability register at 0x58
       type: 0x05 (MSI)
     Capability register at 0xd0
       type: 0x10 (PCI Express)
     Device-dependent header:
         0x40: 0x00000000 0x00000000 0xc0025001 0x64002100
         0x50: 0x00005803 0x00400050 0x0086d005 0x04080000
         0x60: 0x48000800 0x00000420 0x40010088 0x76180000
-       0x70: 0x00001290 0x000000df 0x00006804 0x00000000
+       0x70: 0x00001292 0x000000df 0x00006804 0x00000000
         0x80: 0x10080082 0x00000000 0x00000000 0x00000000
-       0x90: 0x00000000 0x00000000 0x00000000 0x000000ff
-       0xa0: 0x00000000 0x00000000 0x00000000 0x00000000
+       0x90: 0x00000000 0x00000000 0x00000000 0x00000143
+       0xa0: 0x00000000 0x00000044 0x00000000 0x0000002f
         0xb0: 0x00000000 0x00000000 0x00000000 0x00000000
-       0xc0: 0x00000000 0x0000c010 0x0000800e 0x00000000
+       0xc0: 0x00000000 0x0000c010 0x0000000e 0x00000000
         0xd0: 0x00010010 0x00280fa0 0x00105000 0x00036411
         0xe0: 0x00110000 0x00000000 0x00000000 0x00000000
         0xf0: 0x00000000 0x00000000 0x00000000 0x00000000


Log of booting
...
Starting virecover.
bge0: watchdog timeout -- resetting
bge0: pcie mode=0x105000
/netbsd: bge0: link state changed to DOWN
/netbsd: bge0: link state changed to UP
bge0: watchdog timeout -- resetting
bge0: pcie mode=0x105000
/netbsd: bge0: link state changed to DOWN
/netbsd: bge0: link state changed to UP
bge0: watchdog timeout -- resetting
bge0: pcie mode=0x105000
/netbsd: bge0: link state changed to DOWN
/netbsd: bge0: link state changed to UP
Starting local daemons:.
...

# host www.netbsd.org
bge0: watchdog timeout -- resetting
bge0: pcie mode=0x105000
/netbsd: bge0: link state changed to DOWN
/netbsd: bge0: link state changed to UP
bge0: watchdog timeout -- resetting
bge0: pcie mode=0x105000
/netbsd: bge0: link state changed to DOWN
/netbsd: bge0: link state changed to UP
;; connection timed out; no servers could be reached

# ifconfig bge0
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=3f80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx,UDP 
4CSUM_Rx,UDP4CSUM_Tx>
enabled=0
address=00:11:43:xx:xx:xx
media: Ethernet autoselect (100BaseTX full- 
duplex,flowcontrol,rxpause,txpause)
status: active
inet xxx.xxx.xxx.xxx netmask 0xffffff00 broadcast xxx.xxx.xxx.255
inet6 fe80::211:43ff:fexx:xxxx%bge0 prefixlen 64 scopeid 0x1

$ pcictl pci2 dump -d 0 non-working

PCI configuration registers:
  Common header:
    0x00: 0x167714e4 0x00180006 0x02000001 0x00000010

    Vendor Name: Broadcom (0x14e4)
    Device Name: BCM5751 10/100/1000 Ethernet (0x1677)
    Command register: 0x0006
     I/O space accesses: off
     Memory space accesses: on
     Bus mastering: on
     Special cycles: off
     MWI transactions: off
     Palette snooping: off
     Parity error checking: off
     Address/data stepping: off
     System error (SERR): off
     Fast back-to-back transactions: off
    Status register: 0x0018
     Capability List support: on
     66 MHz capable: off
     User Definable Features (UDF) support: off
     Fast back-to-back capable: off
     Data parity error detected: off
     DEVSEL timing: fast (0x0)
      Slave signaled Target Abort: off
      Master received Target Abort: off
      Master received Master Abort: off
      Asserted System Error (SERR): off
      Parity error detected: off
     Class Name: network (0x02)
     Subclass Name: ethernet (0x00)
     Interface: 0x00
     Revision ID: 0x01
     BIST: 0x00
     Header Type: 0x00 (0x00)
     Latency Timer: 0x00
     Cache Line Size: 0x10

    Type 0 ("normal" device) header:
     0x10: 0xdfcf0004 0x00000000 0x00000000 0x00000000
     0x20: 0x00000000 0x00000000 0x00000007 0x01791028
     0x30: 0x00000000 0x00000048 0x00000000 0x0000010b

     Base address register at 0x10
       type: 64-bit nonprefetchable memory
       base: 0x00000000dfcf0000, not sized
     Base address register at 0x18
       not implemented(?)
     Base address register at 0x1c
       not implemented(?)
     Base address register at 0x20
       not implemented(?)
     Base address register at 0x24
       not implemented(?)
     Cardbus CIS PointeR: 0x00000007
     Subsystem vendor ID: 0x1028
     Subsystem ID: 0x0179
     Expansion ROM Base Address: 0x00000000
     Capability list pointer: 0x48
     Reserved @ 0x38: 0x00000000
     Maximum Latency: 0x00
     Minimum Grant: 0x00
     Interrupt pin: 0x01 (pin A)
     Interrupt line: 0x0b

    Capability register at 0x48
      type: 0x01 (Power Management, rev. 1.0)
    Capability register at 0x50
      type: 0x03 (VPD)
    Capability register at 0x48
      type: 0x05 (MSI)
    Capability register at 0xd0
      type: 0x10 (PCI Express)

    Device-dependent header:

	0x40: 0x00000000 0x00000000 0xc0025001 0x64002100
	0x50: 0x00005803 0x00400050 0x0086d005 0x04080000
	0x60: 0x48000800 0x00000420 0x40010088 0x76180000
	0x70: 0x00001290 0x000000df 0x00006804 0x00000000
	0x80: 0x10080082 0x00000000 0x00000000 0x00000000
	0x90: 0x00000000 0x00000000 0x00000000 0x000000ff
	0xa0: 0x00000000 0x00000000 0x00000000 0x00000000
	0xb0: 0x00000000 0x00000000 0x00000000 0x00000000
	0xc0: 0x00000000 0x0000c010 0x0000800e 0x00000000
	0xd0: 0x00010010 0x00280fa0 0x00105000 0x00036411
	0xe0: 0x00110000 0x00000000 0x00000000 0x00000000
	0xf0: 0x00000000 0x00000000 0x00000000 0x00000000


dmesg snippit:
bge0 at pci2 dev 0 function 0: Broadcom BCM5751 Gigabit Ethernet
bge0: interrupting at ioapic0 pin 16 (irq 11)
bge0: pcie mode=0x105000
bge0: ASIC BCM5750 A1 (0x4001), Ethernet address 00:11:43:xx:xx:xx
bge0: setting short Tx thresholds
brgphy0 at bge0 phy 1: BCM5750 1000BASE-T media interface, rev. 0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,  
1000baseT-FDX, auto