>Number:         23876
>Category:       kern
>Synopsis:       ioapic(?) causes problems for my fxp0 NIC on amd64.
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Dec 24 22:14:00 UTC 2003
>Closed-Date:
>Last-Modified:
>Originator:     Richard Rauch
>Release:        NetBSD 1.6ZG (recent CVS and custom kernel)
>Organization:
  "I probably don't know what I'm talking about."  http://www.olib.org/~rkr/
>Environment:
System: NetBSD socrates 1.6ZG NetBSD 1.6ZG (socrates) #0: Tue Dec 23 06:30:20 CST 2003 root@socrates:/usr/netbsd/current/src/sys/arch/amd64/compile/obj.amd64/socrates amd64
Architecture: x86_64
Machine: amd64
>Description:
	This may be amd64-based, but since the bus handling is supposed
	to be largely platform-independant, and the problems went away
	when I removed the ioapic, I am filing this as a kernel PR.

	I've been unable to use my fxp? Intel NIC.  I wasn't sure of the
	problem and assumed that it was caused by funky motherboard
	chipsets.  I was also having trouble with a ~working rtk? NIC
	(which seems like it may in some way have been clashing with the
	motherboard USB; depending on BIOS options and physical card
	positions (IRQs?), one or the other of USB or rtk0 would apparently
	lose an interrupt and the driver would effectively die).

	Symptoms:
		Several things seemed to get timeouts:

		fxp0 ALWAYS had timeouts on data receive, but could send
		packets.  It was effectively useless as far as I could tell.

		Depending, rtk0 would at RANDOM times get a "watchdog"
		timeout, at which point it would practically cease operation.
		It seemed to process about 1 network packet every 1 (or 10?)
		seconds, so that it was possible to gracefully break network
		connections (such as ssh or NFS) if you are patient.

		Alternately, USB would appear to completely die.  This
		would fail with a "usbd_setup_pipe" TIMEOUT error.  The
		TIMEOUT error would only be displayed after USB froze
		(my pointer device is a USB trackball) and then after
		(re)opening a USB device (e.g., exiting and restarting
		X).

	I never found a state where *both* USB and rtk0 would die.

	I never found a state where GENERIC would let me use fxp0.

	It was suggested to me that I try removing ioapic
	from my kernel (at the time a stock GENERIC amd64).  I also
	removed the MPBIOS-related options.

	Since that time, USB has not died.  rtk0 has not died.  fxp0
	WORKS.

	Here is a dmesg with ioapic enabled; it also includes the USB
	failure message:


NetBSD 1.6ZG (GENERIC) #6: Thu Dec 18 20:28:19 CST 2003
	root@socrates:/usr/netbsd/current/src/sys/arch/amd64/compile/obj.amd64/GENERIC
total memory = 511 MB
avail memory = 457 MB
using 6144 buffers containing 26292 KB of memory
mainbus0 (root)
mainbus0: Intel MP Specification (Version 1.4) (OEM00000 PROD00000000)
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD Athlon(tm) 64 Processor 3200+, 1994.98 MHz
cpu0: features: e7dbfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features: e7dbfbff<PGE,MCA,CMOV,PAT,PSE36,MPC,NOX,MMXX,MMX>
cpu0: features: e7dbfbff<FXSR,SSE,SSE2,LONG,3DNOW2,3DNOW>
cpu0: I-cache 64 KB 64b/line 2-way, D-cache 64 KB 64b/line 2-way
cpu0: L2 cache 1 MB 64b/line 16-way
cpu0: ITLB 32 4 KB entries fully associative, 8 4 MB entries fully associative
cpu0: DTLB 32 4 KB entries fully associative, 8 4 MB entries fully associative
cpu0: calibrating local timer
cpu0: apic clock running at 199 MHz
cpu0: 16 page colors
mpbios: bus 0 is type PCI   
mpbios: bus 1 is type PCI   
mpbios: bus 2 is type PCI   
mpbios: bus 3 is type ISA   
ioapic0 at mainbus0 apid 2 (I/O APIC)
ioapic0: pa 0xfec00000, version 11, 24 pins
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: Nvidia Corporation nForce3 Host-PCI bridge (rev. 0xa4)
pcib0 at pci0 dev 1 function 0
pcib0: Nvidia Corporation nForce3 PCI-ISA bridge (rev. 0xa6)
Nvidia Corporation nForce3 SMBus controller (SMBus serial bus, revision 0xa4) at pci0 dev 1 function 1 not configured
ohci0 at pci0 dev 2 function 0: Nvidia Corporation nForce3 USB Host Controller (rev. 0xa5)
pci_intr_map: bus 0 dev 2 func 0 pin 1; line 9
pci_intr_map: no MP mapping found
ohci0: interrupting at irq 9
ohci0: OHCI version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
uhub0 at usb0
uhub0: Nvidia Corporat OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 3 ports with 3 removable, self powered
ohci1 at pci0 dev 2 function 1: Nvidia Corporation nForce3 USB Host Controller (rev. 0xa5)
pci_intr_map: bus 0 dev 2 func 1 pin 2; line 9
pci_intr_map: no MP mapping found
ohci1: interrupting at irq 9
ohci1: OHCI version 1.0, legacy support
usb1 at ohci1: USB revision 1.0
uhub1 at usb1
uhub1: Nvidia Corporat OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 3 ports with 3 removable, self powered
Nvidia Corporation nForce3 USB2 Host Controller (USB serial bus, interface 0x20, revision 0xa2) at pci0 dev 2 function 2 not configured
auich0 at pci0 dev 6 function 0: nForce3 MCP-T AC-97 Audio
auich0: interrupting at ioapic0 pin 11 (irq 11)
auich0: ac97: Avance Logic unknown (0x414c4780) codec; 20 bit DAC, 18 bit ADC, no 3D stereo
auich0: ac97: ext id 9c7<AC97_23,LDAC,SDAC,CDAC,SPDIF,DRA,VRA>
pciide0 at pci0 dev 8 function 0
pciide0: Nvidia Corporation nForce3 ATA133 IDE (rev. 0xa5)
pciide0: bus-master DMA support present, but unused (no driver support)
pciide0: primary channel configured to compatibility mode
pciide0: primary channel interrupting at ioapic0 pin 14 (irq 14)
atabus0 at pciide0 channel 0
pciide0: secondary channel configured to compatibility mode
pciide0: secondary channel interrupting at ioapic0 pin 15 (irq 15)
atabus1 at pciide0 channel 1
ppb0 at pci0 dev 10 function 0: Nvidia Corporation nforce3 PCI-PCI bridge (rev. 0xa2)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled
fxp0 at pci1 dev 7 function 0: i82558 Ethernet, rev 5
fxp0: interrupting at ioapic0 pin 19 (irq 5)
fxp0: Ethernet address 00:03:47:72:7f:87
inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 0
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
rtk0 at pci1 dev 8 function 0: RealTek 8139 10/100BaseTX
rtk0: interrupting at ioapic0 pin 11 (irq 11)
rtk0: Ethernet address 00:90:47:05:bc:75
ukphy0 at rtk0 phy 7: Generic IEEE 802.3u media interface
ukphy0: OUI 0x000000, model 0x0000, rev. 0
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
Realtek Semiconductor 8169 10/100/1000 Ethernet (ethernet network, revision 0x10) at pci1 dev 11 function 0 not configured
Integrated Technology Express, Inc. product 0x8212 (RAID mass storage, revision 0x11) at pci1 dev 12 function 0 not configured
Texas Instruments TSB43AA23 OHCI IEEE 1394 Host Controller (Firewire serial bus, interface 0x10) at pci1 dev 14 function 0 not configured
ppb1 at pci0 dev 11 function 0: Nvidia Corporation product 0x00d2 (rev. 0xa4)
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled
vga0 at pci2 dev 0 function 0: ATI Technologies Radeon 9200SE (rev. 0x01)
wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
pchb1 at pci0 dev 24 function 0
pchb1: Advanced Micro Devices AMD64 HyperTransport configuration (rev. 0x00)
pchb2 at pci0 dev 24 function 1
pchb2: Advanced Micro Devices AMD64 Address Map configuration (rev. 0x00)
pchb3 at pci0 dev 24 function 2
pchb3: Advanced Micro Devices AMD64 DRAM configuration (rev. 0x00)
pchb4 at pci0 dev 24 function 3
pchb4: Advanced Micro Devices AMD64 Miscellaneous configuration (rev. 0x00)
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
lpt0 at isa0 port 0x378-0x37b irq 7
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker
sysbeep0 at pcppi0
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
ioapic0: enabling
auich0: measured ac97 link rate at 16036 Hz, will use 48000 Hz
audio0 at auich0: full duplex, mmap, independent
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
Kernelized RAIDframe activated
uhidev0 at uhub0 port 3 configuration 1 interface 0
uhidev0: Microsoft Microsoft Trackball OpticalM-., rev 1.10/1.21, addr 2, iclass 3/1
ums0 at uhidev0: 5 buttons and Z dir.
wsmouse0 at ums0 mux 0
wd0 at atabus0 drive 0: <WDC WD400BB-75DKA0>
wd0: drive supports 16-sector PIO transfers, LBA48 addressing
wd0: 38146 MB, 77504 cyl, 16 head, 63 sec, 512 bytes/sect x 78125000 sectors
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
atapibus0 at atabus1: 2 targets
cd0 at atapibus0 drive 1: <LG       DVD-ROM DRD-8160B, , 1.01> cdrom removable
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33)
boot device: wd0
root on wd0a dumps on wd0b
WARNING: invalid time in clock chip
WARNING: CHECK AND RESET THE DATE!
root file system type: ffs
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x50, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)
fxp0: device timeout
fxp0: device timeout
usbd_setup_pipe: failed to start endpoint, TIMEOUT

>How-To-Repeat:
	On my system, try to use a GENERIC kernel with one of my
	"CINPRO100PCI-A" fxp0 cards, and/or try to use GENERIC with
	X and this particular rtk0 ethernet card for a protracted
	period.

>Fix:
	Dunno about a *fix*.  The workaround is to build a custom
	kernel without ioapic (and/or MPBIOS).
>Release-Note:
>Audit-Trail:
>Unformatted: