Subject: Re: wm behaviour on NetBSD
To: Chuck Swiger <cswiger@mac.com>
From: Joseph A. Dacuma <jadacuma@ched.gov.ph>
List: netbsd-users
Date: 11/30/2005 12:21:07
Hi Mr. Chuck Swiger! :)

>
> You may be right that the problem in that thread isn't exactly the issue
> affecting NetBSD 3, but it sounds close enough that checking interrupt
> rates
> and/or seeing whether it helps to reduce sharing of IRQs with the em/wm
> Intel
> NIC device is still worth the try.
>
> There seems to be a difference in the way Dell 28[0,5]0 hardware handles
> interrupts from most other APIC hardware, and this seems to affect
> Solaris/Intel and other platforms besides FreeBSD too, so if NetBSD also
> encounted slowdowns or even hardware lockups under load with the onboard
> Intel
> NIC(s), it's not an isolated issue by any means.
>
> Please do post some interrupt numbers if you are seeing the problem, so
> that
> people who know more about NetBSD then I can have specific info to work
> with.
>
> --
> -Chuck
>
> PS: Apparently switching the IRQ from being level triggered to being
> edge-triggered and then masking it until your interrupt handler has
> finished
> running will help reduce the IRQ storm.
>
> Or so I guess...I know what an APIC is, and the difference between edge
> and
> level-triggered interrupts, but how driver re-entrency is handled and
> whether
> NetBSD has a kernel thread per IRQ (aka FreeBSD ithreads), or per driver
> instance (aka IOKit in MacOS X), or uses a more classic
> dispatch/continuation
> model (older BSD) for interrupt handling is not as clear to me.
>
>

I have made another round of checks for the wm driver on NetBSD3_rc5. It
seems to be that this particular model has lots of shared IRQs reading
from its BIOS:

Device		 IRQ
VGA adapter	  7
USB 		 11,10,7,5
Gigabit NIC(0)	 11
Gigabit NIC(1)	 3

When all devices are enabled (including FD controller and Serial Port) I
tried to transfer 53.8 GBs of data. It still generates the same error
"wm0: discarding oversize frame (len -4)".

I tried to disable all USB devices, com port and 1 Gigabit NIc and still
produced the same error continously "wm0: discarding oversize frame (len
-4)" for the entire file transfer.

Below is the entire dmesg output of the machine with all devices enabled:

NetBSD 3.0_RC5 (GENERIC_MP) #0: Wed Nov 30 10:40:21 PHT 2005
	root@andres.ched.gov.ph:/usr/obj/sys/arch/amd64/compile/GENERIC_MP
total memory = 4095 MB
avail memory = 3922 MB
mainbus0 (root)
mainbus0: Intel MP Specification (Version 1.4) (DELL     PE 016E     )
cpu0 at mainbus0: apid 0 (boot processor)
cpu0:                   Intel(R) Xeon(TM) CPU 3.20GHz, 3192.19 MHz
cpu0: features: bffbfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features: bffbfbff<PGE,MCA,CMOV,PAT,PSE36,MPC,NOX,B21,MMXX,MMX>
cpu0: features: bffbfbff<FXSR,SSE,SSE2,B27,B28,LONG,3DNOW>
cpu0: L2 cache 1 MB 64B/line 8-way
cpu0: calibrating local timer
cpu0: apic clock running at 199 MHz
cpu0: 32 page colors
cpu1 at mainbus0: apid 6 (application processor)
cpu1: starting
cpu1:                   Intel(R) Xeon(TM) CPU 3.20GHz, 3192.00 MHz
cpu1: features: bffbfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu1: features: bffbfbff<PGE,MCA,CMOV,PAT,PSE36,MPC,NOX,B21,MMXX,MMX>
cpu1: features: bffbfbff<FXSR,SSE,SSE2,B27,B28,LONG,3DNOW>
cpu1: L2 cache 1 MB 64B/line 8-way
mpbios: bus 0 is type PCI
mpbios: bus 1 is type PCI
mpbios: bus 2 is type PCI
mpbios: bus 3 is type PCI
mpbios: bus 4 is type PCI
mpbios: bus 5 is type PCI
mpbios: bus 6 is type PCI
mpbios: bus 7 is type PCI
mpbios: bus 8 is type PCI
mpbios: bus 9 is type PCI
mpbios: bus 10 is type PCI
mpbios: bus 11 is type PCI
mpbios: bus 12 is type PCI
mpbios: bus 13 is type PCI
mpbios: bus 14 is type PCI
mpbios: bus 15 is type PCI
mpbios: bus 16 is type PCI
mpbios: bus 17 is type ISA
ioapic0 at mainbus0 apid 8 (I/O APIC)
ioapic0: pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 0
ioapic0: remapped to apic 8
ioapic1 at mainbus0 apid 9 (I/O APIC)
ioapic1: pa 0xfec80000, version 20, 24 pins
ioapic1: misconfigured as apic 0
ioapic1: remapped to apic 9
ioapic2 at mainbus0 apid 10 (I/O APIC)
ioapic2: pa 0xfec81000, version 20, 24 pins
ioapic2: misconfigured as apic 0
ioapic2: remapped to apic 10
ioapic3 at mainbus0 apid 11 (I/O APIC)
ioapic3: pa 0xfec83000, version 20, 24 pins
ioapic3: misconfigured as apic 0
ioapic3: remapped to apic 11
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: Intel product 0x3590 (rev. 0x09)
ppb0 at pci0 dev 2 function 0: Intel product 0x3595 (rev. 0x09)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
ppb1 at pci1 dev 0 function 0: Intel product 0x0330 (rev. 0x06)
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
amr0 at pci2 dev 14 function 0: AMI RAID <PERC 4e/Di>
amr0: interrupting at ioapic1 pin 14 (irq 7)
amr0: firmware 513O, BIOS H418, 256MB RAM
ld0 at amr0 unit 0: RAID 5, optimal
ld0: 956 GB, 124843 cyl, 255 head, 63 sec, 512 bytes/sect x 2005606400
sectors
ppb2 at pci1 dev 0 function 2: Intel product 0x0332 (rev. 0x06)
pci3 at ppb2 bus 3
pci3: i/o space, memory space enabled, rd/line, wr/inv ok
ppb3 at pci0 dev 3 function 0: Intel product 0x3596 (rev. 0x09)
pci4 at ppb3 bus 4
pci4: i/o space, memory space enabled, rd/line, wr/inv ok
ppb4 at pci4 dev 0 function 0: Intel product 0x0329 (rev. 0x09)
pci5 at ppb4 bus 5
pci5: i/o space, memory space enabled, rd/line, wr/inv ok
ppb5 at pci4 dev 0 function 2: Intel product 0x032a (rev. 0x09)
pci6 at ppb5 bus 6
pci6: i/o space, memory space enabled, rd/line, wr/inv ok
ppb6 at pci0 dev 4 function 0: Intel product 0x3597 (rev. 0x09)
pci7 at ppb6 bus 7
pci7: i/o space, memory space enabled, rd/line, wr/inv ok
ppb7 at pci0 dev 5 function 0: Intel product 0x3598 (rev. 0x09)
pci8 at ppb7 bus 10
pci8: i/o space, memory space enabled, rd/line, wr/inv ok
ppb8 at pci8 dev 0 function 0: Intel product 0x0329 (rev. 0x09)
pci9 at ppb8 bus 11
pci9: i/o space, memory space enabled, rd/line, wr/inv ok
wm0 at pci9 dev 7 function 0: Intel i82541GI 1000BASE-T Ethernet, rev. 5
wm0: interrupting at ioapic3 pin 0 (irq 11)
wm0: 32-bit 66MHz PCI bus
wm0: 65536 word (16 address bits) SPI EEPROM
wm0: Ethernet address 00:11:43:ec:90:ad
igphy0 at wm0 phy 1: Intel IGP01E1000 Gigabit PHY, rev. 0
igphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
ppb9 at pci8 dev 0 function 2: Intel product 0x032a (rev. 0x09)
pci10 at ppb9 bus 12
pci10: i/o space, memory space enabled, rd/line, wr/inv ok
wm1 at pci10 dev 8 function 0: Intel i82541GI 1000BASE-T Ethernet, rev. 5
wm1: interrupting at ioapic3 pin 1 (irq 3)
wm1: 32-bit 66MHz PCI bus
wm1: 65536 word (16 address bits) SPI EEPROM
wm1: Ethernet address 00:11:43:ec:90:ae
igphy1 at wm1 phy 1: Intel IGP01E1000 Gigabit PHY, rev. 0
igphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
ukphy0 at wm1 phy 2: Generic IEEE 802.3u media interface
ukphy0: OUI 0x000000, model 0x0000, rev. 0
ukphy0: no media present
ppb10 at pci0 dev 6 function 0: Intel product 0x3599 (rev. 0x09)
pci11 at ppb10 bus 13
pci11: i/o space, memory space enabled, rd/line, wr/inv ok
uhci0 at pci0 dev 29 function 0: Intel 82801EB/ER USB UHCI Controller
(rev. 0x02)
uhci0: interrupting at ioapic0 pin 16 (irq 11)
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 29 function 1: Intel 82801EB/ER USB UHCI Controller
(rev. 0x02)
uhci1: interrupting at ioapic0 pin 19 (irq 10)
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2 at pci0 dev 29 function 2: Intel 82801EB/ER USB UHCI Controller
(rev. 0x02)
uhci2: interrupting at ioapic0 pin 18 (irq 7)
usb2 at uhci2: USB revision 1.0
uhub2 at usb2
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
Intel 82801EB/ER USB EHCI Controller (USB serial bus, interface 0x20,
revision 0x02) at pci0 dev 29 function 7 not configured
ppb11 at pci0 dev 30 function 0: Intel 82801BA Hub-PCI Bridge (rev. 0xc2)
pci12 at ppb11 bus 16
pci12: i/o space, memory space enabled
vga0 at pci12 dev 13 function 0: ATI Technologies Radeon 7000/VE (rev. 0x00)
wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
pcib0 at pci0 dev 31 function 0
pcib0: Intel 82801EB LPC Interface Bridge (rev. 0x02)
piixide0 at pci0 dev 31 function 1
piixide0: Intel 82801EB IDE Controller (ICH5) (rev. 0x02)
piixide0: bus-master DMA support present
piixide0: primary channel configured to compatibility mode
piixide0: primary channel interrupting at ioapic0 pin 14 (irq 14)
atabus0 at piixide0 channel 0
piixide0: secondary channel configured to compatibility mode
piixide0: secondary channel interrupting at ioapic0 pin 15 (irq 15)
atabus1 at piixide0 channel 1
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker
sysbeep0 at pcppi0
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
pci13 at mainbus0 bus 8
pci13: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pci14 at mainbus0 bus 9
pci14: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pci15 at mainbus0 bus 14
pci15: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pci16 at mainbus0 bus 15
pci16: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
ioapic0: enabling
ioapic1: enabling
ioapic2: enabling
ioapic3: enabling
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
Kernelized RAIDframe activated
atapibus0 at atabus0: 2 targets
cd0 at atapibus0 drive 0: <HL-DT-STDVD-ROM GDR8082N, , 0106> cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33)
cd0(piixide0:0:0): using PIO mode 4, Ultra-DMA mode 2 (Ultra/33) (using DMA)
uhub3 at uhub1 port 1
uhub3: Dell product 0xa001, class 9/0, rev 2.00/0.00, addr 2
uhub3: 2 ports with 2 removable, self powered
boot device: ld0
root on ld0a dumps on ld0b
root file system type: ffs
cpu1: CPU 6 running
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)


----- end dmesg output-----

Did anything I miss out anything?

Joseph