Re: kern/48733: deadlock in if_output() with interrupt on KERNEL_LOCK

[Wolfgang Stukenbrock <wolfgang.stukenbrock%nagler-company.com@localhost>]

Hi again,

I've found a "hidden" option "MPVERBOSE" and enabled it.
Why the hell are some options missing in the GENERIC (or ALL - if 
present) config file?
What is the expected way to find them ????

Now I've some additional information about the ioapic setup.
Full output of dmesg.boot until interrupts are enabled follow:

s011a# cat /var/run/dmesg.boot
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009, 2010, 2011, 2012
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 6.1 (NSW-S011) #0: Fri May 16 12:33:18 CEST 2014
        root@s011a:/usr/src/sys/arch/amd64/compile/NSW-S011
total memory = 8189 MB
avail memory = 7945 MB
timecounter: Timecounters tick every 10.000 msec
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
Supermicro X7SBi-LN4 (0123456789)
mainbus0 (root)
mainbus0: scanning 0x9dc00 to 0x9dff0 for MP signature
mainbus0: scanning 0x9d800 to 0x9dbf0 for MP signature
mainbus0: scanning 0xf0000 to 0xffff0 for MP signature
mainbus0: MP floating pointer found in bios at 0xf6570
mainbus0: MP config table at 0x9e0a1, 484 bytes long
cpu0 at mainbus0 apid 0cpu0: prelint0 
0x700<vector=0x0,delmode=0x7,dest=0x0> 0x0<target=0x0>
cpu0: prelint1 0x400<vector=0x0,delmode=0x4,dest=0x0> 0x0<target=0x0>
cpu0: timer0 0x10000<vector=0x0,delmode=0x0,masked,dest=0x0> 0x0<target=0x0>
cpu0: pcint0 0x10000<vector=0x0,delmode=0x0,masked,dest=0x0> 0x0<target=0x0>
cpu0: lint0 0x700<vector=0x0,delmode=0x7,dest=0x0> 0x0<target=0x0>
cpu0: lint1 0x400<vector=0x0,delmode=0x4,dest=0x0> 0x0<target=0x0>
cpu0: err0 0x10000<vector=0x0,delmode=0x0,masked,dest=0x0> 0x0<target=0x0>
: Intel(R) Xeon(R) CPU           L3110  @ 3.00GHz, id 0x1067a
cpu0: idle lwp at 0xfffffe822f746840, idle sp at 0xfffffe810e967c50
cpu1 at mainbus0 apid 1: Intel(R) Xeon(R) CPU           L3110  @ 
3.00GHz, id 0x1067a
cpu1: idle lwp at 0xfffffe810e9b78e0, idle sp at 0xfffffe810e994c50
ioapic0 at mainbus0 apid 2: pa 0xfec00000, virtual wire mode, version 
20, 24 pins
ioapic1 at mainbus0 apid 3: pa 0xfecc0000, virtual wire mode, version 
20, 24 pins
acpi0 at mainbus0: Intel ACPICA 20110623
acpi0: X/RSDT: OemId <PTLTD ,    XSDT  ,06040000>, AslId < LTP,00000000>
ioapic0: int9 0x8060<vector=0x60,delmode=0x0,level,dest=0x0> 0x0<target=0x0>
mpacpi: found root PCI bus 0
mpacpi: found subordinate bus 0
mpacpi: found subordinate bus 0
mpacpi: found subordinate bus 0
mpacpi: found subordinate bus 0
mpacpi: found subordinate bus 0
mpacpi: found subordinate bus 0
mpacpi: found subordinate bus 0
mpacpi: 8 PCI busses
mpacpi: ISA interrupt override 0 -> 2 (1/1)
mpacpi: ISA interrupt override 9 -> 9 (1/3)
mpacpi: configuring PCI bus 0 int routing
mpacpi: configuring PCI bus 1 int routing
mpacpi: configuring PCI bus 2 int routing
mpacpi: configuring PCI bus 5 int routing
mpacpi: configuring PCI bus 7 int routing
mpacpi: configuring PCI bus 9 int routing
mpacpi: configuring PCI bus 11 int routing
mpacpi: configuring PCI bus 17 int routing
ioapic0: pin 2 attached to isa0 irq 0 (type 0x0<type=0x0> flags 
0x5<pol=0x1=Act Hi,trig=0x1=Edge>)
ioapic0: pin 1 attached to isa0 irq 1 (type 0x0<type=0x0> flags 
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 3 attached to isa0 irq 3 (type 0x0<type=0x0> flags 
0x0<pol=0x0,trig=0x0>)^M
ioapic0: pin 4 attached to isa0 irq 4 (type 0x0<type=0x0> flags 
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 5 attached to isa0 irq 5 (type 0x0<type=0x0> flags 
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 6 attached to isa0 irq 6 (type 0x0<type=0x0> flags 
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 7 attached to isa0 irq 7 (type 0x0<type=0x0> flags 
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 8 attached to isa0 irq 8 (type 0x0<type=0x0> flags 
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 9 attached to isa0 irq 9 (type 0x0<type=0x0> flags 
0xd<pol=0x1=Act Hi,trig=0x3=Level>)
ioapic0: pin 10 attached to isa0 irq 10 (type 0x0<type=0x0> flags 
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 11 attached to isa0 irq 11 (type 0x0<type=0x0> flags 
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 12 attached to isa0 irq 12 (type 0x0<type=0x0> flags 
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 13 attached to isa0 irq 13 (type 0x0<type=0x0> flags 
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 14 attached to isa0 irq 14 (type 0x0<type=0x0> flags 
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 15 attached to isa0 irq 15 (type 0x0<type=0x0> flags 
0x0<pol=0x0,trig=0x0>)
local apic: pin 1 attached to NMI (type 0x1<type=0x1=NMI> flags 
0x0<pol=0x0,trig=0x0>)
local apic: pin 1 attached to NMI (type 0x1<type=0x1=NMI> flags 
0x0<pol=0x0,trig=0x0>)
ioapic0: pin 16 attached to pci0 device 1 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci0 device 2 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci0 device 3 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 17 attached to pci0 device 3 INT_B (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 18 attached to pci0 device 3 INT_C (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 19 attached to pci0 device 3 INT_D (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci0 device 6 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci0 device 25 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci0 device 26 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 17 attached to pci0 device 26 INT_B (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 18 attached to pci0 device 26 INT_C (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci0 device 27 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci0 device 28 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 17 attached to pci0 device 28 INT_B (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 18 attached to pci0 device 28 INT_C (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 19 attached to pci0 device 28 INT_D (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 23 attached to pci0 device 29 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 22 attached to pci0 device 29 INT_B (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 21 attached to pci0 device 29 INT_C (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 19 attached to pci0 device 29 INT_D (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci0 device 31 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 17 attached to pci0 device 31 INT_B (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 18 attached to pci0 device 31 INT_C (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 19 attached to pci0 device 31 INT_D (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic1: pin 0 attached to pci2 device 1 INT_A (type 0x0<type=0x0> flags 
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic1: pin 1 attached to pci2 device 1 INT_B (type 0x0<type=0x0> flags 
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic1: pin 2 attached to pci2 device 1 INT_C (type 0x0<type=0x0> flags 
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic1: pin 3 attached to pci2 device 1 INT_D (type 0x0<type=0x0> flags 
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic1: pin 4 attached to pci2 device 2 INT_A (type 0x0<type=0x0> flags 
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic1: pin 5 attached to pci2 device 2 INT_B (type 0x0<type=0x0> flags 
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic1: pin 4 attached to pci2 device 2 INT_C (type 0x0<type=0x0> flags 
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic1: pin 5 attached to pci2 device 2 INT_D (type 0x0<type=0x0> flags 
0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci5 device 0 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 17 attached to pci5 device 0 INT_B (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 18 attached to pci5 device 0 INT_C (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 19 attached to pci5 device 0 INT_D (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 17 attached to pci7 device 0 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 18 attached to pci7 device 0 INT_B (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 19 attached to pci7 device 0 INT_C (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci7 device 0 INT_D (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 18 attached to pci9 device 0 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 19 attached to pci9 device 0 INT_B (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci9 device 0 INT_C (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 17 attached to pci9 device 0 INT_D (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 19 attached to pci11 device 0 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 16 attached to pci11 device 0 INT_B (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 17 attached to pci11 device 0 INT_C (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 18 attached to pci11 device 0 INT_D (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 20 attached to pci17 device 0 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 21 attached to pci17 device 0 INT_B (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 22 attached to pci17 device 0 INT_C (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 23 attached to pci17 device 0 INT_D (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 22 attached to pci17 device 3 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
ioapic0: pin 23 attached to pci17 device 4 INT_A (type 0x0<type=0x0> 
flags 0xf<pol=0x3=Act Lo,trig=0x3=Level>)
acpi0: SCI interrupting at int 9
timecounter: Timecounter "ACPI-Fast" frequency 3579545 Hz quality 1000
hpet0 at acpi0: high precision event timer (mem 0xfed00000-0xfed00400)
timecounter: Timecounter "hpet0" frequency 14318180 Hz quality 2000
MBRD (PNP0C02) at acpi0 not configured
pcppi0 at acpi0 (SPKR, PNP0800): io 0x61
sysbeep0 at pcppi0
attimer0 at acpi0 (TIMR, PNP0100): io 0x40-0x43,0x50-0x53
FWH (INT0800) at acpi0 not configured
pckbc0 at acpi0 (KBC0, PNP0303) (kbd port): io 0x60,0x64 irq 1
pckbc1 at acpi0 (MSE0, PNP0F13) (aux port): irq 12
com0 at acpi0 (COM1, PNP0501-1): io 0x3f8-0x3ff irq 4
com0: ns16550a, working fifo
com0: console
ioapic0: int4 0x81<vector=0x81,delmode=0x0,dest=0x0> 0x0<target=0x0>
com1 at acpi0 (COM2, PNP0501-2): io 0x2f8-0x2ff irq 3
com1: ns16550a, working fifo
ioapic0: int3 0x82<vector=0x82,delmode=0x0,dest=0x0> 0x0<target=0x0>
acpibut0 at acpi0 (PWRB, PNP0C0C): ACPI Power Button
apm0 at acpi0: Power Management spec V1.2
attimer0: attached to pcppi0
pckbd0 at pckbc0 (kbd slot)
ioapic0: int1 0x61<vector=0x61,delmode=0x0,dest=0x0> 0x0<target=0x0>
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0 (mux ignored)
pci0 at mainbus0 bus 0: configuration mode 1
mainbus0: added to list as bus 0
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0: Intel product 0x29f0 (rev. 0x01)
ppb0 at pci0 dev 1 function 0: Intel product 0x29f1 (rev. 0x01)
ppb0: PCI Express 2.0 <Root Port of PCI-E Root Complex>
pci1 at ppb0 bus 1
ppb0: added to list as bus 1
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
ppb1 at pci1 dev 0 function 0: Intel product 0x032c (rev. 0x09)
ppb1: PCI Express 1.0 <PCI-E to PCI/PCI-X Bridge>
pci2 at ppb1 bus 2
ppb1: added to list as bus 2
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
wm0 at pci2 dev 1 function 0: Intel i82546GB 1000BASE-T Ethernet (rev. 0x03)
ioapic1: int0 0xa062<vector=0x62,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
wm0: interrupting at ioapic1 pin 0
wm0: 64-bit 133MHz PCIX bus
wm0: 256 word (8 address bits) MicroWire EEPROM
wm0: Ethernet address 00:1b:21:06:12:2a
makphy0 at wm0 phy 1: Marvell 88E1011 Gigabit PHY, rev. 5
makphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
wm1 at pci2 dev 1 function 1: Intel i82546GB 1000BASE-T Ethernet (rev. 0x03)
ioapic1: int1 0xa063<vector=0x63,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
wm1: interrupting at ioapic1 pin 1
wm1: 64-bit 133MHz PCIX bus
wm1: 256 word (8 address bits) MicroWire EEPROM
wm1: Ethernet address 00:1b:21:06:12:2b
makphy1 at wm1 phy 1: Marvell 88E1011 Gigabit PHY, rev. 5
makphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
Intel product 0x0326 (interrupt system, interface 0x20, revision 0x09) 
at pci1 dev 0 function 1 not configured
uhci0 at pci0 dev 26 function 0: Intel 82801I USB UHCI Controller (rev. 
0x02)
ioapic0: int16 0xa064<vector=0x64,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
uhci0: interrupting at ioapic0 pin 16
usb0 at uhci0: USB revision 1.0
uhci1 at pci0 dev 26 function 1: Intel 82801I USB UHCI Controller (rev. 
0x02)
ioapic0: int17 0xa065<vector=0x65,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
uhci1: interrupting at ioapic0 pin 17
usb1 at uhci1: USB revision 1.0
uhci2 at pci0 dev 26 function 2: Intel 82801I USB UHCI Controller (rev. 
0x02)
ioapic0: int18 0xa066<vector=0x66,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
uhci2: interrupting at ioapic0 pin 18
usb2 at uhci2: USB revision 1.0
ehci0 at pci0 dev 26 function 7: Intel 82801I USB EHCI Controller (rev. 
0x02)
ioapic0: int18 0xa066<vector=0x66,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
ehci0: interrupting at ioapic0 pin 18
ehci0: EHCI version 1.0
ehci0: companion controllers, 2 ports each: uhci0 uhci1 uhci2
usb3 at ehci0: USB revision 2.0
ppb2 at pci0 dev 28 function 0: Intel 82801I PCI Express Port #1 (rev. 0x02)
ppb2: PCI Express 1.0 <Root Port of PCI-E Root Complex>
pci3 at ppb2 bus 5
ppb2: added to list as bus 5
pci3: i/o space, memory space enabled, rd/line, wr/inv ok
wm2 at pci3 dev 0 function 0: Intel i82573E IAMT (rev. 0x03)
ioapic0: int16 0xa064<vector=0x64,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
wm2: interrupting at ioapic0 pin 16
wm2: PCI-Express bus
wm2: 256 word (8 address bits) SPI EEPROM
wm2: Ethernet address 00:30:48:fd:4f:8a
makphy2 at wm2 phy 1: Marvell 88E1111 Gigabit PHY, rev. 2
makphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
ppb3 at pci0 dev 28 function 1: Intel 82801I PCI Express Port #2 (rev. 0x02)
ppb3: PCI Express 1.0 <Root Port of PCI-E Root Complex>
pci4 at ppb3 bus 7
ppb3: added to list as bus 7
pci4: i/o space, memory space enabled, rd/line, wr/inv ok
wm3 at pci4 dev 0 function 0: Intel i82573L Gigabit Ethernet (rev. 0x00)
ioapic0: int17 0xa065<vector=0x65,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
wm3: interrupting at ioapic0 pin 17
wm3: PCI-Express bus
wm3: 256 word (8 address bits) SPI EEPROM
wm3: Ethernet address 00:30:48:fd:4f:8b
makphy3 at wm3 phy 1: Marvell 88E1111 Gigabit PHY, rev. 2
makphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
ppb4 at pci0 dev 28 function 2: Intel 82801I PCI Express Port #3 (rev. 0x02)
ppb4: PCI Express 1.0 <Root Port of PCI-E Root Complex>
pci5 at ppb4 bus 9
ppb4: added to list as bus 9
pci5: i/o space, memory space enabled, rd/line, wr/inv ok
wm4 at pci5 dev 0 function 0: Intel i82573L Gigabit Ethernet (rev. 0x00)
ioapic0: int18 0xa066<vector=0x66,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
wm4: interrupting at ioapic0 pin 18
wm4: PCI-Express bus
wm4: 256 word (8 address bits) SPI EEPROM
wm4: Ethernet address 00:30:48:fd:4f:8c
makphy4 at wm4 phy 1: Marvell 88E1111 Gigabit PHY, rev. 2
makphy4: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
ppb5 at pci0 dev 28 function 3: Intel 82801I PCI Express Port #4 (rev. 0x02)
ppb5: PCI Express 1.0 <Root Port of PCI-E Root Complex>
pci6 at ppb5 bus 11
ppb5: added to list as bus 11
pci6: i/o space, memory space enabled, rd/line, wr/inv ok
wm5 at pci6 dev 0 function 0: Intel i82573L Gigabit Ethernet (rev. 0x00)
ioapic0: int19 0xa067<vector=0x67,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
wm5: interrupting at ioapic0 pin 19
wm5: PCI-Express bus
wm5: 256 word (8 address bits) SPI EEPROM
wm5: Ethernet address 00:30:48:fd:4f:8d
makphy5 at wm5 phy 1: Marvell 88E1111 Gigabit PHY, rev. 2
makphy5: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
uhci3 at pci0 dev 29 function 0: Intel 82801I USB UHCI Controller (rev. 
0x02)
ioapic0: int23 0xa068<vector=0x68,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
uhci3: interrupting at ioapic0 pin 23
usb4 at uhci3: USB revision 1.0
uhci4 at pci0 dev 29 function 1: Intel 82801I USB UHCI Controller (rev. 
0x02)
ioapic0: int22 0xa069<vector=0x69,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
uhci4: interrupting at ioapic0 pin 22
usb5 at uhci4: USB revision 1.0
uhci5 at pci0 dev 29 function 2: Intel 82801I USB UHCI Controller (rev. 
0x02)
ioapic0: int21 0xa06a<vector=0x6a,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
uhci5: interrupting at ioapic0 pin 21
usb6 at uhci5: USB revision 1.0
ehci1 at pci0 dev 29 function 7: Intel 82801I USB EHCI Controller (rev. 
0x02)
ioapic0: int23 0xa068<vector=0x68,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
ehci1: interrupting at ioapic0 pin 23
ehci1: EHCI version 1.0
ehci1: companion controllers, 2 ports each: uhci3 uhci4 uhci5
usb7 at ehci1: USB revision 2.0
ppb6 at pci0 dev 30 function 0: Intel 82801BA Hub-PCI Bridge (rev. 0x92)
pci7 at ppb6 bus 17
ppb6: added to list as bus 17
pci7: i/o space, memory space enabled
vga0 at pci7 dev 3 function 0: ATI Technologies ES1000 (rev. 0x02)
wsdisplay0 at vga0 (kbdmux ignored)
drm at vga0 not configured
pciide0 at pci7 dev 4 function 0: Integrated Technology Express product 
0x8213 (rev. 0x00)
pciide0: bus-master DMA support present, but unused (no driver support)
pciide0: primary channel wired to native-PCI mode
ioapic0: int23 0xa068<vector=0x68,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
pciide0: using ioapic0 pin 23 for native-PCI interrupt
atabus0 at pciide0 channel 0
pciide0: secondary channel wired to native-PCI mode
atabus1 at pciide0 channel 1
ichlpcib0 at pci0 dev 31 function 0: Intel 82801IR LPC Interface Bridge 
(rev. 0x02)
timecounter: Timecounter "ichlpcib0" frequency 3579545 Hz quality 1000
ichlpcib0: 24-bit timer
ichlpcib0: TCO (watchdog) timer configured.
gpio0 at ichlpcib0: 64 pins
piixide0 at pci0 dev 31 function 2: Intel 82801I Serial ATA Controller 
(ICH9) (rev. 0x02)
piixide0: bus-master DMA support present
piixide0: primary channel configured to compatibility mode
ioapic0: int14 0x6b<vector=0x6b,delmode=0x0,dest=0x0> 0x0<target=0x0>
piixide0: primary channel interrupting at ioapic0 pin 14
atabus2 at piixide0 channel 0
piixide0: secondary channel configured to compatibility mode
ioapic0: int15 0x6c<vector=0x6c,delmode=0x0,dest=0x0> 0x0<target=0x0>
piixide0: secondary channel interrupting at ioapic0 pin 15
atabus3 at piixide0 channel 1
ichsmb0 at pci0 dev 31 function 3: Intel 82801I SMBus Controller (rev. 0x02)
ioapic0: int17 0xa065<vector=0x65,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
ichsmb0: interrupting at ioapic0 pin 17
iic0 at ichsmb0: I2C bus
spdmem0 at iic0 addr 0x50
spdmem0: DDR2 SDRAM, data ECC, 2GB, 800MHz (PC2-6400)
spdmem0: 14 rows, 10 cols, 2 ranks, 8 banks/chip, 2.50ns cycle time
spdmem0: tAA-tRCD-tRP-tRAS: 5-5-5-18
spdmem0: voltage SSTL 1.8V, refresh time 7.8us (self-refreshing)
spdmem1 at iic0 addr 0x51
spdmem1: DDR2 SDRAM, data ECC, 2GB, 800MHz (PC2-6400)
spdmem1: 14 rows, 10 cols, 2 ranks, 8 banks/chip, 2.50ns cycle time
spdmem1: tAA-tRCD-tRP-tRAS: 5-5-5-18
spdmem1: voltage SSTL 1.8V, refresh time 7.8us (self-refreshing)
spdmem2 at iic0 addr 0x52
spdmem2: DDR2 SDRAM, data ECC, 2GB, 800MHz (PC2-6400)
spdmem2: 14 rows, 10 cols, 2 ranks, 8 banks/chip, 2.50ns cycle time
spdmem2: tAA-tRCD-tRP-tRAS: 5-5-5-18
spdmem2: voltage SSTL 1.8V, refresh time 7.8us (self-refreshing)
spdmem3 at iic0 addr 0x53
spdmem3: DDR2 SDRAM, data ECC, 2GB, 800MHz (PC2-6400)
spdmem3: 14 rows, 10 cols, 2 ranks, 8 banks/chip, 2.50ns cycle time
spdmem3: tAA-tRCD-tRP-tRAS: 5-5-5-18
spdmem3: voltage SSTL 1.8V, refresh time 7.8us (self-refreshing)
piixide1 at pci0 dev 31 function 5: Intel 82801I Serial ATA Controller 
(ICH9) (rev. 0x02)
piixide1: bus-master DMA support present
piixide1: primary channel wired to native-PCI mode
ioapic0: int18 0xa066<vector=0x66,delmode=0x0,actlo,level,dest=0x0> 
0x0<target=0x0>
piixide1: using ioapic0 pin 18 for native-PCI interrupt
atabus4 at piixide1 channel 0
piixide1: secondary channel wired to native-PCI mode
atabus5 at piixide1 channel 1
Intel 82801I Thermal Controller (miscellaneous DASP, revision 0x02) at 
pci0 dev 31 function 6 not configured
isa0 at ichlpcib0
lm0 at isa0 port 0x290-0x297
lm0: Using default temp sensors
lm0: Winbond W83627HF Hardware monitor
acpicpu0 at cpu0: ACPI CPU
acpicpu0: C1: HLT, lat   0 us, pow     0 mW
acpicpu0: P0: FFH, lat  10 us, pow 35000 mW, 3000 MHz
acpicpu0: P1: FFH, lat  10 us, pow 28000 mW, 2667 MHz
acpicpu0: P2: FFH, lat  10 us, pow 22000 mW, 2333 MHz
acpicpu0: P3: FFH, lat  10 us, pow 16000 mW, 2000 MHz
acpicpu0: T0: FFH, lat   1 us, pow 16000 mW, 100 %
acpicpu0: T1: FFH, lat   1 us, pow 14000 mW,  88 %
acpicpu0: T2: FFH, lat   1 us, pow 12000 mW,  75 %
acpicpu0: T3: FFH, lat   1 us, pow 10000 mW,  63 %
acpicpu0: T4: FFH, lat   1 us, pow  8000 mW,  50 %
acpicpu0: T5: FFH, lat   1 us, pow  6000 mW,  38 %
acpicpu0: T6: FFH, lat   1 us, pow  4000 mW,  25 %
acpicpu0: T7: FFH, lat   1 us, pow  2000 mW,  13 %
coretemp0 at cpu0: thermal sensor, 1 C resolution
acpicpu1 at cpu1: ACPI CPU
coretemp1 at cpu1: thermal sensor, 1 C resolution
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
cpu1: prelint0 0x10000<vector=0x0,delmode=0x0,masked,dest=0x0> 
0x0<target=0x0>
cpu1: prelint1 0x10000<vector=0x0,delmode=0x0,masked,dest=0x0> 
0x0<target=0x0>
cpu1: timer0 0x10000<vector=0x0,delmode=0x0,masked,dest=0x0> 0x0<target=0x0>
cpu1: pcint0 0x10000<vector=0x0,delmode=0x0,masked,dest=0x0> 0x0<target=0x0>
cpu1: lint0 0x10700<vector=0x0,delmode=0x7,masked,dest=0x0> 0x0<target=0x0>
cpu1: lint1 0x400<vector=0x0,delmode=0x4,dest=0x0> 0x0<target=0x0>


One interesting thing is, that only pin 0 to 5 of ioapic1 are printed, 
but ioapic1 is reported to have 24 pins ...
May this lead to some "uninitialized" pin-setup and perhaps suprious 
interrupts?

W. Stukenbrock

Wolfgang Stukenbrock wrote:

> The following reply was made to PR kern/48733; it has been noted by GNATS.
>
> From: Wolfgang Stukenbrock <wolfgang.stukenbrock%nagler-company.com@localhost>
> To: gnats-bugs%NetBSD.org@localhost
> Cc: kern-bug-people%netbsd.org@localhost, gnats-admin%netbsd.org@localhost, 
> netbsd-bugs%netbsd.org@localhost
> Subject: Re: kern/48733: deadlock in if_output() with interrupt on KERNEL_LOCK
> Date: Fri, 16 May 2014 10:20:53 +0200
>
>  Hi again,
>  
>  I've already tried LOCK_DEBUG - no change in behaviour and I also don't 
>  see additional usable information. (But now I'm nearly shure, it is not 
>  the KERNEL_LOCK - see below ...)
>  
>  I've still not added the requested curcpu()->ci_biglock_count print, but 
>  I've searched for the irq 6 or something like this.
>  The "surprising" result:
>  
>  s011a# grep irq /var/run/dmesg.boot
>  pckbc0 at acpi0 (KBC0, PNP0303) (kbd port): io 0x60,0x64 irq 1
>  pckbc1 at acpi0 (MSE0, PNP0F13) (aux port): irq 12
>  com0 at acpi0 (COM1, PNP0501-1): io 0x3f8-0x3ff irq 4
>  com1 at acpi0 (COM2, PNP0501-2): io 0x2f8-0x2ff irq 3
>  pckbc0: using irq 1 for kbd slot
>  s011a# grep interrupt !$
>  grep interrupt /var/run/dmesg.boot
>  acpi0: SCI interrupting at int 9
>  wm0: interrupting at ioapic1 pin 0
>  wm1: interrupting at ioapic1 pin 1
>  Intel product 0x0326 (interrupt system, interface 0x20, revision 0x09) 
>  at pci1 dev 0 function 1 not configured
>  uhci0: interrupting at ioapic0 pin 16
>  uhci1: interrupting at ioapic0 pin 17
>  uhci2: interrupting at ioapic0 pin 18
>  ehci0: interrupting at ioapic0 pin 18
>  wm2: interrupting at ioapic0 pin 16
>  wm3: interrupting at ioapic0 pin 17
>  wm4: interrupting at ioapic0 pin 18
>  wm5: interrupting at ioapic0 pin 19
>  uhci3: interrupting at ioapic0 pin 23
>  uhci4: interrupting at ioapic0 pin 22
>  uhci5: interrupting at ioapic0 pin 21
>  ehci1: interrupting at ioapic0 pin 23
>  pciide0: using ioapic0 pin 23 for native-PCI interrupt
>  piixide0: primary channel interrupting at ioapic0 pin 14
>  piixide0: secondary channel interrupting at ioapic0 pin 15
>  ichsmb0: interrupting at ioapic0 pin 17
>  piixide1: using ioapic0 pin 18 for native-PCI interrupt
>  timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
>  s011a# grep ioapic !$
>  grep ioapic /var/run/dmesg.boot
>  ioapic0 at mainbus0 apid 2: pa 0xfec00000, version 20, 24 pins
>  ioapic1 at mainbus0 apid 3: pa 0xfecc0000, version 20, 24 pins
>  wm0: interrupting at ioapic1 pin 0
>  wm1: interrupting at ioapic1 pin 1
>  uhci0: interrupting at ioapic0 pin 16
>  uhci1: interrupting at ioapic0 pin 17
>  uhci2: interrupting at ioapic0 pin 18
>  ehci0: interrupting at ioapic0 pin 18
>  wm2: interrupting at ioapic0 pin 16
>  wm3: interrupting at ioapic0 pin 17
>  wm4: interrupting at ioapic0 pin 18
>  wm5: interrupting at ioapic0 pin 19
>  uhci3: interrupting at ioapic0 pin 23
>  uhci4: interrupting at ioapic0 pin 22
>  uhci5: interrupting at ioapic0 pin 21
>  ehci1: interrupting at ioapic0 pin 23
>  pciide0: using ioapic0 pin 23 for native-PCI interrupt
>  piixide0: primary channel interrupting at ioapic0 pin 14
>  piixide0: secondary channel interrupting at ioapic0 pin 15
>  ichsmb0: interrupting at ioapic0 pin 17
>  piixide1: using ioapic0 pin 18 for native-PCI interrupt
>  
>  
>  
>  
>  Hmmm - no irq6 or pin 6 !?!?!?!
>  The first greating line is in /var/run/dmesg.boot, so the result is 
>  "complete".
>  The board is reported as 'Supermicro X7SBi-LN4 (0123456789)'.
>  The two "additonal" interfaces (wm0 and wm1) are on ioapic1 - no other 
>  HW is on that APIC.
>  
>  
>  
>  The search for not-configured devices say:
>  s011a# grep confi !$
>  grep confi /var/run/dmesg.boot
>  MBRD (PNP0C02) at acpi0 not configured
>  FWH (INT0800) at acpi0 not configured
>  pci0 at mainbus0 bus 0: configuration mode 1
>  Intel product 0x0326 (interrupt system, interface 0x20, revision 0x09) 
>  at pci1 dev 0 function 1 not configured
>  drm at vga0 not configured
>  ichlpcib0: TCO (watchdog) timer configured.
>  piixide0: primary channel configured to compatibility mode
>  piixide0: secondary channel configured to compatibility mode
>  Intel 82801I Thermal Controller (miscellaneous DASP, revision 0x02) at 
>  pci0 dev 31 function 6 not configured
>  
>  
>  Not really a lot - other systems have more unconfigured devices ....
>  Normaly I'm testing all available drivers on a "new" HW, so I'm nearly 
>  100% shure that theese devices are not covered by any driver.
>  
>  After that research it looks like there is something "suddenly" 
>  interrupting without a driver for it.
>  The time it take till "freeze" varies, so a "normal timer timeout" does 
>  not sound reasonable.
>  And why does this occure only after installing two additional interfaces 
>  (wm0 and wm1) that use the second APIC? The system does not show the 
>  problem without them.
>  
>  pci0 at mainbus0 bus 0: configuration mode 1
>  pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
>  pchb0 at pci0 dev 0 function 0: Intel product 0x29f0 (rev. 0x01)
>  ppb0 at pci0 dev 1 function 0: Intel product 0x29f1 (rev. 0x01)
>  ppb0: PCI Express 2.0 <Root Port of PCI-E Root Complex>
>  pci1 at ppb0 bus 1
>  pci1: i/o space, memory space enabled, rd/line, wr/inv ok
>  ppb1 at pci1 dev 0 function 0: Intel product 0x032c (rev. 0x09)
>  ppb1: PCI Express 1.0 <PCI-E to PCI/PCI-X Bridge>
>  pci2 at ppb1 bus 2
>  pci2: i/o space, memory space enabled, rd/line, wr/inv ok
>  wm0 at pci2 dev 1 function 0: Intel i82546GB 1000BASE-T Ethernet (rev. 0x03)
>  wm0: interrupting at ioapic1 pin 0
>  wm0: 64-bit 133MHz PCIX bus
>  wm0: NVM dump:
>  1b00 0621 2a12 0430 ffff ffff ffff ffff
>  c414 2104 460b 1179 8086 1079 8086 34e8
>  000c 1079 0000 2102 10c8 ffff ffff ffff
>  ffff ffff ffff ffff ffff ffff ffff ffff
>  c30c ff63 5004 2102 00c8 ffff ffff ffff
>  ffff ffff ffff ffff ffff ffff ffff 0602
>  ffff ffff ffff ffff ffff ffff ffff ffff
>  ffff ffff ffff ffff ffff ffff ffff 6be8
>  wm0: 256 word (8 address bits) MicroWire EEPROM
>  wm0: Ethernet address 00:1b:21:06:12:2a
>  makphy0 at wm0 phy 1: Marvell 88E1011 Gigabit PHY, rev. 5
>  makphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
>  1000baseT-FDX, auto
>  wm1 at pci2 dev 1 function 1: Intel i82546GB 1000BASE-T Ethernet (rev. 0x03)
>  wm1: interrupting at ioapic1 pin 1
>  wm1: 64-bit 133MHz PCIX bus
>  wm1: NVM dump:
>  1b00 0621 2a12 0430 ffff ffff ffff ffff
>  c414 2104 460b 1179 8086 1079 8086 34e8
>  000c 1079 0000 2102 10c8 ffff ffff ffff
>  ffff ffff ffff ffff ffff ffff ffff ffff
>  c30c ff63 5004 2102 00c8 ffff ffff ffff
>  ffff ffff ffff ffff ffff ffff ffff 0602
>  ffff ffff ffff ffff ffff ffff ffff ffff
>  ffff ffff ffff ffff ffff ffff ffff 6be8
>  wm1: 256 word (8 address bits) MicroWire EEPROM
>  wm1: Ethernet address 00:1b:21:06:12:2b
>  makphy1 at wm1 phy 1: Marvell 88E1011 Gigabit PHY, rev. 5
>  makphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
>  1000baseT-FDX, auto
>  
>  
>  Question: Are unused interrupts enabled? And if "yes", why?
>  I thought I've seen some code that will print a warning about this and 
>  tries to reset the interrupt, but I'm not 100% shure about the 
>  netbsd-version I've looked at.
>  
>  Any hints to go on with debugging?
>  It would take me a lot of time to "understand" the interrupt setup on 
>  the amd64 architecture ...
>  
>  best regards
>  
>  W. Stukenbrock
>  
>  
>  Manuel Bouyer wrote:
>  
>  > The following reply was made to PR kern/48733; it has been noted by GNATS.
>  > 
>  > From: Manuel Bouyer <bouyer%antioche.eu.org@localhost>
>  > To: gnats-bugs%NetBSD.org@localhost
>  > Cc: kern-bug-people%NetBSD.org@localhost, gnats-admin%NetBSD.org@localhost, 
> netbsd-bugs%NetBSD.org@localhost,
>  >         Wolfgang.Stukenbrock%nagler-company.com@localhost
>  > Subject: Re: kern/48733: deadlock in if_output() with interrupt on 
> KERNEL_LOCK
>  > Date: Mon, 12 May 2014 14:23:46 +0200
>  > 
>  >  On Fri, May 09, 2014 at 12:20:00PM +0000, Wolfgang Stukenbrock wrote:
>  >  >  [...]
>  >  >  db{0}> bt/a fffffe822f73f420
>  >  >  trace: pid 0 lid 3 at 0xfffffe810e967760
>  >  >  ether_output() at netbsd:ether_output+0x2b6
>  >  >  ip_output() at netbsd:ip_output+0xa8f
>  >  >  tcp_output() at netbsd:tcp_output+0x1698
>  >  >  tcp_input() at netbsd:tcp_input+0x15d9
>  >  >  ip_input() at netbsd:ip_input+0x3ef
>  >  >  ipintr() at netbsd:ipintr+0x109
>  >  >  softint_dispatch() at netbsd:softint_dispatch+0xd9
>  >  >  DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xfffffe810e967d70
>  >  >  Xsoftintr() at netbsd:Xsoftintr+0x4f
>  >  >  --- interrupt ---
>  >  >  0:
>  >  >  
>  >  >  That is the part that is gooing to send a packet. I see the printout in 
>  >  >  ip_output prior calling 'ifp->if_output()' - not the one behind.
>  >  >  The location pointed to by the backtrace in ether_output() is the call 
>  >  >  to "return ifq_enqueue(...)". I also see the printout I've added in 
>  >  >  front of this call, but not the one behind.
>  >  >  In ifq_enqueue() I see the output of the call to 'ifp->if_start' - the 
>  >  >  wm-driver - in this routine and the printout in front of the splx(s) at 
>  >  >  the end of the routine - not the printout behind it.
>  >  >  This is the localtion where the deadlock happens while processing other 
>  >  >  interrupts in Xspllower.
>  >  >  This always looks the same ....
>  >  
>  >  ether_output() is called with the KERNEL_LOCK held, so at this point cpu0
>  >  already owns KERNEL_LOCK, it won't spin trying to grab it again.
>  >  You can confirm this by printing curcpu()->ci_biglock_count.
>  >  Did you try a kernel with options LOCKDEBUG ?
>  >  
>  >  What's possible here is a loop trying to process the same interrupt
>  >  forever.
>  >  
>  >  >  
>  >  >  
>  >  >  
>  >  >  db{0}> bt/a fffffe822f736440
>  >  >  trace: pid 0 lid 6 at 0xfffffe810e9739c8
>  >  >  breakpoint() at netbsd:breakpoint+0x5
>  >  >  comintr() at netbsd:comintr+0x518
>  >  >  Xintr_ioapic_edge1() at netbsd:Xintr_ioapic_edge1+0xea
>  >  >  --- interrupt ---
>  >  >  bus_space_read_4() at netbsd:bus_space_read_4+0xa
>  >  >  intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x3b
>  >  >  Xintr_ioapic_level6() at netbsd:Xintr_ioapic_level6+0xf2
>  >  >  --- interrupt ---
>  >  >  Xspllower() at netbsd:Xspllower+0xe
>  >  >  DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xfffffe810e973d70
>  >  >  Xsoftintr() at netbsd:Xsoftintr+0x4f
>  >  >  --- interrupt ---
>  >  >  0:
>  >  >  
>  >  >  
>  >  >  Hmmm - not shure about it ...
>  >  >  It looks like that during processing one pending interrupt in Xspllower 
>  >  >  at the end of that routine an interrupt came im that takes the 
>  >  >  KERNEL_LOCK in intr_biglock_wrapper() again and do what? Hangup in 
>  >  >  bus_space_read_4() ???? Busy-loop for whatever reason in that interrupt 
>  >  >  and the location where the DDB-enter occures in bus_space_read_4() is 
>  >  >  just random ????
>  >  >  The comintr looks like the break-interrupt on the serial console of the 
>  >  >  system to enter DDB to me.
>  >  
>  >  it is.
>  >  
>  >  >  Any idea to find out what interrupt routine it is???
>  >  
>  >  dmesg could point to the problem; the interrupt we're looking for is
>  >  level-triggered on pin 6 (so maybe "irq 6")
>  >  
>  >  -- 
>  >  Manuel Bouyer <bouyer%antioche.eu.org@localhost>
>  >       NetBSD: 26 ans d'experience feront toujours la difference
>  >  --
>  >  
>  > 
>  
>  
>  -- 
>  
>  
>  Dr. Nagler & Company GmbH
>  Hauptstraße 9
>  92253 Schnaittenbach
>  
>  Tel. +49 9622/71 97-42
>  Fax +49 9622/71 97-50
>  
>  Wolfgang.Stukenbrock%nagler-company.com@localhost
>  http://www.nagler-company.com
>  
>  
>  Hauptsitz: Schnaittenbach
>  Handelregister: Amberg HRB
>  Gerichtsstand: Amberg
>  Steuernummer: 201/118/51825
>  USt.-ID-Nummer: DE 273143997
>  Geschäftsführer: Dr. Martin Nagler
>  
>  


--


Dr. Nagler & Company GmbH
Hauptstraße 9
92253 Schnaittenbach

Tel. +49 9622/71 97-42
Fax +49 9622/71 97-50

Wolfgang.Stukenbrock%nagler-company.com@localhost
http://www.nagler-company.com


Hauptsitz: Schnaittenbach
Handelregister: Amberg HRB
Gerichtsstand: Amberg
Steuernummer: 201/118/51825
USt.-ID-Nummer: DE 273143997
Geschäftsführer: Dr. Martin Nagler