Re: 5.0: locking panic when running atf-run as root

To: tech-kern%netbsd.org@localhost
Subject: Re: 5.0: locking panic when running atf-run as root
From: buhrow%lothlorien.nfbcal.org@localhost (Brian Buhrow)
Date: Wed, 15 Jul 2009 12:24:26 -0700
        Hello.  I think I'm seeing a related issue under NetBSD-5 here.  I
have a couple of additional data points:

1.  My system is a NetBSD-5 system with netbsd-5 sources as of July 10,
2009.

2.  I have a number of machines running this code in production without a
problem.

3.  this particular machine is doing a lot of web serving.

4.  It is a pair of hyperthreaded Intel cpus, so it shows up as 4 cpus, but
it's actually only 2.

5.  I'm not using WAPBL, in fact, I don't have it built in my kernel.  I
am,however, using ffs with softdep.

6.  Processes which don't write to the filesystem, either directly or
indirectly such as via syslog, do not hang.  Only processes which touch the
filesystem hang.

7. On my system, it takes about 18-21 hours of normal operation for the
problem to show up.

8.  I'm only seeing this behavior on one system.  One difference between
it an the others running 5.x is that it has more logical CPUs than the
others.  

        So,I don't think this is a WAPBL problem.

Here is the dmesg for the afflicted machine, for what it's worth.

        Unfortunately, this is a production machine, so I can't do a lot of
testing with it.  In fact, I hav to drop it back to 4.x today because it's
beginning to impact our operations.  Under 4.x it ran continuously without a 
hitch for over a year.

        Any thoughts or ideas on how to get to the bottom of this would be
greatly appreciated.
-thanks
-Brian

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 5.0_STABLE (RBL) #1: Tue Jul 14 06:26:49 PDT 2009
        buhrow%asterisk.nfbcal.org@localhost:/usr/src/sys/arch/i386/compile/RBL
total memory = 2047 MB
avail memory = 2000 MB
timecounter: Timecounters tick every 10.000 msec
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
Dell Computer Corporation PowerEdge SC1425
PCI BIOS rev. 2.1 found at 0xfc3ee
pcibios: config mechanism [1][x], special cycles [x][x], last bus 4
PCI IRQ Routing Table rev. 1.0 found at 0xfbf50, size 160 bytes (8 entries)
PCI Interrupt Router at 000:31:0 (Intel 82801EB LPC Interface Bridge compatible)
mainbus0 (root)
pci_addr_fixup: 000:31:1 0x8086 0x24db new address 0x00005800
pci_addr_fixup: 000:31:1 0x8086 0x24db new address 0x00005808
pci_addr_fixup: 000:31:1 0x8086 0x24db new address 0x00005810
pci_addr_fixup: 000:31:1 0x8086 0x24db new address 0x0000580c
pci_addr_fixup: 000:31:1 0x8086 0x24db new address 0x80000000
cpu0 at mainbus0 apid 0: Intel 686-class, 3600MHz, id 0xf41
cpu1 at mainbus0 apid 6: Intel 686-class, 3600MHz, id 0xf41
cpu2 at mainbus0 apid 1: Intel 686-class, 3600MHz, id 0xf41
cpu3 at mainbus0 apid 7: Intel 686-class, 3600MHz, id 0xf41
ioapic0 at mainbus0 apid 8: pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0 apid 9: pa 0xfec80000, version 20, 24 pins
ioapic2 at mainbus0 apid 10: pa 0xfec80800, version 20, 24 pins
acpi0 at mainbus0: Intel ACPICA 20080321
acpi0: X/RSDT: OemId <DELL  ,PESC1425,00000001>, AslId <MSFT,0100000a>
acpi0: SCI interrupting at int 9
acpi0: fixed-feature power button present
timecounter: Timecounter "ACPI-Fast" frequency 3579545 Hz quality 1000
ACPI-Fast 24-bit timer
acpi: activated PNP0C0F
acpi: activated PNP0C0F
npx1 at acpi0 (FPU, PNP0C04): io 0xf0-0xff irq 13
npx1: reported by CPUID; using exception 16
pcppi1 at acpi0 (SPK, PNP0800): io 0x61
midi0 at pcppi1: PC speaker (CPU-intensive output)
spkr0 at pcppi1
sysbeep0 at pcppi1
attimer1 at acpi0 (TMR, PNP0100): io 0x40-0x5f irq 0
COMA (PNP0501) [16550A-compatible COM port] at acpi0 not configured
hpet0 at acpi0 (HPET, PNP0103-0): mem 0xfed00000-0xfed003ff
timecounter: Timecounter "hpet0" frequency 14318179 Hz quality 2000
apm0 at acpi0: Power Management spec V1.2
attimer1: attached to pcppi1
ipmi0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: Intel E7525 Memory Controller Hub (rev. 0x09)
ppb0 at pci0 dev 2 function 0: Intel E7525 PCI Express Port A (rev. 0x09)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
ppb1 at pci1 dev 0 function 0: Intel 6700PXH PCI Express-to-PCI Bridge #0 (rev. 
0x09)
ppb1: disabling notification events
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
wm0 at pci2 dev 4 function 0: Intel i82541GI 1000BASE-T Ethernet, rev. 5
wm0: interrupting at ioapic1 pin 0
wm0: 32-bit 66MHz PCI bus
wm0: 65536 word (16 address bits) SPI EEPROM
wm0: Ethernet address 00:11:43:xx:xx:xx
igphy0 at wm0 phy 1: Intel IGP01E1000 Gigabit PHY, rev. 0
igphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
Intel product 0x0326 (interrupt system, interface 0x20, revision 0x09) at pci1 
dev 0 function 1 not configured
ppb2 at pci1 dev 0 function 2: Intel 6700PXH PCI Express-to-PCI Bridge #1 (rev. 
0x09)
ppb2: disabling notification events
pci3 at ppb2 bus 3
pci3: i/o space, memory space enabled, rd/line, wr/inv ok
Intel product 0x0327 (interrupt system, interface 0x20, revision 0x09) at pci1 
dev 0 function 3 not configured
uhci0 at pci0 dev 29 function 0: Intel 82801EB/ER USB UHCI Controller (rev. 
0x02)
uhci0: interrupting at ioapic0 pin 16
usb0 at uhci0: USB revision 1.0
uhci1 at pci0 dev 29 function 1: Intel 82801EB/ER USB UHCI Controller (rev. 
0x02)
uhci1: interrupting at ioapic0 pin 19
usb1 at uhci1: USB revision 1.0
ehci0 at pci0 dev 29 function 7: Intel 82801EB/ER USB EHCI Controller (rev. 
0x02)
ehci0: interrupting at ioapic0 pin 23
ehci0: EHCI version 1.0
ehci0: companion controllers, 2 ports each: uhci0 uhci1
usb2 at ehci0: USB revision 2.0
ppb3 at pci0 dev 30 function 0: Intel 82801BA Hub-PCI Bridge (rev. 0xc2)
pci4 at ppb3 bus 4
pci4: i/o space, memory space enabled
wm1 at pci4 dev 3 function 0: Intel i82541GI 1000BASE-T Ethernet, rev. 5
wm1: interrupting at ioapic0 pin 20
wm1: 32-bit 33MHz PCI bus
wm1: 64 word (6 address bits) MicroWire EEPROM
wm1: Ethernet address 00:11:43:xx:xx:xx
igphy1 at wm1 phy 1: Intel IGP01E1000 Gigabit PHY, rev. 0
igphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
vga1 at pci4 dev 13 function 0: ATI Technologies Radeon 7000/VE QY (rev. 0x00)
wsdisplay0 at vga1 kbdmux 1
wsmux1: connecting to wsdisplay0
drm at vga1 not configured
ichlpcib0 at pci0 dev 31 function 0
ichlpcib0: Intel 82801EB LPC Interface Bridge (rev. 0x02)
timecounter: Timecounter "ichlpcib0" frequency 3579545 Hz quality 1000
ichlpcib0: 24-bit timer
ichlpcib0: TCO (watchdog) timer configured.
ichlpcib0: HPET window not mapped, skipping
piixide0 at pci0 dev 31 function 1
piixide0: Intel 82801EB IDE Controller (ICH5) (rev. 0x02)
piixide0: bus-master DMA support present
piixide0: primary channel configured to compatibility mode
piixide0: primary channel interrupting at ioapic0 pin 14
atabus0 at piixide0 channel 0
piixide0: secondary channel configured to compatibility mode
piixide0: secondary channel ignored (disabled)
piixide1 at pci0 dev 31 function 2
piixide1: Intel 82801EB Serial ATA Controller (rev. 0x02)
piixide1: bus-master DMA support present
piixide1: primary channel configured to native-PCI mode
piixide1: using ioapic0 pin 18 for native-PCI interrupt
atabus1 at piixide1 channel 0
piixide1: secondary channel configured to native-PCI mode
atabus2 at piixide1 channel 1
isa0 at ichlpcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com0: console
pckbc0 at isa0 port 0x60-0x64
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
timecounter: Timecounter "TSC" frequency 3600229770 Hz quality 3000
atapibus0 at atabus0: 2 targets
uhub0 at usb0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhub1 at usb2: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub1: 4 ports with 4 removable, self powered
uhub2 at usb1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
cd0 at atapibus0 drive 0: <HL-DT-STDVD-ROM GDR8082N, , 0106> cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33)
cd0(piixide0:0:0): using PIO mode 4, Ultra-DMA mode 2 (Ultra/33) (using DMA)
wd0 at atabus1 drive 0: <ST3300631AS>
wd0: quirks 2<FORCE_LBA48>
wd0: drive supports 16-sector PIO transfers, LBA48 addressing
wd0: 279 GB, 581421 cyl, 16 head, 63 sec, 512 bytes/sect x 586072368 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
wd0(piixide1:0:0): using PIO mode 4, Ultra-DMA mode 6 (Ultra/133) (using DMA)
wd1 at atabus2 drive 0: <ST3300631AS>
wd1: quirks 2<FORCE_LBA48>
wd1: drive supports 16-sector PIO transfers, LBA48 addressing
wd1: 279 GB, 581421 cyl, 16 head, 63 sec, 512 bytes/sect x 586072368 sectors
wd1: 32-bit data port
wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
wd1(piixide1:1:0): using PIO mode 4, Ultra-DMA mode 6 (Ultra/133) (using DMA)
ipmi0: version 1.5 interface KCS iobase 0xca8/8 spacing 4
Kernelized RAIDframe activated
pad0: outputs: 44100Hz, 16-bit, stereo
audio0 at pad0: half duplex
raid0: RAID Level 1
raid0: Components: /dev/wd0a /dev/wd1a
raid0: Total Sectors: 268435264 (131071 MB)
raid1: RAID Level 1
raid1: Components: /dev/wd0e /dev/wd1e
raid1: Total Sectors: 317636864 (155096 MB)
boot device: raid0
root on raid0a dumps on raid0b
root file system type: ffs
raid0: Device already configured!
raid1: Device already configured!
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)
Accounting started
Follow-Ups:
- Re: 5.0: locking panic when running atf-run as root
  - From: Matthew Mondor
Prev by Date: Re: NetBSD-5 ichlpcib watchdog timer seems to be broken for some revisions of the ichlpcib chip set.
Next by Date: Re: 5.0: locking panic when running atf-run as root
Previous by Thread: Re: 5.0: locking panic when running atf-run as root
Next by Thread: Re: 5.0: locking panic when running atf-run as root
Indexes:
Home | Main Index | Thread Index | Old Index