Subject: Extremely slow IO response times + panic on 3.0_BETA
To: None <port-i386@netbsd.org>
From: Valtteri Vuorikoski <vuori@sci.fi>
List: port-i386
Date: 07/12/2005 20:05:34
I've been running 3.0_BETA CVS'd Jun 30 on a P4 box with two PATA drives
mirrored with raidframe. When doing disk io that involves heavy metadata
operations, (ie. tar xzf pkgsrc.tar.gz) sometimes disk IO response gets
extremely slow. It may take several minutes to do ls -l /usr/bin and
another 30 seconds to get the prompt back once the listing is finished.

One time when i was testing things, the system panic'd while untar of
pkgsrc was in progress and I hit C-c in iostat. Unfortunately I didn't
get a dump (dumps were configured on raid0b, which doesn't seem to work)
and I couldn't reproduce this after I fixed the dumps config.

iostat numbers showed both disks doing a steady ~400 transactions/second
while I was watching. The major slowdown seemed to start at the point
when top showed file cache filling almost the entire physical memory.

Soft-deps are enabled on all filesystems. I've tried with and without
MULTIPROCESSOR and with and without NEW_BUFQ_STRATEGY. They made no
difference. Disk write-back caches are enabled.

Workaround seems to be to disable soft-deps. This makes file operations
just plain slow, but system doesn't go into totally comatose states.

A similar configuration running 2.0.2_STABLE doesn't seem to have this
problem (that I can recall).

Have others seen similar symptons?

 -valtteri


dmesg from the problem system:

NetBSD 3.0_BETA (PUUHAMAA3) #1: Tue Jul 12 15:55:33 UTC 2005
        root@puuhamaa3:/sys/arch/i386/compile/PUUHAMAA3
total memory = 1022 MB
avail memory = 994 MB
BIOS32 rev. 0 found at 0xf0010
mainbus0 (root)
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel (686-class), 2798.79 MHz, id 0xf33
cpu0: features bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features bfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu0: features bfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu0: features2 41d<SSE3,MONITOR,DS-CPL,CID>
cpu0: "Intel(R) Pentium(R) 4 CPU 2.80GHz"
cpu0: I-cache 12K uOp cache 8-way
cpu0: L2 cache 1 MB 64B/line 8-way
cpu0: ITLB 4K/4M: 64 entries
cpu0: DTLB 4K/4M: 64 entries
cpu0: using thermal monitor 1
cpu0: calibrating local timer
cpu0: apic clock running at 199 MHz
cpu0: 32 page colors
cpu1 at mainbus0: apid 1 (application processor)
cpu1: starting
cpu1: Intel (686-class), 2798.66 MHz, id 0xf33
cpu1: features bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu1: features bfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu1: features bfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu1: features2 41d<SSE3,MONITOR,DS-CPL,CID>
cpu1: "Intel(R) Pentium(R) 4 CPU 2.80GHz"
cpu1: I-cache 12K uOp cache 8-way
cpu1: L2 cache 1 MB 64B/line 8-way
cpu1: ITLB 4K/4M: 64 entries
cpu1: DTLB 4K/4M: 64 entries
cpu1: using thermal monitor 1
ioapic0 at mainbus0 apid 2 (I/O APIC)
ioapic0: pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 13
ioapic0: remapped to apic 2
acpi0 at mainbus0
acpi0: using Intel ACPI CA subsystem version 20040211
acpi0: X/RSDT: OemId <A M I ,OEMXSDT ,01000427>, AslId <MSFT,00000097>
acpi0: SCI interrupting at int 9
acpi0: fixed-feature power button present
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
PNP0A03 [PCI Bus] at acpi0 not configured
PNP0000 [AT Interrupt Controller] at acpi0 not configured
PNP0200 [AT DMA Controller] at acpi0 not configured
PNP0100 [AT Timer] at acpi0 not configured
PNP0B00 [AT Real-Time Clock] at acpi0 not configured
PNP0800 [AT-style speaker sound] at acpi0 not configured
npx0 at acpi0 (PNP0C04)
npx0: io 0xf0-0xff irq 13
npx0: using exception 16
com0 at acpi0 (PNP0501-1)
com0: io 0x3f8-0x3ff irq 4
com0: ns16550a, working fifo
com1 at acpi0 (PNP0501-2)
com1: io 0x2f8-0x2ff irq 3
com1: ns16550a, working fifo
PNP0C02 [Plug and Play motherboard register resources] at acpi0 not configured
PNP0C02 [Plug and Play motherboard register resources] at acpi0 not configured
PNP0C02 [Plug and Play motherboard register resources] at acpi0 not configured
PNP0C01 [System Board] at acpi0 not configured
acpibut0 at acpi0 (PNP0C0C-170): ACPI Power Button
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: Intel 82865 Host (rev. 0x02)
agp0 at pchb0: aperture at 0xf8000000, size 0x4000000
ppb0 at pci0 dev 1 function 0: Intel 82865 AGP (rev. 0x02)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled
vga0 at pci1 dev 0 function 0: Matrox MGA G550 AGP (rev. 0x01)
wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
uhci0 at pci0 dev 29 function 0: Intel 82801EB/ER USB UHCI Controller (rev. 0x02)
uhci0: interrupting at ioapic0 pin 16 (irq 10)
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 29 function 1: Intel 82801EB/ER USB UHCI Controller (rev. 0x02)
uhci1: interrupting at ioapic0 pin 19 (irq 5)
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
ehci0 at pci0 dev 29 function 7: Intel 82801EB/ER USB EHCI Controller (rev. 0x02)
ehci0: interrupting at ioapic0 pin 23 (irq 11)
ehci0: EHCI version 1.0
ehci0: companion controllers, 2 ports each: uhci0 uhci1
usb2 at ehci0: USB revision 2.0
uhub2 at usb2
uhub2: Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub2: single transaction translator
uhub2: 4 ports with 4 removable, self powered
ppb1 at pci0 dev 30 function 0: Intel 82801BA Hub-PCI Bridge (rev. 0xc2)
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled
skc0 at pci2 dev 5 function 0: ioapic0 pin 22 (irq 7)
skc0: Marvell Yukon Lite Gigabit Ethernet rev. A3(0x7)
sk0 at skc0 port A: Ethernet address 00:0e:a6:85:71:fa
makphy0 at sk0 phy 0: Marvell 88E1011 Gigabit PHY, rev. 5
makphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
fxp0 at pci2 dev 11 function 0: i82559 Ethernet, rev 8
fxp0: interrupting at ioapic0 pin 23 (irq 11)
fxp0: Ethernet address 00:02:b3:0f:f4:2b
inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 4
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
pcib0 at pci0 dev 31 function 0
pcib0: Intel 82801EB LPC Interface Bridge (rev. 0x02)
piixide0 at pci0 dev 31 function 1
piixide0: Intel 82801EB IDE Controller (ICH5) (rev. 0x02)
piixide0: bus-master DMA support present
piixide0: primary channel wired to compatibility mode
piixide0: primary channel interrupting at ioapic0 pin 14 (irq 14)
atabus0 at piixide0 channel 0
piixide0: secondary channel wired to compatibility mode
piixide0: secondary channel interrupting at ioapic0 pin 15 (irq 15)
atabus1 at piixide0 channel 1
Intel 82801EB/ER SMBus Controller (SMBus serial bus, revision 0x02) at pci0 dev 31 function 3 not configured
isa0 at pcib0
pcppi0 at isa0 port 0x61
sysbeep0 at pcppi0
ioapic0: enabling
Kernelized RAIDframe activated
wd0 at atabus0 drive 0: <WDC WD1600JB-00EVA0>
wd0: drive supports 16-sector PIO transfers, LBA48 addressing
wd0: 149 GB, 310101 cyl, 16 head, 63 sec, 512 bytes/sect x 312581808 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
wd0(piixide0:0:0): using PIO mode 4, Ultra-DMA mode 5 (Ultra/100) (using DMA)
wd1 at atabus1 drive 0: <WDC WD1600JB-98FUA0>
wd1: drive supports 16-sector PIO transfers, LBA48 addressing
wd1: 149 GB, 310101 cyl, 16 head, 63 sec, 512 bytes/sect x 312581808 sectors
wd1: 32-bit data port
wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
wd1(piixide0:1:0): using PIO mode 4, Ultra-DMA mode 5 (Ultra/100) (using DMA)
raid0: RAID Level 1
raid0: Components: /dev/wd0a /dev/wd1a
raid0: Total Sectors: 310484352 (151603 MB)
boot device: raid0
root on raid0a dumps on wd0b
cpu1: CPU 1 running