Subject: Re: Interrupt storm on -current?
To: None <current-users@netbsd.org>
From: Henry Bent <Henry.Bent@oberlin.edu>
List: current-users
Date: 04/02/2007 12:11:22
I just noticed that interrupts are very slowly returning to normal
levels. In the ~5min since I sent the last email, vmstat -i now shows:
interrupt total rate
cpu0 softclock 87194 44
cpu0 softnet 803 0
cpu0 timer 195406 100
cpu0 FPU flush IPI 1 0
cpu0 FPU synch IPI 7 0
cpu0 TLB shootdown IPI 78220 40
cpu1 timer 194700 100
cpu1 FPU synch IPI 8 0
cpu1 TLB shootdown IPI 647074 332
ioapic0 pin 6 1 0
ioapic0 pin 1 1020 0
ioapic0 pin 20 3170 1
ioapic0 pin 14 174943 89
ioapic0 pin 15 6 0
ioapic0 pin 21 280735 144
ioapic0 pin 18 1529 0
Total 1664817 855
but system responsiveness is still pretty sluggish.
-Henry
----- Original Message -----
From: Henry Bent <Henry.Bent@oberlin.edu>
Date: Monday, April 2, 2007 12:05 pm
Subject: Interrupt storm on -current?
To: current-users@netbsd.org
> Hi all,
>
> I am seeing an interrupt storm on i386 -current from yesterday
> afternoon. System responsiveness from the console (I haven't
> started X)
> is noticeably slow. All I did was move some large files from one disk
> to another, then open a new wscons virtual terminal and telnet to
> another machine. My kernel has LOCKDEBUG turned on.
>
> Some info from a totally idle state:
>
> top says:
> CPU0 states: 0.0% user, 0.0% nice, 0.0% system, 96.5% interrupt,
> 3.5% idle
> CPU1 states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt,
> 100% idle
>
> vmstat -i says:
> interrupt total rate
> cpu0 softclock 82551 64
> cpu0 softnet 174 0
> cpu0 timer 128936 100
> cpu0 FPU flush IPI 1 0
> cpu0 FPU synch IPI 6 0
> cpu0 TLB shootdown IPI 78200 60
> cpu1 timer 128230 99
> cpu1 FPU synch IPI 7 0
> cpu1 TLB shootdown IPI 647065 504
> ioapic0 pin 6 1 0
> ioapic0 pin 1 963 0
> ioapic0 pin 20 1723 1
> ioapic0 pin 14 174943 136
> ioapic0 pin 15 6 0
> ioapic0 pin 21 280728 218
> ioapic0 pin 18 263 0
> Total 1523797 1187
>
> cpu1 TLB shootdown IPI seems very high.
>
> dmesg:
> NetBSD 4.99.16 (AELFRICDEBUG) #0: Mon Apr 2 10:22:29 EDT 2007
> root@hbent3:/usr/obj/src-
> current/sys/arch/i386/compile/AELFRICDEBUGtotal memory = 3071 MB
> avail memory = 2993 MB
> sysctl_createv: sysctl_create(machine) returned 17
> timecounter: Timecounters tick every 10.000 msec
> timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
> BIOS32 rev. 0 found at 0xf0010
> mainbus0 (root)
> cpu0 at mainbus0 apid 0: (boot processor)
> cpu0: AMD Dual-Core Opteron or Athlon 64 X2 (686-class), 1808.29
> MHz, id
> 0x20f32
> cpu0: features
> f7dbfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>cpu0:
> features f7dbfbff<PGE,MCA,CMOV,PAT,PSE36,MPC,NOX,MMXX,MMX>
> cpu0: features f7dbfbff<FXSR,SSE,SSE2,HTT,LONG,3DNOW2,3DNOW>
> cpu0: features2 1<SSE3>
> cpu0: "Dual Core AMD Opteron(tm) Processor 165"
> cpu0: I-cache 64 KB 64B/line 2-way, D-cache 64 KB 64B/line 2-way
> cpu0: L2 cache 1 MB 64B/line 16-way
> cpu0: ITLB 32 4 KB entries fully associative, 8 4 MB entries fully
> associative
> cpu0: DTLB 32 4 KB entries fully associative, 8 4 MB entries fully
> associative
> cpu0: AMD Power Management features: f<TTP,VID,FID,TS>
> cpu0: WARNING: AMD errata present, BIOS upgrade may be
> cpu0: WARNING: necessary to ensure reliable operation
> cpu0: calibrating local timer
> cpu0: apic clock running at 200 MHz
> cpu0: 16 page colors
> cpu1 at mainbus0 apid 1: (application processor)
> cpu1: AMD Dual-Core Opteron or Athlon 64 X2 (686-class), 1808.23
> MHz, id
> 0x20f32
> cpu1: features
> f7dbfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>cpu1:
> features f7dbfbff<PGE,MCA,CMOV,PAT,PSE36,MPC,NOX,MMXX,MMX>
> cpu1: features f7dbfbff<FXSR,SSE,SSE2,HTT,LONG,3DNOW2,3DNOW>
> cpu1: features2 1<SSE3>
> cpu1: "Dual Core AMD Opteron(tm) Processor 165"
> cpu1: I-cache 64 KB 64B/line 2-way, D-cache 64 KB 64B/line 2-way
> cpu1: L2 cache 1 MB 64B/line 16-way
> cpu1: ITLB 32 4 KB entries fully associative, 8 4 MB entries fully
> associative
> cpu1: DTLB 32 4 KB entries fully associative, 8 4 MB entries fully
> associative
> cpu1: AMD Power Management features: f<TTP,VID,FID,TS>
> ioapic0 at mainbus0 apid 2
> ioapic0: pa 0xfec00000, version 11, 24 pins
> acpi0 at mainbus0: Advanced Configuration and Power Interface
> acpi0: using Intel ACPI CA subsystem version 20060217
> acpi0: X/RSDT: OemId , AslId <MSFT,00000097>
> acpi0: SCI interrupting at int 9
> acpi0: fixed-feature power button present
> timecounter: Timecounter "ACPI-Safe" frequency 3579545 Hz quality 900
> ACPI-Safe 24-bit timer
> ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
> ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
> PNP0A03 [PCI/PCI-X Host Bridge] at acpi0 not configured
> PNP0000 [AT Interrupt Controller] at acpi0 not configured
> PNP0200 [AT DMA Controller] at acpi0 not configured
> attimer0 at acpi0 (PNP0100): AT Timer
> attimer0: io 0x40-0x43 irq 0
> PNP0B00 [AT Real-Time Clock] at acpi0 not configured
> pcppi0 at acpi0 (PNP0800)
> pcppi0: io 0x61
> midi0 at pcppi0: PC speaker (CPU-intensive output)
> npx0 at acpi0 (PNP0C04)
> npx0: io 0xf0-0xff irq 13
> npx0: reported by CPUID; using exception 16
> fdc0 at acpi0 (PNP0700)
> fdc0: io 0x3f0-0x3f5,0x3f7 irq 6 drq 2
> fdc0: expected BUFFER, got 4
> mpu0 at acpi0 (PNPB006)
> mpu0: io 0x330-0x331 irq 5
> midi1 at mpu0: Roland MPU-401 MIDI UART (CPU-intensive output)
> PNP0C02 [Plug and Play motherboard register resources] at acpi0 not
> configured
> ATK0110 at acpi0 not configured
> PNP0C02 [Plug and Play motherboard register resources] at acpi0 not
> configured
> pckbc0 at acpi0 (PNP0303): kbd port
> pckbc0: io 0x60,0x64 irq 1
> com0 at acpi0 (PNP0501-1)
> com0: io 0x3f8-0x3ff irq 4
> com0: ns16550a, working fifo
> PNP0C02 [Plug and Play motherboard register resources] at acpi0 not
> configured
> PNP0C02 [Plug and Play motherboard register resources] at acpi0 not
> configured
> PNP0C02 [Plug and Play motherboard register resources] at acpi0 not
> configured
> PNP0C01 [System Board] at acpi0 not configured
> acpibut0 at acpi0 (PNP0C0C-170): ACPI Power Button
> PNP0C0F [PCI interrupt link device] at acpi0 not configured
> PNP0C0F [PCI interrupt link device] at acpi0 not configured
> PNP0C0F [PCI interrupt link device] at acpi0 not configured
> PNP0C0F [PCI interrupt link device] at acpi0 not configured
> PNP0C0F [PCI interrupt link device] at acpi0 not configured
> PNP0C0F [PCI interrupt link device] at acpi0 not configured
> PNP0C0F [PCI interrupt link device] at acpi0 not configured
> PNP0C0F [PCI interrupt link device] at acpi0 not configured
> pcppi0: attached to attimer0
> pckbd0 at pckbc0 (kbd slot)
> pckbc0: using irq 1 for kbd slot
> wskbd0 at pckbd0: console keyboard
> pci0 at mainbus0 bus 0: configuration mode 1
> pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
> NVIDIA product 0x02f4 (RAM memory, revision 0xa2) at pci0 dev 0
> function0 not configured
> NVIDIA product 0x02fa (RAM memory, revision 0xa2) at pci0 dev 0
> function1 not configured
> NVIDIA product 0x02fe (RAM memory, revision 0xa2) at pci0 dev 0
> function2 not configured
> NVIDIA product 0x02f8 (RAM memory, revision 0xa2) at pci0 dev 0
> function3 not configured
> NVIDIA product 0x02f9 (RAM memory, revision 0xa2) at pci0 dev 0
> function4 not configured
> NVIDIA product 0x02ff (RAM memory, revision 0xa2) at pci0 dev 0
> function5 not configured
> NVIDIA product 0x027f (RAM memory, revision 0xa2) at pci0 dev 0
> function6 not configured
> NVIDIA product 0x027e (RAM memory, revision 0xa2) at pci0 dev 0
> function7 not configured
> ppb0 at pci0 dev 2 function 0: NVIDIA product 0x02fc (rev. 0xa1)
> pci1 at ppb0 bus 1
> pci1: i/o space, memory space enabled, rd/line, wr/inv ok
> ppb1 at pci0 dev 3 function 0: NVIDIA product 0x02fd (rev. 0xa1)
> pci2 at ppb1 bus 2
> pci2: i/o space, memory space enabled, rd/line, wr/inv ok
> ppb2 at pci0 dev 4 function 0: NVIDIA product 0x02fb (rev. 0xa1)
> pci3 at ppb2 bus 3
> pci3: i/o space, memory space enabled, rd/line, wr/inv ok
> vga0 at pci3 dev 0 function 0: NVIDIA product 0x0141 (rev. 0xa2)
> vga0: WARNING: ignoring 64-bit BAR @ 0x14
> vga0: WARNING: ignoring 64-bit BAR @ 0x1c
> wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation),
> using wskbd0
> wsmux1: connecting to wsdisplay0
> direct rendering for vga0 unsupported
> NVIDIA nForce4 Memory Controller (miscellaneous memory, revision 0xa4)
> at pci0 dev 9 function 0 not configured
> pcib0 at pci0 dev 10 function 0
> pcib0: NVIDIA product 0x0050 (rev. 0xa4)
> NVIDIA nForce4 SMBus (SMBus serial bus, revision 0xa2) at pci0 dev 10
> function 1 not configured
> ohci0 at pci0 dev 11 function 0: NVIDIA nForce4 USB Host Controller
> (rev. 0xa2)
> LUB0: Picked IRQ 20 with weight 0
> ohci0: interrupting at ioapic0 pin 20 (irq 7)
> ohci0: OHCI version 1.0, legacy support
> usb0 at ohci0: USB revision 1.0
> uhub0 at usb0
> uhub0: NVIDIA OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
> uhub0: 10 ports with 10 removable, self powered
> NVIDIA nForce4 USB2 Host Controller (USB serial bus, interface 0x20,
> revision 0xa4) at pci0 dev 11 function 1 not configured
> viaide0 at pci0 dev 15 function 0
> viaide0: NVIDIA nForce4 IDE Controller (rev. 0xf3)
> viaide0: bus-master DMA support present
> viaide0: primary channel configured to compatibility mode
> viaide0: primary channel interrupting at ioapic0 pin 14 (irq 14)
> atabus0 at viaide0 channel 0
> viaide0: secondary channel configured to compatibility mode
> viaide0: secondary channel interrupting at ioapic0 pin 15 (irq 15)
> atabus1 at viaide0 channel 1
> viaide1 at pci0 dev 16 function 0
> viaide1: NVIDIA nForce4 Serial ATA Controller (rev. 0xf3)
> viaide1: bus-master DMA support present
> viaide1: primary channel wired to native-PCI mode
> LSA0: Picked IRQ 21 with weight 0
> viaide1: using ioapic0 pin 21 (irq 11) for native-PCI interrupt
> atabus2 at viaide1 channel 0
> viaide1: secondary channel wired to native-PCI mode
> atabus3 at viaide1 channel 1
> viaide2 at pci0 dev 17 function 0
> viaide2: NVIDIA nForce4 Serial ATA Controller (rev. 0xf3)
> viaide2: bus-master DMA support present
> viaide2: primary channel wired to native-PCI mode
> LSA1: Picked IRQ 22 with weight 0
> viaide2: using ioapic0 pin 22 (irq 11) for native-PCI interrupt
> atabus4 at viaide2 channel 0
> viaide2: secondary channel wired to native-PCI mode
> atabus5 at viaide2 channel 1
> ppb3 at pci0 dev 18 function 0: NVIDIA nForce4 PCI Host Bridge
> (rev. 0xa2)
> pci4 at ppb3 bus 4
> pci4: i/o space, memory space enabled
> bktr0 at pci4 dev 6 function 0
> LNKA: Picked IRQ 16 with weight 0
> bktr0: interrupting at ioapic0 pin 16 (irq 10)
> bktr0: Hauppauge Model 61111 A MB
> bktr0: Detected a MSP3430G-A1 at 0x80
> bktr0: Hauppauge WinCast/TV, Philips NTSC tuner, msp3400c stereo.
> Brooktree Bt878 Video Capture (Audio Section) (miscellaneous
> multimedia,revision 0x02) at pci4 dev 6 function 1 not configured
> emuxki0 at pci4 dev 7 function 0: Creative Labs SB Audigy EMU 10000
> (audio multimedia, revision 0x04)
> LNKB: Picked IRQ 17 with weight 0
> emuxki0: interrupting at ioapic0 pin 17 (irq 7)
> emuxki0: ac97: SigmaTel STAC9721/23 codec; 18 bit DAC, 18 bit ADC,
> SigmaTel 3D
> emuxki0: ac97: ext id 200<AMAP>
> audio0 at emuxki0: full duplex, mmap, independent
> Creative Labs product 0x7003 (miscellaneous input, revision 0x04) at
> pci4 dev 7 function 1 not configured
> fwohci0 at pci4 dev 7 function 2: Creative Labs product 0x4001
> (rev. 0x04)
> LNKC: Picked IRQ 18 with weight 0
> fwohci0: interrupting at ioapic0 pin 18 (irq 7)
> fwohci0: OHCI version 1.10 (ROM=0)
> fwohci0: No. of Isochronous channels is 4.
> fwohci0: EUI64 00:02:3c:00:91:05:73:d9
> fwohci0: Phy 1394a available S400, 2 ports.
> fwohci0: Link S400, max_rec 2048 bytes.
> ieee1394if0 at fwohci0: IEEE1394 bus
> fwip0 at ieee1394if0: IP over IEEE1394
> fwohci0: Initiate bus reset
> fxp0 at pci4 dev 8 function 0: i82559 Ethernet, rev 8
> fxp0: interrupting at ioapic0 pin 18 (irq 7)
> fxp0: Ethernet address 00:90:27:85:a8:7a
> inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 4
> inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
> fwohci1 at pci4 dev 11 function 0: Texas Instruments TSB43AA22/A IEEE
> 1394 Host Controller (rev. 0x00)
> fwohci1: interrupting at ioapic0 pin 16 (irq 10)
> fwohci1: OHCI version 1.10 (ROM=1)
> fwohci1: No. of Isochronous channels is 4.
> fwohci1: EUI64 00:11:d8:00:00:93:6b:a8
> fwohci1: Phy 1394a available S400, 2 ports.
> fwohci1: Link S400, max_rec 2048 bytes.
> ieee1394if1 at fwohci1: IEEE1394 bus
> fwip1 at ieee1394if1: IP over IEEE1394
> fwohci1: Initiate bus reset
> ppb4 at pci0 dev 22 function 0: NVIDIA nForce4 PCIe Host Bridge
> (rev. 0xa3)
> pci5 at ppb4 bus 5
> pci5: i/o space, memory space enabled, rd/line, wr/inv ok
> ppb5 at pci0 dev 23 function 0: NVIDIA nForce4 PCIe Host Bridge
> (rev. 0xa3)
> pci6 at ppb5 bus 6
> pci6: i/o space, memory space enabled, rd/line, wr/inv ok
> pchb0 at pci0 dev 24 function 0
> pchb0: Advanced Micro Devices AMD64 HyperTransport configuration
> (rev. 0x00)
> pchb1 at pci0 dev 24 function 1
> pchb1: Advanced Micro Devices AMD64 Address Map configuration (rev.
> 0x00)pchb2 at pci0 dev 24 function 2
> pchb2: Advanced Micro Devices AMD64 DRAM configuration (rev. 0x00)
> pchb3 at pci0 dev 24 function 3
> pchb3: Advanced Micro Devices AMD64 Miscellaneous configuration
> (rev. 0x00)
> isa0 at pcib0
> it0 at isa0 port 0xd00-0xd07: IT8712F Hardware monitor
> fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode
> ieee1394if0: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
> ieee1394if0: bus manager 0 (me)
> fwohci1: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode
> ieee1394if1: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
> ieee1394if1: bus manager 0 (me)
> timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
> wd1 at atabus0 drive 0: <Maxtor 7Y250P0>
> wd1: drive supports 16-sector PIO transfers, LBA48 addressing
> wd1: 233 GB, 486344 cyl, 16 head, 63 sec, 512 bytes/sect x
> 490234752 sectors
> wd1: 32-bit data port
> wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6
> (Ultra/133)wd2 at atabus0 drive 1: <Maxtor 7Y250P0>
> wd2: drive supports 16-sector PIO transfers, LBA48 addressing
> wd2: 233 GB, 486344 cyl, 16 head, 63 sec, 512 bytes/sect x
> 490234752 sectors
> wd2: 32-bit data port
> wd2: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6
> (Ultra/133)wd1(viaide0:0:0): using PIO mode 4, Ultra-DMA mode 6
> (Ultra/133) (using DMA)
> wd2(viaide0:0:1): using PIO mode 4, Ultra-DMA mode 6 (Ultra/133)
> (using DMA)
> atapibus0 at atabus1: 2 targets
> cd0 at atapibus0 drive 0: <_NEC DVD_RW ND-3550A, , 1.05> cdrom
> removablecd0: 32-bit data port
> cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2
> (Ultra/33)cd0(viaide0:1:0): using PIO mode 4, Ultra-DMA mode 2
> (Ultra/33) (using DMA)
> viaide1 port 0: device present, speed: 3.0Gb/s
> viaide1 port 1: device present, speed: 1.5Gb/s
> wd3 at atabus2 drive 0: <WDC WD2500KS-00MJB0>
> wd3: drive supports 16-sector PIO transfers, LBA48 addressing
> wd3: 232 GB, 484521 cyl, 16 head, 63 sec, 512 bytes/sect x
> 488397168 sectors
> wd3: 32-bit data port
> wd3: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6
> (Ultra/133)wd3(viaide1:0:0): using PIO mode 4, Ultra-DMA mode 6
> (Ultra/133) (using DMA)
> wd0 at atabus3 drive 0: <WDC WD2000JD-22HBC0>
> wd0: drive supports 16-sector PIO transfers, LBA48 addressing
> wd0: 186 GB, 387621 cyl, 16 head, 63 sec, 512 bytes/sect x
> 390721968 sectors
> wd0: 32-bit data port
> wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6
> (Ultra/133)wd0(viaide1:1:0): using PIO mode 4, Ultra-DMA mode 6
> (Ultra/133) (using DMA)
> uhidev0 at uhub0 port 2 configuration 1 interface 0
> uhidev0: Kensington MIAB Optical, rev 1.10/2.00, addr 2, iclass 3/1
> ums0 at uhidev0: 3 buttons and Z dir.
> wsmouse0 at ums0 mux 0
> boot device: wd0
> root on wd0a dumps on wd0b
> root file system type: ffs
> fxp0: Microcode loaded: int delay: 1000 usec, max bundle: 6
> wsdisplay0: screen 1 added (80x25, vt100 emulation)
> wsdisplay0: screen 2 added (80x25, vt100 emulation)
> wsdisplay0: screen 3 added (80x25, vt100 emulation)
> wsdisplay0: screen 4 added (80x25, vt100 emulation)
>
> -Henry Bent
>