Hi all,

I am seeing an interrupt storm on i386 -current from yesterday
afternoon.  System responsiveness from the console (I haven't started X)
is noticeably slow.  All I did was move some large files from one disk
to another, then open a new wscons virtual terminal and telnet to
another machine.  My kernel has LOCKDEBUG turned on.

Some info from a totally idle state:

top says:
CPU0 states:  0.0% user,  0.0% nice,  0.0% system, 96.5% interrupt, 
3.5% idle
CPU1 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt, 
100% idle

vmstat -i says:
interrupt                                     total     rate
cpu0 softclock                                82551       64
cpu0 softnet                                    174        0
cpu0 timer                                   128936      100
cpu0 FPU flush IPI                                1        0
cpu0 FPU synch IPI                                6        0
cpu0 TLB shootdown IPI                        78200       60
cpu1 timer                                   128230       99
cpu1 FPU synch IPI                                7        0
cpu1 TLB shootdown IPI                       647065      504
ioapic0 pin 6                                     1        0
ioapic0 pin 1                                   963        0
ioapic0 pin 20                                 1723        1
ioapic0 pin 14                               174943      136
ioapic0 pin 15                                    6        0
ioapic0 pin 21                               280728      218
ioapic0 pin 18                                  263        0
Total                                       1523797     1187

cpu1 TLB shootdown IPI seems very high.

dmesg:
NetBSD 4.99.16 (AELFRICDEBUG) #0: Mon Apr  2 10:22:29 EDT 2007
        root@hbent3:/usr/obj/src-current/sys/arch/i386/compile/AELFRICDEBUG
total memory = 3071 MB
avail memory = 2993 MB
sysctl_createv: sysctl_create(machine) returned 17
timecounter: Timecounters tick every 10.000 msec
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
BIOS32 rev. 0 found at 0xf0010
mainbus0 (root)
cpu0 at mainbus0 apid 0: (boot processor)
cpu0: AMD Dual-Core Opteron or Athlon 64 X2 (686-class), 1808.29 MHz, id
0x20f32
cpu0: features f7dbfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features f7dbfbff<PGE,MCA,CMOV,PAT,PSE36,MPC,NOX,MMXX,MMX>
cpu0: features f7dbfbff<FXSR,SSE,SSE2,HTT,LONG,3DNOW2,3DNOW>
cpu0: features2 1<SSE3>
cpu0: "Dual Core AMD Opteron(tm) Processor 165"
cpu0: I-cache 64 KB 64B/line 2-way, D-cache 64 KB 64B/line 2-way
cpu0: L2 cache 1 MB 64B/line 16-way
cpu0: ITLB 32 4 KB entries fully associative, 8 4 MB entries fully
associative
cpu0: DTLB 32 4 KB entries fully associative, 8 4 MB entries fully
associative
cpu0: AMD Power Management features: f<TTP,VID,FID,TS>
cpu0: WARNING: AMD errata present, BIOS upgrade may be
cpu0: WARNING: necessary to ensure reliable operation
cpu0: calibrating local timer
cpu0: apic clock running at 200 MHz
cpu0: 16 page colors
cpu1 at mainbus0 apid 1: (application processor)
cpu1: AMD Dual-Core Opteron or Athlon 64 X2 (686-class), 1808.23 MHz, id
0x20f32
cpu1: features f7dbfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu1: features f7dbfbff<PGE,MCA,CMOV,PAT,PSE36,MPC,NOX,MMXX,MMX>
cpu1: features f7dbfbff<FXSR,SSE,SSE2,HTT,LONG,3DNOW2,3DNOW>
cpu1: features2 1<SSE3>
cpu1: "Dual Core AMD Opteron(tm) Processor 165"
cpu1: I-cache 64 KB 64B/line 2-way, D-cache 64 KB 64B/line 2-way
cpu1: L2 cache 1 MB 64B/line 16-way
cpu1: ITLB 32 4 KB entries fully associative, 8 4 MB entries fully
associative
cpu1: DTLB 32 4 KB entries fully associative, 8 4 MB entries fully
associative
cpu1: AMD Power Management features: f<TTP,VID,FID,TS>
ioapic0 at mainbus0 apid 2
ioapic0: pa 0xfec00000, version 11, 24 pins
acpi0 at mainbus0: Advanced Configuration and Power Interface
acpi0: using Intel ACPI CA subsystem version 20060217
acpi0: X/RSDT: OemId <A M I ,OEMXSDT ,04000628>, AslId <MSFT,00000097>
acpi0: SCI interrupting at int 9
acpi0: fixed-feature power button present
timecounter: Timecounter "ACPI-Safe" frequency 3579545 Hz quality 900
ACPI-Safe 24-bit timer
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
PNP0A03 [PCI/PCI-X Host Bridge] at acpi0 not configured
PNP0000 [AT Interrupt Controller] at acpi0 not configured
PNP0200 [AT DMA Controller] at acpi0 not configured
attimer0 at acpi0 (PNP0100): AT Timer
attimer0: io 0x40-0x43 irq 0
PNP0B00 [AT Real-Time Clock] at acpi0 not configured
pcppi0 at acpi0 (PNP0800)
pcppi0: io 0x61
midi0 at pcppi0: PC speaker (CPU-intensive output)
npx0 at acpi0 (PNP0C04)
npx0: io 0xf0-0xff irq 13
npx0: reported by CPUID; using exception 16
fdc0 at acpi0 (PNP0700)
fdc0: io 0x3f0-0x3f5,0x3f7 irq 6 drq 2
fdc0: expected BUFFER, got 4
mpu0 at acpi0 (PNPB006)
mpu0: io 0x330-0x331 irq 5
midi1 at mpu0: Roland MPU-401 MIDI UART (CPU-intensive output)
PNP0C02 [Plug and Play motherboard register resources] at acpi0 not
configured
ATK0110 at acpi0 not configured
PNP0C02 [Plug and Play motherboard register resources] at acpi0 not
configured
pckbc0 at acpi0 (PNP0303): kbd port
pckbc0: io 0x60,0x64 irq 1
com0 at acpi0 (PNP0501-1)
com0: io 0x3f8-0x3ff irq 4
com0: ns16550a, working fifo
PNP0C02 [Plug and Play motherboard register resources] at acpi0 not
configured
PNP0C02 [Plug and Play motherboard register resources] at acpi0 not
configured
PNP0C02 [Plug and Play motherboard register resources] at acpi0 not
configured
PNP0C01 [System Board] at acpi0 not configured
acpibut0 at acpi0 (PNP0C0C-170): ACPI Power Button
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
pcppi0: attached to attimer0
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
NVIDIA product 0x02f4 (RAM memory, revision 0xa2) at pci0 dev 0 function
0 not configured
NVIDIA product 0x02fa (RAM memory, revision 0xa2) at pci0 dev 0 function
1 not configured
NVIDIA product 0x02fe (RAM memory, revision 0xa2) at pci0 dev 0 function
2 not configured
NVIDIA product 0x02f8 (RAM memory, revision 0xa2) at pci0 dev 0 function
3 not configured
NVIDIA product 0x02f9 (RAM memory, revision 0xa2) at pci0 dev 0 function
4 not configured
NVIDIA product 0x02ff (RAM memory, revision 0xa2) at pci0 dev 0 function
5 not configured
NVIDIA product 0x027f (RAM memory, revision 0xa2) at pci0 dev 0 function
6 not configured
NVIDIA product 0x027e (RAM memory, revision 0xa2) at pci0 dev 0 function
7 not configured
ppb0 at pci0 dev 2 function 0: NVIDIA product 0x02fc (rev. 0xa1)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled, rd/line, wr/inv ok
ppb1 at pci0 dev 3 function 0: NVIDIA product 0x02fd (rev. 0xa1)
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
ppb2 at pci0 dev 4 function 0: NVIDIA product 0x02fb (rev. 0xa1)
pci3 at ppb2 bus 3
pci3: i/o space, memory space enabled, rd/line, wr/inv ok
vga0 at pci3 dev 0 function 0: NVIDIA product 0x0141 (rev. 0xa2)
vga0: WARNING: ignoring 64-bit BAR @ 0x14
vga0: WARNING: ignoring 64-bit BAR @ 0x1c
wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation), using wskbd0
wsmux1: connecting to wsdisplay0
direct rendering for vga0 unsupported
NVIDIA nForce4 Memory Controller (miscellaneous memory, revision 0xa4)
at pci0 dev 9 function 0 not configured
pcib0 at pci0 dev 10 function 0
pcib0: NVIDIA product 0x0050 (rev. 0xa4)
NVIDIA nForce4 SMBus (SMBus serial bus, revision 0xa2) at pci0 dev 10
function 1 not configured
ohci0 at pci0 dev 11 function 0: NVIDIA nForce4 USB Host Controller
(rev. 0xa2)
LUB0: Picked IRQ 20 with weight 0
ohci0: interrupting at ioapic0 pin 20 (irq 7)
ohci0: OHCI version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
uhub0 at usb0
uhub0: NVIDIA OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 10 ports with 10 removable, self powered
NVIDIA nForce4 USB2 Host Controller (USB serial bus, interface 0x20,
revision 0xa4) at pci0 dev 11 function 1 not configured
viaide0 at pci0 dev 15 function 0
viaide0: NVIDIA nForce4 IDE Controller (rev. 0xf3)
viaide0: bus-master DMA support present
viaide0: primary channel configured to compatibility mode
viaide0: primary channel interrupting at ioapic0 pin 14 (irq 14)
atabus0 at viaide0 channel 0
viaide0: secondary channel configured to compatibility mode
viaide0: secondary channel interrupting at ioapic0 pin 15 (irq 15)
atabus1 at viaide0 channel 1
viaide1 at pci0 dev 16 function 0
viaide1: NVIDIA nForce4 Serial ATA Controller (rev. 0xf3)
viaide1: bus-master DMA support present
viaide1: primary channel wired to native-PCI mode
LSA0: Picked IRQ 21 with weight 0
viaide1: using ioapic0 pin 21 (irq 11) for native-PCI interrupt
atabus2 at viaide1 channel 0
viaide1: secondary channel wired to native-PCI mode
atabus3 at viaide1 channel 1
viaide2 at pci0 dev 17 function 0
viaide2: NVIDIA nForce4 Serial ATA Controller (rev. 0xf3)
viaide2: bus-master DMA support present
viaide2: primary channel wired to native-PCI mode
LSA1: Picked IRQ 22 with weight 0
viaide2: using ioapic0 pin 22 (irq 11) for native-PCI interrupt
atabus4 at viaide2 channel 0
viaide2: secondary channel wired to native-PCI mode
atabus5 at viaide2 channel 1
ppb3 at pci0 dev 18 function 0: NVIDIA nForce4 PCI Host Bridge (rev. 0xa2)
pci4 at ppb3 bus 4
pci4: i/o space, memory space enabled
bktr0 at pci4 dev 6 function 0
LNKA: Picked IRQ 16 with weight 0
bktr0: interrupting at ioapic0 pin 16 (irq 10)
bktr0: Hauppauge Model 61111 A MB
bktr0: Detected a MSP3430G-A1 at 0x80
bktr0: Hauppauge WinCast/TV, Philips NTSC tuner, msp3400c stereo.
Brooktree Bt878 Video Capture (Audio Section) (miscellaneous multimedia,
revision 0x02) at pci4 dev 6 function 1 not configured
emuxki0 at pci4 dev 7 function 0: Creative Labs SB Audigy EMU 10000
(audio multimedia, revision 0x04)
LNKB: Picked IRQ 17 with weight 0
emuxki0: interrupting at ioapic0 pin 17 (irq 7)
emuxki0: ac97: SigmaTel STAC9721/23 codec; 18 bit DAC, 18 bit ADC,
SigmaTel 3D
emuxki0: ac97: ext id 200<AMAP>
audio0 at emuxki0: full duplex, mmap, independent
Creative Labs product 0x7003 (miscellaneous input, revision 0x04) at
pci4 dev 7 function 1 not configured
fwohci0 at pci4 dev 7 function 2: Creative Labs product 0x4001 (rev. 0x04)
LNKC: Picked IRQ 18 with weight 0
fwohci0: interrupting at ioapic0 pin 18 (irq 7)
fwohci0: OHCI version 1.10 (ROM=0)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:02:3c:00:91:05:73:d9
fwohci0: Phy 1394a available S400, 2 ports.
fwohci0: Link S400, max_rec 2048 bytes.
ieee1394if0 at fwohci0: IEEE1394 bus
fwip0 at ieee1394if0: IP over IEEE1394
fwohci0: Initiate bus reset
fxp0 at pci4 dev 8 function 0: i82559 Ethernet, rev 8
fxp0: interrupting at ioapic0 pin 18 (irq 7)
fxp0: Ethernet address 00:90:27:85:a8:7a
inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 4
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fwohci1 at pci4 dev 11 function 0: Texas Instruments TSB43AA22/A IEEE
1394 Host Controller (rev. 0x00)
fwohci1: interrupting at ioapic0 pin 16 (irq 10)
fwohci1: OHCI version 1.10 (ROM=1)
fwohci1: No. of Isochronous channels is 4.
fwohci1: EUI64 00:11:d8:00:00:93:6b:a8
fwohci1: Phy 1394a available S400, 2 ports.
fwohci1: Link S400, max_rec 2048 bytes.
ieee1394if1 at fwohci1: IEEE1394 bus
fwip1 at ieee1394if1: IP over IEEE1394
fwohci1: Initiate bus reset
ppb4 at pci0 dev 22 function 0: NVIDIA nForce4 PCIe Host Bridge (rev. 0xa3)
pci5 at ppb4 bus 5
pci5: i/o space, memory space enabled, rd/line, wr/inv ok
ppb5 at pci0 dev 23 function 0: NVIDIA nForce4 PCIe Host Bridge (rev. 0xa3)
pci6 at ppb5 bus 6
pci6: i/o space, memory space enabled, rd/line, wr/inv ok
pchb0 at pci0 dev 24 function 0
pchb0: Advanced Micro Devices AMD64 HyperTransport configuration (rev. 0x00)
pchb1 at pci0 dev 24 function 1
pchb1: Advanced Micro Devices AMD64 Address Map configuration (rev. 0x00)
pchb2 at pci0 dev 24 function 2
pchb2: Advanced Micro Devices AMD64 DRAM configuration (rev. 0x00)
pchb3 at pci0 dev 24 function 3
pchb3: Advanced Micro Devices AMD64 Miscellaneous configuration (rev. 0x00)
isa0 at pcib0
it0 at isa0 port 0xd00-0xd07: IT8712F Hardware monitor
fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode
ieee1394if0: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
ieee1394if0: bus manager 0 (me)
fwohci1: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode
ieee1394if1: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
ieee1394if1: bus manager 0 (me)
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
wd1 at atabus0 drive 0: <Maxtor 7Y250P0>
wd1: drive supports 16-sector PIO transfers, LBA48 addressing
wd1: 233 GB, 486344 cyl, 16 head, 63 sec, 512 bytes/sect x 490234752 sectors
wd1: 32-bit data port
wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
wd2 at atabus0 drive 1: <Maxtor 7Y250P0>
wd2: drive supports 16-sector PIO transfers, LBA48 addressing
wd2: 233 GB, 486344 cyl, 16 head, 63 sec, 512 bytes/sect x 490234752 sectors
wd2: 32-bit data port
wd2: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
wd1(viaide0:0:0): using PIO mode 4, Ultra-DMA mode 6 (Ultra/133) (using DMA)
wd2(viaide0:0:1): using PIO mode 4, Ultra-DMA mode 6 (Ultra/133) (using DMA)
atapibus0 at atabus1: 2 targets
cd0 at atapibus0 drive 0: <_NEC DVD_RW ND-3550A, , 1.05> cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33)
cd0(viaide0:1:0): using PIO mode 4, Ultra-DMA mode 2 (Ultra/33) (using DMA)
viaide1 port 0: device present, speed: 3.0Gb/s
viaide1 port 1: device present, speed: 1.5Gb/s
wd3 at atabus2 drive 0: <WDC WD2500KS-00MJB0>
wd3: drive supports 16-sector PIO transfers, LBA48 addressing
wd3: 232 GB, 484521 cyl, 16 head, 63 sec, 512 bytes/sect x 488397168 sectors
wd3: 32-bit data port
wd3: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
wd3(viaide1:0:0): using PIO mode 4, Ultra-DMA mode 6 (Ultra/133) (using DMA)
wd0 at atabus3 drive 0: <WDC WD2000JD-22HBC0>
wd0: drive supports 16-sector PIO transfers, LBA48 addressing
wd0: 186 GB, 387621 cyl, 16 head, 63 sec, 512 bytes/sect x 390721968 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
wd0(viaide1:1:0): using PIO mode 4, Ultra-DMA mode 6 (Ultra/133) (using DMA)
uhidev0 at uhub0 port 2 configuration 1 interface 0
uhidev0: Kensington MIAB Optical, rev 1.10/2.00, addr 2, iclass 3/1
ums0 at uhidev0: 3 buttons and Z dir.
wsmouse0 at ums0 mux 0
boot device: wd0
root on wd0a dumps on wd0b
root file system type: ffs
fxp0: Microcode loaded: int delay: 1000 usec, max bundle: 6
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)

-Henry Bent