Subject: Re: IFQ_MAXLEN: How large can it be?
To: None <tech-net@NetBSD.org>
From: Christoph Kaegi <kgc@zhwin.ch>
List: tech-net
Date: 11/16/2006 14:49:24
On 16.11-14:04, Manuel Bouyer wrote:
> On Thu, Nov 16, 2006 at 11:58:23AM +0100, Christoph Kaegi wrote:
> > On 16.11-11:49, Manuel Bouyer wrote:
> > > >
> > > > The dropped packets I reported where all on the ipintrq.
> > >
> > > OK, so maybe increasing the input queue size can be of some help to
> > > absorb bursts. how many interfaces with lots of input traffic do you have ?
> > >
> >
> > I'd say three. Ca. 1 Minute of nload says:
>
> OK, so 4096 should be more than enough. There are 256 receive descriptors per
> wm interface; so even if the ring is full when the hardware interrupt is
> processed, it will fill only 768 slots in the IP input queue.
>
> Maybe your problem is lack of CPU power then. What kind of hardware is it ?
It's a Transtec 1002L (aka Supermicro 6023P-8), dmesg output is below.
vmstat 1 looks like this at the moment:
---------------------------- 8< ----------------------------
intfw (primary)# vmstat 1
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr m0 c0 s0 in sy cs us sy id
0 0 0 268124 593708 74 0 0 0 0 0 0 0 0 4294965249 127 8 0 11 89
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6813 30 7 0 28 72
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 7394 30 7 0 13 87
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6687 30 7 0 26 74
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6326 30 7 0 16 84
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6951 30 7 0 19 81
1 0 0 268124 593708 5 0 0 0 0 0 0 0 1 6739 30 12 0 6 94
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 7274 35 7 0 7 93
0 0 0 268124 593708 813 0 0 0 0 0 0 0 0 6847 1164 27 0 22 78
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6658 30 8 0 12 88
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6628 30 7 0 36 64
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6784 30 7 0 42 58
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6776 30 7 0 64 36
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 7266 30 7 0 14 86
0 0 0 268124 593708 1447 0 0 0 0 0 0 0 0 7115 113 14 0 29 71
0 0 0 268124 593708 8 0 0 0 0 0 0 0 0 7293 48 7 0 14 86
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 7530 30 6 0 21 79
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 7502 30 7 0 15 85
0 0 0 268124 593708 811 0 0 0 0 0 0 0 1 7744 1170 28 0 16 84
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 7401 30 7 0 20 80
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 8295 30 8 0 11 89
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 8049 30 7 0 21 79
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 7751 30 7 0 8 92
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6594 36 8 0 9 91
0 0 0 268124 593708 5 0 0 0 0 0 0 0 0 6511 32 8 0 20 80
---------------------------- 8< ----------------------------
I always wondered, if the system is able to handle the
10'000 to 20'000 interrupts it sometimes gets.
(according to sysstat vm 1)
Sometimes it even gets more than that.
Can I somehow measure if interrupts get lost?
Thanks alot for your attention Manuel!
Chris
---------------------------- 8< ----------------------------
NetBSD 3.1_RC4 (gw) #7: Mon Nov 6 13:07:45 CET 2006
root@gw:/usr/src/NetBSD-3/src/sys/arch/i386/compile/INTFW
total memory = 1023 MB
avail memory = 998 MB
BIOS32 rev. 0 found at 0xfd651
mainbus0 (root)
cpu0 at mainbus0: (uniprocessor)
cpu0: Intel Xeon (686-class), 2666.01 MHz, id 0xf29
cpu0: features bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features bfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu0: features bfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu0: features2 4400<CID,xTPR>
cpu0: "Intel(R) Xeon(TM) CPU 2.66GHz"
cpu0: I-cache 12K uOp cache 8-way, D-cache 8 KB 64B/line 4-way
cpu0: L2 cache 512 KB 64B/line 8-way
cpu0: ITLB 4K/4M: 64 entries
cpu0: DTLB 4K/4M: 64 entries
cpu0: running without thermal monitor!
cpu0: 16 page colors
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: Intel E7501 MCH Host (rev. 0x01)
Intel E7500 MCH DRAM Controller (undefined subclass 0x00, revision 0x01) at pci0 dev 0 function 1 not configured
ppb0 at pci0 dev 2 function 0: Intel E7500 MCH HI_B vppb 1 (rev. 0x01)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled
Intel 82870P2 P64H2 IOxAPIC (interrupt system, interface 0x20, revision 0x04) at pci1 dev 28 function 0 not configured
ppb1 at pci1 dev 29 function 0: Intel 82870P2 P64H2 PCI-PCI Bridge (rev. 0x04)
pci2 at ppb1 bus 2
pci2: i/o space, memory space enabled
Intel 82870P2 P64H2 IOxAPIC (interrupt system, interface 0x20, revision 0x04) at pci1 dev 30 function 0 not configured
ppb2 at pci1 dev 31 function 0: Intel 82870P2 P64H2 PCI-PCI Bridge (rev. 0x04)
pci3 at ppb2 bus 3
pci3: i/o space, memory space enabled
wm0 at pci3 dev 2 function 0: Intel i82546EB 1000BASE-T Ethernet, rev. 1
wm0: interrupting at irq 11
wm0: 64-bit 133MHz PCIX bus
wm0: 256 word (8 address bits) MicroWire EEPROM
wm0: Ethernet address 00:30:48:2a:9c:d4
makphy0 at wm0 phy 1: Marvell 88E1011 Gigabit PHY, rev. 3
makphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
wm1 at pci3 dev 2 function 1: Intel i82546EB 1000BASE-T Ethernet, rev. 1
wm1: interrupting at irq 11
wm1: 64-bit 133MHz PCIX bus
wm1: 256 word (8 address bits) MicroWire EEPROM
wm1: Ethernet address 00:30:48:2a:9c:d5
makphy1 at wm1 phy 1: Marvell 88E1011 Gigabit PHY, rev. 3
makphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
ppb3 at pci0 dev 3 function 0: Intel E7500 MCH HI_C vppb 1 (rev. 0x01)
pci4 at ppb3 bus 4
pci4: i/o space, memory space enabled
Intel 82870P2 P64H2 IOxAPIC (interrupt system, interface 0x20, revision 0x04) at pci4 dev 28 function 0 not configured
ppb4 at pci4 dev 29 function 0: Intel 82870P2 P64H2 PCI-PCI Bridge (rev. 0x04)
pci5 at ppb4 bus 5
pci5: i/o space, memory space enabled
wm2 at pci5 dev 1 function 0: Intel i82546GB 1000BASE-T Ethernet, rev. 3
wm2: interrupting at irq 11
wm2: 64-bit 66MHz PCIX bus
wm2: 256 word (8 address bits) MicroWire EEPROM
wm2: Ethernet address 00:04:23:a7:b9:cc
makphy2 at wm2 phy 1: Marvell 88E1011 Gigabit PHY, rev. 5
makphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
wm3 at pci5 dev 1 function 1: Intel i82546GB 1000BASE-T Ethernet, rev. 3
wm3: interrupting at irq 11
wm3: 64-bit 66MHz PCIX bus
wm3: 256 word (8 address bits) MicroWire EEPROM
wm3: Ethernet address 00:04:23:a7:b9:cd
makphy3 at wm3 phy 1: Marvell 88E1011 Gigabit PHY, rev. 5
makphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
wm4 at pci5 dev 2 function 0: Intel i82546GB 1000BASE-T Ethernet, rev. 3
wm4: interrupting at irq 11
wm4: 64-bit 66MHz PCIX bus
wm4: 256 word (8 address bits) MicroWire EEPROM
wm4: Ethernet address 00:04:23:a7:f5:c2
makphy4 at wm4 phy 1: Marvell 88E1011 Gigabit PHY, rev. 5
makphy4: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
wm5 at pci5 dev 2 function 1: Intel i82546GB 1000BASE-T Ethernet, rev. 3
wm5: interrupting at irq 11
wm5: 64-bit 66MHz PCIX bus
wm5: 256 word (8 address bits) MicroWire EEPROM
wm5: Ethernet address 00:04:23:a7:f5:c3
makphy5 at wm5 phy 1: Marvell 88E1011 Gigabit PHY, rev. 5
makphy5: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
Intel 82870P2 P64H2 IOxAPIC (interrupt system, interface 0x20, revision 0x04) at pci4 dev 30 function 0 not configured
ppb5 at pci4 dev 31 function 0: Intel 82870P2 P64H2 PCI-PCI Bridge (rev. 0x04)
pci6 at ppb5 bus 6
pci6: i/o space, memory space enabled
ahd0 at pci6 dev 2 function 0
ahd0: interrupting at irq 11
ahd0: aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
scsibus0 at ahd0: 16 targets, 8 luns per target
ahd1 at pci6 dev 2 function 1
ahd1: interrupting at irq 11
ahd1: aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI-X 101-133Mhz, 512 SCBs
scsibus1 at ahd1: 16 targets, 8 luns per target
uhci0 at pci0 dev 29 function 0: Intel 82801CA USB Controller (rev. 0x02)
uhci0: interrupting at irq 11
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 29 function 1: Intel 82801CA USB Controller (rev. 0x02)
uhci1: interrupting at irq 10
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2 at pci0 dev 29 function 2: Intel 82801CA USB Controller (rev. 0x02)
uhci2: interrupting at irq 5
usb2 at uhci2: USB revision 1.0
uhub2 at usb2
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
ppb6 at pci0 dev 30 function 0: Intel 82801BA Hub-PCI Bridge (rev. 0x42)
pci7 at ppb6 bus 7
pci7: i/o space, memory space enabled
vga1 at pci7 dev 1 function 0: ATI Technologies Rage XL (rev. 0x27)
wsdisplay0 at vga1 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
pcib0 at pci0 dev 31 function 0
pcib0: Intel 82801CA LPC Interface (rev. 0x02)
piixide0 at pci0 dev 31 function 1
piixide0: Intel 82801CA IDE Controller (ICH3) (rev. 0x02)
piixide0: bus-master DMA support present
piixide0: primary channel configured to compatibility mode
piixide0: primary channel interrupting at irq 14
atabus0 at piixide0 channel 0
piixide0: secondary channel configured to compatibility mode
piixide0: secondary channel interrupting at irq 15
atabus1 at piixide0 channel 1
Intel 82801CA SMBus Controller (SMBus serial bus, revision 0x02) at pci0 dev 31 function 3 not configured
isa0 at pcib0
lpt0 at isa0 port 0x378-0x37b irq 7
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
lm0 at isa0 port 0x290-0x297: W83627HF
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
sysbeep0 at pcppi0
npx0 at isa0 port 0xf0-0xff: using exception 16
apm0 at mainbus0: Power Management spec V1.2
Kernelized RAIDframe activated
scsibus0: waiting 2 seconds for devices to settle...
scsibus1: waiting 2 seconds for devices to settle...
atapibus0 at atabus1: 2 targets
cd0 at atapibus0 drive 0: <CD-224E, , 1.9A> cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33)
cd0(piixide0:1:0): using PIO mode 4, Ultra-DMA mode 2 (Ultra/33) (using DMA)
sd0 at scsibus0 target 0 lun 0: <SEAGATE, ST336607LC, 0007> disk fixed
sd0: 35003 MB, 49855 cyl, 2 head, 718 sec, 512 bytes/sect x 71687372 sectors
sd0: sync (6.25ns offset 63), 16-bit (320.000MB/s) transfers, tagged queueing
sd1 at scsibus0 target 1 lun 0: <SEAGATE, ST336607LC, 0007> disk fixed
sd1: 35003 MB, 49855 cyl, 2 head, 718 sec, 512 bytes/sect x 71687372 sectors
sd1: sync (6.25ns offset 63), 16-bit (320.000MB/s) transfers, tagged queueing
ses0 at scsibus0 target 6 lun 0: <SUPER, GEM318, 0> processor fixed
ses0: SAF-TE Compliant Device
ses0: async, 8-bit transfers
raid0: RAID Level 1
raid0: Components: /dev/sd0a /dev/sd1a
raid0: Total Sectors: 71687168 (35003 MB)
boot device: raid0
root on raid0a dumps on raid0b
root file system type: ffs
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)
---------------------------- 8< ----------------------------
--
----------------------------------------------------------------------
Christoph Kaegi kgc@zhwin.ch
----------------------------------------------------------------------