At 09:43 PM 11/30/2006, Thor Lancelot Simon wrote:

>I note that you snipped the text where I noted that because you're
>testing the wm card with mismatched kernel and ifconfig, you're not
>using its hardware checksum offload.  That's one thing you should
>definitely fix, and if you don't have that turned on for other
>kernels you're testing, of course you should probably fix it there too.

OK, I updated the base as well and rebuilt the kernel. There doesnt 
seem to be much difference, perhaps +5Kpps by turning it on.  But it 
seems to be the driver, as I get FAR better results with the bge nic 
(see below)


# ifconfig wm0
wm0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
         capabilities=7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx,TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
         enabled=300<IP4CSUM_Rx,IP4CSUM_Tx>
         address: 00:15:17:0b:70:98
         media: Ethernet autoselect (1000baseT 
full-duplex,flowcontrol,rxpause,txpause)
         status: active
         inet 192.168.88.223 netmask 0xffffff00 broadcast 192.168.88.255
         inet6 fe80::215:17ff:fe0b:7098%wm0 prefixlen 64 scopeid 0x5
# ifconfig wm1
wm1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
         capabilities=7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx,TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
         enabled=300<IP4CSUM_Rx,IP4CSUM_Tx>
         address: 00:15:17:0b:70:99
         media: Ethernet autoselect (1000baseT 
full-duplex,flowcontrol,rxpause,txpause)
         status: active
         inet 192.168.44.223 netmask 0xffffff00 broadcast 192.168.44.255
         inet6 fe80::215:17ff:fe0b:7099%wm1 prefixlen 64 scopeid 0x6
# netstat -ni
Name  Mtu   Network       Address              Ipkts Ierrs    Opkts Oerrs Colls
wm0   1500  <Link>        00:15:17:0b:70:98 32226898 
281780       15     0     0
wm0   1500  192.168.88/24 192.168.88.223    32226898 
281780       15     0     0
wm0   1500  fe80::/64     fe80::215:17ff:fe 32226898 
281780       15     0     0
wm1   1500  <Link>        00:15:17:0b:70:99       34     0  7117358     0     0
wm1   1500  192.168.44/24 192.168.44.223          34     0  7117358     0     0
wm1   1500  fe80::/64     fe80::215:17ff:fe       34     0  7117358     0     0


There are no errors on the switchport.

And in SMP which is GERNIC.MP with

options   GATEWAY         # packet forwarding





NetBSD 4.99.4 (ROUTER) #1: Thu Nov 30 19:23:52 EST 2006
         mdtancsa@r2-netbsd.sentex.ca:/usr/obj/sys/arch/i386/compile/ROUTER
total memory = 2047 MB
avail memory = 2002 MB
timecounter: Timecounters tick every 10.000 msec
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
BIOS32 rev. 0 found at 0xf21a0
mainbus0 (root)
mainbus0: Intel MP Specification (Version 1.4) (OEM00000 PROD00000000)
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD Unknown K7 (Athlon) (686-class), 2015.10 MHz, id 0x20fb1
cpu0: features f7dbfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features f7dbfbff<PGE,MCA,CMOV,PAT,PSE36,MPC,NOX,MMXX,MMX>
cpu0: features f7dbfbff<FXSR,SSE,SSE2,HTT,LONG,3DNOW2,3DNOW>
cpu0: features2 1<SSE3>
cpu0: "AMD Athlon(tm) 64 X2 Dual Core Processor 3800+"
cpu0: I-cache 64 KB 64B/line 2-way, D-cache 64 KB 64B/line 2-way
cpu0: L2 cache 512 KB 64B/line 16-way
cpu0: ITLB 32 4 KB entries fully associative, 8 4 MB entries fully associative
cpu0: DTLB 32 4 KB entries fully associative, 8 4 MB entries fully associative
cpu0: AMD Power Management features: f<TTP,VID,FID,TS>
cpu0: calibrating local timer
cpu0: apic clock running at 201 MHz
cpu0: 8 page colors
cpu1 at mainbus0: apid 1 (application processor)
cpu1: starting
cpu1: AMD Unknown K7 (Athlon) (686-class), 2015.00 MHz, id 0x20fb1
cpu1: features f7dbfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu1: features f7dbfbff<PGE,MCA,CMOV,PAT,PSE36,MPC,NOX,MMXX,MMX>
cpu1: features f7dbfbff<FXSR,SSE,SSE2,HTT,LONG,3DNOW2,3DNOW>
cpu1: features2 1<SSE3>
cpu1: "AMD Athlon(tm) 64 X2 Dual Core Processor 3800+"
cpu1: I-cache 64 KB 64B/line 2-way, D-cache 64 KB 64B/line 2-way
cpu1: L2 cache 512 KB 64B/line 16-way
cpu1: ITLB 32 4 KB entries fully associative, 8 4 MB entries fully associative
cpu1: DTLB 32 4 KB entries fully associative, 8 4 MB entries fully associative
cpu1: AMD Power Management features: f<TTP,VID,FID,TS>
mpbios: bus 0 is type PCI
mpbios: bus 1 is type PCI
mpbios: bus 2 is type PCI
mpbios: bus 3 is type PCI
mpbios: bus 4 is type PCI
mpbios: bus 5 is type PCI
mpbios: bus 6 is type ISA
ioapic0 at mainbus0 apid 2 (I/O APIC)
ioapic0: pa 0xfec00000, version 11, 24 pins
ioapic0: misconfigured as apic 0
ioapic0: remapped to apic 2
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
NVIDIA nForce4 Memory Controller (miscellaneous memory, revision 
0xa3) at pci0 dev 0 function 0 not configured
pcib0 at pci0 dev 1 function 0
pcib0: NVIDIA product 0x0050 (rev. 0xa3)
NVIDIA nForce4 SMBus (SMBus serial bus, revision 0xa2) at pci0 dev 1 
function 1 not configured
viaide0 at pci0 dev 6 function 0
viaide0: NVIDIA nForce4 IDE Controller (rev. 0xf2)
viaide0: bus-master DMA support present
viaide0: primary channel configured to compatibility mode
viaide0: primary channel interrupting at ioapic0 pin 14 (irq 14)
atabus0 at viaide0 channel 0
viaide0: secondary channel configured to compatibility mode
viaide0: secondary channel interrupting at ioapic0 pin 15 (irq 15)
atabus1 at viaide0 channel 1
ppb0 at pci0 dev 9 function 0: NVIDIA nForce4 PCI Host Bridge (rev. 0xa2)
pci1 at ppb0 bus 5
pci1: i/o space, memory space enabled
vga1 at pci1 dev 8 function 0: ATI Technologies Rage XL (AGP) (rev. 0x65)
wsdisplay0 at vga1 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
nfe0 at pci0 dev 10 function 0: ioapic0 pin 3 (irq 3), address 
00:13:d4:ae:9b:6b
makphy0 at nfe0 phy 1: Marvell 88E1111 Gigabit PHY, rev. 2
makphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
ppb1 at pci0 dev 11 function 0: NVIDIA nForce4 PCIe Host Bridge (rev. 0xa3)
pci2 at ppb1 bus 4
pci2: i/o space, memory space enabled, rd/line, wr/inv ok
bge0 at pci2 dev 0 function 0: Broadcom BCM5751 Gigabit Ethernet
bge0: interrupting at ioapic0 pin 11 (irq 11)
bge0: pcie mode=0x105000
bge0: ASIC BCM5750 A1 (0x4001), Ethernet address 00:10:18:14:15:12
bge0: setting short Tx thresholds
brgphy0 at bge0 phy 1: BCM5750 1000BASE-T media interface, rev. 0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
ppb2 at pci0 dev 12 function 0: NVIDIA nForce4 PCIe Host Bridge (rev. 0xa3)
pci3 at ppb2 bus 3
pci3: i/o space, memory space enabled, rd/line, wr/inv ok
bge1 at pci3 dev 0 function 0: Broadcom BCM5751 Gigabit Ethernet
bge1: interrupting at ioapic0 pin 10 (irq 10)
bge1: pcie mode=0x105000
bge1: ASIC BCM5750 A1 (0x4001), Ethernet address 00:10:18:14:27:d5
bge1: setting short Tx thresholds
brgphy1 at bge1 phy 1: BCM5750 1000BASE-T media interface, rev. 0
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
ppb3 at pci0 dev 13 function 0: NVIDIA nForce4 PCIe Host Bridge (rev. 0xa3)
pci4 at ppb3 bus 2
pci4: i/o space, memory space enabled, rd/line, wr/inv ok
bge2 at pci4 dev 0 function 0: Broadcom BCM5751 Gigabit Ethernet
bge2: interrupting at ioapic0 pin 5 (irq 5)
bge2: pcie mode=0x105000
bge2: ASIC BCM5750 A1 (0x4001), Ethernet address 00:10:18:14:38:d2
bge2: setting short Tx thresholds
brgphy2 at bge2 phy 1: BCM5750 1000BASE-T media interface, rev. 0
brgphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
ppb4 at pci0 dev 14 function 0: NVIDIA nForce4 PCIe Host Bridge (rev. 0xa3)
pci5 at ppb4 bus 1
pci5: i/o space, memory space enabled, rd/line, wr/inv ok
wm0 at pci5 dev 0 function 0: Intel PRO/1000 PT (82571EB), rev. 6
wm0: interrupting at ioapic0 pin 7 (irq 7)
wm0: PCI-Express bus
wm0: 65536 word (16 address bits) SPI EEPROM
wm0: Ethernet address 00:15:17:0b:70:98
igphy0 at wm0 phy 1: Intel IGP01E1000 Gigabit PHY, rev. 0
igphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
wm1 at pci5 dev 0 function 1: Intel PRO/1000 PT (82571EB), rev. 6
wm1: interrupting at ioapic0 pin 5 (irq 5)
wm1: PCI-Express bus
wm1: 65536 word (16 address bits) SPI EEPROM
wm1: Ethernet address 00:15:17:0b:70:99
igphy1 at wm1 phy 1: Intel IGP01E1000 Gigabit PHY, rev. 0
igphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
pchb0 at pci0 dev 24 function 0
pchb0: Advanced Micro Devices AMD64 HyperTransport configuration (rev. 0x00)
pchb1 at pci0 dev 24 function 1
pchb1: Advanced Micro Devices AMD64 Address Map configuration (rev. 0x00)
pchb2 at pci0 dev 24 function 2
pchb2: Advanced Micro Devices AMD64 DRAM configuration (rev. 0x00)
pchb3 at pci0 dev 24 function 3
pchb3: Advanced Micro Devices AMD64 Miscellaneous configuration (rev. 0x00)
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
attimer0 at isa0 port 0x40-0x43: AT Timer
pcppi0 at isa0 port 0x61
pcppi0: children must have an explicit unit
midi0 at pcppi0: PC speaker (CPU-intensive output)
sysbeep0 at pcppi0
isapnp0 at isa0 port 0x279: ISA Plug 'n Play device support
npx0 at isa0 port 0xf0-0xff
npx0: reported by CPUID; using exception 16
pcppi0: attached to attimer0
isapnp0: no ISA Plug 'n Play devices found
ioapic0: enabling
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
Kernelized RAIDframe activated
wd0 at atabus0 drive 0: <ST340014A>
wd0: drive supports 16-sector PIO transfers, LBA48 addressing
wd0: 38166 MB, 77545 cyl, 16 head, 63 sec, 512 bytes/sect x 78165360 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
wd0(viaide0:0:0): using PIO mode 4, Ultra-DMA mode 5 (Ultra/100) (using DMA)
atapibus0 at atabus1: 2 targets
cd0 at atapibus0 drive 1: <AOPEN 8X8 DVD Dual AAN, , 1.4A> cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33)
wd1 at atabus1 drive 0: <ST340014A>
wd1: drive supports 16-sector PIO transfers, LBA48 addressing
wd1: 38166 MB, 77545 cyl, 16 head, 63 sec, 512 bytes/sect x 78165360 sectors
wd1: 32-bit data port
wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
wd1(viaide0:1:0): using PIO mode 4, Ultra-DMA mode 5 (Ultra/100) (using DMA)
cd0(viaide0:1:1): using PIO mode 4, Ultra-DMA mode 2 (Ultra/33) (using DMA)
boot device: wd0
root on wd0a dumps on wd0b
root file system type: ffs
cpu1: CPU 1 running
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)
#



The best I can get is about 125Kpps


However, if I switch to the 2 bge nics (ie NON trunked mode), I get 
close to 600 Kpps on the one stream and a max of 360Kpps when I have 
the stream in the opposite direction going.  This is comparable to 
the other boxes.  However, the driver did wedge and I had to ifconfig 
down/up it to recover once during testing.

Nov 30 19:36:21 r2-netbsd /netbsd: bge1: pcie mode=0x105000
Nov 30 19:38:00 r2-netbsd /netbsd: bge2: pcie mode=0x105000
Nov 30 19:54:18 r2-netbsd /netbsd: bge: failed on len 52?
Nov 30 19:54:49 r2-netbsd last message repeated 10930 times
Nov 30 19:55:55 r2-netbsd last message repeated 14526 times
Nov 30 19:56:11 r2-netbsd /netbsd: ed on len 52?
Nov 30 19:56:11 r2-netbsd /netbsd: bge: failed on len 52?
Nov 30 19:56:12 r2-netbsd last message repeated 719 times
Nov 30 19:56:20 r2-netbsd /netbsd: ed on len 52?
Nov 30 19:56:20 r2-netbsd /netbsd: bge: failed on len 52?
Nov 30 19:56:21 r2-netbsd last message repeated 717 times

         ---Mike