Subject: bridge(4) and silent data corruption :-(
To: None <tech-net@netbsd.org, thorpej@netbsd.org>
From: Sean Doran <smd@ab.use.net>
List: tech-net
Date: 04/29/2002 03:40:30
ALTQ was a red herring, bridge(4) in a pure GENERIC corrupts data. :-(

Setup:

Efficient
Networks    ---[10baseT]--- NetBSD(bridge) --- Netgear FS105 switch --- "in"
ADSL router

("in" is a variety of machines)

The kernel is a snapshot:

NetBSD 1.5ZC (GENERIC) #0: Wed Apr 24 17:36:26 EDT 2002
    autobuild@tgm.wasabisystems.com:/home/tv/scratch/obj.i386/sys/arch/i386/compile/GENERIC

There is nothing funky done in BIOS.

Note that the NetBSD box can easily be removed, and the switch and
router be directly connected using the same or a different port.
The direct connection between switch and router is a tried and
tested combination, although the Efficient Networks box does really
really lousy queue management.

When the NetBSD box is installed with bridging enabled, the "in"
boxes cannot sustain a large (multimegabyte) scp protocol 2 
connection with the outside world; the boxes complain about
MAC corruption on input.  There are other signs of
silent corruption of otherwise valid TCP segments received
across the bridge (bugged-out jpegs and so on from the web).

This is a little disturbing.

Any ideas?

(I've tried the setup with and without configuring inet addrs
into the ethernet interfaces, with and without discovery,
with and without spanning tree, using various speeds and
duplexes, swapping leads, and so on, without any obvious
impact.  (However, cranking down the speed on the connection
to the FE switch to 10 Mbps gets rid of most (not all) of
the occasional "exN: uplistptr was 0" message).

	Sean.
- --

bridge0: flags=41<UP,RUNNING>
        Configuration:
                priority 32768 hellotime 2 fwddelay 15 maxage 20
        Interfaces:
                ex1 flags=3<LEARNING,DISCOVER>
                        port 2 priority 128
                ex0 flags=3<LEARNING,DISCOVER>
                        port 1 priority 128
        Address cache (max cache: 100, timeout: 1200):
                00:04:76:9a:0f:13 ex1 562 flags=0<>
                00:20:6f:0f:d9:38 ex0 538 flags=0<>
                00:03:93:8e:4d:e4 ex1 538 flags=0<>
                00:30:65:ed:34:f2 ex1 537 flags=0<>

ex0: flags=8963<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        capabilities=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
        enabled=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
        address: 00:04:76:de:ba:da
        media: Ethernet autoselect (10baseT)
        status: active
        inet xxx.xxx.xxx.xxx netmask 0xfffffff8 broadcast xxx.xxx.xxx.xxx
        inet6 ex0 prefixlen 64 scopeid 0x1
ex1: flags=8963<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        capabilities=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
        enabled=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
        address: 00:04:75:80:72:a2
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet xxx.xxx.xxx.xxx netmask 0xfffffff8 broadcast xxx.xxx.xxx.xxx
        inet6 ex1 prefixlen 64 scopeid 0x2
bridge0: flags=41<UP,RUNNING> mtu 1500

NetBSD 1.5ZC (GENERIC) #0: Wed Apr 24 17:36:26 EDT 2002
    autobuild@tgm.wasabisystems.com:/home/tv/scratch/obj.i386/sys/arch/i386/compile/GENERIC
cpu0: Intel Pentium III (Coppermine) Celeron (686-class), 851.98 MHz
cpu0: I-cache 16 KB 32b/line 4-way, D-cache 16 KB 32b/line 2-way
cpu0: L2 cache 128 KB 32b/line 4-way
cpu0: features 383f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR>
cpu0: features 383f9ff<PGE,MCA,CMOV,FGPAT,PSE36,MMX>
cpu0: features 383f9ff<FXSR,SSE>
total memory = 126 MB
avail memory = 109 MB
using 1645 buffers containing 6580 KB of memory
BIOS32 rev. 0 found at 0xfdb70
mainbus0 (root)
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: Intel 82810E Memory Controller Hub (rev. 0x03)
agp0 at pchb0: aperture at 0xdc000000, size 0x4000000
vga1 at pci0 dev 1 function 0: Intel 82810E Graphics Controller (rev. 0x03)
wsdisplay0 at vga1 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
ppb0 at pci0 dev 30 function 0: Intel 82801AA Hub-to-PCI Bridge (rev. 0x02)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled
ex0 at pci1 dev 3 function 0: 3Com 3c905C-TX 10/100 Ethernet with mngmt (rev. 0x74)
ex0: interrupting at irq 11
ex0: MAC address 00:04:76:de:ba:da
bmtphy0 at ex0 phy 24: Broadcom 3c905C internal PHY, rev. 6
bmtphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
ex1 at pci1 dev 4 function 0: 3Com 3c905C-TX 10/100 Ethernet with mngmt (rev. 0x30)
ex1: interrupting at irq 10
ex1: MAC address 00:04:75:80:72:a2
ukphy0 at ex1 phy 24: Generic IEEE 802.3u media interface
ukphy0: OUI 0x0006b8, model 0x0035, rev. 0
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
pcib0 at pci0 dev 31 function 0
pcib0: Intel 82801AA LPC Interface Bridge (rev. 0x02)
pciide0 at pci0 dev 31 function 1: Intel 82801AA IDE Controller (ICH) (rev. 0x02)
pciide0: bus-master DMA support present
pciide0: primary channel wired to compatibility mode
wd0 at pciide0 channel 0 drive 0: <ST320410A>
wd0: drive supports 16-sector PIO transfers, LBA addressing
wd0: 19092 MB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 39102336 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
pciide0: primary channel interrupting at irq 14
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 4 (Ultra/66) (using DMA data transfers)
pciide0: secondary channel wired to compatibility mode
atapibus0 at pciide0 channel 1: 2 targets
cd0 at atapibus0 drive 0: <SAMSUNG CDRW/DVD SM-308B, , BB04> type 5 cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2
pciide0: secondary channel interrupting at irq 15
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2 (using DMA data transfers)
uhci0 at pci0 dev 31 function 2: Intel 82801AA USB Controller (rev. 0x02)
uhci0: interrupting at irq 12
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
Intel 82801AA SMBus Controller (SMBus serial bus, revision 0x02) at pci0 dev 31 function 3 not configured
auich0 at pci0 dev 31 function 5: i82801AA (ICH) AC-97 Audio
auich0: interrupting at irq 10
auich0: SigmaTel STAC9721/23 codec; 18 bit DAC, 18 bit ADC, SigmaTel 3D
auich0: warning, fixed rate codec
audio0 at auich0: full duplex, mmap, independent
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
lpt0 at isa0 port 0x378-0x37b irq 7
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker
sysbeep0 at pcppi0
isapnp0 at isa0 port 0x279: ISA Plug 'n Play device support
npx0 at isa0 port 0xf0-0xff: using exception 16
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
isapnp0: no ISA Plug 'n Play devices found
biomask f36d netmask ff6d ttymask ffef
Kernelized RAIDframe activated
cd0: no disk label
uhub1 at uhub0 port 2
uhub1: vendor 0x0416 USB HUB, class 9/0, rev 1.10/0.06, addr 2
uhub1: 4 ports with 4 removable, self powered
uhub2 at uhub1 port 1
uhub2: BTC USB Keyboard, class 9/0, rev 1.10/0.03, addr 3
uhub2: 2 ports with 1 removable, bus powered
uhidev0 at uhub2 port 1 configuration 1 interface 0
uhidev0: BTC USB Keyboard, rev 1.10/0.03, addr 4, iclass 3/1
ukbd0 at uhidev0
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub2 port 1 configuration 1 interface 1
uhidev1: BTC USB Keyboard, rev 1.10/0.03, addr 4, iclass 3/0
uhidev1: 7 report ids
uhid0 at uhidev1 reportid 1: input=1, output=0, feature=0
uhid1 at uhidev1 reportid 2: input=1, output=0, feature=0
uhid2 at uhidev1 reportid 3: input=2, output=0, feature=0
uhid3 at uhidev1 reportid 7: input=3, output=0, feature=0
boot device: wd0
root on wd0a dumps on wd0b
root file system type: ffs
in6_ifattach: bridge0 is not multicast capable, IPv6 not enabled
in6_ifattach: bridge0 is not multicast capable, IPv6 not enabled
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)