Subject: Re: bridge(4) and silent data corruption :-(
To: None <mcr@sandelman.ottawa.on.ca, smd@ab.use.net>
From: Sean Doran <smd@ab.use.net>
List: tech-net
Date: 04/29/2002 18:24:16 
mcr writes:

|   Presumeably, if packets are getting corrupted, that TCP checksumming should
| really be catching this, unless the bridge is "fixing" them in some way...

It sure looks like that...  However, I haven't figured out why local traffic
through the bridge works OK but non-local doesn't.

Topology 1:

Router ---- hub --- NetBSDbridge --- switch --- station-b
             |
           station-a

Working:

	scp2 from world to station-a
	scp2 from station-a to world
	scp2 from station-a to station-b
	scp2 from station-b to station-a

Not working (meaning "Disconnected: Corrupted MAC on input."): 

	scp2 from from world to station-b
	scp2 from station-b to world


Topology 2: 

Router ---- hub --- NetBSDbridge --- switch --- station-b
					|
				     station-a

Working:

	scp2 from station-a to station-b (and vice-versa)
	
Not working:

	scp2 from world to station-a (and vice-versa)
	scp2 from world to station-b (and vice-versa)


Topology 3:

Router --- NetBSD-bridge --- hub --- station-b
			      |
			    station-a

Working:

	scp2 from station-a to station-b (and vice-versa)

Not working:

	scp2 from world to station-a (and vice-versa)
	scp2 from world to station-b (and vice-versa)


|   Maybe a tcpdump from both sides of the bridge? Too bad there isn't a good
| program to coorelate and compare packets!

No kidding...  Here's a complete dump of world->station-a (it fails
nice and quickly) during Topology 3.

(tcpdump -w x -vvv -s 1500 -i en0 -ttt host world; tcpdump -r x
 yeah, station-a is a mac os x laptoy...).

Note that it gets through the challenge/response phase and into
the bulk transfer phase for a few segments before ssh2 on station-a
complains about corrupted MAC on input.

The file is a couple hundred meg, and completes successfully in the
"working" tests above.

	Sean.

01:42:53.624319 station-b.49159 > world.22: S 2163028226:2163028226(0) win 32768 <mss 1460,nop,wscale 0,nop,nop,timestamp 4383 0> (DF)
01:42:53.703212 world.22 > station-b.49159: S 3523436887:3523436887(0) ack 2163028227 win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 0 4383> (DF) [tos 0x20] 
01:42:53.703276 station-b.49159 > world.22: . ack 1 win 33304 <nop,nop,timestamp 4383 0> (DF)
01:42:53.778311 world.22 > station-b.49159: . ack 1 win 65535 <nop,nop,timestamp 0 0> (DF) [tos 0x20] 
01:42:53.781180 world.22 > station-b.49159: P 1:51(50) ack 1 win 65535 <nop,nop,timestamp 0 0> (DF) [tos 0x20] 
01:42:53.781873 station-b.49159 > world.22: P 1:25(24) ack 51 win 33304 <nop,nop,timestamp 4383 0> (DF)
01:42:53.865764 world.22 > station-b.49159: P 51:683(632) ack 25 win 65535 <nop,nop,timestamp 0 4383> (DF) [tos 0x20] 
01:42:53.865863 station-b.49159 > world.22: P 25:657(632) ack 683 win 32672 <nop,nop,timestamp 4384 0> (DF)
01:42:54.162237 world.22 > station-b.49159: . ack 657 win 65535 <nop,nop,timestamp 1 4384> (DF) [tos 0x20] 
01:42:54.162340 station-b.49159 > world.22: P 657:681(24) ack 683 win 33304 <nop,nop,timestamp 4384 1> (DF)
01:42:54.243357 world.22 > station-b.49159: P 683:835(152) ack 681 win 65535 <nop,nop,timestamp 1 4384> (DF) [tos 0x20] 
01:42:54.267953 station-b.49159 > world.22: P 681:825(144) ack 835 win 33304 <nop,nop,timestamp 4384 1> (DF)
01:42:54.378737 world.22 > station-b.49159: P 835:1499(664) ack 825 win 65535 <nop,nop,timestamp 1 4384> (DF) [tos 0x20] 
01:42:54.434982 station-b.49159 > world.22: P 825:841(16) ack 1499 win 33304 <nop,nop,timestamp 4385 1> (DF)
01:42:54.702187 world.22 > station-b.49159: . ack 841 win 65535 <nop,nop,timestamp 2 4385> (DF) [tos 0x20] 
01:42:54.702265 station-b.49159 > world.22: P 841:889(48) ack 1499 win 33304 <nop,nop,timestamp 4385 2> (DF)
01:42:54.777240 world.22 > station-b.49159: P 1499:1547(48) ack 889 win 65535 <nop,nop,timestamp 2 4385> (DF) [tos 0x20] 
01:42:54.777495 station-b.49159 > world.22: P 889:953(64) ack 1547 win 33304 <nop,nop,timestamp 4385 2> (DF)
01:42:54.861250 world.22 > station-b.49159: P 1547:1627(80) ack 953 win 65535 <nop,nop,timestamp 2 4385> (DF) [tos 0x20] 
01:42:54.861623 station-b.49159 > world.22: P 953:1193(240) ack 1627 win 33304 <nop,nop,timestamp 4386 2> (DF)
01:42:54.948328 world.22 > station-b.49159: P 1627:1819(192) ack 1193 win 65535 <nop,nop,timestamp 2 4386> (DF) [tos 0x20] 
01:42:55.069853 station-b.49159 > world.22: . ack 1819 win 33304 <nop,nop,timestamp 4386 2> (DF)
01:42:56.996843 station-b.49159 > world.22: P 1193:1577(384) ack 1819 win 33304 <nop,nop,timestamp 4390 2> (DF)
01:42:57.090147 world.22 > station-b.49159: P 1819:1851(32) ack 1577 win 65535 <nop,nop,timestamp 7 4390> (DF) [tos 0x20] 
01:42:57.090951 station-b.49159 > world.22: P 1577:1641(64) ack 1851 win 33304 <nop,nop,timestamp 4390 7> (DF)
01:42:57.168158 world.22 > station-b.49159: P 1851:1899(48) ack 1641 win 65535 <nop,nop,timestamp 7 4390> (DF) [tos 0x20] 
01:42:57.168422 station-b.49159 > world.22: P 1641:1721(80) ack 1899 win 33304 <nop,nop,timestamp 4390 7> (DF) [tos 0x8] 
01:42:57.246210 world.22 > station-b.49159: P 1899:1947(48) ack 1721 win 65535 <nop,nop,timestamp 7 4390> (DF) [tos 0x28] 
01:42:57.246420 station-b.49159 > world.22: P 1721:1769(48) ack 1947 win 33304 <nop,nop,timestamp 4390 7> (DF) [tos 0x8] 
01:42:57.321185 world.22 > station-b.49159: P 1947:2011(64) ack 1769 win 65535 <nop,nop,timestamp 7 4390> (DF) [tos 0x28] 
01:42:57.323007 station-b.49159 > world.22: P 1769:1817(48) ack 2011 win 33304 <nop,nop,timestamp 4390 7> (DF) [tos 0x8] 
01:42:57.415753 world.22 > station-b.49159: . 2011:3459(1448) ack 1817 win 65535 <nop,nop,timestamp 7 4390> (DF) [tos 0x28] 
01:42:57.421768 world.22 > station-b.49159: . 3459:4907(1448) ack 1817 win 65535 <nop,nop,timestamp 7 4390> (DF) [tos 0x28] 
01:42:57.421861 station-b.49159 > world.22: . ack 4907 win 33304 <nop,nop,timestamp 4391 7> (DF) [tos 0x8] 
01:42:57.427702 world.22 > station-b.49159: . 4907:6355(1448) ack 1817 win 65535 <nop,nop,timestamp 7 4390> (DF) [tos 0x28] 
01:42:57.433517 world.22 > station-b.49159: . 6355:7803(1448) ack 1817 win 65535 <nop,nop,timestamp 7 4390> (DF) [tos 0x28] 
01:42:57.433610 station-b.49159 > world.22: . ack 7803 win 33304 <nop,nop,timestamp 4391 7> (DF) [tos 0x8] 
01:42:57.511957 world.22 > station-b.49159: . 7803:9251(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.517708 world.22 > station-b.49159: . 9251:10699(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.517782 station-b.49159 > world.22: . ack 10699 win 33304 <nop,nop,timestamp 4391 8> (DF) [tos 0x8] 
01:42:57.523732 world.22 > station-b.49159: . 10699:12147(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.529810 world.22 > station-b.49159: . 12147:13595(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.529882 station-b.49159 > world.22: . ack 13595 win 33304 <nop,nop,timestamp 4391 8> (DF) [tos 0x8] 
01:42:57.535654 world.22 > station-b.49159: . 13595:15043(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.541489 world.22 > station-b.49159: . 15043:16491(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.541577 station-b.49159 > world.22: . ack 16491 win 33304 <nop,nop,timestamp 4391 8> (DF) [tos 0x8] 
01:42:57.607787 world.22 > station-b.49159: . 16491:17939(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.613742 world.22 > station-b.49159: . 17939:19387(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.613824 station-b.49159 > world.22: . ack 19387 win 33304 <nop,nop,timestamp 4391 8> (DF) [tos 0x8] 
01:42:57.619720 world.22 > station-b.49159: . 19387:20835(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.625759 world.22 > station-b.49159: . 20835:22283(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.625834 station-b.49159 > world.22: . ack 22283 win 33304 <nop,nop,timestamp 4391 8> (DF) [tos 0x8] 
01:42:57.631564 world.22 > station-b.49159: . 22283:23731(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.640740 world.22 > station-b.49159: . 23731:25179(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.640824 station-b.49159 > world.22: . ack 25179 win 33304 <nop,nop,timestamp 4391 8> (DF) [tos 0x8] 
01:42:57.646761 world.22 > station-b.49159: . 25179:26627(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.652760 world.22 > station-b.49159: . 26627:28075(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.652834 station-b.49159 > world.22: . ack 28075 win 33304 <nop,nop,timestamp 4391 8> (DF) [tos 0x8] 
01:42:57.658490 world.22 > station-b.49159: . 28075:29523(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.670089 station-b.49159 > world.22: . ack 29523 win 33304 <nop,nop,timestamp 4391 8> (DF) [tos 0x8] 
01:42:57.703672 world.22 > station-b.49159: . 29523:30971(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.709711 world.22 > station-b.49159: . 30971:32419(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.709786 station-b.49159 > world.22: . ack 32419 win 33304 <nop,nop,timestamp 4391 8> (DF) [tos 0x8] 
01:42:57.718811 world.22 > station-b.49159: . 32419:33867(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.721645 world.22 > station-b.49159: . 33867:35315(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.721720 station-b.49159 > world.22: . ack 35315 win 33304 <nop,nop,timestamp 4391 8> (DF) [tos 0x8] 
01:42:57.722356 station-b.49159 > world.22: P 1817:1865(48) ack 35315 win 33304 <nop,nop,timestamp 4391 8> (DF) [tos 0x8] 
01:42:57.730705 world.22 > station-b.49159: . 35315:36763(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.736776 world.22 > station-b.49159: . 36763:38211(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.737597 station-b.49159 > world.22: . ack 38211 win 33304 <nop,nop,timestamp 4391 8> (DF) [tos 0x8] 
01:42:57.742743 world.22 > station-b.49159: . 38211:39659(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.748774 world.22 > station-b.49159: . 39659:41107(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.748861 station-b.49159 > world.22: . ack 41107 win 33304 <nop,nop,timestamp 4391 8> (DF) [tos 0x8] 
01:42:57.754777 world.22 > station-b.49159: . 41107:42555(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.760671 world.22 > station-b.49159: . 42555:44003(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.760742 station-b.49159 > world.22: . ack 44003 win 33304 <nop,nop,timestamp 4391 8> (DF) [tos 0x8] 
01:42:57.761302 station-b.49159 > world.22: FP 1865:1929(64) ack 44003 win 33304 <nop,nop,timestamp 4391 8> (DF) [tos 0x8] 
01:42:57.767060 world.22 > station-b.49159: . 44003:45451(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.767197 station-b.49159 > world.22: R 2163030043:2163030043(0) win 0
01:42:57.772713 world.22 > station-b.49159: . 45451:46899(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.772786 station-b.49159 > world.22: R 2163030043:2163030043(0) win 0
01:42:57.778728 world.22 > station-b.49159: . 46899:48347(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.778773 station-b.49159 > world.22: R 2163030043:2163030043(0) win 0
01:42:57.784413 world.22 > station-b.49159: . 48347:49795(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.784446 station-b.49159 > world.22: R 2163030043:2163030043(0) win 0
01:42:57.799703 world.22 > station-b.49159: . 49795:51243(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.799738 station-b.49159 > world.22: R 2163030043:2163030043(0) win 0
01:42:57.805806 world.22 > station-b.49159: . 51243:52691(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.805917 station-b.49159 > world.22: R 2163030043:2163030043(0) win 0
01:42:57.811448 world.22 > station-b.49159: . 52691:54139(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.811501 station-b.49159 > world.22: R 2163030043:2163030043(0) win 0
01:42:57.829804 world.22 > station-b.49159: . 54139:55587(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.829905 station-b.49159 > world.22: R 2163030043:2163030043(0) win 0
01:42:57.835800 world.22 > station-b.49159: . 55587:57035(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.835842 station-b.49159 > world.22: R 2163030043:2163030043(0) win 0
01:42:57.841684 world.22 > station-b.49159: . 57035:58483(1448) ack 1817 win 65535 <nop,nop,timestamp 8 4390> (DF) [tos 0x28] 
01:42:57.841718 station-b.49159 > world.22: R 2163030043:2163030043(0) win 0
01:42:57.847719 world.22 > station-b.49159: . 58483:59931(1448) ack 1865 win 65535 <nop,nop,timestamp 8 4391> (DF) [tos 0x28] 
01:42:57.847756 station-b.49159 > world.22: R 2163030091:2163030091(0) win 0
01:42:57.853743 world.22 > station-b.49159: . 59931:61379(1448) ack 1865 win 65535 <nop,nop,timestamp 8 4391> (DF) [tos 0x28] 
01:42:57.853778 station-b.49159 > world.22: R 2163030091:2163030091(0) win 0
01:42:57.859764 world.22 > station-b.49159: . 61379:62827(1448) ack 1865 win 65535 <nop,nop,timestamp 8 4391> (DF) [tos 0x28] 
01:42:57.859800 station-b.49159 > world.22: R 2163030091:2163030091(0) win 0
01:42:57.865841 world.22 > station-b.49159: . 62827:64275(1448) ack 1865 win 65535 <nop,nop,timestamp 8 4391> (DF) [tos 0x28] 
01:42:57.865874 station-b.49159 > world.22: R 2163030091:2163030091(0) win 0
01:42:57.874665 world.22 > station-b.49159: . 64275:65723(1448) ack 1865 win 65535 <nop,nop,timestamp 8 4391> (DF) [tos 0x28] 
01:42:57.874699 station-b.49159 > world.22: R 2163030091:2163030091(0) win 0
01:42:57.880720 world.22 > station-b.49159: . 65723:67171(1448) ack 1865 win 65535 <nop,nop,timestamp 8 4391> (DF) [tos 0x28] 
01:42:57.880760 station-b.49159 > world.22: R 2163030091:2163030091(0) win 0
01:42:57.886748 world.22 > station-b.49159: . 67171:68619(1448) ack 1865 win 65535 <nop,nop,timestamp 8 4391> (DF) [tos 0x28] 
01:42:57.886800 station-b.49159 > world.22: R 2163030091:2163030091(0) win 0
01:42:57.892706 world.22 > station-b.49159: . 68619:70067(1448) ack 1865 win 65535 <nop,nop,timestamp 8 4391> (DF) [tos 0x28] 
01:42:57.892719 world.22 > station-b.49159: . ack 1930 win 65535 <nop,nop,timestamp 8 4391> (DF) [tos 0x28] 
01:42:57.892750 station-b.49159 > world.22: R 2163030091:2163030091(0) win 0
01:42:57.892765 station-b.49159 > world.22: R 2163030156:2163030156(0) win 0
01:42:57.898468 world.22 > station-b.49159: . 70067:71515(1448) ack 1865 win 65535 <nop,nop,timestamp 8 4391> (DF) [tos 0x28] 
01:42:57.898500 station-b.49159 > world.22: R 2163030091:2163030091(0) win 0