yamt@mwd.biglobe.ne.jp wrote:

> btw, i vaguely remember the bug in question is only triggered with
> very small packets like ip header + a few octets.
> ie. the threshold is not ETHER_MIN_LEN.

Finally I've managed to confirm this value today
(well, I was hit by many other bugs when I tried to do this :-/),
then the threshold seems 26 bytes.
i.e. packets with have IP header and less than 6 octets,
which can be reproduced by ping -s 1473~1477. (examples attached)

If a TX packet less than the threshold is passed,
re(4) doesn't send out any packet at the moment.
If once it happens, the following TX packet becomes mangled,
i.e. no IP checksum is set in the packet so it will be zero.

If the following TX packet is 34 bytes or larger,
only one next packet is mangled.

If the following TX packets are 29~33 bytes, it seems
about next five packets will have wrong (zero) checksums.
In some case they would have corrupted IP headers
(wrong destination etc), so re(4) might set IP checksums
at wrong place.

I'm not sure what will happen if the following packets are also
less than 29 bytes (no idea how I can test it other than ping(8)).


Then, what is the best way to workaround this
(nasty but likely ;-p) hardware bug?
---

Example (A):
Commands on re(4) host (with IFCAP_CSUM_IPv4_Tx is enabled in rtl8169.c):

% ping -s 1473 -n -c 1 192.168.20.28
% ping -s 6 -n -c 1 192.168.20.28
% ping -s 6 -n -c 1 192.168.20.28

--- tcpdump -vn output at destination ---
20:16:34.012469 IP (tos 0x0, ttl 255, id 4099, offset 0, flags [+], length: 1500) 192.168.20.210 > 192.168.20.28: icmp 1480: echo request seq 0
20:16:41.206804 IP (tos 0x0, ttl 255, id 4100, offset 0, flags [none], length: 34, bad cksum 0 (->198)!) 192.168.20.210 > 192.168.20.28: icmp 14: echo request seq 0
20:16:43.070557 IP (tos 0x0, ttl 255, id 4101, offset 0, flags [none], length: 34) 192.168.20.210 > 192.168.20.28: icmp 14: echo request seq 0
20:16:43.070792 IP (tos 0x0, ttl 255, id 60992, offset 0, flags [none], length: 34) 192.168.20.28 > 192.168.20.210: icmp 14: echo reply seq 0

---
Example (B):
Commands on re(4) host (with IFCAP_CSUM_IPv4_Tx is enabled in rtl8169.c):

% ping -s 1473 -n -c 1 192.168.20.28
% ping -s 1 -n -c 1 192.168.20.28
% ping -s 1 -n -c 1 192.168.20.28
% ping -s 1 -n -c 1 192.168.20.28
% ping -s 1 -n -c 1 192.168.20.28
% ping -s 1 -n -c 1 192.168.20.28
% ping -s 1 -n -c 1 192.168.20.28

--- tcpdump -vn output at destination ---
20:13:06.930529 IP (tos 0x0, ttl 255, id 4092, offset 0, flags [+], length: 1500) 192.168.20.210 > 192.168.20.28: icmp 1480: echo request seq 0
20:14:49.384280 IP (tos 0x0, ttl 255, id 4093, offset 0, flags [none], length: 29, bad cksum 0 (->1a4)!) 192.168.20.210 > 192.168.20.28: icmp 9: echo request seq 0
20:14:52.631823 IP (tos 0x0, ttl 255, id 4094, offset 0, flags [none], length: 29, bad cksum 0 (->1a3)!) 192.168.20.210 > 192.168.20.28: icmp 9: echo request seq 0
20:14:53.722353 IP (tos 0x0, ttl 255, id 4095, offset 0, flags [none], length: 29, bad cksum 0 (->1a2)!) 192.168.20.210 > 192.168.20.28: icmp 9: echo request seq 0
20:14:55.806126 IP (tos 0x0, ttl 255, id 4096, offset 0, flags [none], length: 29, bad cksum 0 (->1a1)!) 192.168.20.210 > 192.168.20.28: icmp 9: echo request seq 0
20:14:57.549817 IP (tos 0x0, ttl 255, id 4097, offset 0, flags [none], length: 29, bad cksum 0 (->1a0)!) 192.168.20.210 > 192.168.20.28: icmp 9: echo request seq 0
20:14:59.085900 IP (tos 0x0, ttl 255, id 4098, offset 0, flags [none], length: 29) 192.168.20.210 > 192.168.20.28: icmp 9: echo request seq 0
20:14:59.086130 IP (tos 0x0, ttl 255, id 60986, offset 0, flags [none], length: 29) 192.168.20.28 > 192.168.20.210: icmp 9: echo reply seq 0

----

Example (C):
Commands on re(4) host (with IFCAP_CSUM_IPv4_Tx is enabled in rtl8169.c):

% ping -s 1473 -n -c 1 192.168.20.28
% ping -s 5 -n -c 1 192.168.20.28
% ping -s 5 -n -c 1 192.168.20.28
% ping -s 5 -n -c 1 192.168.20.28
% ping -s 5 -n -c 1 192.168.20.28
% ping -s 5 -n -c 1 192.168.20.28
% ping -s 5 -n -c 1 192.168.20.28

--- tcpdump -vn output at destination ---
20:17:51.455558 IP (tos 0x0, ttl 255, id 4104, offset 0, flags [+], length: 1500) 192.168.20.210 > 192.168.20.28: icmp 1480: echo request seq 0
20:17:56.842909 IP (tos 0x0, ttl 255, id 4105, offset 0, flags [none], length: 33, bad cksum 0 (->194)!) 192.168.20.210 > 192.168.20.28: icmp 13: echo request seq 0
20:17:59.659083 IP (tos 0x0, ttl 255, id 4106, offset 0, flags [none], length: 33, bad cksum 0 (->193)!) 192.168.20.210 > 192.168.20.28: icmp 13: echo request seq 0
20:18:01.441536 IP (tos 0x0, ttl 255, id 4107, offset 0, flags [none], length: 33, bad cksum 0 (->192)!) 192.168.20.210 > 192.168.20.28: icmp 13: echo request seq 0
20:18:03.492525 IP (tos 0x0, ttl 255, id 4108, offset 0, flags [none], length: 33, bad cksum 0 (->191)!) 192.168.20.210 > 192.168.20.28: icmp 13: echo request seq 0
20:18:04.778551 IP (tos 0x0, ttl 255, id 4109, offset 0, flags [none], length: 33, bad cksum 0 (->190)!) 192.168.20.210 > 192.168.20.28: icmp 13: echo request seq 0
20:18:06.675099 IP (tos 0x0, ttl 255, id 4110, offset 0, flags [none], length: 33) 192.168.20.210 > 192.168.20.28: icmp 13: echo request seq 0
20:18:06.675339 IP (tos 0x0, ttl 255, id 60999, offset 0, flags [none], length: 33) 192.168.20.28 > 192.168.20.210: icmp 13: echo reply seq 0
---

Example (D):

Commands on re(4) host (with IFCAP_CSUM_IPv4_Tx is enabled in rtl8169.c):

% ping -s 1473 -n -c 1 192.168.20.28
% ping -s 1 -n -c 1 192.168.20.28
% ping -s 6 -n -c 1 192.168.20.28
% ping -s 6 -n -c 1 192.168.20.28
% ping -s 6 -n -c 1 192.168.20.28
% ping -s 6 -n -c 1 192.168.20.28

--- tcpdump -vn output at destination ---
20:19:13.432362 IP (tos 0x0, ttl 255, id 4114, offset 0, flags [+], length: 1500) 192.168.20.210 > 192.168.20.28: icmp 1480: echo request seq 0
20:19:17.769722 IP (tos 0x0, ttl 255, id 4115, offset 0, flags [none], length: 29, bad cksum 0 (->18e)!) 192.168.20.210 > 192.168.20.28: icmp 9: echo request seq 0
20:19:21.651856 IP (tos 0x0, ttl 255, id 4116, offset 0, flags [none], length: 34, bad cksum 0 (->188)!) 192.168.20.210 > 192.168.20.28: icmp 14: echo request seq 0
20:19:23.466803 IP (tos 0x0, ttl 255, id 4117, offset 0, flags [none], length: 34, bad cksum 0 (->9c42)!) 192.168.20.210 > 192.168.121.96: icmp 14: echo request seq 0 (wrong icmp cksum 0 (->f7ff)!)
20:19:26.803880 IP (tos 0x0, ttl 255, id 4118, offset 0, flags [none], length: 34, bad cksum 0 (->186)!) 192.168.20.210 > 192.168.20.28: icmp 14: echo request seq 0
20:19:29.508109 IP (tos 0x0, ttl 255, id 4119, offset 0, flags [none], length: 34) 192.168.20.210 > 192.168.20.28: icmp 14: echo request seq 0
20:19:29.508345 IP (tos 0x0, ttl 255, id 61008, offset 0, flags [none], length: 34) 192.168.20.28 > 192.168.20.210: icmp 14: echo reply seq 0

---

Example (E):

Commands on re(4) host (with IFCAP_CSUM_IPv4_Tx is enabled in rtl8169.c):

% ping -s 1478 -n -c 1 192.168.20.28
% ping -s 1 -n -c 1 192.168.20.28
% ping -s 1 -n -c 1 192.168.20.28

--- tcpdump -vn output at destination ---
20:28:21.410209 IP (tos 0x0, ttl 255, id 4123, offset 0, flags [+], length: 1500) 192.168.20.210 > 192.168.20.28: icmp 1480: echo request seq 0
20:28:21.410265 IP (tos 0x0, ttl 255, id 4123, offset 1480, flags [none], length: 26) 192.168.20.210 > 192.168.20.28: icmp
20:28:21.410703 IP (tos 0x0, ttl 255, id 61095, offset 0, flags [+], length: 1500) 192.168.20.28 > 192.168.20.210: icmp 1480: echo reply seq 0
20:28:21.410753 IP (tos 0x0, ttl 255, id 61095, offset 1480, flags [none], length: 26) 192.168.20.28 > 192.168.20.210: icmp
20:28:25.096612 IP (tos 0x0, ttl 255, id 4124, offset 0, flags [none], length: 29) 192.168.20.210 > 192.168.20.28: icmp 9: echo request seq 0
20:28:25.096857 IP (tos 0x0, ttl 255, id 61096, offset 0, flags [none], length: 29) 192.168.20.28 > 192.168.20.210: icmp 9: echo reply seq 0
20:28:26.146356 IP (tos 0x0, ttl 255, id 4125, offset 0, flags [none], length: 29) 192.168.20.210 > 192.168.20.28: icmp 9: echo request seq 0
20:28:26.146564 IP (tos 0x0, ttl 255, id 61097, offset 0, flags [none], length: 29) 192.168.20.28 > 192.168.20.210: icmp 9: echo reply seq 0

---
Izumi Tsutsui