>Number:         12671
>Category:       kern
>Synopsis:       NAT and some kind of fragmented packets break TCP connections
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Apr 15 13:49:01 PDT 2001
>Closed-Date:
>Last-Modified:
>Originator:     Martin Husemann
>Release:        1.5U, cvs update from April 14
>Organization:
	
>Environment:
	
System: NetBSD night-porter.duskware.de 1.5U NetBSD 1.5U (PORTER) #0: Sun Apr 15 12:04:33 MEST 2001 martin@night-porter.duskware.de:/usr/src/sys/arch/i386/compile/PORTER i386
Architecture: i386
Machine: i386
>Description:

I have the following network setup:

  [ISP]----<DSL>---[DSL-Modem]--<PPPoE>-----[NAT]-------[Client]
                                           ^     ^     ^
                                           |     |     |
                                           |     |     tlp0, MTU 1500
                                           |     |
                                           |    ep0, MTU 1500
                                           |
                                          ne0<->pppoe0, MTU 1492

Both NAT and Client are running NetBSD/i386 -current (kernel source from
a cvs update less than 12 hours ago).

NAT (for the moment) does not use any filter rules, and the obvious ipnat
setup:

map pppoe0 192.168.150.0/24 -> 0/32 proxy port ftp ftp/tcp
map pppoe0 192.168.150.0/24 -> 0/32 portmap tcp/udp 43000:49999
map pppoe0 192.168.150.0/24 -> 0/32

This setup exhibits severe problems as soon as it receives fragmented
packets for TCP sessions behind the NAT box. I notices with fetchmail doing
POP3 to an external mail server and with cvs doing larger updates.

Seems the data delivered in the fragments is not ACKd properly and repeated
by the peer until it gives up (and the connections breaks).

A simple example to demonstrate the effect is:

   ftp -a ftp.netbsd.org

It works on the NAT box (see tcpdump attached), but doing the same on Client
doesn't work, it stops midway in the banner message and then times out (see
second tcpdump attached below). I have PMTU discovery off on NAT and on on
Client (can't remember why, NAT is a quite new default config, and I probably
turned it on on Client for testing some time ago). But all combinations of
net.inet.ip.mtudisc on both machines do not make any difference. 

Now for the tcpdumps:

First try: "ftp -a ftp.netbsd.org" on NAT (works):

tcpdump: listening on pppoe0
14:41:10.642286 pD9506F36.dip.t-dialin.net.65533 > ftp.netbsd.org.ftp: S [tcp sum ok] 1566079173:1566079173(0) win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0> (ttl 64, id 14299)
14:41:10.867318 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.65533: S [tcp sum ok] 59445291:59445291(0) ack 1566079174 win 32768 <mss 1460,nop,wscale 0,nop,nop,timestamp 11154264 0> (ttl 52, id 20530)
14:41:10.867841 pD9506F36.dip.t-dialin.net.65533 > ftp.netbsd.org.ftp: . [tcp sum ok] 1:1(0) ack 1 win 17520 <nop,nop,timestamp 0 11154264> (ttl 64, id 14300)
14:41:11.104858 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.65533: P [tcp sum ok] 1:7(6) ack 1 win 33580 <nop,nop,timestamp 11154265 0> [tos 0x10] (ttl 52, id 20648)
14:41:11.289843 pD9506F36.dip.t-dialin.net.65533 > ftp.netbsd.org.ftp: . [tcp sum ok] 1:1(0) ack 7 win 17520 <nop,nop,timestamp 1 11154265> [tos 0x10] (ttl 64, id 14306)
14:41:11.509923 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.65533: P [tcp sum ok] 7:68(61) ack 1 win 33580 <nop,nop,timestamp 11154265 0> [tos 0x10] (ttl 52, id 20825)
14:41:11.514001 pD9506F36.dip.t-dialin.net.65533 > ftp.netbsd.org.ftp: P [tcp sum ok] 1:17(16) ack 68 win 17520 <nop,nop,timestamp 1 11154265> [tos 0x10] (ttl 64, id 14307)
14:41:11.866619 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.65533: P [tcp sum ok] 68:117(49) ack 17 win 33580 <nop,nop,timestamp 11154266 1> [tos 0x10] (ttl 52, id 21009)
14:41:11.869385 pD9506F36.dip.t-dialin.net.65533 > ftp.netbsd.org.ftp: P [tcp sum ok] 17:29(12) ack 117 win 17520 <nop,nop,timestamp 2 11154266> [tos 0x10] (ttl 64, id 14308)
14:41:12.101881 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.65533: P [tcp sum ok] 117:123(6) ack 29 win 33580 <nop,nop,timestamp 11154267 2> [tos 0x10] (ttl 52, id 21135)
14:41:12.120354 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.65533: . [bad tcp cksum 88b4!] 123:1563(1440) ack 29 win 33580 <nop,nop,timestamp 11154267 2> (frag 21136:1472@0+) [tos 0x10] (ttl 51)
14:41:12.120488 ftp.netbsd.org > pD9506F36.dip.t-dialin.net: (frag 21136:8@1472) [tos 0x10] (ttl 51)
14:41:12.121146 pD9506F36.dip.t-dialin.net.65533 > ftp.netbsd.org.ftp: . [tcp sum ok] 29:29(0) ack 1571 win 16072 <nop,nop,timestamp 3 11154267> [tos 0x10] (ttl 64, id 14309)
14:41:12.351359 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.65533: P [tcp sum ok] 1571:2496(925) ack 29 win 33580 <nop,nop,timestamp 11154267 2> [tos 0x10] (ttl 52, id 21288)
14:41:12.489916 pD9506F36.dip.t-dialin.net.65533 > ftp.netbsd.org.ftp: . [tcp sum ok] 29:29(0) ack 2496 win 17520 <nop,nop,timestamp 3 11154267> [tos 0x10] (ttl 64, id 14310)
14:41:12.667134 pD9506F36.dip.t-dialin.net.65533 > ftp.netbsd.org.ftp: P [tcp sum ok] 29:35(6) ack 2496 win 17520 <nop,nop,timestamp 4 11154267> [tos 0x10] (ttl 64, id 14311)
14:41:12.887927 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.65533: P [tcp sum ok] 2496:2545(49) ack 35 win 33580 <nop,nop,timestamp 11154268 4> [tos 0x10] (ttl 52, id 21573)
14:41:12.889837 pD9506F36.dip.t-dialin.net.65533 > ftp.netbsd.org.ftp: . [tcp sum ok] 35:35(0) ack 2545 win 17520 <nop,nop,timestamp 4 11154268> [tos 0x10] (ttl 64, id 14312)
14:41:12.891959 pD9506F36.dip.t-dialin.net.65533 > ftp.netbsd.org.ftp: P [tcp sum ok] 35:41(6) ack 2545 win 17520 <nop,nop,timestamp 4 11154268> [tos 0x10] (ttl 64, id 14313)
14:41:13.117669 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.65533: P [tcp sum ok] 2545:2569(24) ack 41 win 33580 <nop,nop,timestamp 11154269 4> [tos 0x10] (ttl 52, id 21697)
14:41:13.289832 pD9506F36.dip.t-dialin.net.65533 > ftp.netbsd.org.ftp: . [tcp sum ok] 41:41(0) ack 2569 win 17520 <nop,nop,timestamp 5 11154269> [tos 0x10] (ttl 64, id 14314)
14:41:13.509247 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.65533: P [tcp sum ok] 2569:2655(86) ack 41 win 33580 <nop,nop,timestamp 11154269 4> [tos 0x10] (ttl 52, id 21881)
14:41:13.512738 pD9506F36.dip.t-dialin.net.65533 > ftp.netbsd.org.ftp: P [tcp sum ok] 41:46(5) ack 2655 win 17520 <nop,nop,timestamp 5 11154269> [tos 0x10] (ttl 64, id 14315)
14:41:13.733896 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.65533: P [tcp sum ok] 2655:2690(35) ack 46 win 33580 <nop,nop,timestamp 11154270 5> [tos 0x10] (ttl 52, id 21985)
14:41:13.889838 pD9506F36.dip.t-dialin.net.65533 > ftp.netbsd.org.ftp: . [tcp sum ok] 46:46(0) ack 2690 win 17520 <nop,nop,timestamp 6 11154270> [tos 0x10] (ttl 64, id 14317)
14:41:15.431707 pD9506F36.dip.t-dialin.net.65533 > ftp.netbsd.org.ftp: P [tcp sum ok] 46:52(6) ack 2690 win 17520 <nop,nop,timestamp 9 11154270> [tos 0x10] (ttl 64, id 14318)
14:41:15.652941 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.65533: P [tcp sum ok] 2690:2696(6) ack 52 win 33580 <nop,nop,timestamp 11154274 9> [tos 0x10] (ttl 52, id 22621)
14:41:15.655920 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.65533: FP [tcp sum ok] 2696:2882(186) ack 52 win 33580 <nop,nop,timestamp 11154274 9> [tos 0x10] (ttl 52, id 22622)
14:41:15.656418 pD9506F36.dip.t-dialin.net.65533 > ftp.netbsd.org.ftp: . [tcp sum ok] 52:52(0) ack 2883 win 17334 <nop,nop,timestamp 10 11154274> [tos 0x10] (ttl 64, id 14319)
14:41:15.664086 pD9506F36.dip.t-dialin.net.65533 > ftp.netbsd.org.ftp: F [tcp sum ok] 52:52(0) ack 2883 win 17520 <nop,nop,timestamp 10 11154274> [tos 0x10] (ttl 64, id 14320)
14:41:15.881928 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.65533: . [tcp sum ok] 2883:2883(0) ack 53 win 33580 <nop,nop,timestamp 11154274 10> [tos 0x10] (ttl 52, id 22705)

209 packets received by filter
0 packets dropped by kernel

Second try from Client (stops after thre banner lines):

tcpdump: listening on pppoe0
14:41:48.416748 pD9506F36.dip.t-dialin.net.64484 > ftp.netbsd.org.ftp: S [tcp sum ok] 1302952455:1302952455(0) win 16384 <mss 1460> (DF) (ttl 63, id 23248)
14:41:48.634386 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.64484: S [tcp sum ok] 544866333:544866333(0) ack 1302952456 win 32768 <mss 1460> (ttl 52, id 38146)
14:41:48.635405 pD9506F36.dip.t-dialin.net.64484 > ftp.netbsd.org.ftp: . [tcp sum ok] 1:1(0) ack 1 win 17520 (DF) (ttl 63, id 23252)
14:41:48.875199 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.64484: P [tcp sum ok] 1:7(6) ack 1 win 33580 [tos 0x10] (ttl 52, id 38270)
14:41:48.920065 pD9506F36.dip.t-dialin.net.64484 > ftp.netbsd.org.ftp: . [tcp sum ok] 1:1(0) ack 7 win 17520 (DF) [tos 0x10] (ttl 63, id 23253)
14:41:49.139744 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.64484: P [tcp sum ok] 7:68(61) ack 1 win 33580 [tos 0x10] (ttl 52, id 38390)
14:41:49.152034 pD9506F36.dip.t-dialin.net.64484 > ftp.netbsd.org.ftp: P [tcp sum ok] 1:17(16) ack 68 win 17520 (DF) [tos 0x10] (ttl 63, id 23259)
14:41:49.485966 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.64484: P [tcp sum ok] 68:117(49) ack 17 win 33580 [tos 0x10] (ttl 52, id 38480)
14:41:49.489370 pD9506F36.dip.t-dialin.net.64484 > ftp.netbsd.org.ftp: P [tcp sum ok] 17:31(14) ack 117 win 17520 (DF) [tos 0x10] (ttl 63, id 23260)
14:41:49.854281 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.64484: P [tcp sum ok] 117:123(6) ack 31 win 33580 [tos 0x10] (ttl 52, id 38611)
14:41:49.871302 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.64484: . [bad tcp cksum c7bf!] 123:1575(1452) ack 31 win 33580 (frag 38612:1472@0+) [tos 0x10] (ttl 51)
14:41:49.871431 ftp.netbsd.org > pD9506F36.dip.t-dialin.net: (frag 38612:8@1472) [tos 0x10] (ttl 51)
14:41:49.920618 pD9506F36.dip.t-dialin.net.64484 > ftp.netbsd.org.ftp: . [tcp sum ok] 31:31(0) ack 123 win 17520 (DF) [tos 0x10] (ttl 63, id 23261)
14:41:50.571080 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.64503: . [bad tcp cksum c7bf!] 1380507519:1380508971(1452) ack 3992673956 win 33580 (frag 38897:1472@0+) [tos 0x10] (ttl 51)
14:41:50.571180 ftp.netbsd.org > pD9506F36.dip.t-dialin.net: (frag 38897:8@1472) [tos 0x10] (ttl 51)
14:41:51.677823 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.64484: . [bad tcp cksum c7bf!] 123:1575(1452) ack 31 win 33580 (frag 39354:1472@0+) [tos 0x10] (ttl 51)
14:41:51.677918 ftp.netbsd.org > pD9506F36.dip.t-dialin.net: (frag 39354:8@1472) [tos 0x10] (ttl 51)
14:41:53.570337 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.64484: . [bad tcp cksum c7bf!] 123:1575(1452) ack 31 win 33580 (frag 40241:1472@0+) [tos 0x10] (ttl 51)
14:41:53.570441 ftp.netbsd.org > pD9506F36.dip.t-dialin.net: (frag 40241:8@1472) [tos 0x10] (ttl 51)
14:41:53.587104 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.64495: . [bad tcp cksum c7bf!] 1826399912:1826401364(1452) ack 2190447758 win 33580 (frag 40242:1472@0+) [tos 0x10] (ttl 51)
14:41:53.587208 ftp.netbsd.org > pD9506F36.dip.t-dialin.net: (frag 40242:8@1472) [tos 0x10] (ttl 51)
14:41:56.584554 pD9506F36.dip.t-dialin.net.64484 > ftp.netbsd.org.ftp: F [tcp sum ok] 31:31(0) ack 123 win 17520 (DF) [tos 0x10] (ttl 63, id 23274)
14:41:56.802802 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.64484: . [tcp sum ok] 1583:1583(0) ack 32 win 33580 [tos 0x10] (ttl 52, id 41544)
14:41:57.570460 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.64484: . [bad tcp cksum c7bf!] 123:1575(1452) ack 32 win 33580 (frag 41894:1472@0+) [tos 0x10] (ttl 51)
14:41:57.570632 ftp.netbsd.org > pD9506F36.dip.t-dialin.net: (frag 41894:8@1472) [tos 0x10] (ttl 51)
14:42:05.570386 ftp.netbsd.org.ftp > pD9506F36.dip.t-dialin.net.64484: . [bad tcp cksum c7bf!] 123:1575(1452) ack 32 win 33580 (frag 45525:1472@0+) [tos 0x10] (ttl 51)
14:42:05.570531 ftp.netbsd.org > pD9506F36.dip.t-dialin.net: (frag 45525:8@1472) [tos 0x10] (ttl 51)

259 packets received by filter
0 packets dropped by kernel

>How-To-Repeat:

Configure the outgoing interface of a NAT router to MTU 1492 and do a
ftp -a ftp.netbsd.org from the NATed network.

>Fix:
n/a

But note that downgrading to:

sys/netinet/ip_nat.c r1.37
sys/netinet/ip_nat.h r1.21
sys/netinet/ip_ftp_pxy.c r1.17
sys/netinet/ip_rcmd_pxy.c r1.5

fixes the problem. The first is probably enough, but won't compile without
the others. Revision 1.38 of ip_nat.c is the first one to exhibit the problem.
(This somehow breaks ftp proxy, but the underlying fragmentation problem goes
away.)
>Release-Note:
>Audit-Trail:
>Unformatted: