Subject: fragmentation by NetBSD routers vs. reassembly on other systems....
To: NetBSD Networking Technical Discussion List <tech-net@NetBSD.ORG>
From: Greg A. Woods <woods@weird.com>
List: tech-net
Date: 09/01/2000 23:07:09
So I've had this GIF tunnel I've been running to transit my network over
a cable modem link.

The problem started when I noticed that net.inet.ip.mtudisc is now on by
default and since, I though to myself, "This is really stupid!" and I
turned it off on at least the system where I generally send e-mail from
these days.

The result was that my little NetBSD/i386-1.4V running the
GIF-over-cable-modem tunnel was then forced to fragment my packets.  I
knew this would happen so I set "net.inet.tcp.mssdflt = 1200" in an
effort to convince my system so send properly sized packets..

Not too many days later I noticed that I was not able to send any e-mail
to a select few sites if it was over ~1KB (i.e. if it caused a
fragmented packet).  I contacted a few of the admins of the sites in
question, and was contacted by at least one who had noticed my system's
failing SMTP connects.  Various experiments proved that it was not
Path-MTU-discovery or firewalls on their end causing any problems, and
indeed tcpdumps on my router showed that my system was simply getting
stuck retranmitting the larger packets without getting any kind of
response from anyone at all.

Unfortunately at the time I was not able to do tcpdumps on the upstream
router to see if the fragments looked OK or not, and none of the admins
I contacted were able to capture meaningful packet traces for me either.
It didn't help that I was suffering enormous latency and packet loss
problems on my cable modem for the first week of this issue either!

I first experimented with setting "net.inet.tcp.mssdflt" down lower and
lower and then really low at 128 (remember I had set it to 1200 so that
I though I'd be sending the maximum size packets for the tunnel).
However for these sites no amount of adjusting my MSS default seemed to
have any effect (while for other tests I've done adjusting mssdflt
improves the troughput of high-bandwidth routed connections, even on the
fly!).

Eventually when I was able to have tcpdumps done on the upstream router,
and confirm that fragments were indeed still being generated, I finally
gave in and turned mtudisc back on (and also mss_ifmtu for good measure).

Voila!  Immediately all the e-mail still stuck in my queue was off on
its merry way!

One person I was having trouble connecting to was using a linuxrouter
and said that /proc/sys/net/ipv4/ip_always_defrag was on (by default, I
think!) but turning it off didn't seem to help any.

What's also weird is that I can send large ICMP echo requests to at
least some of the sites I've had similar troubles with and all the
fragmentation and reassembly works fine for those packets and their
corresponding replies.  So, the problem seems to be restricted to TCP.
(Routers shouldn't be doing fragment reassembly, should they?)

What's stunning though were the wide variety of sites I had trouble
sending to, and the even wider variety of sites I had no trouble sending
to.  Is it possible that there are a significant number of broken
fragment reassembly implementations out there?  Or is it more likely
that there's some subtle bug in GIF tunnels and/or NetBSD's ability to
do TCP fragmentation?

In any case I'm certainly more confused now than I was a while ago!

I'm willing to do more testing, and I will be upgrading my kernels on
both the server and router to -current again soon I hope.....  If anyone
has any further suggestions or clues, please post them!

-- 
							Greg A. Woods

+1 416 218-0098      VE3TCP      <gwoods@acm.org>      <robohack!woods>
Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>