Subject: kern/37281: ip_len byte order in fr_fastroute
To: None <kern-bug-people@netbsd.org, gnats-admin@netbsd.org,>
From: None <raeburn@raeburn.org>
List: netbsd-bugs
Date: 11/05/2007 04:00:01
>Number:         37281
>Category:       kern
>Synopsis:       ip_len byte order in fr_fastroute
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Nov 05 04:00:01 +0000 2007
>Originator:     Ken Raeburn
>Release:        NetBSD 4.0_RC2
>Organization:
>Environment:
System: NetBSD raeburn.org 4.0_RC2 NetBSD 4.0_RC2 (GENERIC) #0: Mon Oct 1 20:52:07 PDT 2007 builds@wb34:/home/builds/ab/netbsd-4-0-RC2/i386/200710012132Z-obj/home/builds/ab/netbsd-4-0-RC2/src/sys/arch/i386/compile/GENERIC i386
Architecture: i386
Machine: i386
>Description:

My gateway machine at home has two ethernet links active, one to my
local network and one to my ISP.  There's also a tunnel to a server
that routes traffic for several globally-reachable addresses to me.

I'm concerned about the ISP blocking outgoing packets with source
addresses it hasn't assigned.  Using ipfilter rules like "pass out
quick on tlp2 to gif4 from <subnet> to any" to get around ISP
filtering helps with that, or did up through NetBSD 2.0.  When I
upgraded to 3.1 (and shortly thereafter to 4.0RC2), the tunnel stopped
working; much of the outgoing traffic was corrupted.  It turns out
that the corruption seems to be very specific: The length field in the
encapsulated packet is in host byte order rather than network byte
order.

Some hardware watchpoint hacks show that fr_fastroute is restoring the
host byte ordering that was in use when it was called, around line
1252 in 4.0RC2:

     	if (i) {
/* --> */    ip->ip_len = ntohs(ip->ip_len);
       	     ip->ip_off = ntohs(ip->ip_off);
    	}

The call stack is:

fr_fastroute+0x18c
fr_check+0x605
fr_check_wrapper+0x97
pfil_run_hooks+0x91
ip_input+0x5dd
ipintr+0x24

In fr_check_wrapper, the length and offset fields are switched to host
order, and switched back after fr_check returns, if it doesn't return
an error or zero out the mbuf pointer.

In fr_fastroute, we switch back to network order, call the output
routine (gif_output enqueues a packet which includes a reference to
the original packet data), and then restore the data to host byte
order (before the gif soft interrupt service routine has sent the
encapsulated packet).

(See also the "ip_len byte order in fr_fastroute" thread started
November 4 in http://mail-index.netbsd.org/tech-net/2007/11/ .)

>How-To-Repeat:

Set up a gif tunnel, use ipfilter force some traffic to use the
tunnel.  It may need to be in a machine that is already acting as a
router between multiple ethernets, and the traffic to be corrupted
might need to originate on another local machine.

Look at tcpdump on the interface where outgoing tunnel traffic would
be seen.  I saw lots of "#### bytes missing" reports, where the
difference is the difference between the host-order and net-order
representations of the length.

Also, Hans Rosenfeld says he's seeing similar problems, and referred
to an earlier report (on current-users in early 2005) of what sounds
like the same or a similar problem, in connection with an ipfilter
update and use of NAT.  (I have NAT configured as well, but use a
tunnel instead for the particular traffic where I've been noticing the
problem.)

Meanwhile, Tracy says (in a conversation over Zephyr several days ago)
that an acquaintance of hers is using gif tunnels but normal routing
tables to get traffic into them, and isn't seeing this problem.

>Fix:

??

Possible workarounds:
 - hack gif output routines to force correct byte order
 - copy packet in gif_output (might degrade performance)
 - check and see if the ISP isn't actually blocking such traffic these days :-)
   and stop sending through the tunnel if that's the case
 - use NAT for more things (no good for servers, or protocols like Zephyr that
   set up listening UDP sockets locally to receive from multiple servers)