Never mind, I believe I figured it out.

The "toggle the last bit" hack assumes that the next hop for the two
addresses is the same.  In my case they are directly connected together,
so the next hop has a different MAC address.  

I should be able to route packets directly with a static route, but I
think NAT keeps getting in the way.  Does anyone know how I can tell NAT
NOT to translate packets if they are going to the 192.168.78 sub-net?
Since they are going out on my external interface, and their source
address is on the 192.168.149 subnet, I'm assuming their source address is
getting rewritten.

Thanks,
	Rick

On Wed, 19 May 1999, Rick Byers wrote:

> Date: Wed, 19 May 1999 01:10:00 -0400 (EDT)
> From: Rick Byers <rickb@iaw.on.ca>
> To: current-users@netbsd.org
> Cc: pmhood@uwaterloo.ca
> Subject: IP over IP broken?
> 
> Hi, I'm trying to setup a IP over IP tunnel using the ipip device.  I'm
> having problem that I believe to be related to the "toggling of the last
> bit of the IP address" as mentioned in the BUGS section of ipip(4).
> 
> I'm trying to establish a tunnel between two local networks connected to
> the internet behind a NetBSD-1.4 box doing NAT. 
> 
> Here is a picture of my network:
> 
> 192.168.149.16   My PC (NetBSD-current)
>      |
> 192.168.149.1    Gateway machine, NetBSD-1.4 running NAT
> 24.112.22.6     
>      |
>  Rogers@Home network
>      |
> 24.112.22.29     Gateway machine, NetBSD-1.4 running NAT
> 192.168.78.5
>      |
> 192.168.78.1     Destination machine.
> 
> I would like to be able to establish connections FROM 192.168.149.16 to
> 192.168.78.1.  So, on my gateway machine (192.168.149.1) I did:
> 
> ifconfig ipip0 24.112.22.6 24.112.22.29 netmask 0xffffffff up
> route add -net 192.168.78 24.112.22.29
> 
> and the mirror image on the other gateway machine.
> ifconfig -a and netstat -nr output look good.  However, if I try to ping
> from 192.168.149.1 to 192.168.78.5 my packets go off into never-never
> land.  It appears as if the gateway machine is actually trying to send the
> packets to 24.112.22.28 (intead of .29). 
> 
> wormhole:/usr/home/rick# tcpdump -s 128 -e -vv -n proto 4
> tcpdump: listening on le0
> 21:34:44.173209 2:7:1:16:7:2f 0:e0:29:16:68:1d 0800 118: 24.112.22.6 >
> 24.112.22.29: 192.168.149.16 > 192.168.78.5: icmp: echo request (ttl 254,
> id 11407) (ttl 254, id 30335)
> 
> This looked good to me, until I checked the MAC addresses:
> 
> arp -n -a
> ? (24.112.22.1) at 00:e0:14:bd:b7:08         
> ? (24.112.22.6) at 02:07:01:16:07:2f permanent
> ? (24.112.22.28) at 00:e0:29:16:68:1d
> ? (24.112.22.29) at 00:4f:4c:04:99:22
> ...
> 
> The encapsulated packet is supposed to be going to 24.112.22.29 (and that
> is what's in the IP dst), but the link level header is directing it to the
> ethernet address of 24.112.22.28!! (no, that's not my gateway, 24.112.22.1
> is).
> 
> I expect this is a problem in ipip_compute_route()
> (sys/netinet/ip_ipip.c).  Specifically:
> 
>         /*
>          * Toggle the last bit on the address so our interface won't
>          * match.
>          *
>          * XXX What we want here is a flag to exclude picking routes
>          * XXX with this interface.
>          */
>         ((struct sockaddr_in *)&ro->ro_dst)->sin_addr.s_addr =
>             htonl(ntohl(sc->sc_dst.s_addr) ^ 1);
> 
>         rtalloc(ro);
> 
>         /*
>          * Now change it back, or else ip_ouput() will just drop
>          * the route and find the one for the ipip interface.
>          */
>         ((struct sockaddr_in *)&ro->ro_dst)->sin_addr = sc->sc_dst;
> 
> So I expect either the toggling actually changes sc->sc_dst (so that it
> never gets undone), or the rtalloc() command leaves something around.  I
> looked at it for a while, but it's getting late and I wasn't able to make
> any progress.
> 
> I have the exact same problem if I use the gre driver instead of the ipip
> driver.  Which one should I be using anyway?
> 
> Any ideas on how to fix this problem (in NetBSD-1.4)?
> 
> Thanks,
> 	Rick
> 
> =========================================================================
> Rick Byers                       University of Waterloo, Computer Science
> rickb@iaw.on.ca                               http://www.iaw.on.ca/rickb/
> 
> 
>