Re: Two NetBSD networking stack questions

[Andrew Collins <bsderandrew%gmail.com@localhost>]

On Wed, Nov 9, 2011 at 11:38 AM, David Young <dyoung%pobox.com@localhost> wrote:
> On Tue, Nov 08, 2011 at 04:22:46PM -0700, Andrew Collins wrote:
>> I'm working on an embedded NetBSD product, and I've hit two issues
>> with the networking stack that I'm curious about.
>>
>> The first question is related to ether_output's route lookups.  In
>> cases where a route has RTF_GATEWAY is set, and there is no rt_gwroute
>> or rt_gwroute isn't RTF_UP, a route lookup is done, and after
>> obtaining a usable route for the gateway the rt_ifp is checked against
>> the current ifp to make sure the route is on the same interface we're
>> trying to send from.  What I find odd is that this is *not* done for
>> cases where we find a usable rt_gwroute right off the bat, and in
>> practice, I see certain situations where a cloned route on another
>> interface is picked up for rt_gwroute, then used during the ARP
>> request for the gateway.  It ends up sort of working, because
>> arpresolve uses the current ifp to make the ARP request so it gets
>> sent out the right interface, but we end up in this weird situation
>> where we have a route on a different ifp holding information about a
>> link level address for a host off of our ifp.  Is there a reason that
>> rt_gwroute isn't checked to ensure rt_ifp == ifp even when we don't do
>> a lookup?
>
> How does one reproduce this situation where rt_gwroute->rt_ifp != ifp?
>

The way I ran into was it was with a default route which was an
interface route on an interface capable of arp, and hence was cloning.
 A packet happened to get
sent to an address that later on ends up being the gateway for a route
on a separate interface.  When this packet gets sent we create a
cloned route on the original
interface.  Shortly afterwards, the route on the 2nd interface gets
created with a gateway pointing to that address, and rt_gwroute picks
up the cloned route on other
interface.

I can repro in a similar fashion:

Destination        Gateway            Flags    Refs      Use    Mtu  Interface
default            link#10            UCS         1        0      -  vlan1
192.168.0/24       link#10            UC          0        0      -  vlan1
192.168.10/24       link#7             UC          2        0      -  vlan2

In this setup, ping out to 192.168.20.100, get:

Destination        Gateway            Flags    Refs      Use    Mtu  Interface
default            link#10            UCS         1        0      -  vlan1
192.168.0/24       link#10            UC          0        0      -  vlan1
192.168.10/24       link#7             UC          2        0      -  vlan2
192.168.20.100       link#10            UHLc        1       38      -  vlan1

Now bring up an interface at 192.168.20.1/24, and then set up a
gateway route for an address reachable through this interface:

Destination        Gateway            Flags    Refs      Use    Mtu  Interface
default            link#10            UCS         1        0      -  vlan1
192.168.0/24       link#10            UC          0        0      -  vlan1
192.168.10/24       link#7             UC          2        0      -  vlan2
192.168.20/24      link#12            UC          0        0      -  vlan3
192.168.20.100       link#10            UHLc        1       38      -  vlan1
192.168.30.1       192.168.20.100      UGHS        0        0      -  vlan3

After doing this, if you ping 192.168.30.1,  you'll see arps out on
vlan3 for 192.168.20.100, but after dropping into ddb and looking in
arptab:

rtentry=0x8669b990 flags=0x807 refcnt=0 use=4 expire=0
 key=[16,2,0,0,192,168,30,1,0,0,0,0,0,0,0,0]
 mask=[NULL] gw=[16,2,0,0,192,168,20,100,0,0,0,0,0,0,0,0]
 ifp=0xcaf8e020 (vlan3) ifa=0xc01bce00
  ifa_addr=[16,2,0,0,192,168,20,1,0,0,0,0,0,0,0,0]
  ifa_dsta=[16,2,0,0,192,168,20,255,0,0,0,0,0,0,0,0]
  ifa_mask=[7,2,0,0,255,255,255]
  flags=0x101,refcnt=5,metric=0
 gwroute=0x8669baa0 llinfo=0x0

rtentry=0x8669baa0 flags=0x2405 refcnt=1 use=1 expire=1320944433
 key=[16,2,0,0,192,168,20,100,0,0,0,0,0,0,0,0]
 mask=[NULL] gw=[19,18,10,0,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
 ifp=0xc01bb800 (vlan1) ifa=0xc01bca00
  ifa_addr=[16,2,0,0,192,168,0,1,0,0,0,0,0,0,0,0]
  ifa_dsta=[16,2,0,0,192,168,0,255,0,0,0,0,0,0,0,0]
  ifa_mask=[7,2,0,0,255,255,255]
  flags=0x101,refcnt=6,metric=0
 gwroute=0x0 llinfo=0xc00253e0
  la_rt=0x8669baa0 la_hold=0x862b5300, la_asked=0x3

The gwroute of our route points to a route on vlan1, and we have the
cloned route on vlan1 holding llinfo and la_hold for information
meant for vlan3.

This is obviously a pretty weird setup, but since we check the other
cases for rt_ifp == ifp in ether_output, and it's *possible* to get
into this state,
I don't see why we don't check this for all cases.

>> The second question is related to PPP.  I'm running into a case where
>> packets getting sent to a local address are actually sent out on the
>> wire for PPP devices.  For example:
>>
>> pppoe1: flags=8851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST> mtu 1492
>>         inet 192.168.150.1 -> 192.168.149.1 netmask 0xff000000
>>
>> If I ping 192.168.150.1, I actually see packets getting emitted on the
>> wire where the src addr == dst addr.  Is this espected behavior?
>
> What does 'route -n get 192.168.150.1' say?

# route -n get 192.168.150.1
   route to: 192.168.150.1
destination: default
       mask: default
    gateway: 192.168.149.1
 local addr: 192.168.150.1
  interface: pppoe1
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    rttvar  hopcount      mtu     expire
       0         0         0         0         0         0         0         0

# netstat -r -n
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use    Mtu  Interface
default            192.168.149.1      UGS         1        1      -  pppoe1
127.0.0.1          127.0.0.1          UH          2      458  33176  lo0
192.168.149.1      192.168.150.1      UH          1        0      -  pppoe1

I tried this is as an interface route as well and got the same result
(packets on the wire that is).  Am I just screwing up the default
route?

>
> Dave
>
> --
> David Young             OJC Technologies is now Pixo
> dyoung%pixotech.com@localhost     Urbana, IL   (217) 344-0444 x24
>