kern/39572: ARP resolution fails when MSG_DONTROUTE flag is set

[puneet.mehta%wipro.com@localhost]

>Number:         39572
>Category:       kern
>Synopsis:       ARP resolution fails when MSG_DONTROUTE flag is set
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Sep 17 15:10:00 +0000 2008
>Originator:     Puneet
>Release:        NetBSD 1.6.1
>Organization:
Wipro Technologies
>Environment:
NetBSD  1.6.1 NetBSD 1.6.1 (GENERIC) #1: Thu Jun 14 04:52:29 PDT 2007     
root@:/usr/src/sys/arch/i386/compile/GENERIC i386

>Description:
I have the following two routes installed on kernel routing table of my NetBSD 
machine, for the same destination:

1. A directly connected route, with outgoing interface as Inf1
2. A host specific ?/32? route via an alternate path with outgoing interface as 
Inf2.

In this scenario, if a packet is sent to the destination, the packets egresses 
from Inf2.(longest prefix match)
If a packet is sent on interface Inf1 setting the MSG_DONTROUTE flag (to bypass 
the routing table), arp resolution is failing and I am getting the following 
error message logged in /var/log/messages:

>>>>>>>>>>>> /netbsd: arpresolve: can't allocate llinfo on fxp2 for 23.23.23.23


This problem exists only when we have a directly connected route and a '/32' 
route (via an alternate path) to the same destination. ARP resolution fails for 
the directly connected interface, an the packet gets dropped.
>How-To-Repeat:
+--------+ 21.21.21.22                 21.21.21.21 +----------+
|  PC1   |-----------------------------------------|   PC2    |
+--------+                                         +----------+
     \  23.23.23.24                          25.25.25.26 /
      \                                                 /
       \                                               /
        \                                             /
         \                                           /
          \                                         /
           \                                       /
            \                                     /
             \                                   /
              \                                 /
               \                               /
                \                             /
                 \                           /
                  \                         /
                   \                       /
                    \                     /
                     \                   /
                      \                 /
                       \               /
                        \             /
                         \           /
                          \         /
                           \       /
                            \     /
                 23.23.23.23 \   / 25.25.25.25
                         +----------+
                         |   PC3    |
                         +----------+ 

On PC1
======
bash-3.2# route show
Routing tables

Internet:
Destination       Gateway            Flags
10.11.0.0         link#1             U
10.11.1.59        00:10:18:82:09:66  UH
10.11.6.150       00:19:d1:5e:62:46  UH
10.11.8.204       00:10:18:82:02:f7  UH
21.21.21.0        link#2             U
21.21.21.21       link#2             UH
21.21.21.22       00:02:b3:28:52:61  UH
23.23.23.0        link#3             U
23.23.23.23       21.21.21.21        UG
localhost         127.0.0.1          UH

bash-3.2# traceroute 23.23.23.23
traceroute to 23.23.23.23 (23.23.23.23), 30 hops max, 40 byte packets
 1  21.21.21.21 (21.21.21.21)  0.243 ms  0.120 ms  0.113 ms
 2  23.23.23.23 (23.23.23.23)  0.332 ms  0.206 ms  0.233 ms


On PC2
======
bash-3.2# route show
Routing tables

Internet:
Destination       Gateway            Flags
default           100.0.2.3          UG
1.0.0.1           1.0.0.1            UH
1.0.0.1           link#5             U
10.11.0.0         link#1             U
10.11.6.150       00:19:d1:5e:62:46  UH
21.21.21.0        link#2             U
21.21.21.22       link#2             UH
23.23.23.0        21.21.21.22        UG
23.23.23.23       25.25.25.25        UG
25.25.25.0        link#4             U
25.25.25.25       link#4             UH
100.0.2.0         link#3             U
loopback          link#5             U
localhost         127.0.0.1          UH


On PC3
======
bash-3.2# route show
Routing tables

Internet:
Destination       Gateway            Flags
10.11.0.0         link#1             U
10.11.6.11        00:02:a5:84:95:62  UH
10.11.6.150       00:19:d1:5e:62:46  UH
21.21.21.0        25.25.25.26        UG
23.23.23.0        link#2             U
25.25.25.0        link#3             U
25.25.25.26       link#3             UH
loopback          link#5             U
localhost         127.0.0.1          UH


ping 23.23.23.23 is successful from PC1.
ping -r 23.23.23.23 is unsuccessful from PC1.

(ping -r is used to bypass the routing table and send the packet through 
interface 23.23.23.24).

tail -f /var/log/messages (on PC1)
==================================
arpresolve: can't allocate llinfo on fxp2 for 23.23.23.23
arplookup: unable to enter address for 
23.23.23.23@01:01:46:5c:17:17:17:18:17:17:17:17:08:00:7d:45 on null (host is 
not on local network)
arpresolve: can't allocate llinfo on fxp2 for 23.23.23.23
arplookup: unable to enter address for 
23.23.23.23@01:01:46:5a:17:17:17:18:17:17:17:17:08:00:7c:44 on null (host is 
not on local network)
arpresolve: can't allocate llinfo on fxp2 for 23.23.23.23
arplookup: unable to enter address for 
23.23.23.23@01:01:46:58:17:17:17:18:17:17:17:17:08:00:7b:43 on null (host is 
not on local network)
>Fix: