A while back I mentioned that under 1.0, I was running into a lot of problems
with not being able to connect to certain hosts on my subnet, and when I
tried, it would fail with errors like

Jan 30 16:08:55 netbsd4me /netbsd: arptnew failed on 8095180b
Jan 30 16:08:55 netbsd4me /netbsd: arpresolve: can't allocate llinfo

I went into if_ether.c and decided to dig a little further.

It looks like some kind of problem adding LLC entries to the routing table
and ARP tables for some hosts of ours which are dual-homed gateways, and which
run "gated" in such a manner as to provide host routes to themselves for both
of their addresses onto both interfaces.  This allows us to reference the same
name for all interfaces and it doesn't matter which name to address
translation is chosen, because the route to that host will always choose the
locally attached interface (or, in case the source host is *also* dual-homed,
the metrics are setup to choose the faster FDDI/CDDI subnet interface).

Here is an example.  I just rebooted, and here are the Link entries in my
routing table:

netbsd4me:1:29 % netstat -rn | egrep UHL
128.149.24.1     link#1             UHL         1        0  le0
128.149.24.3     8:0:20:8:ac:1f     UHL         3      198  le0
128.149.24.4     link#1             UHL         2        0  le0
128.149.24.11    128.149.24.11      UHL         1        5  le0
128.149.24.30    8:0:20:9:4a:7      UHL         1        0  le0
128.149.24.45    0:0:18:0:7:24      UHL         0        0  le0
128.149.24.106   8:0:20:e:4e:a4     UHL         3      426  le0
128.149.24.108   128.149.24.108     UHL         1        0  le0
128.149.24.109   8:0:9:19:ab:26     UHL         1        0  le0
128.149.24.123   8:0:20:12:53:c5    UHL         1       14  le0
128.149.24.126   128.149.24.126     UHL         1        0  le0
128.149.24.127   128.149.24.127     UHL         1        2  le0
128.149.24.128   8:0:20:12:95:a8    UHL         1        0  le0
128.149.24.165   8:0:69:2:35:c6     UHL         1        9  le0
128.149.24.255   link#1             UHL         1        8  le0

Notice that 4 entries don't have any Ethernet address assigned to them.

Before the ARP table entries were timed out, the entries for these hosts
were bogus.  e.g.:

	mipl7.jpl.nasa.gov (128.149.24.11) at 0:0:0:f8:72:4f

This is a SPARCstation-2; it definitely *doesn't* have an Ethernet address
that is anything other than "8:0:20:xx:yy:zz".  I don't know where this
"0:0:0:f8:72:4f" came from.  In-memory junk, perhaps?

During bootup, right about the time "routed" is started, I see

Feb  1 18:47:35 netbsd4me /netbsd: arp_rtrequest: bad gateway value
Feb  1 18:47:35 netbsd4me /netbsd: arp_rtrequest: bad gateway value<7>arp_rtrequest: bad gateway value<7>arp_rtrequest: bad gateway value

The checks in /usr/src/sys/netinet/if_ether.c that produce these errors are
the same:

void
arp_rtrequest(req, rt, sa)
        int req;
        register struct rtentry *rt;
        struct sockaddr *sa;
{
        register struct sockaddr *gate = rt->rt_gateway;
        register struct llinfo_arp *la = (struct llinfo_arp *)rt->rt_llinfo;
        static struct sockaddr_dl null_sdl = {sizeof(null_sdl), AF_LINK};

...
        case RTM_RESOLVE:
                if (gate->sa_family != AF_LINK ||
                    gate->sa_len < sizeof(null_sdl)) {
                        log(LOG_DEBUG, "arp_rtrequest: bad gateway value");
                        break;
                }

and down in arplookup():

/*
 * Lookup or enter a new address in arptab.
 */
static struct llinfo_arp *
arplookup(addr, create, proxy)
        u_long addr;
        int create, proxy;
{
        register struct rtentry *rt;
        static struct sockaddr_inarp sin;

...
        rt = rtalloc1((struct sockaddr *)&sin, create);
...
        if ((rt->rt_flags & RTF_GATEWAY) || (rt->rt_flags & RTF_LLINFO) == 0 ||
            rt->rt_gateway->sa_family != AF_LINK) {
                if (create)
                        log(LOG_DEBUG, "arptnew failed on %x\n", ntohl(addr));
                return (0);

Indeed, upon adding debug statements in this block above, I found that both
syslog messages were caused by the condition

	gate->sa_family != AF_LINK 
a.k.a.
	rt->rt_gateway->sa_family != AF_LINK

being true (i.e., sa_family not being of type AF_LINK).

As might be expected, when I try to connect to any of these 4 hosts, I hit
the "rt->rt_gateway->sa_family != AF_LINK" clause and the result is that the
ARP table entry can't be added and I can't connect to them.

At this point I'm at a thin brick wall.  It's almost like as if "routed"
starts up, and it gets these 2 routes from a dual-homed host, and the lower
level code says "Waitasec, I'm looking for link-level info first, I don't want
yer routes yet!" and it fails because it doesn't have the LLC stuff first.
(Sorry; I'm not too familiar with the 4.4BSD networking code and things like
 arpresolve() et al. look drastically different from the SunOS 4.1.3 version.)

On one hand, I can't believe I'm the only person using NetBSD 1.0 on a subnet
with other gateways.  On the other hand, maybe I'm the only one on a subnet
where the other dual-homed hosts are emitting these 2 host routes for each
of their interfaces, so perhaps it is something in this kind of environment
that is confusing the networking code.

Any ideas for other things to test/instrument to try and solve this problem
would be appreciated.  As we tend to make our most important machines
dual-homed, it's a pain to have to do 3rd party logins through "working"
hosts to get to them.  (Not to mention the fact that I keep forgetting and
try to send mail to "somebody@somedualhomedhost", and it ends up sitting in
the mail queue if I forget that the destination can't be directly reached.)

Thanks,

	- Greg