Sorry.  Sent that message before I intended to.

Yes, as der Mouse just said, that's not enough.

The problem is that entries in the routing table are reference
counted.

When the ip_forward() function allocated the route and cached
it in ipforward_rt, it bumped the reference count up.  At this
point, nothing userland can do can cause the kernel to stop
using the route, even if it deletes the route from the routing
table.

As der Mouse suggested in PR3508, one possible solution is to place
a "generation number" on the routing table itself, and consider
the route needs to be reallocated, if the generation number changed,
or if the existing test returns true.

I'm not sure what a cache of size 1 really buys you anyway, so I
just rewrote ip_forward() and ip_rtaddr() to always allocate.  If
caching really makes such a big difference, the cache size should
probably be configurable.

ip_forward()
{
	...
	if (ipforward_rt.ro_rt) {
		RTFREE(ipforward_rt.ro_rt);
		ipforward_rt.ro_rt = 0;
	}
	sin = satosin(&ipforward_rt.ro_dst);
	sin->sin_family = AF_INET;
	sin->sin_len = sizeof(struct sockaddr_in);
	sin->sin_addr = ip->ip_dst;

	rtalloc(&ipforward_rt);
	if (ipforward_rt.ro_rt == 0) {
		icmp_error(m, ICMP_UNREACH, ICMP_UNREACH_HOST, dest, 0);
		return;
	}

	rt = ipforward_rt.ro_rt;
	...
}


In message <20021112180853.J9551@wooj.nexthop.com>, you said:
;
;
;On Tue, Nov 12, 2002 at 02:53:30PM -0800, Tad Hunt wrote:
;> In message <20021112173218.C18167@wooj.nexthop.com>, you said:
;> ;A program such as gated(8) watches for interface changes like this, and
;> ;will update e.g. static routes as necessary.  So in this case, without
;> ;alternate paths available, when ifc1 goes down, the route is deleted.
;> ;When it comes back up, the route is re-added.  At this point ip_output()
;> ;notices that the cached route is dead and fetches the new route.
;> 
;> Except it doesn't.
;
;I guess I'm not sure what you're referring to.  Do you mean that ip_output()
;doesn't notice that a cached route has been deleted?
;
;
;	/*
;         * If there is a cached route,
;         * check that it is to the same destination
;         * and is still up.  If not, free it and try again.
;         * The address family should also be checked in case of sharing the
;         * cache with IPv6.
;         */
;        if (ro->ro_rt && ((ro->ro_rt->rt_flags & RTF_UP) == 0 ||
;            dst->sin_family != AF_INET ||
;            !in_hosteq(dst->sin_addr, ip->ip_dst))) {
;                RTFREE(ro->ro_rt);
;                ro->ro_rt = (struct rtentry *)0;
;        }
;
;
;Nick