>>[This is not machine dependent.  Redirecting from port-i386 to tech-net]
>>On Mon, 22 Oct 2001, Alicia da Conceicao wrote:
>>> Sorry to bother everyone again, but I was wondering if it is possible
>>> for a single NetBSD server to have multiple defaults routes on different
>>> network interfaces, that each have their own separate gateways?
>>
>>Check the "ifdefault" work done by Paul Vixie and Ted Lemon a few years ago.
>>ftp://ftp.vix.com/pub/vixie/ifdefault/
>	kame tree includes sys/net/radix.c modified for equal cost multipath
>	routing support.  you can RTM_ADD multiple times for the same
>	destination/gateway pair, and outgoing traffic will be routed to
>	either of the route based on some hash function.
>	check www.kame.net.

	some more details.

itojun


--
7.2 Multipath routing support

KAME/NetBSD includes kernel extension for multipath routing support.  Users can
install multiple routing entries toward the same destination address prefix
(rt_key).  If combined with routing daemons support, it can provide equal
cost multipath routing, as described in the following two RFCs:
	RFC2991: Multipath Issues in Unicast and Multicast Next-Hop Selection
	RFC2992: Analysis of an Equal-Cost Multi-Path Algorithm

Multipath support will be enabled for inet and inet6 radix table,
if "options RADIX_MPATH" is specified.  Even with multipath support
enabled, there's no behavior change if you operate it normally with
single routing entry per rt_key.

If you issue multiple RTM_ADD requests with the same rt_key, they
will coexist in the routing table (4.4BSD behavior: the latter one
gets rejected with EEXIST).  First one to get added will appear
at the top of rn_dupedkey, so on normal lookups the first one 
will be picked up (fewer behavior change).

On RTM_DELETE/CHANGE/LOCK, it is now required to specify
rt_gateway, if there are multiple route entries with the same
rt_key.  If there's only one entry, no need for rt_gateway, and the original
behavior for non-multipath case is preserved.

On RTM_GET, you can identify single route out of multipaths
if you specify rt_gateway.  rt_gateway is still optional,
and the first route is returned if it is not specified.

IPv4 and IPv6 output logic will pick a route out of multiple
gateways, based on Modulo-N Hash (RFC2991) using lowermost 32bits
from ip_src ^ ip_dst (IPv6 case uses ip6_dst only due to some codepath
differences).  For example, it is likely that packets
to 3ffe::1 and 3ffe::2 (or 10.0.0.1 and 10.0.0.2) will go through a different
gateway.

DEMO:
(adding multipath routes)
# route add -inet 10.1.0.0 -netmask 0xffffff00 10.0.0.1
# route add -inet 10.1.0.0 -netmask 0xffffff00 10.0.0.2
# route add -inet 10.1.0.0 -netmask 0xffffff00 10.0.0.3
(use different gateways based on ip_dst)
# ping -n -c1 10.1.0.1
# ping -n -c1 10.1.0.2
# ping -n -c1 10.1.0.3
# ping -n -c1 10.1.0.4
(removing only one of them)
# route delete -inet 10.1.0.0 10.0.0.2
(it will raise error, as there are two routes with the same rt_key and the
command argument is considered ambiguous)
# route delete -inet 10.1.0.0
(it is not ambiguous, it works)
# route delete -inet 10.1.0.0 10.0.0.3
(now you can erase the route without rt_gateway, as there's only one)
# route delete -inet 10.1.0.0

TODO:
- L3: more clever hash/gw selection policy (-> RFC2991/2992)
- L3: avoid faulty gateways/nonworking paths
- L3: revisit issues with cached route on struct route/sendto(3)

RADIX_MPATH kernel compilation option conflicts with RADIX_ART.