That's interesting, the tunneling.. I have a very large block of IPv6 
tunnels that come out at my ISP, so I don't think I would need colo 
routers. I think that just might work.

Someone might find this interesting: So, I implemented a bit of what I 
was babbeling about earlier. I have the two interfaces rtk0 and bce0 
where rtk0 is directly connected to a DSL modem and bce0 is goes through 
the lan 192.168.0.* to a NAT box. I brought up bridge0 and added the two 
interfaces to it (it'll become apparent why). Then I brought up agr0 and 
added the two interfaces. Then I set the IP of agr0 to the external IP 
for that DSL modem and then also added a LAN addres via aliasing. I 
don't beleive adding that aliasing is possible without the bridge0, but 
I may be wrong; although, my testing does seem to indicate that it's 
necessary. Anyhow, with this setup I can switch to set either of the NAT 
box or the Internet gateway as my default. The only problem now is 
balancing network traffic over the two routes. I've done this in OpenBSD 
like I said, but I'll have to look at ipf.

So here's my big question: can ALTQ decide on which route to take? Can I 
say, send packets to this route 50% of the time and this other route 
another 50% of the time?

Hmm, another question about that other guys tunneling. Did all of his 
tunnels come out on the other end as the same IP? I see why that would 
allow for a TCP stream to be spread over the two interfaces now. That 
makes sense. And I see now why my way would still only allow for a TCP 
connection to be on one interface/route or the other but not spread out 
across. I do think that with some crafty packet magling it's still 
theoretically possible. This tunneling seems more promising though, I'll 
try that later.

For those interested (maybe nobody is, but i find it quite neat):

 % ifconfig -a
rtk0: 
flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
        address: 00:40:05:8c:75:4c
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
bce0: 
flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
        address: 00:40:05:8c:75:4c
        media: Ethernet autoselect (100baseTX)
        status: active
lo0: flags=8009<UP,LOOPBACK,MULTICAST> mtu 33192
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
agr0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        agrport: rtk0, flags=0x3<COLLECTING,DISTRIBUTING>
        agrport: bce0, flags=0x3<COLLECTING,DISTRIBUTING>
        address: 00:40:05:8c:75:4c
        inet 192.168.0.12 netmask 0xffffff00 broadcast 192.168.0.255
        inet alias 64.142.7x.x netmask 0xff000000 broadcast 64.255.255.255
        inet6 fe80::dakjldkjla5fdadad:754c%agr0 prefixlen 64 scopeid 0x9
bridge0: flags=41<UP,RUNNING> mtu 1500


 % brconfig -a
bridge0: flags=41<UP,RUNNING>
        Configuration:
                priority 32768 hellotime 2 fwddelay 15 maxage 20
                ipfilter disabled flags 0x0
        Interfaces:
                rtk0 flags=3<LEARNING,DISCOVER>
                        port 1 priority 128
                bce0 flags=3<LEARNING,DISCOVER>
                        port 2 priority 128
        Address cache (max cache: 100, timeout: 1200):
                00:40:05:43:a4:a3 bce0 1169 flags=0<>
                00:10:67:00:7a:b0 rtk0 1155 flags=0<>
                00:e0:4c:77:6c:01 bce0 1153 flags=0<>
                00:04:a8:37:20:b6 rtk0 1139 flags=0<>

% route add default 64.142.7x.1
% route show
Routing tables

Internet:
Destination       Gateway            Flags
default           64.142.7x.1        UG
      ....                  ....                     ...
PING sonic.net (209.204.190.64): 56 data bytes
64 bytes from 209.204.190.64: icmp_seq=0 ttl=250 time=12.430 ms
% route flush
    ....
% route add default 192.168.0.1
% route show
Routing tables

Internet:
Destination       Gateway            Flags
default           192.168.0.1        UG
          ....            ....      ...
% ping sonic.net
% ping sonic.net
PING sonic.net (209.204.190.64): 56 data bytes
64 bytes from 209.204.190.64: icmp_seq=0 ttl=250 time=12.430 ms

% traceroute sonic.net
traceroute to sonic.net (209.204.190.64), 64 hops max, 40 byte packets
 1  192.168.0.1 (192.168.0.1)  0.415 ms  0.183 ms  0.147 ms
 2  *****.dsl.static.sonic.net (6x.12.13x.x)  9.838 ms  9.176 ms  9.530 ms

% traceroute sonic.net
traceroute to sonic.net (209.204.190.64), 64 hops max, 40 byte packets
 1  *****sl.static.sonic.net (64.142.71.1)  14.153 ms  13.815 ms  14.423 ms
 2  1.fe-1-1-0.gw4.200p-sf.sonic.net (64.142.32.8)  15.142 ms  14.343 
ms  15.461 ms


Miles Nordin wrote:

>>>>>>"adal" == Anthony de Almeida Lopes <guerrilla_thought@gmx.de> writes:
>>>>>>            
>>>>>>
>
>  adal> Both connections have their own gateways and their IP
>  adal> addresses are on different subnets.
>
>I think that means you have to use round-robin NAT.  None of the other
>multipath stuff will work for you.
>
>The neatest ISP aggregation I've heard of is some guy that colocates a
>proprietary router like Cisco and another proprietary router at the
>site with two ISP connections, then makes two gre tunnels, one over
>each ISP, and uses equal-cost multipath routing to spread packets over
>the two gre tunnels.  That way, a single TCP flow can be spread over
>the two last-mile connections.  I don't think that level of balancing
>is possible without using tunnels and also having a colocated router
>at which to terminate the tunnels, and also I don't think there is any
>equal-cost multipath in free unix.  There is some multipath, but it is
>radix multipath based on a hash of the destination (KAME) or multipath
>that caches routes so a single destination always takes just one
>interface (Linux iproute2).
>  
>