carp and routing

[Stephen Borrill <netbsd%precedence.co.uk@localhost>]

I'm trying to set up redundant firewalls using carp(4) as detailed in 
section 28.5 here:
https://www.netbsd.org/docs/guide/en/chap-carp.html

The examples ignore routing, especially setting a default gateway.

Machine 1:
carp0: 	flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
	enabled=0
	carp: MASTER carpdev xennet0 vhid 1 advbase 1 advskew 0
	address: 00:00:5e:00:01:01
	inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255 
carp1: 	flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
	enabled=0
	carp: MASTER carpdev xennet1 vhid 2 advbase 1 advskew 0
	address: 00:00:5e:00:01:02
	inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63

Machine 2:
carp0:	flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
	enabled=0
	carp: BACKUP carpdev xennet0 vhid 1 advbase 1 advskew 100
	address: 00:00:5e:00:01:01
	inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
carp1:	flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
	enabled=0
	carp: BACKUP carpdev xennet1 vhid 2 advbase 1 advskew 100
	address: 00:00:5e:00:01:02
	inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63

My first attempt just set the default gateway in /etc/mygate with just 
mahcine 1 running

The routes looked OK the face of it:

Internet:
Destination        Gateway            Flags    Refs      Use    Mtu 
Interface
default            80.x.y.62          UGS         -        -      -  carp1
80.x.y.0/26        link#5             UC          -        -      -  carp1
80.x.y.62          c4:71:fe:65:53:61  UHLc        -        -      -  carp1
127/8              127.0.0.1          UGRS        -        -  33648  lo0
127.0.0.1          127.0.0.1          UH          -        -  33648  lo0
192.168.1/24       link#4             UC          -        -      -  carp0

But it didn't work:
# ping -n 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
^C
----8.8.8.8 PING Statistics----
2 packets transmitted, 0 packets received, 100.0% packet loss

Guessing at some sort of race condition, between setting up carp and the 
route, I added the "route add default" command to /etc/rc.local after a 
sleep 5. This fixes it with a single machine. The routing table in both 
cases looks identical.

I then started up the second machine and looked its routing table:
Internet:
Destination        Gateway            Flags    Refs      Use    Mtu 
Interface
default            80.x.y.62          UGS         -        -      -  carp1
80.x.y.0/26        80.x.y.20          U           -        -      -  carp1
127/8              127.0.0.1          UGRS        -        -  33648  lo0
127.0.0.1          127.0.0.1          UH          -        -  33648  lo0
192.168.1/24       192.168.1.88       U           -        -      -  carp0

If I forced machine 1 down (ifconfig carp0 down;ifconfig carp1 down), 
machine 2 shows its interfaces as MASTER, but again, no route to hosts 
even though MAC address of the router does appear in the routing table 
after a while:

Internet:
Destination        Gateway            Flags    Refs      Use    Mtu 
Interface
default            80.x.y.62          UGS         -        -      -  carp1
80.x.y.0/26        link#5             UC          -        -      -  carp1
80.x.y.62          c4:71:fe:65:53:61  UHLc        -        -      -  carp1
127/8              127.0.0.1          UGRS        -        -  33648  lo0
127.0.0.1          127.0.0.1          UH          -        -  33648  lo0
192.168.1/24       link#4             UC          -        -      -  carp0
# ping -c1 80.x.y.62
PING 80.x.y.62 (80.x.y.62): 56 data bytes
64 bytes from 80.x.y.62: icmp_seq=0 ttl=255 time=0.875988 ms

----80.x.y.62 PING Statistics----
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.875988/0.875988/0.875988/0.000000 ms
# ping -c1 8.8.8.8
PING google-public-dns-a.google.com (8.8.8.8): 56 data bytes
ping: sendto: No route to host
^C
----google-public-dns-a.google.com PING Statistics----
1 packets transmitted, 0 packets received, 100.0% packet loss

A similar problem happens at failback to the master. FreeBSD and OpenBSD 
have similar problems reported too, but with no clear answers.

--
Stephen