tech-net archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: carp and routing

On Tue, 14 Mar 2017, Stephen Borrill wrote:
I'm trying to set up redundant firewalls using carp(4) as detailed in section 28.5 here: 
https://www.netbsd.org/docs/guide/en/chap-carp.html

The examples ignore routing, especially setting a default gateway.

Machine 1:
carp0: 	flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
	enabled=0
	carp: MASTER carpdev xennet0 vhid 1 advbase 1 advskew 0
	address: 00:00:5e:00:01:01
	inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
carp1:     flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
	enabled=0
	carp: MASTER carpdev xennet1 vhid 2 advbase 1 advskew 0
	address: 00:00:5e:00:01:02
	inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63

Machine 2:
carp0:	flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
	enabled=0
	carp: BACKUP carpdev xennet0 vhid 1 advbase 1 advskew 100
	address: 00:00:5e:00:01:01
	inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
carp1:	flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
	enabled=0
	carp: BACKUP carpdev xennet1 vhid 2 advbase 1 advskew 100
	address: 00:00:5e:00:01:02
	inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
My first attempt just set the default gateway in /etc/mygate with just mahcine 1 running 

The routes looked OK the face of it:

Internet:
Destination        Gateway            Flags    Refs      Use    Mtu Interface
default            80.x.y.62          UGS         -        -      -  carp1
80.x.y.0/26        link#5             UC          -        -      -  carp1
80.x.y.62          c4:71:fe:65:53:61  UHLc        -        -      -  carp1
127/8              127.0.0.1          UGRS        -        -  33648  lo0
127.0.0.1          127.0.0.1          UH          -        -  33648  lo0
192.168.1/24       link#4             UC          -        -      -  carp0

But it didn't work:
# ping -n 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
^C
----8.8.8.8 PING Statistics----
2 packets transmitted, 0 packets received, 100.0% packet loss
Guessing at some sort of race condition, between setting up carp and the route, I added the "route add default" command to /etc/rc.local after a sleep 5. This fixes it with a single machine. The routing table in both cases looks identical. 

I then started up the second machine and looked its routing table:
Internet:
Destination        Gateway            Flags    Refs      Use    Mtu Interface
default            80.x.y.62          UGS         -        -      -  carp1
80.x.y.0/26        80.x.y.20          U           -        -      -  carp1
127/8              127.0.0.1          UGRS        -        -  33648  lo0
127.0.0.1          127.0.0.1          UH          -        -  33648  lo0
192.168.1/24       192.168.1.88       U           -        -      -  carp0
If I forced machine 1 down (ifconfig carp0 down;ifconfig carp1 down), machine 2 shows its interfaces as MASTER, but again, no route to hosts even though MAC address of the router does appear in the routing table after a while: 

Internet:
Destination        Gateway            Flags    Refs      Use    Mtu Interface
default            80.x.y.62          UGS         -        -      -  carp1
80.x.y.0/26        link#5             UC          -        -      -  carp1
80.x.y.62          c4:71:fe:65:53:61  UHLc        -        -      -  carp1
127/8              127.0.0.1          UGRS        -        -  33648  lo0
127.0.0.1          127.0.0.1          UH          -        -  33648  lo0
192.168.1/24       link#4             UC          -        -      -  carp0
# ping -c1 80.x.y.62
PING 80.x.y.62 (80.x.y.62): 56 data bytes
64 bytes from 80.x.y.62: icmp_seq=0 ttl=255 time=0.875988 ms

----80.x.y.62 PING Statistics----
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.875988/0.875988/0.875988/0.000000 ms
# ping -c1 8.8.8.8
PING google-public-dns-a.google.com (8.8.8.8): 56 data bytes
ping: sendto: No route to host
^C
----google-public-dns-a.google.com PING Statistics----
1 packets transmitted, 0 packets received, 100.0% packet loss
I have a workaround which is to watch out for the carrier status using ifwatchd and then refresh the default route. So: 

/etc/kickroute contains:
#!/bin/sh
route change default `cat /etc/mygate`

rc.conf contains:
ifwatchd=YES
ifwatchd_flags="-c /etc/kickroute carp1"

rc.local contains:
sleep 5
/etc/kickroute
A similar problem happens at failback to the master. FreeBSD and OpenBSD have similar problems reported too, but with no clear answers.
I tried FreeBSD 11 (which is rather different as carp is no longer a pseudo-device, carp is configured on the hardware interface directly). It works fine without any of the above hacks. 

--
Stephen
Home | Main Index | Thread Index | Old Index