Re: carp and routing

To: Stephen Borrill <netbsd%precedence.co.uk@localhost>
Subject: Re: carp and routing
From: Ryota Ozaki <ozaki-r%netbsd.org@localhost>
Date: Fri, 19 May 2017 11:43:32 +0900
On Fri, May 19, 2017 at 12:35 AM, Stephen Borrill
<netbsd%precedence.co.uk@localhost> wrote:
> On Thu, 18 May 2017, Ryota Ozaki wrote:
>>
>> On Thu, May 18, 2017 at 8:44 PM, Stephen Borrill
>> <netbsd%precedence.co.uk@localhost> wrote:
>>>
>>> On Thu, 18 May 2017, Ryota Ozaki wrote:
>>>>
>>>>
>>>> I fixed an issue of CARP in -current, which is a regression between
>>>> -7 and -current, but I'm not sure the fix solves your problem
>>>> completely. Could you try the latest source code and report how the
>>>> fix changes the situation (or not).
>>>
>>>
>>>
>>> Prior to your changes I found that with -current, I could not send any
>>> packets at all even on the LAN (a regression from -7). With your changes
>>> I
>>> find:
>>> 1) When first booted, machine 1 (Master) can send packets on LAN and via
>>> default route (regression fixed)
>>> 2) Upon failover to machine 2, machine 2 can now send packets via default
>>> route without having to do a "route change default" (fixed problem)
>>
>>
>> Good.
>>
>>> 3) Upon failback to machine 1, machine 1 can no longer send packets
>>> (regression still present)
>
>
> I noticed that pinging after failback to machine 1 only failed if I had
> pinged on machine 2. So I reasoned that the problem was because switches,
> etc.  hadn't noticed the change. I proved this by using arping to send a
> gratuitous arp reply:
> arping -c 1 -A -I carp0 192.168.1.88
> arping -c 1 -A -I carp1 80.x.y.20
>
> If the above commands are run after the interface becomes a master, then it
> works. Are we missing out on sending a gratuitous arp after becomg master? I
> notice OpenBSD send two, the second after a small delay:
> http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/netinet/ip_carp.c.diff?r1=1.127&r2=1.128&f=h

Thank you for the investigation.

I've written a patch to fix the issue in a different way from OpenBSD.
Can you try the patch?
  http://www.netbsd.org/~ozaki-r/fix-carp-garp.diff

This is my guess how the regression was introduced:
(1) DAD for IPv4 and IN_IFF_DETACHED flag were introduced (pre -7)
    - If IN_IFF_DETACHED flag is on an IP address, any packets won't
      be sent via the IP address (including GARP packets)
    - The flag is cleared by DAD that is kicked by say an event of
      a link state change
(2) The link state change handler was changed to run in softint
    (after -7)
(3) CARP was changed to use the handler (after -7)
    - This allows CARP to kick DAD and clear IN_IFF_DETACHED flag
      *eventually*
    - OTOH, by the change, some operations are executed in reverse
    - For example, CARP tries to send a GARP packet before the handler
      is executed and fails to send it

And my patch allows CARP to execute the handler directly
(not via softint) before sending a GARP packet.

>
>> Could you show me the results of the following command on machine 1?
>> - ifconfig carp0
>> - ifconfig xennet0
>> - netstat -nr -f inet
>> - arp -na
>
>
> I'll skip these because of the above arp tests.
>
>>> I tried to apply your if_arp.c patch to -7. but the code is just too
>>> different.
>>
>>
>> The patch can be applicable to -current and is even unnecessary to -7
>> because it fixes a regression introduced recently.
>
>
> I don't think it is as simple as that. My email below (and elaborated on
> here:  http://mail-index.netbsd.org/tech-net/2017/05/15/msg006331.html )
> describes a problem with -7 here you cannot route via the default gateway
> without doing a "route change default" after becoming master. It appears
> that with the correct gratuitous arps, -current works OK.

I guess pulling up the commit ip_carp.c,v 1.88 to -7 would fix the issue.
The commit is (3) in the above list and (2) isn't in -7 so it just fixes
the issue that CARP doesn't send GARP packets, without the regression
introduced by (3) in -current.

  ozaki-r

>
>
>>>> On Wed, Mar 15, 2017 at 4:15 AM, Stephen Borrill
>>>> <netbsd%precedence.co.uk@localhost> wrote:
>>>>>
>>>>>
>>>>> I'm trying to set up redundant firewalls using carp(4) as detailed in
>>>>> section 28.5 here:
>>>>> https://www.netbsd.org/docs/guide/en/chap-carp.html
>>>>>
>>>>> The examples ignore routing, especially setting a default gateway.
>>>>>
>>>>> Machine 1:
>>>>> carp0:  flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>>>>>         capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
>>>>>         enabled=0
>>>>>         carp: MASTER carpdev xennet0 vhid 1 advbase 1 advskew 0
>>>>>         address: 00:00:5e:00:01:01
>>>>>         inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
>>>>> carp1:
>>>>> flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>>>>>         capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
>>>>>         enabled=0
>>>>>         carp: MASTER carpdev xennet1 vhid 2 advbase 1 advskew 0
>>>>>         address: 00:00:5e:00:01:02
>>>>>         inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
>>>>>
>>>>> Machine 2:
>>>>> carp0:  flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>>>>>         capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
>>>>>         enabled=0
>>>>>         carp: BACKUP carpdev xennet0 vhid 1 advbase 1 advskew 100
>>>>>         address: 00:00:5e:00:01:01
>>>>>         inet 192.168.1.88 netmask 0xffffff00 broadcast 192.168.1.255
>>>>> carp1:  flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>>>>>         capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
>>>>>         enabled=0
>>>>>         carp: BACKUP carpdev xennet1 vhid 2 advbase 1 advskew 100
>>>>>         address: 00:00:5e:00:01:02
>>>>>         inet 80.x.y.20 netmask 0xffffffc0 broadcast 80.71.28.63
>>>>>
>>>>> My first attempt just set the default gateway in /etc/mygate with just
>>>>> mahcine 1 running
>>>>>
>>>>> The routes looked OK the face of it:
>>>>>
>>>>> Internet:
>>>>> Destination        Gateway            Flags    Refs      Use    Mtu
>>>>> Interface
>>>>> default            80.x.y.62          UGS         -        -      -
>>>>> carp1
>>>>> 80.x.y.0/26        link#5             UC          -        -      -
>>>>> carp1
>>>>> 80.x.y.62          c4:71:fe:65:53:61  UHLc        -        -      -
>>>>> carp1
>>>>> 127/8              127.0.0.1          UGRS        -        -  33648
>>>>> lo0
>>>>> 127.0.0.1          127.0.0.1          UH          -        -  33648
>>>>> lo0
>>>>> 192.168.1/24       link#4             UC          -        -      -
>>>>> carp0
>>>>>
>>>>> But it didn't work:
>>>>> # ping -n 8.8.8.8
>>>>> PING 8.8.8.8 (8.8.8.8): 56 data bytes
>>>>> ping: sendto: No route to host
>>>>> ping: sendto: No route to host
>>>>> ^C
>>>>> ----8.8.8.8 PING Statistics----
>>>>> 2 packets transmitted, 0 packets received, 100.0% packet loss
>>>>>
>>>>> Guessing at some sort of race condition, between setting up carp and
>>>>> the
>>>>> route, I added the "route add default" command to /etc/rc.local after a
>>>>> sleep 5. This fixes it with a single machine. The routing table in both
>>>>> cases looks identical.
>>>>>
>>>>> I then started up the second machine and looked its routing table:
>>>>> Internet:
>>>>> Destination        Gateway            Flags    Refs      Use    Mtu
>>>>> Interface
>>>>> default            80.x.y.62          UGS         -        -      -
>>>>> carp1
>>>>> 80.x.y.0/26        80.x.y.20          U           -        -      -
>>>>> carp1
>>>>> 127/8              127.0.0.1          UGRS        -        -  33648
>>>>> lo0
>>>>> 127.0.0.1          127.0.0.1          UH          -        -  33648
>>>>> lo0
>>>>> 192.168.1/24       192.168.1.88       U           -        -      -
>>>>> carp0
>>>>>
>>>>> If I forced machine 1 down (ifconfig carp0 down;ifconfig carp1 down),
>>>>> machine 2 shows its interfaces as MASTER, but again, no route to hosts
>>>>> even
>>>>> though MAC address of the router does appear in the routing table after
>>>>> a
>>>>> while:
>>>>>
>>>>> Internet:
>>>>> Destination        Gateway            Flags    Refs      Use    Mtu
>>>>> Interface
>>>>> default            80.x.y.62          UGS         -        -      -
>>>>> carp1
>>>>> 80.x.y.0/26        link#5             UC          -        -      -
>>>>> carp1
>>>>> 80.x.y.62          c4:71:fe:65:53:61  UHLc        -        -      -
>>>>> carp1
>>>>> 127/8              127.0.0.1          UGRS        -        -  33648
>>>>> lo0
>>>>> 127.0.0.1          127.0.0.1          UH          -        -  33648
>>>>> lo0
>>>>> 192.168.1/24       link#4             UC          -        -      -
>>>>> carp0
>>>>> # ping -c1 80.x.y.62
>>>>> PING 80.x.y.62 (80.x.y.62): 56 data bytes
>>>>> 64 bytes from 80.x.y.62: icmp_seq=0 ttl=255 time=0.875988 ms
>>>>>
>>>>> ----80.x.y.62 PING Statistics----
>>>>> 1 packets transmitted, 1 packets received, 0.0% packet loss
>>>>> round-trip min/avg/max/stddev = 0.875988/0.875988/0.875988/0.000000 ms
>>>>> # ping -c1 8.8.8.8
>>>>> PING google-public-dns-a.google.com (8.8.8.8): 56 data bytes
>>>>> ping: sendto: No route to host
>>>>> ^C
>>>>> ----google-public-dns-a.google.com PING Statistics----
>>>>> 1 packets transmitted, 0 packets received, 100.0% packet loss
>>>>>
>>>>> A similar problem happens at failback to the master. FreeBSD and
>>>>> OpenBSD
>>>>> have similar problems reported too, but with no clear answers.
>>>>>
>>>>> --
>>>>> Stephen
>>>>>
>>>>
>>>
>>
>
Follow-Ups:
- Re: carp and routing
  - From: Stephen Borrill
References:
- carp and routing
  - From: Stephen Borrill
- Re: carp and routing
  - From: Ryota Ozaki
- Re: carp and routing
  - From: Stephen Borrill
- Re: carp and routing
  - From: Ryota Ozaki
- Re: carp and routing
  - From: Stephen Borrill
Prev by Date: Re: carp and routing
Next by Date: Re: carp and routing
Previous by Thread: Re: carp and routing
Next by Thread: Re: carp and routing
Indexes:
Home | Main Index | Thread Index | Old Index