kern/47013: carp device misbehaving with ipv6 alias (ip6_output failed: 65 / HOSTUNREACH)

To: kern-bug-people%netbsd.org@localhost,gnats-admin%netbsd.org@localhost,netbsd-bugs%netbsd.org@localhost
Subject: kern/47013: carp device misbehaving with ipv6 alias (ip6_output failed: 65 / HOSTUNREACH)
From: support@net01.malaiwah.local
Date: Thu, 27 Sep 2012 19:40:00 +0000 (UTC)
>Number:         47013
>Category:       kern
>Synopsis:       ipv4+ipv6 carp0 interface is trying to get MASTER status 
>although master is still advertising
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Sep 27 19:40:00 +0000 2012
>Originator:     Michel Belleau
>Release:        NetBSD 6.0_RC2
>Organization:
>Environment:
System: NetBSD net01.malaiwah.local 6.0_RC2 NetBSD 6.0_RC2 (SHEEVAPLUG) #3: Mon 
Sep 17 00:57:10 UTC 2012 
root@60448eae-930e-41f9-acf1-32776a7758b9.local:/zones/60448eae-930e-41f9-acf1-32776a7758b9/data/src/sys/arch/evbarm/compile/obj/SHEEVAPLUG
 evbarm
Architecture: arm
Machine: evbarm
>Description:
        I have two servers configured with a carp interface that has both IPv4 
and IPv6 addresses.
        The second server (advskew 100) is trying to take over the first one 
every couple of seconds; it looks like this is happenning since I added the 
IPv6 alias to the carp interface.
        The network traces are showing that the first server is really 
advertising every second, but somehow the second server is still trying to take 
over the interface IPs.

        sysctl value:
        # sysctl -a | grep net.inet.carp
        net.inet.carp.preempt = 0
        net.inet.carp.arpbalance = 0
        net.inet.carp.allow = 1
        net.inet.carp.log = 1

        tcpdump trace:
    14:28:01.821864 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:02.845864 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:03.869987 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:04.893894 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:05.918201 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:06.941902 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:07.966179 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:08.990252 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:10.013908 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:11.037912 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:12.061905 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:13.085945 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:14.109952 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:15.133944 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:15.953660 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:16.977089 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:18.001099 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:19.025135 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:20.049238 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:21.073148 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:22.097185 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:23.121147 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:24.145164 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:25.169164 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:26.193162 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:27.217174 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:28.241185 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:29.265196 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:30.289178 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:31.108400 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:32.132388 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:33.156677 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:34.180702 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:35.204388 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:36.228681 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:37.252399 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:38.276716 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:39.300696 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:40.324420 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:41.348657 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:42.372402 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:43.396424 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36
    14:28:44.420417 IP 192.168.15.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 
2, prio 0, authtype none, intvl 1s, length 36

>How-To-Repeat:
        On a NetBSD 6.0 RC2 machine that has carp device enabled (I was trying 
on ARM / sheevaplug, but that might be relevant to other ports as well) issue 
(with the correct carpdev and IP addresses):

        # ifconfig carp0 create
        # ifconfig carp0 vhid 3 pass mekmitasdigoat carpdev mvgbe0 192.168.15.3 
netmask 255.255.255.128
        # ifconfig carp0 inet6 2607:fa48:6e63:eb05::2 alias

        Watch dmesg (or /var/log/message) complaining about this at regular 
intervals:
        carp0: ip6_output failed: 65
        carp0: ip6_output failed: 65
        carp0: ip6_output failed: 65
        carp0: ip6_output failed: 65
        carp0: ip6_output failed: 65

        Furthermore, if you set up a second machine (I got a second sheevaplug) 
the same way but with "advskew 100", you will find the same error messages on 
its dmesg AND it will try to takeover the carp IP address once in a while 
(BACKUP status becomes MASTER for a tiny amount of time if you check "ifconfig 
carp0" continually).

        I enabled the sysctl carp log to find this out (from the second 
machine):

Sep 27 18:52:58 net02 /netbsd: carp0: INIT -> MASTER (preempting)
Sep 27 18:52:58 net02 /netbsd: carp0: state transition from: BACKUP -> to: 
MASTER
Sep 27 18:52:58 net02 /netbsd: carp0: ip6_output failed: 65
Sep 27 18:52:59 net02 /netbsd: carp0: MASTER -> BACKUP (more frequent 
advertisement received)
Sep 27 18:52:59 net02 /netbsd: carp0: state transition from: MASTER -> to: 
BACKUP
Sep 27 18:53:02 net02 /netbsd: carp0: INIT -> MASTER (preempting)
Sep 27 18:53:03 net02 /netbsd: carp0: state transition from: BACKUP -> to: 
MASTER
Sep 27 18:53:03 net02 /netbsd: carp0: ip6_output failed: 65
Sep 27 18:53:03 net02 /netbsd: carp0: MASTER -> BACKUP (more frequent 
advertisement received)
Sep 27 18:53:03 net02 /netbsd: carp0: state transition from: MASTER -> to: 
BACKUP
Sep 27 18:53:06 net02 /netbsd: carp0: INIT -> MASTER (preempting)
Sep 27 18:53:06 net02 /netbsd: carp0: state transition from: BACKUP -> to: 
MASTER
Sep 27 18:53:06 net02 /netbsd: carp0: ip6_output failed: 65
Sep 27 18:53:07 net02 /netbsd: carp0: MASTER -> BACKUP (more frequent 
advertisement received)
Sep 27 18:53:07 net02 /netbsd: carp0: state transition from: MASTER -> to: 
BACKUP
Sep 27 18:53:10 net02 /netbsd: carp0: INIT -> MASTER (preempting)
Sep 27 18:53:10 net02 /netbsd: carp0: state transition from: BACKUP -> to: 
MASTER
Sep 27 18:53:10 net02 /netbsd: carp0: ip6_output failed: 65
Sep 27 18:53:11 net02 /netbsd: carp0: MASTER -> BACKUP (more frequent 
advertisement received)
Sep 27 18:53:11 net02 /netbsd: carp0: state transition from: MASTER -> to: 
BACKUP
Sep 27 18:53:14 net02 /netbsd: carp0: INIT -> MASTER (preempting)
Sep 27 18:53:14 net02 /netbsd: carp0: state transition from: BACKUP -> to: 
MASTER
Sep 27 18:53:14 net02 /netbsd: carp0: ip6_output failed: 65
Sep 27 18:53:15 net02 /netbsd: carp0: MASTER -> BACKUP (more frequent 
advertisement received)
Sep 27 18:53:15 net02 /netbsd: carp0: state transition from: MASTER -> to: 
BACKUP
Sep 27 18:53:18 net02 /netbsd: carp0: INIT -> MASTER (preempting)
Sep 27 18:53:18 net02 /netbsd: carp0: state transition from: BACKUP -> to: 
MASTER
Sep 27 18:53:18 net02 /netbsd: carp0: ip6_output failed: 65
Sep 27 18:53:19 net02 /netbsd: carp0: MASTER -> BACKUP (more frequent 
advertisement received)
Sep 27 18:53:19 net02 /netbsd: carp0: state transition from: MASTER -> to: 
BACKUP

        It looks like every time the "ip6_output" error message comes in, the 
interface tries to preempt and become the master, shortly thereafter it comes 
back to BACKUP state (as it should have stayed from the start).

>Fix:
        No known fix yet.
Follow-Ups:
- Re: kern/47013: carp device misbehaving with ipv6 alias (ip6_output failed: 65 / HOSTUNREACH)
  - From: Manuel Bouyer
Prev by Date: NetBSD Nightly Trouble Ticket Report
Next by Date: PR/46402 CVS commit: src/sys
Previous by Thread: kern/47012: "shutdown now" from multiuser causes kernel panic (-current)
Next by Thread: Re: kern/47013: carp device misbehaving with ipv6 alias (ip6_output failed: 65 / HOSTUNREACH)
Indexes:
Home | Main Index | Thread Index | Old Index