Re: kern/51877: carp related panic during shutdown

[Hauke Fath <hf%spg.tu-darmstadt.de@localhost>]

The following reply was made to PR kern/51877; it has been noted by GNATS.

From: Hauke Fath <hf%spg.tu-darmstadt.de@localhost>
To: Ryota Ozaki <ozaki-r%netbsd.org@localhost>
Cc: gnats-bugs%NetBSD.org@localhost, kern-bug-people%NetBSD.org@localhost, gnats-admin%netbsd.org@localhost
Subject: Re: kern/51877: carp related panic during shutdown
Date: Mon, 16 Jan 2017 13:14:38 +0100

 On 01/16/17 07:58, Ryota Ozaki wrote:
 > Can you try with DEBUG && LOCKDEBUG if not enabled?
 >
 > And can you show me states of carp0 and routes just before shutdown?
 > (ifconfig carp0 and netstat -nr -f inet)
 
 Booting a 7.99.59 pf DEBUG/LOCKDEBUG/DIAGNOSTIC kernel from today's 
 sources on the carp(4) secondary machine, dmesg has:
 
 [...]
 IPv6 mode: router
 Configuring network interfaces: wm0 ixg0 wm4wm4: link state DOWN (was 
 UNKNOWN)
   vlan2 vlan3 vlan7 vlan8 vlan9 vlan10 vlan11 vlan12 carp0ifconfig: 
 SIOCAIFADDR_IN6: Can'tcarp2: state transition from: I
   assign requested address
 carp3: state transition from: INIT -> to: BACKUP
   carp2 carp3 carp7carp7: state transition from: INIT -> to: BACKUP
   carp8carp8: state transition from: INIT -> to: BACKUP
   carp9carp9: state transition from: INIT -> to: BACKUP
   carp10carp10: state transition from: INIT -> to: BACKUP
   carp11carp11: state transition from: INIT -> to: BACKUP
   carp12carp12: state transition from: INIT -^@> to: BACKUP
   pfsync0.
 [...]
 
 - note the mangled "Can't assign requested address" message - the -7 
 kernel doesn't have that.
 
 # ifconfig  carp0
 carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
          capabilities=7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>
          capabilities=7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
          capabilities=7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
          enabled=0
          carp: MASTER carpdev wm0 vhid 1 advbase 1 advskew 192
          address: 00:00:5e:00:01:01
          inet 130.83.42.73 netmask 0xfffffff8 broadcast 130.83.42.79
 # netstat -nr -f inet
 Routing tables
 
 Internet:
 Destination        Gateway            Flags    Refs      Use    Mtu 
 Interface
 default            130.83.42.78       UGS         -        -      -L wm0
 10.0.49/24         link#14            UC          -        -      -  vlan10
 10.0.49.252        link#14            UHL         -        -      -  lo0
 127/8              127.0.0.1          UGRS        -        -  33624  lo0
 127.0.0.1          lo0                UH          -        -  33624  lo0
 130.83.18.0/26     link#15            UC          -        -      -  vlan11
 130.83.18.60       link#15            UHL         -        -      -  lo0
 130.83.18.64/26    link#16            UC          -        -      -  vlan12
 130.83.18.124      link#16            UHL         -        -      -  lo0
 130.83.18.128/26   link#13            UC          -        -      -  vlan9
 130.83.18.188      link#13            UHL         -        -      -  lo0
 130.83.18.192/26   link#12            UC          -        -      -  vlan8
 130.83.18.252      link#12            UHL         -        -      -  lo0
 130.83.42.72/29    link#3             UC          -        -      -  wm0
 130.83.42.73       130.83.42.73       UH          -        -      -  carp0
 130.83.42.75       link#3             UHL         -        -      -  lo0
 130.83.197.0/28    link#10            UC          -        -      -  vlan3
 130.83.197.0/27    link#18            UC          -        -      -  carp2
 130.83.197.11      link#10            UHL         -        -      -  lo0
 130.83.197.16/28   link#9             UC          -        -      -  vlan2
 130.83.197.28      link#9             UHL         -        -      -  lo0
 130.83.228.0/26    link#11            UC          -        -      -  vlan7
 130.83.228.60      link#11            UHL         -        -      -  lo0
 192.168.27.0/28    link#7             UC          -        -      -  wm4
 192.168.27.12      link#7             UHL         -        -      -  lo0
 # shutdown -r now
 Shutdown  NOW!
 
 [...]
 
 Done running shutdown hooks.
 Jan 16 12:55:32 Zinnenwand syslogd[433]: Exiting on signal 15
 carp0: incorrect hash from 130.83.42.74
 carp0: incorrect hash from 130.83.42.74
 carp0: incorrect hash from 130.83.42.74
 syncing disks... done
 
 [...]
 
 igphy3: detached
 wm3: detached
 igphy2: detached
 wm2: detached
 igphy1: detached
 wm1: detached
 igphy0: detached
 carp0: state transition from: MASTER -> to: INIT
 Mutex error: lockdebug_barrier: spin lock held
 
 lock address : 0xfffffe821e74f400 type     :               spin
 initialized  : 0xffffffff80426c5a
 shared holds :                  0 exclusive:                  1
 shares wanted:                  0 exclusive:                  0
 current cpu  :                  2 last held:                  2
 current lwp  : 0xfffffe810fc42000 last held: 0xfffffe810fc42000
 last locked* : 0xffffffff8044b97a unlocked : 0xffffffff8046cfc4
 owner field  : 0x0000000000010700 wait/spin:                0/1
 
 Skipping crash dump on recursive panic
 panic: LOCKDEBUG: Mutex error: lockdebug_barrier: spin lock held
 cpu2: Begin traceback...
 vpanic() at netbsd:vpanic+0x140
 snprintf() at netbsd:snprintf
 lockdebug_more() at netbsd:lockdebug_more
 rw_enter() at netbsd:rw_enter+0x5fe
 uvm_fault_internal() at netbsd:uvm_fault_internal+0x161
 trap() at netbsd:trap+0x30a
 --- trap (number 6) ---
 mutex_tryenter() at netbsd:mutex_tryenter+0x12
 lwp_trylock() at netbsd:lwp_trylock+0x17
 turnstile_block() at netbsd:turnstile_block+0x238
 mutex_enter() at netbsd:mutex_enter+0x36c
 rt_update_wait() at netbsd:rt_update_wait+0x10
 _rt_free() at netbsd:_rt_free+0x11
 rtrequest1() at netbsd:rtrequest1+0x5ef
 rtrequest() at netbsd:rtrequest+0x3e
 carp_setroute() at netbsd:carp_setroute+0xd3
 carp_setrun() at netbsd:carp_setrun+0x56
 carp_carpdev_state() at netbsd:carp_carpdev_state+0x6c
 if_down() at netbsd:if_down+0x17d
 if_detach() at netbsd:if_detach+0x1d8
 wm_detach() at netbsd:wm_detach+0xc1
 config_detach() at netbsd:config_detach+0xf8
 config_detach_all() at netbsd:config_detach_all+0x97
 cpu_reboot() at netbsd:cpu_reboot+0x176
 sys_reboot() at netbsd:sys_reboot+^@0x75
 syscall() at netbsd:syscall+0x1e8
 --- syscall (number 208) ---
 7413ea83c99a:
 cpu2: End traceback...
 rebooting...
 
 
 -- the primary carp(4) machine was still running -7 at the time. Maybe 
 the "carp0: incorrect hash from 130.83.42.74" messages (which the -7 
 kernel does not show) was due to this offset.
 
 OTOH, there is no similar message from the other carp* interfaces, but 
 then they are all on vlans; carp0 is the only carp on a physical interface.
 
 HTH,
 hauke