Re: "ro->_ro_rt ==NULL || ro->_ro_rt->rt_refcnt > 0" failed

[Ryota Ozaki <ozaki-r%netbsd.org@localhost>]

Hi Paul and Jun,

Thank you for your reporting!

Now I can reproduce the issue quickly using openvpn.
So I would provide a fix soon (hopefully).

  ozaki-r

On Tue, Sep 1, 2015 at 8:14 PM, Paul Goyette <paul%vps1.whooppee.com@localhost> wrote:
> On Mon, 31 Aug 2015, Ryota Ozaki wrote:
>
>> Hi,
>>
>> I've committed a fix for rt_refcnt. Could you try again
>> with -current? (though I'm not sure the fix is related to
>> the issue...)
>
>
> The fix is not helping on my situation, either!
>
> Here is the latest info, based on a kernel from today's sources...
>
> (gdb) target kvm netbsd.0.core
> 0xffffffff8069eda5 in cpu_reboot (howto=howto@entry=260,
>     bootstr=bootstr@entry=0x0)
>     at /build/netbsd-local/src/sys/arch/amd64/amd64/machdep.c:671
> 671                     dumpsys();
> #0  0xffffffff8069eda5 in cpu_reboot (howto=howto@entry=260,
>     bootstr=bootstr@entry=0x0)
>     at /build/netbsd-local/src/sys/arch/amd64/amd64/machdep.c:671
> #1  0xffffffff808e4cc4 in vpanic (
>     fmt=0xffffffff80d2d8b8 "kernel %sassertion \"%s\" failed: file \"%s\",
> line %d ", ap=ap@entry=0xfffffe810f5a48a8)
>     at /build/netbsd-local/src/sys/kern/subr_prf.c:342
> #2  0xffffffff80a7d763 in kern_assert (
>     fmt=fmt@entry=0xffffffff80d2d8b8 "kernel %sassertion \"%s\" failed: file
> \"%s\", line %d ") at
> /build/netbsd-local/src/sys/lib/libkern/kern_assert.c:51
> #3  0xffffffff80a8421a in rtcache_invariants (ro=0xfffffe821c830060)
>     at /build/netbsd-local/src/sys/net/route.h:441
> #4  0xffffffff8082b2a5 in rtcache_invariants (ro=0xfffffe821c830060)
>     at /build/netbsd-local/src/sys/net/route.h:441
> #5  rtcache_getdst (ro=0xfffffe821c830060)
>     at /build/netbsd-local/src/sys/net/route.h:467
> #6  rtcache_lookup2 (ro=0xfffffe821c830060,
> dst=dst@entry=0xfffffe810f5a495c,
>     clone=clone@entry=1, hitp=hitp@entry=0xfffffe810f5a4958)
>     at /build/netbsd-local/src/sys/net/route.c:1493
> #7  0xffffffff805070f7 in rtcache_lookup1 (clone=1, dst=0xfffffe810f5a495c,
>     ro=<optimized out>) at /build/netbsd-local/src/sys/net/route.h:449
> #8  selectroute (dstsock=dstsock@entry=0xfffffe810f5a4bc4,
>     opts=opts@entry=0xfffffe81f7af67d0, mopts=<optimized out>,
>     ro=ro@entry=0x0, retifp=retifp@entry=0xfffffe810f5a4a10,
>     retrt=retrt@entry=0xfffffe810f5a4a18, clone=1,
> norouteok=norouteok@entry=1)
>     at /build/netbsd-local/src/sys/netinet6/in6_src.c:665
> #9  0xffffffff8050723a in in6_selectif (retifp=0xfffffe810f5a4a10, ro=0x0,
>     mopts=<optimized out>, opts=0xfffffe81f7af67d0,
> dstsock=0xfffffe810f5a4bc4)
>     at /build/netbsd-local/src/sys/netinet6/in6_src.c:724
> #10 in6_selectsrc (dstsock=dstsock@entry=0xfffffe810f5a4bc4,
>     opts=opts@entry=0xfffffe81f7af67d0, mopts=<optimized out>,
>     ro=ro@entry=0xfffffe821c830060, laddr=laddr@entry=0xfffffe821c8300a0,
>     ifpp=ifpp@entry=0xfffffe810f5a4ae0,
> errorp=errorp@entry=0xfffffe810f5a4adc)
>     at /build/netbsd-local/src/sys/netinet6/in6_src.c:204
> #11 0xffffffff80800869 in rip6_output (m=m@entry=0xfffffe821c249400,
>     so=so@entry=0xfffffe8219325db0,
> dstsock=dstsock@entry=0xfffffe810f5a4bc4,
>     control=control@entry=0x0)
>     at /build/netbsd-local/src/sys/netinet6/raw_ip6.c:447
> #12 0xffffffff80800e08 in rip6_send (l=<optimized out>, control=0x0,
>     nam=0xfffffe81effcd638, m=0xfffffe821c249400, so=0xfffffe8219325db0)
>     at /build/netbsd-local/src/sys/netinet6/raw_ip6.c:893
> #13 rip6_send_wrapper (a=0xfffffe8219325db0, b=0xfffffe821c249400,
>     c=0xfffffe81effcd638, d=0x0, e=<optimized out>)
>     at /build/netbsd-local/src/sys/netinet6/raw_ip6.c:966
> #14 0xffffffff80999251 in sosend (so=0xfffffe8219325db0,
>     addr=0xfffffe81effcd638, uio=0xfffffe810f5a4d10, top=0xfffffe821c249400,
>     control=0x0, flags=<optimized out>, l=0xfffffe81f455c4c0)
>     at /build/netbsd-local/src/sys/kern/uipc_socket.c:1064
> #15 0xffffffff809a0785 in do_sys_sendmsg_so (l=l@entry=0xfffffe81f455c4c0,
>     s=s@entry=4, so=<optimized out>, fp=0xfffffe81f1ac6380,
>     mp=mp@entry=0xfffffe810f5a4e58, flags=flags@entry=0,
>     retsize=retsize@entry=0xfffffe810f5a4eb8)
>     at /build/netbsd-local/src/sys/kern/uipc_syscalls.c:622
> #16 0xffffffff809a0ad2 in do_sys_sendmsg (l=l@entry=0xfffffe81f455c4c0, s=4,
>     mp=mp@entry=0xfffffe810f5a4e58, flags=0,
>     retsize=retsize@entry=0xfffffe810f5a4eb8)
>     at /build/netbsd-local/src/sys/kern/uipc_syscalls.c:672
> #17 0xffffffff809a0b9b in sys_sendmsg (l=0xfffffe81f455c4c0,
>     uap=0xfffffe810f5a4f00, retval=0xfffffe810f5a4eb8)
>     at /build/netbsd-local/src/sys/kern/uipc_syscalls.c:528
> #18 0xffffffff80901f6c in sy_call (rval=0xfffffe810f5a4eb8,
>     uap=0xfffffe810f5a4f00, l=0xfffffe81f455c4c0,
>     sy=0xffffffff810ef240 <sysent+672>)
>     at /build/netbsd-local/src/sys/sys/syscallvar.h:65
> #19 sy_invoke (code=28, rval=0xfffffe810f5a4eb8, uap=0xfffffe810f5a4f00,
>     l=0xfffffe81f455c4c0, sy=0xffffffff810ef240 <sysent+672>)
>     at /build/netbsd-local/src/sys/sys/syscallvar.h:94
> #20 syscall (frame=0xfffffe810f5a4f00)
>     at /build/netbsd-local/src/sys/arch/x86/x86/syscall.c:156
> #21 0xffffffff80100691 in Xsyscall ()
> #3  0xffffffff80a8421a in rtcache_invariants (ro=0xfffffe821c830060)
>     at /build/netbsd-local/src/sys/net/route.h:441
> 441             KASSERT(ro->_ro_rt == NULL || ro->_ro_rt->rt_refcnt > 0);
> (gdb) print ro->_ro_rt
> $1 = (struct rtentry *) 0xfffffe81fc5c39a0
> (gdb) print ro->_ro_rt->rt_refcnt
> $2 = 0
> (gdb)
>
>
>
> To reproduce this, I simply bring up my openvpn tunnel, which has IPv6
> addresses on both ends.  I can successfully ping6 the remote tunnel
> end-point (ie, the "directly attached neighbor").  However, as soon as I try
> to ping6 to something more than 1 hop away (I've been using ftp.netbet.org
> for testing!) it crashes as above.
>
> This is on a GENERIC amd64 kernel, built from sources updated via anoncvs on
> 2015-09-01 at 06:53:13 UTC.
>
>
>
> -------------------------------------------------------------------------
> | Paul Goyette     | PGP Key fingerprint:     | E-mail addresses:       |
> | (Retired)        | FA29 0E3B 35AF E8AE 6651 | paul at whooppee.com    |
> | Kernel Developer | 0786 F758 55DE 53BA 7731 | pgoyette at netbsd.org  |
> -------------------------------------------------------------------------