Re: "ro->_ro_rt ==NULL || ro->_ro_rt->rt_refcnt > 0" failed

[Ryota Ozaki <ozaki-r%netbsd.org@localhost>]

On Wed, Sep 2, 2015 at 3:16 PM, Ryota Ozaki <ozaki-r%netbsd.org@localhost> wrote:
> Hi Paul and Jun,
>
> Thank you for your reporting!
>
> Now I can reproduce the issue quickly using openvpn.
> So I would provide a fix soon (hopefully).

Oops. The tested kernel was built at 8/24. A kernel built today
doesn't reproduce the issue...

  ozaki-r

>
>   ozaki-r
>
> On Tue, Sep 1, 2015 at 8:14 PM, Paul Goyette <paul%vps1.whooppee.com@localhost> wrote:
>> On Mon, 31 Aug 2015, Ryota Ozaki wrote:
>>
>>> Hi,
>>>
>>> I've committed a fix for rt_refcnt. Could you try again
>>> with -current? (though I'm not sure the fix is related to
>>> the issue...)
>>
>>
>> The fix is not helping on my situation, either!
>>
>> Here is the latest info, based on a kernel from today's sources...
>>
>> (gdb) target kvm netbsd.0.core
>> 0xffffffff8069eda5 in cpu_reboot (howto=howto@entry=260,
>>     bootstr=bootstr@entry=0x0)
>>     at /build/netbsd-local/src/sys/arch/amd64/amd64/machdep.c:671
>> 671                     dumpsys();
>> #0  0xffffffff8069eda5 in cpu_reboot (howto=howto@entry=260,
>>     bootstr=bootstr@entry=0x0)
>>     at /build/netbsd-local/src/sys/arch/amd64/amd64/machdep.c:671
>> #1  0xffffffff808e4cc4 in vpanic (
>>     fmt=0xffffffff80d2d8b8 "kernel %sassertion \"%s\" failed: file \"%s\",
>> line %d ", ap=ap@entry=0xfffffe810f5a48a8)
>>     at /build/netbsd-local/src/sys/kern/subr_prf.c:342
>> #2  0xffffffff80a7d763 in kern_assert (
>>     fmt=fmt@entry=0xffffffff80d2d8b8 "kernel %sassertion \"%s\" failed: file
>> \"%s\", line %d ") at
>> /build/netbsd-local/src/sys/lib/libkern/kern_assert.c:51
>> #3  0xffffffff80a8421a in rtcache_invariants (ro=0xfffffe821c830060)
>>     at /build/netbsd-local/src/sys/net/route.h:441
>> #4  0xffffffff8082b2a5 in rtcache_invariants (ro=0xfffffe821c830060)
>>     at /build/netbsd-local/src/sys/net/route.h:441
>> #5  rtcache_getdst (ro=0xfffffe821c830060)
>>     at /build/netbsd-local/src/sys/net/route.h:467
>> #6  rtcache_lookup2 (ro=0xfffffe821c830060,
>> dst=dst@entry=0xfffffe810f5a495c,
>>     clone=clone@entry=1, hitp=hitp@entry=0xfffffe810f5a4958)
>>     at /build/netbsd-local/src/sys/net/route.c:1493
>> #7  0xffffffff805070f7 in rtcache_lookup1 (clone=1, dst=0xfffffe810f5a495c,
>>     ro=<optimized out>) at /build/netbsd-local/src/sys/net/route.h:449
>> #8  selectroute (dstsock=dstsock@entry=0xfffffe810f5a4bc4,
>>     opts=opts@entry=0xfffffe81f7af67d0, mopts=<optimized out>,
>>     ro=ro@entry=0x0, retifp=retifp@entry=0xfffffe810f5a4a10,
>>     retrt=retrt@entry=0xfffffe810f5a4a18, clone=1,
>> norouteok=norouteok@entry=1)
>>     at /build/netbsd-local/src/sys/netinet6/in6_src.c:665
>> #9  0xffffffff8050723a in in6_selectif (retifp=0xfffffe810f5a4a10, ro=0x0,
>>     mopts=<optimized out>, opts=0xfffffe81f7af67d0,
>> dstsock=0xfffffe810f5a4bc4)
>>     at /build/netbsd-local/src/sys/netinet6/in6_src.c:724
>> #10 in6_selectsrc (dstsock=dstsock@entry=0xfffffe810f5a4bc4,
>>     opts=opts@entry=0xfffffe81f7af67d0, mopts=<optimized out>,
>>     ro=ro@entry=0xfffffe821c830060, laddr=laddr@entry=0xfffffe821c8300a0,
>>     ifpp=ifpp@entry=0xfffffe810f5a4ae0,
>> errorp=errorp@entry=0xfffffe810f5a4adc)
>>     at /build/netbsd-local/src/sys/netinet6/in6_src.c:204
>> #11 0xffffffff80800869 in rip6_output (m=m@entry=0xfffffe821c249400,
>>     so=so@entry=0xfffffe8219325db0,
>> dstsock=dstsock@entry=0xfffffe810f5a4bc4,
>>     control=control@entry=0x0)
>>     at /build/netbsd-local/src/sys/netinet6/raw_ip6.c:447
>> #12 0xffffffff80800e08 in rip6_send (l=<optimized out>, control=0x0,
>>     nam=0xfffffe81effcd638, m=0xfffffe821c249400, so=0xfffffe8219325db0)
>>     at /build/netbsd-local/src/sys/netinet6/raw_ip6.c:893
>> #13 rip6_send_wrapper (a=0xfffffe8219325db0, b=0xfffffe821c249400,
>>     c=0xfffffe81effcd638, d=0x0, e=<optimized out>)
>>     at /build/netbsd-local/src/sys/netinet6/raw_ip6.c:966
>> #14 0xffffffff80999251 in sosend (so=0xfffffe8219325db0,
>>     addr=0xfffffe81effcd638, uio=0xfffffe810f5a4d10, top=0xfffffe821c249400,
>>     control=0x0, flags=<optimized out>, l=0xfffffe81f455c4c0)
>>     at /build/netbsd-local/src/sys/kern/uipc_socket.c:1064
>> #15 0xffffffff809a0785 in do_sys_sendmsg_so (l=l@entry=0xfffffe81f455c4c0,
>>     s=s@entry=4, so=<optimized out>, fp=0xfffffe81f1ac6380,
>>     mp=mp@entry=0xfffffe810f5a4e58, flags=flags@entry=0,
>>     retsize=retsize@entry=0xfffffe810f5a4eb8)
>>     at /build/netbsd-local/src/sys/kern/uipc_syscalls.c:622
>> #16 0xffffffff809a0ad2 in do_sys_sendmsg (l=l@entry=0xfffffe81f455c4c0, s=4,
>>     mp=mp@entry=0xfffffe810f5a4e58, flags=0,
>>     retsize=retsize@entry=0xfffffe810f5a4eb8)
>>     at /build/netbsd-local/src/sys/kern/uipc_syscalls.c:672
>> #17 0xffffffff809a0b9b in sys_sendmsg (l=0xfffffe81f455c4c0,
>>     uap=0xfffffe810f5a4f00, retval=0xfffffe810f5a4eb8)
>>     at /build/netbsd-local/src/sys/kern/uipc_syscalls.c:528
>> #18 0xffffffff80901f6c in sy_call (rval=0xfffffe810f5a4eb8,
>>     uap=0xfffffe810f5a4f00, l=0xfffffe81f455c4c0,
>>     sy=0xffffffff810ef240 <sysent+672>)
>>     at /build/netbsd-local/src/sys/sys/syscallvar.h:65
>> #19 sy_invoke (code=28, rval=0xfffffe810f5a4eb8, uap=0xfffffe810f5a4f00,
>>     l=0xfffffe81f455c4c0, sy=0xffffffff810ef240 <sysent+672>)
>>     at /build/netbsd-local/src/sys/sys/syscallvar.h:94
>> #20 syscall (frame=0xfffffe810f5a4f00)
>>     at /build/netbsd-local/src/sys/arch/x86/x86/syscall.c:156
>> #21 0xffffffff80100691 in Xsyscall ()
>> #3  0xffffffff80a8421a in rtcache_invariants (ro=0xfffffe821c830060)
>>     at /build/netbsd-local/src/sys/net/route.h:441
>> 441             KASSERT(ro->_ro_rt == NULL || ro->_ro_rt->rt_refcnt > 0);
>> (gdb) print ro->_ro_rt
>> $1 = (struct rtentry *) 0xfffffe81fc5c39a0
>> (gdb) print ro->_ro_rt->rt_refcnt
>> $2 = 0
>> (gdb)
>>
>>
>>
>> To reproduce this, I simply bring up my openvpn tunnel, which has IPv6
>> addresses on both ends.  I can successfully ping6 the remote tunnel
>> end-point (ie, the "directly attached neighbor").  However, as soon as I try
>> to ping6 to something more than 1 hop away (I've been using ftp.netbet.org
>> for testing!) it crashes as above.
>>
>> This is on a GENERIC amd64 kernel, built from sources updated via anoncvs on
>> 2015-09-01 at 06:53:13 UTC.
>>
>>
>>
>> -------------------------------------------------------------------------
>> | Paul Goyette     | PGP Key fingerprint:     | E-mail addresses:       |
>> | (Retired)        | FA29 0E3B 35AF E8AE 6651 | paul at whooppee.com    |
>> | Kernel Developer | 0786 F758 55DE 53BA 7731 | pgoyette at netbsd.org  |
>> -------------------------------------------------------------------------