kern/52554: IPv6 connections not routing to default gateway

[rcbixler%nyx.net@localhost]

>Number:         52554
>Category:       kern
>Synopsis:       IPv6 connections not routing to default gateway
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Sep 18 19:20:00 +0000 2017
>Originator:     Roy Bixler
>Release:        8.99.2
>Organization:
>Environment:
NetBSD localhost 8.99.2 NetBSD 8.99.2 (GENERIC) #0: Mon Sep 18 17:56:40 UTC 2017  root@localhost:/usr/src/sys/arch/amd64/compile/obj/GENERIC amd64
>Description:
Problem system has IP addresses assigned via DHCP.  IPv4 works as expected, but IPv6 connections to hosts outside the LAN hang.  The IPv6 address is assigned as expected from DHCP and the routing table looks fine as well.  The test system is in a QEMU KVM and I've verified that other operating systems (Debian and Gentoo Linux, NetBSD 7.1) don't have the IPv6 routing problem.  The issue only occurs when the guest OS is NetBSD-current or NetBSD-8.

My test connection was to do a command like the following:

$ telnet netbsd.org www
Trying 2001:470:a085:999::80...

and see whether it hangs.  When I do a "tcpdump" command on an affected system, I only see the outbound connection:

17:30:28.031786 IP6 2620:105:c000:3:60da:5ed:9c53:69cf.65531 > 2001:470:a085:999::80.http: Flags [S], seq 3456172076, win 32768, options [mss 1440,nop,wscale 3,sackOK,TS val 1 ecr 0], length 0
1

There are no neighbor solicitations.  I do see router advertisements separately.  To compare, I did a "tcpdump" on a good configuration, like NetBSD 7.1, and I see a neighbor solicitation and then an advertisement in response before the outbound connection:

17:36:30.890177 IP6 2620:105:c000:3:9ae2:df9b:70eb:1894 > ff02::1:ffda:2df3: ICMP6, neighbor solicitation, who has fe80::ec4:7aff:feda:2df3, length 32
17:36:30.890609 IP6 fe80::ec4:7aff:feda:2df3 > 2620:105:c000:3:9ae2:df9b:70eb:1894: ICMP6, neighbor advertisement, tgt is fe80::ec4:7aff:feda:2df3, length 32
17:36:30.890744 IP6 2620:105:c000:3:9ae2:df9b:70eb:1894.65535 > 2001:470:a085:999::80.http: Flags [S], seq 3981027458, win 32768, options [mss 1440,nop,wscale 3,sackOK,nop,nop,nop,nop,TS val 1 ecr 0], length 0

I decided to try the following change (on a slightly older version of NetBSD-current):

net.inet6.icmp6.nd6_useloopback=0

and, when I tried the "telnet" command above, I got a kernel panic soon afterwards.  Here is the traceback from it:

crash> bt
_KERNEL_OPT_NARCNET() at 0
_KERNEL_OPT_ACPI_SCANPCI() at _KERNEL_OPT_ACPI_SCANPCI+0x1
vpanic() at vpanic+0x149
snprintf() at snprintf
trap() at trap+0xc4c
--- trap (number 6) ---
rw_write_held() at rw_write_held+0x1a
nd6_resolve() at nd6_resolve+0x104
ether_output() at ether_output+0x596
ip6_output() at ip6_output+0x1a8c
icmp6_reflect() at icmp6_reflect+0x1a8
icmp6_error() at icmp6_error+0x2f4
nd6_llinfo_timer() at nd6_llinfo_timer+0x3d7
callout_softclock() at callout_softclock+0x201
softint_dispatch() at softint_dispatch+0xd3
DDB lost frame for Xsoftintr+0x4f, trying 0xffffe400034c0ff0
Xsoftintr() at Xsoftintr+0x4f
--- interrupt ---
0:

That was only a one-time change and all other testing was done with the default setting.

>How-To-Repeat:
Install NetBSD-8 or NetBSD-current in a QEMU KVM (with a bridge and tap set up for full network access) and get dynamically assigned addresses with dhcpcd.  Notice that any IPv6 connections outside the LAN stall and time out.  I do have NetBSD-8 set up elsewhere to work with IPv6, so I'm aware that there may be something special about the LAN setup here.  My best guess is according to a thread I started earlier on a different and much less severe IPv6-related problem.  For instance, see the following message:

http://mail-index.netbsd.org/netbsd-users/2017/09/12/msg020112.html

where I describe how Linux clients on the network get set up with multiple default IPv6 gateways on the same interface.  On the networks where I have no problems with IPv6 and NetBSD-8, there is only one IPv6 router.
>Fix:
Downgrade to NetBSD 7.1 or use a different IPv6 router setup.