tech-net archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: UDP_ENCAP_ESPINUDP_NON_IKE



Well, the crash is repeatable on the one week old daily snapshot current kernel. Again, here is the current kernel I am using:

NetBSD 8.99.17 (XEN3_DOMU) #0: Wed May 16 21:54:38 UTC 2018
mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/xen/compile/XEN3_DOMU

What is happening is ... crazy.

With the current kernel, when the remote client connects, we get caught in an endless loop of creating ipsec security associations. The log shows phase1 is created, then the phase2 associations, then we respond to negotiate a new phase1 and two new phase 2's, and I think this loop just continued until we ran out of memory. The windows client actually thought we were connected and showed it was connected in the network control panel, but the racoon log never reported that a ppp interface was up. When you look at the attached snippets from the logs, I bet you will agree that many ppp interfaces and ipsec SAs were created and when we finally ran out of memory to create another one, we crashed. I say this because the trace indicated the crash occurred at this branch. [1]. From the console at the start of the crash report, I got this:

[ 334.5292103] panic: kernel diagnostic assertion "IFNET_LOCKED(ifp)" failed: file "/usr/src/sys/net/if.c", line 3595

I don't understand line 3595 because if.c only has 661 lines, unless there was a mistake in how I copied it from the log.

It is a problem with the ifp structure in if.c, probably when it was trying to create or attach a new ppp interface.

The attached log snippets showed it created 8 phase 1 security associations when it crashed. Normally, for one connection, there is only one phase 1 SA.

Chuck

[1] https://nxr.netbsd.org/xref/src/sys/net/if.c?r=1.14


On 05/24/2018 01:23 AM, Maxime Villard wrote:
Le 23/05/2018 à 23:56, Chuck Zmudzinski a écrit :
Last update on my testing of the proposed racoon patch:

I tested a NetBSD 8.0 RC1 kernel with the attached patch to udp_usrreq.c that
comments out the branch that processes packets with the
UDP_ENCAP_ESPINUDP_NON_IKE socket option to test what would happen if we
remove that from the kernel, and ran my NetBSD 7 system on it with the
unpatched racoon and with our racoon that has support for ENABLE_NATT_00
removed. As expected, with the old racoon, the connection attempt fails on this kernel because of the bug in racoon that mistakenly causes the kernel
to use that branch of the kernel that I removed in this kernel. Also, as
expected, our patch to racoon that removes support for ENABLE_NATT_00 fixes the problem on this kernel without UDP_ENCAP_ESPINUDP_NON_IKE so I think this
solution will work on NetBSD 8.x. This is good news.

Alright, thanks. Note however that your patch is not correct, you also need
to replace INP_ESPINUDP_ALL by INP_ESPINUDP in udp4_realinput().

The bad news: I started testing with a recent current kernel downloaded from daily snapshots. It is about a week old. I ran my NetBSD 7 system on that current kernel with the new racoon without support for ENABLE_NATT_00, and as
expected it connected fine. However, as soon as I disconnected the VPN
connection on the remote host, the current kernel crashed. I could not
recover the log to see what happened when I rebooted after the crash.

I think I have done enough testing to show that our patch to racoon is a good
place to begin, but if you want to test this on the current kernel, be
prepared to deal with kernel crashes. I guess that is always true when using
current kernels...

Hum, no, current is not supposed to crash. You tested a kernel downloaded from
the snapshots without patching it, right?

If possible, please re-test; set the following sysctl

    sysctl -w ddb.onpanic=1

and then try to trigger the crash, it should give you a log immediately.

Thanks,
Maxime

From the console when it crashed:

[ 334.5292103] panic: kernel diagnostic assertion "IFNET_LOCKED(ifp)" failed: file "/usr/src/sys/net/if.c", line 3595
[ 334.5292103] cpu0: Begin traceback...
[ 334.5292103] ?() at ffffffff804e3d48
[ 334.5292103] ?() at ffffffff805e0c25
[ 334.5292103] ?() at ffffffff80573ac9
[ 334.5292103] ?() at ffffffff802cc6ac
[ 334.5292103] ?() at ffffffff802d106f
[ 334.5292103] ?() at ffffffff805a41a6
[ 334.5292103] ?() at ffffffff805a75bc
[ 334.5292103] ?() at ffffffff805a579a
[ 334.5292103] ?() at ffffffff80515e49
[ 334.5292103] ?() at ffffffff804fb3f8
[ 334.5292103] ?() at ffffffff804efb53
[ 334.5292103] ?() at ffffffff804efc59
[ 334.5292103] ?() at ffffffff8020f28c
[ 334.5292103] cpu0: End traceback...
[ 334.5292103] fatal breakpoint trap in supervisor mode
[ 334.5292103] trap type 1 code 0 rip 0xffffffff80205845 cs 0x8 rflags 0x202 cr2 0x7772c0508000 ilevel 0x4 rsp 0xffffa00
026a94670
[ 334.5292103] curlwp 0xffffa00000dbd420 pid 1946.1 lowest kstack 0xffffa00026a902c0
Stopped in pid 1946.1 (route) at        ffffffff80205845:       leave
ds          0
es          0
fs          4680
gs          4620
rdi         4
rsi         ffffffff80a83000
rbp         ffffa00026a94670
rbx         104
rdx         1
rcx         0
rax         0
r8          ffffffff807003c0
r9          cccccccccccccccd
r10         ffffa00026a945e0
r11         e02b
r12         ffffffff80621f68
r13         ffffa00026a946b8
r14         0
r15         0
rip         ffffffff80205845
cs          8
rflags      202
rsp         ffffa00026a94670
ss          e02b
ffffffff80205845:       leave
db{0}>

From /var/log/messages, showing what was logged just before rebooting:

May 24 13:40:13 ave racoon: INFO: IPsec-SA established: ESP/Transport 192.168.137.5[4500]->192.168.1.80[4500] spi=13178033(0xc914b1) 
May 24 13:40:13 ave racoon: INFO: IPsec-SA established: ESP/Transport 192.168.137.5[4500]->192.168.1.80[4500] spi=1996696044(0x770329ec) 
May 24 13:40:43 ave racoon: INFO: purged IPsec-SA proto_id=ESP spi=1996696044. 
May 24 13:40:43 ave racoon: INFO: ISAKMP-SA expired 192.168.137.5[4500]-192.168.1.80[4500] spi:002796146c2e118e:ed91d65dc650f66d 
May 24 13:40:43 ave racoon: INFO: ISAKMP-SA deleted 192.168.137.5[4500]-192.168.1.80[4500] spi:002796146c2e118e:ed91d65dc650f66d 
May 24 13:40:43 ave racoon: INFO: ISAKMP-SA expired 192.168.137.5[4500]-192.168.1.80[4500] spi:7a86601aa1b28599:a47403492942ede9 
May 24 13:40:43 ave racoon: INFO: ISAKMP-SA deleted 192.168.137.5[4500]-192.168.1.80[4500] spi:7a86601aa1b28599:a47403492942ede9 
May 24 13:40:43 ave racoon: INFO: ISAKMP-SA expired 192.168.137.5[4500]-192.168.1.80[4500] spi:8a104a04a360878c:3a3f67500531a2ad 
May 24 13:40:43 ave racoon: INFO: ISAKMP-SA deleted 192.168.137.5[4500]-192.168.1.80[4500] spi:8a104a04a360878c:3a3f67500531a2ad 
May 24 13:40:43 ave racoon: INFO: ISAKMP-SA expired 192.168.137.5[4500]-192.168.1.80[4500] spi:2dc96d1a78494751:be13ab93ec159ff5 
May 24 13:40:43 ave racoon: INFO: ISAKMP-SA deleted 192.168.137.5[4500]-192.168.1.80[4500] spi:2dc96d1a78494751:be13ab93ec159ff5 
May 24 13:40:43 ave racoon: INFO: unsupported PF_KEY message REGISTER 
May 24 13:40:43 ave racoon: ERROR: no iph2 found: ESP 192.168.1.80[500]->192.168.137.5[500] spi=13178033(0xc914b1) 
May 24 13:40:43 ave racoon: INFO: unsupported PF_KEY message REGISTER 
May 24 13:40:58 ave syslogd[270]: last message repeated 6 times
May 24 09:53:08 ave syslogd[273]: restart
May 24 09:53:08 ave /netbsd: [   1.0000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
May 24 09:53:08 ave /netbsd: [   1.0000000]     2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017,
May 24 09:53:08 ave /netbsd: [   1.0000000]     2018 The NetBSD Foundation, Inc.  All rights reserved.
May 24 09:53:08 ave /netbsd: [   1.0000000] Copyright (c) 1982, 1986, 1989, 1991, 1993
May 24 09:53:08 ave /netbsd: [   1.0000000]     The Regents of the University of California.  All rights reserved.
May 24 09:53:08 ave /netbsd: 
May 24 09:53:08 ave /netbsd: [   1.0000000] NetBSD 8.99.17 (XEN3_DOMU) #0: Wed May 16 21:54:38 UTC 2018
May 24 09:53:08 ave /netbsd: [   1.0000000] 	mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/xen/compile/XEN3_DOMU



Home | Main Index | Thread Index | Old Index