kern/48098: panic: kernel diagnostic assertion "cred != NULL": sys/kern/kern_auth.c

[marcotte%panix.com@localhost]

>Number:         48098
>Category:       kern
>Synopsis:       panic: kernel diagnostic assertion "cred != NULL": 
>sys/kern/kern_auth.c
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Jul 30 07:50:00 +0000 2013
>Originator:     Brian Marcotte
>Release:        6.1
>Organization:
Public Access Networks, Corp.
>Environment:
NetBSD dw6.panix.com 6.1 NetBSD 6.1 (PANIX-XEN-WEB-DEBUG) #1: Tue Jul 23 
23:10:42 EDT 2013  
root%juggler.panix.com@localhost:/misc/obj/misc/devel/netbsd/6.1/src/sys/arch/i386/compile/PANIX-XEN-WEB-DEBUG
 i386
>Description:
We're occasionally seeing these panics:

  panic: kernel diagnostic assertion "cred != NULL" failed: file 
"/misc/devel/netbsd/6.1/src/sys/kern/kern_auth.c", line 300
  cpu3: Begin traceback...
  
kern_assert(c03e74d4,c03e7515,c03f5ff1,c03f53f0,12c,3,e2433c14,c030f30d,0,12355bc6)
 at netbsd:kern_assert+0x23
  kauth_cred_geteuid(0,12355bc6,25ef,e50854a6,2500,0,0,e2433b2c,c02b017e,2) at 
netbsd:kauth_cred_geteuid+0x43
  
sysctl_net_inet_tcp_ident(e2433c94,0,bf7fdde4,e2433cb4,bf7fdf4c,100,e2433c84,ca35cd40,c86f58a0,4)
 at netbsd:sysctl_net_inet_tcp_ident+0x2fd
  
sysctl_dispatch(e2433c84,4,bf7fdde4,e2433cb4,bf7fdf4c,100,e2433c84,ca35cd40,c86f58a0,e2433cb4)
 at netbsd:sysctl_dispatch+0xc7
  
sys___sysctl(ca35cd40,e2433d00,e2433d28,0,c9d40374,ca,0,bb782000,e2433d00,c972e52c)
 at netbsd:sys___sysctl+0xea
  
syscall(e2433d48,bb7c00b3,bb7d00ab,bb74001f,bb7f001f,4,bf7fdde4,bf7fddb0,bb7c65bc,bf7fdde0)
 at netbsd:syscall+0xaa
  cpu3: End traceback...

Possibly related, we've noticed an "ESTABLISHED socket leak". These
servers run Apache and often the output of "netstat -f inet" shows
ESTABLISHED connections to port 80 which lsof doesn't. Restarting
Apache will clear them all.

If tcpdrop is used on one of the mystery connections, we get this panic:

  panic: kernel diagnostic assertion "cred != NULL" failed: file 
"/misc/devel/netbsd/6.1/src/sys/kern/kern_auth.c", line 292
  cpu3: Begin traceback...
  
kern_assert(c03e74d4,c03e7515,c03f5ff1,c03f53f0,124,0,e278da40,c01f0c7d,0,ca7ced80)
 at netbsd:kern_assert+0x23
  
kauth_cred_getuid(0,ca7ced80,2aba24,0,c03969e3,ca195e54,ca7ced80,c86d7268,e278da54,c033d75c)
 at netbsd:kauth_cred_getuid+0x43
  proc_uidmatch(ca7ced80,0,c86d80a8,e278da90,c01cd59a,ca7ced80,8,0,1a,ca195e54) 
at netbsd:proc_uidmatch+0x2d
  socket_listener_cb(ca7ced80,8,0,1a,ca195e54,c9ee27f4,0,1,0,8) at 
netbsd:socket_listener_cb+0x3c
  
kauth_authorize_action_internal(1a,ca195e54,c9ee27f4,0,e278dacc,c01cf3fd,c86d80a8,ca7ced80,8,1a)
 at netbsd:kauth_authorize_action_internal+0x9a
  
kauth_authorize_action(c86d80a8,ca7ced80,8,1a,ca195e54,c9ee27f4,0,e278dc14,c030f461,ca7ced80)
 at netbsd:kauth_authorize_action+0x2f
  
kauth_authorize_network(ca7ced80,8,1a,ca195e54,c9ee27f4,0,0,e278db2c,c02b017e,2)
 at netbsd:kauth_authorize_network+0x3d
  
sysctl_net_inet_tcp_ident(e278dc94,0,0,e278dcb4,bf7fec74,100,e278dc84,ca8fda80,c86f59c0,4)
 at netbsd:sysctl_net_inet_tcp_ident+0x451
  
sysctl_dispatch(e278dc84,4,0,e278dcb4,bf7fec74,100,e278dc84,ca8fda80,c86f59c0,e278dcb4)
 at netbsd:sysctl_dispatch+0xc7
  
sys___sysctl(ca8fda80,e278dd00,e278dd28,0,c96ed6cc,ca,0,bb78b000,e278dd00,c8c8aa3c)
 at netbsd:sys___sysctl+0xea
  syscall(e278dd48,b3,ab,bf7f001f,806001f,4,0,bf7fe400,bb7c65bc,0) at 
netbsd:syscall+0xaa
  cpu3: End traceback...


>How-To-Repeat:
I don't know how to cause the mystery connections, but they are almost
always present on at least two of our web servers.

Running tcpdrop on the mystery connections will always cause the second
panic.

>Fix:
Unknown