kern/47598: Kernel crash in kauth_cred_uidmatch caused by netstat

[luke%maurits.id.au@localhost]

>Number:         47598
>Category:       kern
>Synopsis:       Kernel crash in kauth_cred_uidmatch caused by netstat
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Feb 26 17:35:00 +0000 2013
>Originator:     Luke Maurits
>Release:        6.0 STABLE
>Organization:
>Environment:
NetBSD <hostname> 6.0_STABLE NetBSD 6.0_STABLE (MYKERNEL) #2: Mon Feb  4 
03:42:25 UTC 2013  
luke%miku.maurits.id.au@localhost:/usr/obj/sys/arch/i386/compile/MYKERNEL i386
>Description:
For many months now I have had irregular, random kernel crashes on one of my 
machines.  The most recent case yielded the following backtrace:

uvm_fault(0xc0af5bd0, 0, 1) -> 0xe
fatal page fault in supervisor mode
trap type 6 code 0 eip c0203a7b cs 9 eflags 10296 cr2 40 ilevel 0
kernel: supervisor trap page fault, code=0                       
Stopped in pid 8877.1 (netstat) at      netbsd:kauth_cred_uidmatch
+0x1b:        m ovl     40(%esi),%
edx kauth_cred_uidmatch
(c12f50c0,0,c12f50c0,c7a378e4,c031ed1b,c0a01120,c12f50c0,c7a3
791c,c02041cd,c12f50c0) at netbsd:kauth_cred_uidmatch+0x1b
secmodel_extensions_network_cb
(c12f50c0,8,0,19,c12dbbb0,0,0,1,0,c1993078) at net
bsd:secmodel_extensions_network_cb+0x5b kauth_authorize_action
(c0a02060,c12f50c0,8,19,c12dbbb0,0,0,c7a37c1c,c0377511,c12 f50c0) at
netbsd:kauth_authorize_action+0x7d kauth_authorize_network
(c12f50c0,8,19,c12dbbb0,0,0,c7a3798c,c05c6ce0,0,6) at net
bsd:kauth_authorize_network+0x3d sysctl_inpcblist
(c7a37c9c,4,0,c7a37cbc,0,0,c7a37c8c,c0f8c7e0,c0a13b40,4) at netb
sd:sysctl_inpcblist+0x171 sysctl_dispatch
(c7a37c8c,8,0,c7a37cbc,0,0,c7a37c8c,c0f8c7e0,c0a13b40,c7a37cbc) a t
netbsd:sysctl_dispatch+0xb7 sys___sysctl
(c0f8c7e0,c7a37d00,c7a37d28,ca,abd17000,0,c7a37d00,c0af0884,2,abf48c
67) at netbsd:sys___sysctl
+0xea syscall
(c7a37d48,b6fb00b3,ab,beb0001f,b6fb001f,8,0,bebfeb40,abf687bc,bebfef98)
a t netbsd:syscall+0xaa

I've recorded 3 of these now, and the backtrace is always through the same 
series of functions, only the particular pointer values change.

This seems possibly related to kern/43290.  That bug is also caused by a kauth 
problem in netstat, but it is on kath_cred_getuid where mine is on 
kauth_cred_uidmatch.

These are happening on a Xen domU (VPS).  Right now it is running NetBSD
6.0_STABLE.  The kernel configuration is derived from the standard XEN3PAE_DOMU 
with the addition of

no options      INSECURE
options         PAX_MPROTECT=1 
options         PAX_SEGVGUARD=1
options         PAX_ASLR=1
options         FILEASSOC
options         VERIFIED_EXEC_FP_MD5
options         VERIFIED_EXEC_FP_SHA1
options         VERIFIED_EXEC_FP_RMD160
options         VERIFIED_EXEC_FP_SHA512
options         VERIFIED_EXEC_FP_SHA384
options         VERIFIED_EXEC_FP_SHA256

However, I am pretty certain I got the earliest instances of this crash
earlier on, when it was running 5.1.2 and the stock XEN3PAE_DOMU kernel with no 
modifications.

The machine in question is primarily a web server, with fairly low traffic.  
Things which are typically running all the time are imapproxy, ossec, sshd, 
php, lighttpd and mysql.  The crash is happening in netstat, but I'm never 
running it myself at the time of the crashes, so it must be being invoked by 
something else, most likely one of the above or one of the daily cron scripts.  
netstat does not crash if I just run it myself after ssh'ing in, at least not 
when I pass it no args and it does whatever its defaults are.

I have this machine set to drop to the debugger when it crashes, and I
can access that via my VPS provider's console system, so if anybody
needs me to the next time this happens I can try to provide the values
of variables or anything else which may be necessary.
>How-To-Repeat:

>Fix: