Subject: kern/32409: NetBSD-3.0/i386 vnlock deadlock
To: None <kern-bug-people@netbsd.org, gnats-admin@netbsd.org,>
From: None <soda@netbsd.org>
List: netbsd-bugs
Date: 12/29/2005 09:05:00
>Number:         32409
>Category:       kern
>Synopsis:       NetBSD-3.0/i386 vnlock deadlock
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Dec 29 09:05:00 +0000 2005
>Originator:     SODA Noriyuki
>Release:        NetBSD 3.0_BETA
>Organization:
The NetBSD project
>Environment:
This is observed on ftp.netbsd.org.
Architecture: i386
Machine: i386
>Description:

On ftp.netbsd.org, all processes(?) which touch filesystem hang,
and other processes which only touch network stack do work.

The followings are some information collected in DDB.

Is there any other information I could collect?

-------------
DDB ps output
-------------

db> ps/w

(I removed some processes due to security reason.
 but I think those processes aren't relevant.
 please mail me, if you need full output)

 PID          COMMAND     EMUL  PRI UTIME STIME WAIT-MSG    WAIT-CHANNEL
 7466           inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 19749          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 20255          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 6942           inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 21789          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 4378           inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 14103          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 24845          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 24324          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 5888           inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 8440           inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 18159          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 26345          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 4320           inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 24779          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 2762           inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 20169          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 3783           inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 9414           inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
db757           inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 28337          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait

 22187          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 12960          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 10397          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 8856           inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 24208          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 5260           inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 5256           inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 23685          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 19579          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 2168           inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 20590          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 4198           inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 18532          inetd   netbsd   24   0.0   0.0 poll        netbsd:selwait
 8283           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22611          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 19026          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 28752          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 9289           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 14407          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22566          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 2564           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
--db_more--     inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 5411           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50

 11291          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 12302           cron   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 12778           cron   netbsd   32   0.0   0.0 ppwait      0x94eff338
 1362           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 215            inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 23544          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 17390          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 13290          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 5599           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 12250          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 5584           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8124           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 18875          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 19896          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 17847          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 11698          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 1456           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 5546           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22950          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 7588           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 24477          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
--db_more--     inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 27537          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50

 25483          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 18824          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 378            inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 9586           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 23408          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22383          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8558           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 3949           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 876            inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 5482           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 12648          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 5981           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 2905           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8536           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 11088          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 1865           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 23368          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 26430          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 2871           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22325           cron   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 29492           cron   netbsd   32   0.0   0.0 ppwait      0x7751d1a4
--db_more--     inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 1319           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50

 20258          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 17177          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 7432           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 29959          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 1790           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 1262           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 7907           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 13020          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8410           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 9425           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 27343          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8397           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 26312          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 706            inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 15039          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 4792           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 7340           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 26262          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 4238           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 19083          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 29834          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
--db_more--     inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 28286          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50

 4212           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 21103          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 10348          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 14443          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 5731           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 9821           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 15957          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 7246           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 27212          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 5184           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 23103          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 26174          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 29757          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 7739           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8762           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 6201           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 12343          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 3123           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 18980          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 23584          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 25119          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
--db_more--     inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 5138           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50

 527            inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 5129           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 1542           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 1539           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 20973          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 26435          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22261          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 12696          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 25807           tcsh   netbsd   17   0.0   0.0 biowait     0x66589138
 13931          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 11751          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 20444          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 12136           sshd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 347             tcsh   netbsd   17   0.0   0.0 biowait     0x665890bc
 10616          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 20047          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 6704           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 17591          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 3557           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 21755          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 13730          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
--db_more--     inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 20008          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50

 8621           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 14325          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22103          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 18759          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 26145          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 15854          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 2713           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 27014          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 4800           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22175          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 16643          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 6394           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 29485           sshd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8889           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 15452          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 6481           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 18043          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 81             inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 25307          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 11543          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 17335          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
--db_more--     inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8094           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 4943 
 28086          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 5905           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 12592          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 14723          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 29231          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 29031          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 19384          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 6726           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 5072           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 16083          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 2185           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 12299          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 16182           sshd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 19265          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 9060           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 26229          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 7037           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 16375          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 16830          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8871           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 18095          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
--db_more-- 
 8024           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50

 24700          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 17678           tcsh   netbsd   17   0.0   0.0 biowait     0x66001e3c
 750            inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 26800          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 18722          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 26001          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 19668          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 11746          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 27830          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 2027           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 17918          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22595          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 19567          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22941           sshd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 24638          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 27216          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 14510          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 9552           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22438          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 4970           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 13439          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
--db_more--     inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 25003          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50

 12681          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 20318          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 7408           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 21846          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 13636          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 23526          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 28648          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 4876           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 24708          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 20452          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 10652          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 11831          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 19122          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 27030          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 29866          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8486           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 332            inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 2347            sshd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 21499          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 7385           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 15641          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
--db_more--     inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 16086          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50

 25676     supfilesrv   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 12457           cron   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 2484            cron   netbsd   32   0.0   0.0 ppwait      0x922187fc
 22287          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 13632          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 16466          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 7252           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 29711          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 17188          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 1154           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 4730           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8335           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 24606          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 14847          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 4706           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 25558          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 7898           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8046           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 28775          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 26267          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 19054          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
--db_more--     inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8148           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50

 20638           sshd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8733           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 12871          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 2583           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 4382           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8113           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 3909           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 1369           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 4039           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 7666           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 18312          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 19910          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 1465           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 21872          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 23309          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 5210           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 19627          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 3918           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 16939          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 13278          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 23907          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
--db_more--     inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 7226           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50

 24804          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 389             sshd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 24807          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 6074           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 29036          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 2842           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 24776          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 333            inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 19711          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8010           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 17594          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 7878           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 1564           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 29238          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 14136          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22053          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 25474          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 18484          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 21030          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 15599          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 15721          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
--db_more--     inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 23243          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50

 7044           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 10406          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 5689           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22389          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 13407           sshd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 3807           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 28563          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8853           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 27372          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 23072          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 3096           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 5652           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 21932          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 10840          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 5034           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8429           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 3724           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22009          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 23785          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 3256           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22090          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
--db_more--     inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 25584          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50

 14421          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 18837          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 10487          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 17444          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 6296           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 9313           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 17908          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 18823          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22013          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 29245     supfilesrv   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 18234          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22149     supfilesrv   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 20430          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 25641          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 23848     supfilesrv   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 6828           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 9074      supfilesrv   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 202            inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8006      supfilesrv   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 6588      supfilesrv   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 19179          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
--db_more--     inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 25164          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 3689 
 4875           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 4224           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 17868          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 7076           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 9288           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 15191           cron   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 24769           cron   netbsd   32   0.0   0.0 ppwait      0x8537a808
 4493            sshd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 6734           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 29322          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 14797          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 24894          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22987          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 7050           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 22242          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 6877           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 26302          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 10714           sshd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 20847           sshd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 14720          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 707            inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
--db_more-- 
 18720          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50

 27714          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 4891           inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 19452          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 13686          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 21648          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 17783          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 13666          inetd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 18213           sshd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 14226           ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 15516           ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 28558           ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 2911            ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 2611            ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 13555           ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 9750            ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8285            ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 8153            ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 12629           ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 18983           ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 12017           ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 10400           ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
--db_more--      ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 9538            ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50

 870           uptime   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 2212              sh   netbsd   24   0.0   0.0 piperd      0x854163f0
 23947       pkg_info   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 6203              sh   netbsd   32   0.0   0.0 wait        0x71ea6e60
 27587           ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 16179           ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 21075           ftpd   netbsd   20   0.0   0.0 vnlock      0x71ae6ef0
 22204           ftpd   netbsd   20   0.0   0.0 vnlock      0x71ae6ef0
 329             ftpd   netbsd    4   0.0   0.0 uvn_fp2     0x611e4540
 7412            ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 29447           ftpd   netbsd   20   0.0   0.0 vnlock      0x7184f8f8
 17576           ftpd   netbsd   20   0.0   0.0 vnlock      0x71ae6ef0
 6371            ftpd   netbsd   20   0.0   0.0 vnlock      0x71ae6ef0
 23412             sh   netbsd   24   0.0   0.0 piperd      0x85416750
 29656           ftpd   netbsd   20   0.0   0.0 vnlock      0x70e91e50
 27725           make   netbsd   24   0.0   0.0 piperd      0x74232634
 7753              sh   netbsd   32   0.0   0.0 wait        0x7360e338
 22867           make   netbsd   24   0.1   0.0 piperd      0x74232a24
 801             ftpd   netbsd   20   0.0   0.0 vnlock      0x71ae6ef0
 5738            ftpd   netbsd   20   0.0   0.0 vnlock      0x71a9f9a8
 19698           ftpd   netbsd   20   0.0   0.0 vnlock      0x71ae6ef0
--db_more--      ftpd   netbsd   20   0.0   0.0 vnlock      0x71ae6ef0
 25006           ftpd   netbsd    4   0.0   0.0 uvn_fp2     0x61566500

 23440           ftpd   netbsd   20   0.0   0.1 vnlock      0x71ae6ef0
 6920            ftpd   netbsd   20   0.0   0.2 vnlock      0x71ae6ef0
 6402            ftpd   netbsd    4   0.0   0.0 uvn_fp2     0x604c0880
 27196             sh   netbsd   32   0.0   0.0 wait        0x85b46990
 927             make   netbsd   32   0.0   0.0 wait        0x93d5233c
 6460            ftpd   netbsd   20   0.0   0.0 vnlock      0x71a9f9a8
 16886           ftpd   netbsd   20   0.0   0.0 vnlock      0x71a9f9a8
 17609           ftpd   netbsd   20   0.0   0.0 vnlock      0x71ae6ef0
 20242           ftpd   netbsd   20   0.0   0.0 vnlock      0x71ae6ef0
 28513           ftpd   netbsd   20   0.0   0.0 vnlock      0x71ae6ef0
 13737           ftpd   netbsd   20   0.0   0.0 vnlock      0x71ae6ef0
 15797           ftpd   netbsd   20   0.0   0.0 vnlock      0x71ae6ef0
 3700            ftpd   netbsd    4   0.0   1.7 uvn_fp2     0x6190b2c0
 21764             sh   netbsd   32   0.0   0.0 wait        0x7360e4d0
 18478            ssh   netbsd   24   1.0   0.2 select      netbsd:selwait
 26133           perl   netbsd   24   0.0   0.0 piperd      0x70e80480
 3720            make   netbsd   32   0.0   0.0 wait        0x7360eb30
 15973            cvs   netbsd   17   5.9  13.4 biowait     0x64e06290
 27956             sh   netbsd   32   0.0   0.0 wait        0x71b84198
--db_more--        sh   netbsd   32   0.0   0.0 wait        0x85b46198

 22051             sh   netbsd   32   0.0   0.0 wait        0x7316dcc4
 13936             sh   netbsd   32   0.0   0.0 wait        0x85b46000
 908             time   netbsd   32   0.0   0.0 wait        0x71b844c8
 5916              sh   netbsd   32   0.0   0.0 wait        0x71b84660
 4773              sh   netbsd   32   0.0   0.0 wait        0x73081000
 23686           cron   netbsd   24   0.0   0.0 piperd      0x742323f4
 20928           cron   netbsd   24   0.0   0.0 piperd      0x74232d84
 3726            ftpd   netbsd   20   0.0   0.0 vnlock      0x71ae6ef0
 10369           ftpd   netbsd   20   0.0   0.0 vnlock      0x71ae6ef0
 10609           ftpd   netbsd   20   0.0   0.0 vnlock      0x71ae6ef0
 5065            ftpd   netbsd   20   0.0   0.1 vnlock      0x71a9f9a8
 14124           ftpd   netbsd   20   0.0   0.2 vnlock      0x71a9f9a8
 28202           ftpd   netbsd   20   0.0   0.2 vnlock      0x71a9f9a8
 24253           ftpd   netbsd   20   0.0   0.2 vnlock      0x71a9f9a8
 10555           ftpd   netbsd   20   0.0   0.2 vnlock      0x71a9f9a8
 16488           ftpd   netbsd   20   0.0   0.2 vnlock      0x71a9f9a8
 7465            ftpd   netbsd   20   0.0   0.2 vnlock      0x71a9f9a8
 12963           ftpd   netbsd   20   0.0   0.2 vnlock      0x71a9f9a8
 29562           ftpd   netbsd   20   0.0   0.1 vnlock      0x71ae6ef0
 21567           ftpd   netbsd   20   0.0   0.2 vnlock      0x71a9f9a8
 489             ftpd   netbsd   17   0.0   0.6 biowait     0x6582b9ec
--db_more--      tcsh   netbsd   17   0.0   0.0 biowait     0x66589610
 4617            tcsh   netbsd   32   0.0   0.0 ppwait      0x853a6660

 5085            tcsh   netbsd   40   0.0   0.0 pause       0x86e125f4
 22042           sshd   netbsd   24   0.1   0.0 select      netbsd:selwait
 9210            sshd   netbsd   24   0.0   0.0 netio       0x65588cf0
 29890           ftpd   netbsd    4   0.0   0.1 uvn_fp2     0x6369dfc0
 11643          rsync   netbsd    4  22.9  10.7 uvn_fp2     0x6105a080
 7885            ftpd   netbsd   20   0.0   2.3 vnlock      0x71ae6ef0
 4672            ftpd   netbsd    4   0.0   0.1 uvn_fp2     0x611e0e00
 23008            ksh   netbsd   40   0.0   0.0 pause       0x995c4508
 26365           sshd   netbsd   24   0.0   0.0 select      netbsd:selwait
 10387           sshd   netbsd   24   0.0   0.0 netio       0x658e36e0
 15165           ftpd   netbsd    4   0.0   3.4 uvn_fp2     0x61333740
 26929           ftpd   netbsd   20   0.0   3.6 vnlock      0x71ae6ef0
 11378           ftpd   netbsd   20   0.0   0.8 vnlock      0x71a9f9a8
 17680           ftpd   netbsd    4   0.0   3.1 uvn_fp2     0x6224a6c0
 3328            ftpd   netbsd   20   0.0   0.1 vnlock      0x71a9f9a8
 11647           ftpd   netbsd   20   0.0   0.0 vnlock      0x71a9f9a8
--db_more--      ftpd   netbsd   20   0.0   0.0 vnlock      0x71a9f9a8
 23667           ftpd   netbsd   20   0.0   0.0 vnlock      0x71a9f9a8

 10981           ftpd   netbsd   20   0.0   0.1 vnlock      0x71a9f9a8
 11186           ftpd   netbsd   20   0.0   0.0 vnlock      0x71ae6ef0
 13573           ftpd   netbsd   17   1.5   2.0 biowait     0x663d1c3c
 13858          rsync   netbsd   20  11.8   8.3 vnlock      0x70e91e50
 17102           ftpd   netbsd   20   0.0   0.2 vnlock      0x71a9f9a8
 20789           ftpd   netbsd   20   0.0   0.3 vnlock      0x71a9f9a8
 25982           ftpd   netbsd   20   0.0   0.2 vnlock      0x71a9f9a8
 12436           ftpd   netbsd   20   0.0   0.3 vnlock      0x71a9f9a8
 14457           ftpd   netbsd   20   0.0   0.2 vnlock      0x71a9f9a8
 10152           ftpd   netbsd    4   1.6  33.9 uvn_fp2     0x62944d80
 25777          rsync   netbsd   17   3.9   4.5 biowait     0x65790900
 6904            ftpd   netbsd   17   1.7  17.5 biowait     0x6657e7fc
 24786           ftpd   netbsd   24   0.3  18.0 netio       0x66593c14
 26750          rsync   netbsd    4  51.6  44.7 uvn_fp2     0x611ef900
 15295           ftpd   netbsd   17   5.6  14.0 biowait     0x65158c9c
 12033           ftpd   netbsd   20   0.0   1.6 vnlock      0x71a9f9a8
 18664           ftpd   netbsd    4   0.0   0.0 uvn_fp2     0x61fc5b40
 18363           ftpd   netbsd    4   1.7   9.3 uvn_fp2     0x623e6380
 27261           ftpd   netbsd   17   1.5  11.0 biowait     0x6582b41c
 24114           ftpd   netbsd   17  18.9  13.3 biowait     0x65790a74
 25495           tcsh   netbsd   32   0.0   0.0 ppwait      0x71b84990
--db_more--      tcsh   netbsd   40   0.0   0.0 pause       0x7184c34c
 2368            tcsh   netbsd   40   0.0   0.0 pause       0x7184c34c
  0420           sshd   netbsd   24   0.0   0.0 select      netbsd:selwait
 7697            sshd   netbsd   24   0.0   0.0 netio       0x64bbb928
 7860            tcsh   netbsd   25   0.0   0.0 ttyin       0x705e1a5c
 8147            sshd   netbsd   24   0.0   0.0 select      netbsd:selwait
 18029           sshd   netbsd   24   0.0   0.0 netio       0x64a3c0f0
 18081           tcsh   netbsd   25   0.0   0.0 ttyin       0x705e184c
 6193            sshd   netbsd   24   0.0   0.0 select      netbsd:selwait
 24474           sshd   netbsd   24   0.0   0.0 netio       0x64918e30
 28065           ftpd   netbsd    4   0.6  18.9 uvn_fp2     0x622e4b80
 24987           ftpd   netbsd   17   5.1   7.2 biowait     0x6657eae4
 23929           ftpd   netbsd   24  52.3  24.1 netio       0x64bdc31c
 8861           rsync   netbsd   24   0.9   2.2 select      netbsd:selwait
 5145            ftpd   netbsd   17  36.7  65.1 biowait     0x65790428
 24882          rsync   netbsd    4 185.5 105.2 uvn_fp2     0x61641480
 1223            bash   netbsd   32   0.0   0.0 wait        0x90efdb34
 17695           bash   netbsd   32   0.0   0.0 wait        0x90ef04d4
 20253           sshd   netbsd   24   0.1   0.0 select      netbsd:selwait
 8913            sshd   netbsd   24   0.0   0.0 netio       0x64ec6afc
 27766            cat   netbsd   24   0.0   0.0 piperd      0x742327e4

 28368           sshd   netbsd   24   0.0   0.0 select      netbsd:selwait
 12628           sshd   netbsd   24   0.0   0.0 netio       0x64a3c378
 23003             sh   netbsd   25   0.0   0.0 ttyin       0x716e6118
 10225           sshd   netbsd   24   0.0   0.0 select      netbsd:selwait
 9305            sshd   netbsd   24   0.0   0.0 netio       0x64cdf914
 14688             sh   netbsd   32   0.0   0.0 wait        0x72203808
 3818             csh   netbsd   40   0.0   0.0 pause       0x730e4180

 18573           tcsh   netbsd   40   0.0   0.0 pause       0x730f118c
 19569           sshd   netbsd   24   0.0   0.2 select      netbsd:selwait
 18266           sshd   netbsd   24   0.0   0.0 netio       0x64c1409c
 4626           getty   netbsd   17   0.0   0.0 biowait     0x66589974

 8563            tcsh   netbsd   17   0.0   0.0 biowait     0x65158938
 7639            tcsh   netbsd   40   0.0   0.0 pause       0x73167198
 5930            tcsh   netbsd   40   0.0   0.0 pause       0x730eb184
 6747            sshd   netbsd   24   0.3   0.1 select      netbsd:selwait
 6396            sshd   netbsd   24   0.0   0.0 netio       0x64b3ee9c
 5087            ftpd   netbsd   17  28.7  18.6 biowait     0x65790ce0
 4445            ftpd   netbsd   24  28.1  18.9 netio       0x649c5d7c
 4195            ftpd   netbsd   24  27.1  19.2 netio       0x64966780
 2662             ksh   netbsd   40   0.0   0.0 pause       0x71f82140
 1682             ksh   netbsd   40   0.0   0.0 pause       0x71af5074
 1469            sshd   netbsd   24   0.3   0.2 select      netbsd:selwait
 1493            sshd   netbsd   24   0.0   0.0 netio       0x64bb28c8
 197             bash   netbsd   32   0.0   0.1 wait        0x71ca04cc
 196             sshd   netbsd   24   0.4   0.3 select      netbsd:selwait
 606             sshd   netbsd   24   0.0   0.0 netio       0x6496dae8
 691            getty   netbsd   25   0.0   0.0 ttyin       0x716f3758
 944            getty   netbsd   25   0.0   0.0 ttyin       0x716f3650
--db_more--     getty   netbsd   25   0.0   0.0 ttyin       0x716f3548
 1335            cron   netbsd   32   0.1   0.1 nanosleep   netbsd:nanowait.0

 423            inetd   netbsd   24   0.2   1.7 kqread      0x6491a000
 1273          master   netbsd   24   0.0   0.2 select      netbsd:selwait
 921             sshd   netbsd   24   0.0   0.3 select      netbsd:selwait
 528           rtsold   netbsd   24   0.2   0.8 poll        netbsd:selwait
 632             ntpd   netbsd   20   0.7   0.8 vnlock      0x70e91e50
 466               sh   netbsd   32   0.0   0.7 wait        0x71932668
 303        python2.4   netbsd   20 433.2 165.4 vnlock      0x71921a58
 302               sh   netbsd   32   0.0   0.0 wait        0x70e94cc4
 277       supfilesrv   netbsd   24   0.0   0.0 netcon      0x63c7c2c2
 275       supfilesrv   netbsd   24   0.0   0.0 netcon      0x63c7c1ea
 276       supfilesrv   netbsd   24   0.0   0.0 netcon      0x63c71cde
 254       supfilesrv   netbsd   24   0.0   0.0 netcon      0x63c71e8e
 145          syslogd   netbsd    4   2.0   6.6 uvn_fp2     0x61532480
 7           aiodoned   netbsd    4   0.0  19.9 aiodoned    netbsd:uvm+0x38
 6            ioflush   netbsd    4   0.0 250.1 genput      0x62538d80
 5         pagedaemon   netbsd    4   0.0  53.5 pgdaemon    netbsd:uvm+0x2c
 4          atapibus0   netbsd   16   0.0   0.0 sccomp      0x64752b78
 3            atabus1   netbsd   16   0.0   0.0 atath       0x64752b9c
--db_more--   atabus0   netbsd   16   0.0   0.0 atath       0x64752a30
 423            inetd   netbsd   24   0.2   1.7 kqread      0x6491a000
 1273          master   netbsd   24   0.0   0.2 select      netbsd:selwait
 921             sshd   netbsd   24   0.0   0.3 select      netbsd:selwait
 528           rtsold   netbsd   24   0.2   0.8 poll        netbsd:selwait
 632             ntpd   netbsd   20   0.7   0.8 vnlock      0x70e91e50
 466               sh   netbsd   32   0.0   0.7 wait        0x71932668
 303        python2.4   netbsd   20 433.2 165.4 vnlock      0x71921a58
 302               sh   netbsd   32   0.0   0.0 wait        0x70e94cc4
 277       supfilesrv   netbsd   24   0.0   0.0 netcon      0x63c7c2c2
 275       supfilesrv   netbsd   24   0.0   0.0 netcon      0x63c7c1ea
 276       supfilesrv   netbsd   24   0.0   0.0 netcon      0x63c71cde
 254       supfilesrv   netbsd   24   0.0   0.0 netcon      0x63c71e8e
 145          syslogd   netbsd    4   2.0   6.6 uvn_fp2     0x61532480
 7           aiodoned   netbsd    4   0.0  19.9 aiodoned    netbsd:uvm+0x38
 6            ioflush   netbsd    4   0.0 250.1 genput      0x62538d80
 5         pagedaemon   netbsd    4   0.0  53.5 pgdaemon    netbsd:uvm+0x2c
 4          atapibus0   netbsd   16   0.0   0.0 sccomp      0x64752b78
 3            atabus1   netbsd   16   0.0   0.0 atath       0x64752b9c
 1               init   netbsd   32   0.0   0.1 wait        0x705e2000

db>           swapper   netbsd    4   0.0   0.7 scheduler   netbsd:proc0


/* So, it seems most processes hang at vnlock 0x70e91e50 */

-----------
back traces
-----------

db> trace
cpu_Debugger(647dd400,6038cd8c,f420,1b,60315a20) at netbsd:cpu_Debugger+0x1
wskbd_translate(603324c0,2,1,1e,6464fb60) at netbsd:wskbd_translate+0x74
wskbd_input(647dd400,2,1,6038cdd4,6464fb00) at netbsd:wskbd_input+0x133
pckbd_input(63c54300,1,246,1,60340e60) at netbsd:pckbd_input+0x72
pckbportintr(60332600,0,1,6038ce34,1) at netbsd:pckbportintr+0x4f
pckbcintr(63c54380,0,95100010,60380030,10) at netbsd:pckbcintr+0xad
Xintr_legacy1() at netbsd:Xintr_legacy1+0xa9
--- interrupt ---
cpu_switch(60343840,0,0,206,950e277c) at netbsd:cpu_switch+0x9f
ltsleep(60343680,4,602ed6a7,0,0) at netbsd:ltsleep+0x355
uvm_scheduler(60343660,0,603472bc,602dd893,0) at netbsd:uvm_scheduler+0x83
main(0,0,0,0,0) at netbsd:main+0x6c9

db> trace/t 0t12136
trace: pid 12136 swapped out
db> trace/t 0t29485
trace: pid 29485 swapped out
db> trace/t 0t16182
trace: pid 16182 swapped out
db> trace/t 0t303
trace: pid 303 swapped out
db> trace/t 0t632
trace: pid 632 swapped out

db> trace/t 0t8283
trace: pid 8283  at 0x81c77a9c
ltsleep(70e91e50,14,602df049,0,70e91e50) at netbsd:ltsleep+0x355
acquire(81c77b54,81c77b3c,0,0,600) at netbsd:acquire+0x112
lockmgr(70e91e50,10002,70e91de0,0,0) at netbsd:lockmgr+0x350
genfs_lock(81c77b94,705ec2a0,62020,0,3) at netbsd:genfs_lock+0x25
vn_lock(70e91de0,10002,81c77d8c,6023dbad,7f) at netbsd:vn_lock+0x90
vget(70e91de0,10002,70e91de0,6483d000,1305) at netbsd:vget+0xa9
ufs_ihashget(1305,2,2,0,40500) at netbsd:ufs_ihashget+0x5f
ffs_vget(6483d000,2,81c77c98,6483d000,70e91750) at netbsd:ffs_vget+0x2f
ufs_root(6483d000,81c77cec,0,a51c01c9,0) at netbsd:ufs_root+0x21
lookup(81c77e34,95175400,400,81c77e4c,81c77e58) at netbsd:lookup+0x526
namei(81c77e34,0,e,65eede90,0) at netbsd:namei+0x11f
unp_connect(64a23c28,6514fd00,79cff66c,0,0) at netbsd:unp_connect+0x83
uipc_usrreq(64a23c28,4,0,6514fd00,0) at netbsd:uipc_usrreq+0xad
soconnect(64a23c28,6514fd00,79cff66c,3,79cff66c) at netbsd:soconnect+0xbf
sys_connect(950e277c,81c77f64,81c77f5c,79cff66c,4de647cd) at netbsd:sys_connect+
0xb3
syscall_plain() at netbsd:syscall_plain+0x85
--- syscall (number 98) ---
0x5db2e1af:

db> trace/t 0t22611
trace: pid 22611  at 0x8162ba9c
ltsleep(70e91e50,14,602df049,0,70e91e50) at netbsd:ltsleep+0x355
acquire(8162bb54,8162bb3c,0,0,600) at netbsd:acquire+0x112
lockmgr(70e91e50,10002,70e91de0,85000,0) at netbsd:lockmgr+0x350
genfs_lock(8162bb94,705ec2a0,0,0,3) at netbsd:genfs_lock+0x25
vn_lock(70e91de0,10002,8162bd8c,6023dbad,705ecd20) at netbsd:vn_lock+0x90
vget(70e91de0,10002,70e91de0,6483d000,1305) at netbsd:vget+0xa9
ufs_ihashget(1305,2,2,0,40500) at netbsd:ufs_ihashget+0x5f
ffs_vget(6483d000,2,8162bc98,6483d000,70e91750) at netbsd:ffs_vget+0x2f
ufs_root(6483d000,8162bcec,0,60207b6c,603466d8) at netbsd:ufs_root+0x21
lookup(8162be34,95175000,400,8162be4c,8162be58) at netbsd:lookup+0x526
namei(8162be34,0,e,65eed470,0) at netbsd:namei+0x11f
unp_connect(6572f400,65153900,79cff4d4,0,0) at netbsd:unp_connect+0x83
uipc_usrreq(6572f400,4,0,65153900,0) at netbsd:uipc_usrreq+0xad
Hsoconnect(6572f400,65153900,79cff4d4,3,79cff4d4) at netbsd:soconnect+0xbf
sys_connect(950e26f8,8162bf64,8162bf5c,79cff4d4,0) at netbsd:sys_connect+0xb3
syscall_plain() at netbsd:syscall_plain+0x85
--- syscall (number 98) ---
0x5db2e1af:

------------------------------
look at the vnode (0x70e91de0)
------------------------------

db> show vnode/f 70e91de0       
OBJECT 0x70e91de0: locked=0, pgops=0x603158d0, npages=0, refs=380
  PAGES <pg,offset>:

VNODE flags 81<ROOT,VLOCKSWORK>
mp 0x6483d000 numoutput 0 size 0x200
data 0x70e93de4 usecount 380 writecount 0 holdcnt 1 numoutput 0
tag VT_UFS(1) type VDIR(2) mount 0x6483d000 typedata 0x0
clean bufs:
 bp 0x664d9114
  vp 0x70e91de0 lblkno 0x0 blkno 0xb40 dev 0x1305
  error 0 flags 0x120200<DONE,CACHE,READ>
  bufsize 0x400 bcount 0x400 resid 0x0
  data 0x70563000 saveaddr 0x0 dep 0x0
  iodone 0x0
dirty bufs:

db> x/bx 70e91de0,0t168
0x70e91de0:     0   0   0   0   d0  58  31  60  0   0   0   0   e8  1d  e9  70
0x70e91df0:     0   0   0   0   7c  1   0   0   0   2   0   0   0   0   0   0
0x70e91e00:     81  0   0   0   0   0   0   0   0   0   0   0   1   0   0   0
0x70e91e10:     0   d0  83  64  0   d9  c8  63  90  8c  84  71  70  8d  84  71
0x70e91e20:     38  1d  e9  70  c8  1e  e9  70  14  91  4d  66  0   0   0   0
0x70e91e30:     0   0   0   0   0   0   0   0   ec  97  d8  75  0   0   0   0
0x70e91e40:     0   0   0   0   0   0   0   0   2   0   0   0   1   0   0   0
0x70e91e50:     0   0   0   0   0   4   8   0   0   0   0   0   1   0   0   0
0x70e91e60:     7b  1   0   0   49  f0  2d  60  7   73  0   0   1   0   0   0
0x70e91e70:     14  0   0   0   0   0   0   0   0   0   0   0   50  1e  e9  70
0x70e91e80:     e4  3d  e9  70  0   0   0   0

/* look at the inode */

db> x/bx 70e93de4,0t192
0x70e93de4:     d4  86  2c  60  0   0   0   0   0   0   0   0   0   0   0   0
0x70e93df4:     0   0   0   0   0   0   0   0   b9  cc  2d  60  ff  ff  ff  ff
0x70e93e04:     0   0   0   0   8   0   0   0   0   0   0   0   0   0   0   0
0x70e93e14:     0   0   0   0   0   0   0   0   d8  f4  11  83  0   0   0   0
0x70e93e24:     0   0   0   0   e0  1d  e9  70  0   52  82  64  60  13  e9  70
0x70e93e34:     0   0   0   0   5   13  0   0   2   0   0   0   0   a0  82  64
0x70e93e44:     0   0   0   0   0   0   0   0   0   0   0   0   fa  52  49  b8
0x70e93e54:     51  e   b3  43  0   0   0   0   0   0   0   0   0   0   0   0
0x70e93e64:     0   0   0   0   ac  0   0   0   10  0   0   0   15  0   0   0
0x70e93e74:     0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
0x70e93e84:     ed  41  15  0   0   2   0   0   0   0   0   0   0   0   0   0
0x70e93e94:     8d  0   7c  0   0   0   0   0   0   0   0   0   0   0   0   0

------------------------------
look at the lock (70e91e50)
------------------------------
db> x/bx 70e91e50,0t44
0x70e91e50:     0   0   0   0   0   4   8   0   0   0   0   0   1   0   0   0
0x70e91e60:     7b  1   0   0   49  f0  2d  60  7   73  0   0   1   0   0   0
0x70e91e70:     14  0   0   0   0   0   0   0   0   0   0   0

/* 0x70e91de0 + offsetof(struct vnode, v_vnlock) == 70e91e7c */

db> x/lx 70e91e7c
0x70e91e7c:     70e91e50

/* 0x70e91e50+offsetof(struct lock, lk_lockholder) == 70e91e68 */
db> x/lu 70e91e68
0x70e91e68:     29447

db> trace/t 0t29447
trace: pid 29447 swapped out

/* Ugh! */

>How-To-Repeat:

not known.

>Fix:

N/A.