Subject: 2.0G -current MP debug info ..
To: None <port-alpha@netbsd.org>
From: Stephen M. Jones <smj@cirr.com>
List: port-alpha
Date: 07/19/2004 14:34:30
I hope that this is helpful (DIAGNOSTIC and DEBUG="-g" options set).  
The hang occured an NFS client went down for a few minutes and
came back up.  Within about 10 minutes of that, the machine hung.
I've also included the last 'top' and 'systat vmstat'
output.  There were 44979 active vnodes at the time of the hang.
I was unable to take a dump.

db{0}> trace
cpu_Debugger() at netbsd:cpu_Debugger+0x4
panic() at netbsd:panic+0x1e8
console_restart() at netbsd:console_restart+0x74
XentRestart() at netbsd:XentRestart+0x90
--- console restart (from ipl 3) ---
wakeup() at netbsd:wakeup+0x68
nfsrv_wakenfsd() at netbsd:nfsrv_wakenfsd+0xc8
nfsrv_rcv() at netbsd:nfsrv_rcv+0x138
sowakeup() at netbsd:sowakeup+0xd8
udp4_sendup() at netbsd:udp4_sendup+0x14c
udp4_realinput() at netbsd:udp4_realinput+0x2ac
udp_input() at netbsd:udp_input+0x2ec
ip_input() at netbsd:ip_input+0xaa4
ipintr() at netbsd:ipintr+0xa0
netintr() at netbsd:netintr+0x88
softintr_dispatch() at netbsd:softintr_dispatch+0x194
exception_return() at netbsd:exception_return+0x7c
--- root of call graph ---

 PID          COMMAND      STRUCT PROC *            UAREA *     VMSPACE/VM_MAP
 12030             dd 0xfffffc000531ac70 0xfffffe000e24a000 0xfffffc00049feb08
 3215          systat 0xfffffc000531aee8 0xfffffe000e25a000 0xfffffc00049ff088
 11770            ksh 0xfffffc000531b650 0xfffffe000e26e000 0xfffffc00049ff4a8
 2991             top 0xfffffc000531a9f8 0xfffffe000e266000 0xfffffc00049ffa28
 11753            ksh 0xfffffc000531bb40 0xfffffe000e25e000 0xfffffc00049ff8c8
 7784         telnetd 0xfffffc000531a780 0xfffffe000e256000 0xfffffc00049fedc8
 2848             ksh 0xfffffc000531b3d8 0xfffffe000e252000 0xfffffc00049ff348
 7077             ksh 0xfffffc000531b160 0xfffffe000e246000 0xfffffc00049ff1e8
 7075         telnetd 0xfffffc0001c593c0 0xfffffe000e23e000 0xfffffc0001c642c0
 2963       mount_mfs 0xfffffc000531b8c8 0xfffffe000e26a000 0xfffffc00049ff608
 4581             ksh 0xfffffc0001c59638 0xfffffe000e286000 0xfffffc0001c64b00
 11434            ksh 0xfffffc00049cdb38 0xfffffe000ded6000 0xfffffc0001c64160
 7327        upclient 0xfffffc0006624c78 0xfffffe000e27a000 0xfffffc0034c32010
 394            inetd 0xfffffc000531a018 0xfffffe000deda000 0xfffffc00049fe6e8
 337            timed 0xfffffc00049cd648 0xfffffe000e24e000 0xfffffc00049fe848
 102        rpc.lockd 0xfffffc00049cd8c0 0xfffffe000e242000 0xfffffc00049fe588
 100        rpc.statd 0xfffffc00025a8770 0xfffffe000e1e6000 0xfffffc00049fe428
 98              nfsd 0xfffffc00049cd3d0 0xfffffe000e23a000 0xfffffc00049fe2c8
 97              nfsd 0xfffffc00049cd158 0xfffffe000e236000 0xfffffc00049fe168
 96              nfsd 0xfffffc00049ccee0 0xfffffe000e232000 0xfffffc00049fe008
 247             nfsd 0xfffffc00049ccc68 0xfffffe000e22e000 0xfffffc0001c65e40
 220             nfsd 0xfffffc00049cc9f0 0xfffffe000e22a000 0xfffffc0001c65ce0
 253             nfsd 0xfffffc00049cc778 0xfffffe000e226000 0xfffffc0001c65b80
 211             nfsd 0xfffffc00049cc500 0xfffffe000e222000 0xfffffc0001c65a20
 241             nfsd 0xfffffc00049cc288 0xfffffe000e21e000 0xfffffc0001c658c0
 240             nfsd 0xfffffc00049cc010 0xfffffe000e21a000 0xfffffc0001c65760
 246             nfsd 0xfffffc00025a9b30 0xfffffe000e216000 0xfffffc0001c65600
 237             nfsd 0xfffffc00025a98b8 0xfffffe000e212000 0xfffffc0001c654a0
 239             nfsd 0xfffffc00025a9640 0xfffffe000e20e000 0xfffffc0001c65340
 238             nfsd 0xfffffc00025a93c8 0xfffffe000e20a000 0xfffffc0001c651e0
 218             nfsd 0xfffffc00025a9150 0xfffffe000e206000 0xfffffc0001c65080
 242             nfsd 0xfffffc00025a8ed8 0xfffffe000e202000 0xfffffc0001c64f20
 217             nfsd 0xfffffc00025a8c60 0xfffffe000e1f6000 0xfffffc0001c64dc0
 235             nfsd 0xfffffc00025a89e8 0xfffffe000e1ea000 0xfffffc0001c64c60
 236             nfsd 0xfffffc0001c598b0 0xfffffe000e1f2000 0xfffffc0001c646e0
 212           mountd 0xfffffc00025a84f8 0xfffffe000e1fe000 0xfffffc0001c649a0
 184           ypbind 0xfffffc00025a8280 0xfffffe000e1fa000 0xfffffc0001c64840
 158          rpcbind 0xfffffc00025a8008 0xfffffe000e1ee000 0xfffffc0001c64580
 127          syslogd 0xfffffc0001c59b28 0xfffffe000dede000 0xfffffc0001c64420
 8           aiodoned 0xfffffc0001c59148 0xfffffe000ded2000 0xfffffc0000623738
 7            ioflush 0xfffffc0001c58ed0 0xfffffe000dece000 0xfffffc0000623738
>6         pagedaemon 0xfffffc0001c58c58 0xfffffe000deca000 0xfffffc0000623738
 5            atabus0 0xfffffc0001c589e0 0xfffffe000dec6000 0xfffffc0000623738
 4           scsibus1 0xfffffc0001c58768 0xfffffe000dec2000 0xfffffc0000623738
 3         isp0:fc_th 0xfffffc0001c584f0 0xfffffe000debe000 0xfffffc0000623738
 2           scsibus0 0xfffffc0001c58278 0xfffffe000deba000 0xfffffc0000623738
 1               init 0xfffffc0001c58000 0xfffffe000deb6000 0xfffffc0001c64000
 0            swapper 0xfffffc0000623a58 0xfffffc0000200000 0xfffffc0000623738
db{0}> ps/l
 PID         LID S     FLAGS       STRUCT LWP *            UAREA * WAIT
 12030         1 3       0x4 0xfffffc0004a325a8 0xfffffe000e24a000 biowait
 3215          1 3      0x84 0xfffffc0004a32e18 0xfffffe000e25a000 ttyin
 11770         1 3      0x80 0xfffffc0004a32c38 0xfffffe000e26e000 pause
 2991          1 3      0x84 0xfffffc0004a330e8 0xfffffe000e266000 poll
 11753         1 3      0x80 0xfffffc0004a32f08 0xfffffe000e25e000 pause
 7784          1 3      0x84 0xfffffc0004a32878 0xfffffe000e256000 poll
 2848          1 3      0x80 0xfffffc0004a32b48 0xfffffe000e252000 pause
 7077          1 3      0x80 0xfffffc0004a32a58 0xfffffe000e246000 pause
 7075          1 3      0x84 0xfffffc0004a32968 0xfffffe000e23e000 poll
 2963          1 2       0x4 0xfffffc0004a32008 0xfffffe000e26a000
 4581          1 3      0x80 0xfffffc0001c68870 0xfffffe000e286000 ttyin
 11434         1 3      0x80 0xfffffc0001c68780 0xfffffe000ded6000 pause
 7327          1 3      0x80 0xfffffc0004a332c8 0xfffffe000e27a000 pause
 394           1 3      0x80 0xfffffc0004a321e8 0xfffffe000deda000 kqread
 337           1 3      0x80 0xfffffc0004a322d8 0xfffffe000e24e000 poll
 102           1 3      0x80 0xfffffc0004a320f8 0xfffffe000e242000 select
 100           1 3      0x84 0xfffffc0001c68e10 0xfffffe000e1e6000 select
 98            1 3      0x84 0xfffffc0001c69e00 0xfffffe000e23a000 nfsd
 97            1 3      0x84 0xfffffc0001c69d10 0xfffffe000e236000 nfsd
 96            1 3      0x84 0xfffffc0001c69c20 0xfffffe000e232000 nfsd
 247           1 3      0x84 0xfffffc0001c69b30 0xfffffe000e22e000 nfsd
 220           1 3      0x84 0xfffffc0001c69a40 0xfffffe000e22a000 nfsd
 253           1 3      0x84 0xfffffc0001c69950 0xfffffe000e226000 nfsd
 211           1 3      0x84 0xfffffc0001c69860 0xfffffe000e222000 nfsd
 241           1 3      0x84 0xfffffc0001c69770 0xfffffe000e21e000 nfsd
 240           1 3      0x84 0xfffffc0001c69680 0xfffffe000e21a000 nfsd
 246           1 3      0x84 0xfffffc0001c69590 0xfffffe000e216000 nfsd
 237           1 3      0x84 0xfffffc0001c694a0 0xfffffe000e212000 nfsd
 239           1 3      0x84 0xfffffc0001c693b0 0xfffffe000e20e000 nfsd
 238           1 3      0x84 0xfffffc0001c692c0 0xfffffe000e20a000 nfsd
 218           1 3      0x84 0xfffffc0001c691d0 0xfffffe000e206000 nfsd
 242           1 3      0x84 0xfffffc0001c690e0 0xfffffe000e202000 nfsd
 217           1 3      0x84 0xfffffc0001c68ff0 0xfffffe000e1f6000 nfsd
 235           1 3      0x84 0xfffffc0001c68f00 0xfffffe000e1ea000 nfsd
 236           1 3      0x84 0xfffffc0001c68b40 0xfffffe000e1f2000 nfsd
 212           1 3      0x84 0xfffffc0001c68d20 0xfffffe000e1fe000 select
 184           1 3      0x84 0xfffffc0001c68c30 0xfffffe000e1fa000 select
 158           1 3      0x80 0xfffffc0001c68a50 0xfffffe000e1ee000 poll
 127           1 3      0x80 0xfffffc0001c68960 0xfffffe000dede000 poll
 8             1 3       0x4 0xfffffc0001c68690 0xfffffe000ded2000 aiodoned
 7             1 3       0x4 0xfffffc0001c685a0 0xfffffe000dece000 syncer
>6         >   1 7       0x4 0xfffffc0001c684b0 0xfffffe000deca000
 5             1 3       0x4 0xfffffc0001c683c0 0xfffffe000dec6000 atath
 4             1 3       0x4 0xfffffc0001c682d0 0xfffffe000dec2000 sccomp
 3             1 3       0x4 0xfffffc0001c681e0 0xfffffe000debe000 fcclnup
 2             1 3       0x4 0xfffffc0001c680f0 0xfffffe000deba000 sccomp
 1             1 3      0x80 0xfffffc0001c68000 0xfffffe000deb6000 wait
 0             1 3       0x4 0xfffffc0000623ce0 0xfffffc0000200000 scheduler
db{0}> ps/n
 PID           PPID     PGRP        UID S   FLAGS LWPS          COMMAND    WAIT
 12030        11770    11770          0 2  0x4002    1               dd biowait
 3215          2848     3215          0 2  0x4102    1           systat   ttyin
 11770         4581    11770          0 2  0x4002    1              ksh   pause
 2991         11753     2991        100 2  0x4002    1              top    poll
 11753         7784    11753        100 2  0x4002    1              ksh   pause
 7784           394      394          0 2  0x4000    1          telnetd    poll
 2848          7077     2848          0 2  0x4002    1              ksh   pause
 7077          7075     7077        100 2  0x4002    1              ksh   pause
 7075           394      394          0 2  0x4000    1          telnetd    poll
 2963             1     2963          0 2       0    1        mount_mfs
 4581         11434     4581          0 2  0x4002    1              ksh   ttyin
 11434            1    11434        100 2  0x4002    1              ksh   pause
 7327             1     7435      32767 2     0x2    1         upclient   pause
 394              1      394          0 2       0    1            inetd  kqread
 337              1      337          0 2       0    1            timed    poll
 102              1      102          0 2       0    1        rpc.lockd  select
 100              1      100          0 2       0    1        rpc.statd  select
 98               1      219          0 2       0    1             nfsd    nfsd
 97               1      219          0 2       0    1             nfsd    nfsd
 96               1      219          0 2       0    1             nfsd    nfsd
 247              1      219          0 2       0    1             nfsd    nfsd
 220              1      219          0 2       0    1             nfsd    nfsd
 253              1      219          0 2       0    1             nfsd    nfsd
 211              1      219          0 2       0    1             nfsd    nfsd
 241              1      219          0 2       0    1             nfsd    nfsd
 240              1      219          0 2       0    1             nfsd    nfsd
 246              1      219          0 2       0    1             nfsd    nfsd
 237              1      219          0 2       0    1             nfsd    nfsd
 239              1      219          0 2       0    1             nfsd    nfsd
 238              1      219          0 2       0    1             nfsd    nfsd
 218              1      219          0 2       0    1             nfsd    nfsd
 242              1      219          0 2       0    1             nfsd    nfsd
 217              1      219          0 2       0    1             nfsd    nfsd
 235              1      219          0 2       0    1             nfsd    nfsd
 236              1      219          0 2       0    1             nfsd    nfsd
 212              1      212          0 2       0    1           mountd  select
 184              1      184          0 2       0    1           ypbind  select
 158              1      158          0 2       0    1          rpcbind    poll
 127              1      127          0 2       0    1          syslogd    poll
 8                0        0          0 2 0x20200    1         aiodoned aiodone
 7                0        0          0 2 0x20200    1          ioflush  syncer
>6                0        0          0 2 0x20200    1       pagedaemon
 5                0        0          0 2 0x20200    1          atabus0   atath
 4                0        0          0 2 0x20200    1         scsibus1  sccomp
 3                0        0          0 2 0x20200    1     isp0:fc_thrd fcclnup
 2                0        0          0 2 0x20200    1         scsibus0  sccomp
 1                0        1          0 2  0x4000    1             init    wait
 0               -1        0          0 2 0x20200    1          swapper schedul
db{0}> ps/w
 PID          COMMAND     EMUL  PRI UTIME STIME WAIT-MSG    WAIT-CHANNEL
 12030             dd   netbsd   17   0.0   5.2 biowait     netbsd:ddb_regp+0x9a
f7cd8
 3215          systat   netbsd   25   0.5   0.4 ttyin       netbsd:ddb_regp+0x15
c9070
 11770            ksh   netbsd   40   0.0   0.0 pause       0xfffffc003ab42c98
 2991             top   netbsd   24   1.8   1.2 poll        netbsd:selwait
 11753            ksh   netbsd   40   0.0   0.0 pause       netbsd:ddb_regp+0x25
cee08
 7784         telnetd   netbsd   24   1.0   0.7 poll        netbsd:selwait
 2848             ksh   netbsd   40   0.0   0.1 pause       0xfffffc003ab42090
 7077             ksh   netbsd   40   0.0   0.1 pause       0xfffffc002010c0c0
 7075         telnetd   netbsd   24   0.1   0.3 poll        netbsd:selwait
 2963       mount_mfs   netbsd   17   0.0  26.2
 4581             ksh   netbsd   25   0.0   0.1 ttyin       netbsd:ddb_regp+0x15
c8d40
 11434            ksh   netbsd   40   0.0   0.0 pause       netbsd:ddb_regp+0x16
31930
 7327        upclient   netbsd   40   0.1   0.9 pause       0xfffffc002010ccc8
 394            inetd   netbsd   24   0.0   1.1 kqread      netbsd:ddb_regp+0x56
3ed28
 337            timed   netbsd   24   0.0   0.1 poll        netbsd:selwait
 102        rpc.lockd   netbsd   24   0.0   0.1 select      netbsd:selwait
 100        rpc.statd   netbsd   24   0.0   0.2 select      netbsd:selwait
 98              nfsd   netbsd   24   0.0 813.7 nfsd        0xfffffe00004e9000
 97              nfsd   netbsd   24   0.0 801.1 nfsd        0xfffffe00004e9200
 96              nfsd   netbsd   24   0.0 783.0 nfsd        0xfffffe00004e9400
 247             nfsd   netbsd   24   0.0 840.6 nfsd        0xfffffe00004e9600
 220             nfsd   netbsd   24   0.0 806.0 nfsd        0xfffffe00004e9800
 253             nfsd   netbsd   24   0.0 794.0 nfsd        0xfffffe00004e9a00
 211             nfsd   netbsd   24   0.0 792.9 nfsd        0xfffffe00004e9c00
 241             nfsd   netbsd   24   0.0 829.5 nfsd        0xfffffe00004e9e00
 240             nfsd   netbsd   24   0.0 794.2 nfsd        0xfffffe0000011600
 246             nfsd   netbsd   24   0.0 802.9 nfsd        0xfffffe0000490600
 237             nfsd   netbsd   24   0.0 802.9 nfsd        0xfffffe0000490800
 239             nfsd   netbsd   24   0.0 807.0 nfsd        0xfffffe0000491400
 238             nfsd   netbsd   24   0.0 818.2 nfsd        0xfffffe0000490a00
 218             nfsd   netbsd   24   0.0 795.1 nfsd        0xfffffe0000490e00
 242             nfsd   netbsd   24   0.0 826.3 nfsd        0xfffffe0000491600
 217             nfsd   netbsd   24   0.0 810.5 nfsd        0xfffffe0000491000
 235             nfsd   netbsd   24   0.0 799.0 nfsd        0xfffffe0000491200
 236             nfsd   netbsd   24   0.0 795.4 nfsd        0xfffffe0000490c00
 212           mountd   netbsd   24   0.0   0.2 select      netbsd:selwait
 184           ypbind   netbsd   24   0.6  12.7 select      netbsd:selwait
 158          rpcbind   netbsd   24   0.2   2.8 poll        netbsd:selwait
 127          syslogd   netbsd   24   0.0   1.1 poll        netbsd:selwait
 8           aiodoned   netbsd    4   0.0 567.7 aiodoned    netbsd:uvm+0x54
 7            ioflush   netbsd   40   0.0 188.4 syncer      netbsd:rushjob
>6         pagedaemon   netbsd    4   0.0  10.2
 5            atabus0   netbsd   16   0.0   0.0 atath       0xfffffe000005f6f0
 4           scsibus1   netbsd   16   0.0   0.0 sccomp      0xfffffe000006c130
 3         isp0:fc_thrd   netbsd   16   0.0   0.0 fcclnup     0xfffffe000006c300

 2           scsibus0   netbsd   16   0.0   1.2 sccomp      0xfffffe0000068130
 1               init   netbsd   32   0.0   0.0 wait        netbsd:ddb_regp+0x16
28d28
 0            swapper   netbsd    4   0.0   1.5 scheduler   netbsd:proc0

load averages:  3.60,  3.94,  4.85                                        19:02:27
48 processes:  47 sleeping, 1 on processor
CPU0 states:  0.0% user,  0.0% nice,  1.2% system,  4.2% interrupt, 94.5% idle
CPU1 states:  0.0% user,  0.0% nice, 12.4% system,  0.0% interrupt, 87.6% idle
Memory: 503M Act, 252M Inact, 552K Wired, 3624K Exec, 356M File, 1328K Free
Swap: 1028M Total, 455M Used, 572M Free

  PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
  238 root       2    0    80K  584K nfsd/1    13:42  9.72%  9.72% nfsd
 2963 root      -5    0   216K  393M biowai/1   0:26  0.05%  0.05% mount_mfs
12030 root      -5    0  2096K 1688K biowai/0   0:05  0.05%  0.05% dd
  247 root       2    0    80K  584K nfsd/0    14:04  0.00%  0.00% nfsd
  241 root       2    0    80K  584K nfsd/1    13:53  0.00%  0.00% nfsd
  242 root       2    0    80K  584K nfsd/0    13:50  0.00%  0.00% nfsd
   98 root       2    0    80K  584K nfsd/1    13:37  0.00%  0.00% nfsd
  217 root       2    0    80K  584K nfsd/1    13:34  0.00%  0.00% nfsd
  239 root       2    0    80K  584K nfsd/1    13:31  0.00%  0.00% nfsd
  220 root       2    0    80K  584K nfsd/0    13:30  0.00%  0.00% nfsd
  237 root       2    0    80K  584K nfsd/1    13:27  0.00%  0.00% nfsd
  246 root       2    0    80K  584K nfsd/1    13:27  0.00%  0.00% nfsd
   97 root       2    0    80K  584K nfsd/0    13:25  0.00%  0.00% nfsd
  235 root       2    0    80K  584K nfsd/1    13:23  0.00%  0.00% nfsd
  218 root       2    0    80K  584K nfsd/1    13:19  0.00%  0.00% nfsd
  236 root       2    0    80K  584K nfsd/1    13:19  0.00%  0.00% nfsd
  253 root       2    0    80K  584K nfsd/1    13:18  0.00%  0.00% nfsd
  240 root       2    0    80K  584K nfsd/0    13:18  0.00%  0.00% nfsd
  211 root       2    0    80K  584K nfsd/1    13:17  0.00%  0.00% nfsd
   96 root       2    0    80K  584K nfsd/1    13:07  0.00%  0.00% nfsd
    8 root     -18    0     0K   26M aiodon/1   9:30  0.00%  0.00% [aiodoned]
    7 root      18    0     0K   26M syncer/1   3:09  0.00%  0.00% [ioflush]
  184 root       2    0   320K  904K select/1   0:14  0.00%  0.00% ypbind
    6 root     -18    0     0K   26M pgdaem/0   0:07  0.00%  0.00% [pagedaemon]
 2991 smj       28    0   488K 1296K CPU/1      0:03  0.00%  0.00% top
  158 root       2    0   432K    0K poll/1     0:03  0.00%  0.00% <rpcbind>
 7327 nobody    18    0   192K    0K pause/0    0:01  0.00%  0.00% <upclient>
 7784 root       2    0   232K 1760K poll/0     0:01  0.00%  0.00% telnetd
  394 root       2    0   336K    0K kqread/1   0:01  0.00%  0.00% <inetd>
  127 root       2    0   232K    0K poll/0     0:01  0.00%  0.00% <syslogd>
    2 root      -6    0     0K   26M sccomp/1   0:01  0.00%  0.00% [scsibus0]
    0 root     -18    0     0K   26M schedu/1   0:01  0.00%  0.00% [swapper]
 7077 smj       18    0   384K    0K pause/0    0:00  0.00%  0.00% <ksh>
11753 smj       18    0   384K    0K pause/0    0:00  0.00%  0.00% <ksh>
11434 smj       18    0   376K    0K pause/1    0:00  0.00%  0.00% <ksh>
 2848 root      18    0   368K    0K pause/0    0:00  0.00%  0.00% <ksh>
11770 root      18    0   360K    0K pause/1    0:00  0.00%  0.00% <ksh>
    1 root      10    0   120K    0K wait/1     0:00  0.00%  0.00% <init>
 3215 root       3    0   632K 1496K ttyin/1    0:00  0.00%  0.00% systat
 4581 root       3    0   360K    0K ttyin/1    0:00  0.00%  0.00% <ksh>
 7075 root       2    0   232K 1760K poll/0     0:00  0.00%  0.00% telnetd
  212 root       2    0   528K  392K select/1   0:00  0.00%  0.00% mountd
  100 root       2    0   320K  296K select/1   0:00  0.00%  0.00% rpc.statd
  337 root       2    0   472K    0K poll/0     0:00  0.00%  0.00% <timed>
  102 root       2    0   344K    0K select/1   0:00  0.00%  0.00% <rpc.lockd>
    3 root      -6    0     0K   26M fcclnu/0   0:00  0.00%  0.00% [isp0:fc_thr
    4 root      -6    0     0K   26M sccomp/0   0:00  0.00%  0.00% [scsibus1]
    5 root      -6    0     0K   26M atath/0    0:00  0.00%  0.00% [atabus0]

    3 users    Load  3.60  3.94  4.85                  Mon Jul 19 19:02:28

Proc:r  d  s  w    Csw   Trp   Sys  Int  Sof   Flt            PAGING   SWAPPING
        2  8       873             6812 3930    62            in  out   in  out
                                                      ops     36    7
   6.8% Sy   0.1% Us   0.0% Ni   2.1% In  91.0% Id    pages        47
|    |    |    |    |    |    |    |    |    |    |
===%%                                                                     forks
                                                                          fkppw
          memory totals (in kB)            15428 Interrupts               fksvm
         real   virtual    free                  soft serial              pwait
Active 515976    982312     512             3899 soft net              36 relck
All   1028384   1494720  586424               28 soft clock            36 rlkok
                                            1024 cpu0 clock               noram
Namei         Sys-cache     Proc-cache      4909 cpu0 device              ndcpy
    Calls     hits    %     hits     %      2140 cpu0 ipi                 fltcp
        6        6  100                     2140 shootdown ipi            zfod
                                                 cpu0 imb ipi             cow
Disks:   md0   sd0                               synch fpu ipi         32 fmin
 seeks                                           discard fpu ipi       42 ftarg
 xfers         129                          1024 cpu1 clock         32265 itarg
 bytes       5863K                           132 cpu1 ipi              69 wired
 %busy        99.8                             1 microset ipi             pdfre
                                             131 shootdown ipi        688 pdscn
                                                 cpu1 imb ipi
                                                 synch fpu ipi
                                                 discard fpu ipi
                                                 isa irq 4
                                                 cpu1 ast ipi