NetBSD-Bugs archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx network interfaces in use



The following reply was made to PR port-amd64/53155; it has been noted by GNATS.

From: Havard Eidnes <he%NetBSD.org@localhost>
To: gnats-bugs%NetBSD.org@localhost
Cc: 
Subject: Re: port-amd64/53155: OS wedges after <12h uptime when >2 bnx
 network interfaces in use
Date: Wed, 04 Apr 2018 16:24:58 +0200 (CEST)

 Hi,
 
 a follow-up with a new hang with similar symptoms.
 
 This time fewer processes in "tstile" and a few in biowait.
 Doing "c" once and waiting a few seconds and collecting new info
 appears to indicate it's not making any progress.
 
 Again, edited DDB console log follows.
 
 fatal breakpoint trap in supervisor mode
 trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0x767a01b11cb0
  ilevel 0x8 rsp 0xffff80008f69bc88
 curlwp 0xfffffe822f6fd440 pid 0.6 lowest kstack 0xffff80008f6972c0
 Stopped in pid 0.6 (system) at  netbsd:breakpoint+0x5:  leave
 db{0}: trace
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 bus_space_read_2() at netbsd:bus_space_read_2+0x10
 uhci_intr() at netbsd:uhci_intr+0x8f
 intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
 Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}: c
 fatal breakpoint trap in supervisor mode
 trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0x767a01b11cb0
  ilevel 0x8 rsp 0xffff80008f69bc88
 curlwp 0xfffffe822f6fd440 pid 0.6 lowest kstack 0xffff80008f6972c0
 Stopped in pid 0.6 (system) at  netbsd:breakpoint+0x5:  leave
 db{0}: tra
 breakpoint() at netbsd:breakpoint+0x5
 comintr() at netbsd:comintr+0x562
 Xhandle_ioapic_edge13() at netbsd:Xhandle_ioapic_edge13+0x66
 --- interrupt ---
 bus_space_read_2() at netbsd:bus_space_read_2+0x10
 uhci_intr() at netbsd:uhci_intr+0x8f
 intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x1d
 Xhandle_ioapic_level5() at netbsd:Xhandle_ioapic_level5+0x6a
 --- interrupt ---
 Xspllower() at netbsd:Xspllower+0xe
 DDB lost frame for netbsd:Xsoftintr+0x4f, trying 0xffff80008f69bff0
 Xsoftintr() at netbsd:Xsoftintr+0x4f
 --- interrupt ---
 0:
 db{0}: ps
 PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 5868     1 3   5         0   fffffe8215553960           postdrop biowait
 11951    1 3   5        80   fffffe81da398080           sendmail netio
 6001     1 3   7         0   fffffe82180044a0             vmstat biowait
 9372     1 3   2         0   fffffe81f0125420               tail tstile
 8217     1 3   6         0   fffffe81efc4d320               tail tstile
 9862     1 3   3        80   fffffe81d8509060               cron wait
 4632     1 3   0        80   fffffe822b3bb080                ssh select
 4405     2 3   4        80   fffffe8224c45b40             expect select
 4405     1 3   0        80   fffffe821f14e540             expect parked
 4495     1 3   0        80   fffffe8226a1e8a0                 sh wait
 9356     1 3   5        80   fffffe821b5d0200               perl pipe_rd
 6548     1 3   5        80   fffffe8221863a60                 sh wait
 10228:   2 7   0         0   fffffe81d9f12440             expect
 10228    1 3   7        80   fffffe820e8f9160             expect parked
 7679     1 3   2        80   fffffe822c143460                 sh wait
 12514    1 3   2        80   fffffe821daaa100               perl pipe_rd
 10055    1 3   2        80   fffffe82140c9140                 sh wait
 11571    1 3   0        80   fffffe8221215460                ssh select
 7806     1 3   4        80   fffffe82205b9280                 sh wait
 9266     1 3   0        80   fffffe8221215880                ssh select
 10356    1 3   5        80   fffffe8229b8e760                 sh wait
 7934     1 3   0        80   fffffe8225e1f6e0                ssh select
 10592    1 3   1        80   fffffe82170dc500                 sh wait
 9681     1 3   2        80   fffffe820e8f9580               perl wait
 9305     1 3   1        80   fffffe8219f67b60                 sh wait
 9908     1 3   0        80   fffffe8225fad840                ssh select
 8779     1 3   0        80   fffffe82205b96a0                 sh wait
 6856     1 3   0        80   fffffe8215553120                ssh select
 8803     1 3   1        80   fffffe81eb91a620                 sh wait
 8994     1 3   0        80   fffffe81d9f12860                ssh select
 10470    1 3   6        80   fffffe8225fad420                 sh wait
 9662     1 3   6        80   fffffe821a15d8e0               perl wait
 5076     1 3   5        80   fffffe82140c9980                 sh wait
 7123     1 3   7         0   fffffe81f997ca80               perl biowait
 8956     1 3   5        80   fffffe8225f1c920                 sh wait
 9975     1 3   0        80   fffffe81f0b356e0                ssh select
 5270     1 3   2        80   fffffe822b3bb4a0                 sh wait
 8794     1 3   0        80   fffffe822b9d6760                ssh select
 5168     1 3   2        80   fffffe8215553540                 sh wait
 8565     1 3   0        80   fffffe821951e020                ssh select
 8522     1 3   0        80   fffffe822a8f4100                 sh wait
 8946     1 3   0        80   fffffe820dddda20                ssh select
 8677     1 3   5        80   fffffe822d94a560                 sh wait
 6875     1 3   0        80   fffffe821951e440                ssh select
 8644     1 3   7        80   fffffe821d24b660                 sh wait
 7694     1 3   0        80   fffffe82251dfb20                ssh select
 7939     1 3   3        80   fffffe820354a5a0                 sh wait
 5953     1 3   5        80   fffffe81f997c240                ssh select
 5952     1 3   6        80   fffffe8224c45720                 sh wait
 7773     1 3   0        80   fffffe822140b680                ssh select
 6643     1 3   7        80   fffffe822c5ab680                 sh wait
 6757     1 3   0        80   fffffe81f0125840                ssh select
 8943     1 3   4        80   fffffe81fc3e5640                 sh wait
 7329     1 3   0        80   fffffe822d946780                ssh select
 6222     1 3   0        80   fffffe820af7d1a0                 sh wait
 6930     1 3   0        80   fffffe821a2012a0                ssh select
 6708     1 3   6        80   fffffe821a15d0a0                 sh wait
 5857     1 3   6        80   fffffe8223ee1320                ssh select
 7463     1 3   2        80   fffffe821d4af220                 sh wait
 5817     1 3   0        80   fffffe8222cfd540                ssh select
 6855     1 3   5        80   fffffe8225a53780                 sh wait
 7144     1 3   0        80   fffffe821b5d0620                ssh select
 7222     1 3   5        80   fffffe8221863640                 sh wait
 6589     1 3   0        80   fffffe820af7d5c0                ssh select
 7157     1 3   3        80   fffffe822d94a140                 sh wait
 6877     1 3   2        80   fffffe82039c65e0                ssh select
 6057     1 3   7        80   fffffe821d4af640                 sh wait
 6392     1 3   5        80   fffffe822d71a6e0                ssh select
 4924     1 3   4        80   fffffe8222a64100                 sh wait
 5490     1 3   0        80   fffffe8218a57000                ssh select
 5061     1 3   6        80   fffffe81fbd2daa0                 sh wait
 6508     1 3   0        80   fffffe82251df2e0                ssh select
 4832     1 3   6        80   fffffe81fbd2d680                 sh wait
 5990     1 3   0        80   fffffe8225e1f2c0                ssh select
 5243     1 3   2        80   fffffe820dddd600                 sh wait
 3871     1 3   5        80   fffffe8221476b20                par pause
 2696     1 3   5        80   fffffe822aad3900                 sh wait
 4142     1 3   0        80   fffffe821a62c060                 sh wait
 4763     1 3   0        80   fffffe82299ed1a0                 sh wait
 5220     1 3   0        80   fffffe8222a64520                 sh wait
 4858     1 3   3        80   fffffe821daba920               cron pipe_rd
 3551     1 3   0        80   fffffe821de8b9e0                ssh select
 4159     1 3   5        80   fffffe8226fc48e0                 sh wait
 5091     1 3   0        80   fffffe8221863220                ssh select
 3848     1 3   0        80   fffffe822c143880                 sh wait
 4429     1 3   0        80   fffffe82140c9560                ssh select
 3178     1 3   0        80   fffffe81eb91aa40                 sh wait
 4329     1 3   0        80   fffffe820354a180                ssh select
 4193     1 3   1        80   fffffe820dddd1e0                 sh wait
 3408     1 3   0        80   fffffe82251df700                ssh select
 4951     1 3   0        80   fffffe8221ea5660               bash wait
 4422     1 3   5        80   fffffe820af7d9e0               sshd select
 3919     1 3   0        80   fffffe82299aa6a0               sshd select
 3312     1 3   0        80   fffffe8229b8eb80                ssh select
 3607     1 3   7        80   fffffe8225a53ba0               bash wait
 4021     1 3   0        80   fffffe821e8b72e0               sshd select
 3876     1 3   0        80   fffffe821daba0e0               sshd select
 3633     1 3   1        80   fffffe822aad34e0                ssh select
 3679     1 3   7        80   fffffe822a8f4520               bash wait
 3250     1 3   0        80   fffffe8224c45300               sshd select
 3676     1 3   0        80   fffffe821ffb8200               sshd select
 3306     1 3   0        80   fffffe82253fc020                ssh select
 3556     1 3   2        80   fffffe821951e860                 sh wait
 2753     1 3   0        80   fffffe8221215040                ssh select
 2752     1 3   0        80   fffffe821a201ae0               bash wait
 3789     1 3   0        80   fffffe8228146640               sshd select
 2376     1 3   5        80   fffffe820e8f99a0               sshd select
 1466     1 3   7        80   fffffe8221476700               bash wait
 1782     1 3   0        80   fffffe821daba500               tcsh ttyraw
 2030     1 3   2        80   fffffe821ffb8620               sshd select
 1515     1 3   5        80   fffffe822c5abaa0               sshd select
 1238     1 3   0        80   fffffe821d4afa60                ssh select
 853      1 3   3        80   fffffe822d772580                 sh wait
 1740     1 3   0        80   fffffe821e6ec1e0                ssh select
 459      1 3   7        80   fffffe821e6ec600                 sh wait
 458      1 3   0        80   fffffe821daaa940                ssh select
 457      1 3   1        80   fffffe821e6eca20                 sh wait
 1222     1 3   0        80   fffffe821dd2b1c0                ssh select
 1093     1 3   0        80   fffffe821dd2b5e0                 sh wait
 1092     1 3   0        80   fffffe821dd2ba00                ssh select
 835      1 3   0        80   fffffe821de8b1a0                 sh wait
 1212     1 3   0        80   fffffe821d19a9c0                ssh select
 1467     1 3   0        80   fffffe821e0db160          conserver nanoslp
 1082     1 3   5        80   fffffe821e0db580                 sh wait
 1081     1 3   0        80   fffffe821e0db9a0                ssh select
 824      1 3   5        80   fffffe821ef36140                 sh wait
 1079     1 3   0        80   fffffe821ef36560                ssh select
 1206     1 3   5        80   fffffe821ef36980                 sh wait
 1203     1 3   6        80   fffffe821f14e960          conserver select
 1324     1 3   0        80   fffffe821df920c0                ssh select
 299      1 3   0        80   fffffe821df924e0                 sh wait
 298      1 3   0        80   fffffe821df92900                ssh select
 297      1 3   0        80   fffffe821e2910a0                 sh wait
 296      1 3   0        80   fffffe821e2914c0                ssh select
 1703     1 3   0        80   fffffe821e2918e0                 sh wait
 1190     1 3   0        80   fffffe821e442080                ssh select
 1061     1 3   0        80   fffffe821e4424a0                 sh wait
 1188     1 3   0        80   fffffe821e4428c0                ssh select
 1187     1 3   0        80   fffffe821e6e9060                 sh wait
 674      1 3   0        80   fffffe821e6e9480                ssh select
 289      1 3   0        80   fffffe821e6e98a0                 sh wait
 288      1 3   0        80   fffffe821ff14040                ssh select
 927      1 3   0        80   fffffe821ff14460                 sh wait
 798      1 3   0        80   fffffe821ff14880                ssh select
 1181     1 3   0        80   fffffe822129d020                 sh wait
 1052     1 3   0        80   fffffe822129d440                ssh select
 1051     1 3   0        80   fffffe822dafd120                 sh wait
 1178     1 3   0        80   fffffe821f670280                ssh select
 1177     1 3   0        80   fffffe82283bd680                 sh wait
 1176     1 3   0        80   fffffe8225e1fb00                ssh select
 1431     1 3   7        80   fffffe822129d860                 sh wait
 790      1 3   0        80   fffffe821edea000          conserver select
 1173     1 3   0        80   fffffe821edea420          conserver nanoslp
 1044     1 3   0        80   fffffe82299aaac0                ssh select
 402      1 3   7        80   fffffe822140b260                 sh wait
 400      1 3   0        80   fffffe82214d7360          conserver select
 1293     1 3   0        80   fffffe822d1cc5a0                ssh select
 1292     1 3   0        80   fffffe821f670ac0                 sh wait
 267      1 3   0        80   fffffe82214d7780                ssh select
 266      1 3   4        80   fffffe82214d7ba0                 sh wait
 1415     1 3   5        80   fffffe821f5f2760          conserver select
 1030     1 3   0        80   fffffe821f5f2b80                ssh select
 1157     1 3   3        80   fffffe82299ed9e0                 sh wait
 1156     1 3   0        80   fffffe82299ed5c0                ssh select
 1155     1 3   0        80   fffffe8229e40ae0                 sh wait
 386      1 3   0        80   fffffe821f9a1320                ssh select
 385      1 3   7        80   fffffe821f9a1740                 sh wait
 384      1 3   0        80   fffffe821f9a1b60                ssh select
 447      1 3   7        80   fffffe82211cb300                 sh wait
 830      1 3   0        80   fffffe82211cb720                ssh select
 1021     1 3   7        80   fffffe82299aa280                 sh wait
 888      1 3   0        80   fffffe82211cbb40                ssh select
 1143     1 3   1        80   fffffe82214762e0                 sh wait
 176      1 3   0        80   fffffe8220dab2c0                ssh select
 815      1 3   0        80   fffffe8220dab6e0                 sh wait
 878      1 3   0        80   fffffe8220dabb00                ssh select
 1069     1 3   0        80   fffffe8220f432a0                 sh wait
 1196     1 3   7        80   fffffe8220f436c0                ssh select
 363      1 3   5        80   fffffe8220f43ae0                 sh wait
 361      1 3   0        80   fffffe821f6706a0          conserver select
 1115     1 3   0        80   fffffe8221cb31e0                ssh select
 986      1 3   5        80   fffffe8221cb3600                 sh wait
 985      1 3   0        80   fffffe8221cb3a20                ssh select
 984      1 3   5        80   fffffe822261e1c0                 sh wait
 1239     1 3   0        80   fffffe822261e5e0                ssh select
 854      1 3   5        80   fffffe822261ea00                 sh wait
 789      1 3   0        80   fffffe822a8ba1a0                ssh select
 1108     1 3   5        80   fffffe822a8ba5c0                 sh wait
 851      1 3   0        80   fffffe822a8ba9e0                ssh select
 210      1 3   5        80   fffffe8220918180                 sh wait
 913      1 3   0        80   fffffe82209185a0                ssh select
 208      1 3   5        80   fffffe82209189c0                 sh wait
 783      1 3   0        80   fffffe8222d0a160                ssh select
 974      1 3   0        80   fffffe8222d0a580                 sh wait
 1165     1 3   0        80   fffffe8222d0a9a0                ssh select
 1356     1 3   0        80   fffffe822287f140                 sh wait
 139      1 3   0        80   fffffe822287f560                ssh select
 138      1 3   0        80   fffffe822287f980                 sh wait
 137      1 3   5        80   fffffe8222cfd120          conserver select
 168      1 3   1        80   fffffe822dafd960          conserver select
 1127     1 3   5        80   fffffe8228f78620          conserver select
 933      1 3   5        80   fffffe822dda2920              getty ttyraw
 1031     1 3   0        80   fffffe822dd49100          conserver select
 797      1 3   5        80   fffffe8225a53360          conserver select
 829      1 3   3        80   fffffe82283bd260               cron nanoslp
 1019     1 3   3         0   fffffe822bcd0a00          python2.7 biowait
 112      1 3   0        80   fffffe8229b8e340              inetd kqueue
 772      1 3   0        80   fffffe8223ee1740              nginx kqueue
 803      1 3   0        80   fffffe8223ee1b60              nginx pause
 773      1 3   5        80   fffffe8229e406c0               qmgr kqueue
 951      1 3   5        80   fffffe8228146a60             pickup kqueue
 794      1 3   5        80   fffffe8229e402a0             master kqueue
 696      1 3   0        80   fffffe822d1cc180               sshd select
 725      1 3   3        80   fffffe822d772160           tac_plus select
 686      1 3   6        80   fffffe822806d240             powerd kqueue
 659      1 3   0        80   fffffe822806d660               ntpd pause
 606      4 3   6        80   fffffe822806da80              dhcpd parked
 606      3 3   6        80   fffffe8228146220              dhcpd kqueue
 606      2 3   6        80   fffffe8228f78200              dhcpd parked
 606      1 3   0        80   fffffe822986b1e0              dhcpd parked
 383  :   1 7   7         0   fffffe8228f78a40            syslogd
 1        1 3   3        80   fffffe810f45a0a0               init wait
 0      123 3   3       200   fffffe822986ba20            npfgc-0 xchicv
 0      122 3   5       200   fffffe822d1cc9c0              ipmi0 ipmi0
 0      121 3   0       200   fffffe822dda2500            physiod physiod
 0      120 3   0       200   fffffe822dd49520           aiodoned aiodoned
 0      119 3   0       200   fffffe822dd49940            ioflush syncer
 0      118 3   0       200   fffffe822dda20e0           pgdaemon pgdaemon
 0      115 3   0       280   fffffe810f4260c0              spkr1 bellcv
 0      114 3   4       280   fffffe810f4264e0           audiomix play
 0      113 3   4       280   fffffe810ed322a0           audiorec record
 0      112 3   5       200   fffffe810f2c9720          atapibus0 sccomp
 0      110 3   0       200   fffffe810f2ba700               usb2 usbevt
 0      109 3   0       200   fffffe810f2ba2e0               usb1 usbevt
 0      108 3   0       200   fffffe810f2bab20               usb3 usbevt
 0      107 3   0       200   fffffe810f2c9300               usb0 usbevt
 0      106 3   5       200   fffffe810f2e0b60               usb4 usbevt
 0      105 3   5       200   fffffe810f426900               usb5 usbevt
 0      104 3   0       200   fffffe810f45a4c0            rt_free rt_free
 0      103 3   0       200   fffffe810f45a8e0              unpgc unpgc
 0      102 3   3       200   fffffe810f463080    key_timehandler key_timehandler
 
 0      101 3   7       200   fffffe810f4634a0    icmp6_wqinput/7 icmp6_wqinput
 0      100 3   6       200   fffffe810f4638c0    icmp6_wqinput/6 icmp6_wqinput
 0       99 3   5       200   fffffe810f3ec060    icmp6_wqinput/5 icmp6_wqinput
 0       98 3   4       200   fffffe810f3ec480    icmp6_wqinput/4 icmp6_wqinput
 0       97 3   3       200   fffffe810f3ec8a0    icmp6_wqinput/3 icmp6_wqinput
 0       96 3   2       200   fffffe810f38d040    icmp6_wqinput/2 icmp6_wqinput
 0       95 3   1       200   fffffe810f38d460    icmp6_wqinput/1 icmp6_wqinput
 0       94 3   0       200   fffffe810f2c9b40    icmp6_wqinput/0 icmp6_wqinput
 0       93 3   0       200   fffffe810f38d880          nd6_timer nd6_timer
 0       92 3   7       200   fffffe810f34e020     icmp_wqinput/7 icmp_wqinput
 0       91 3   6       200   fffffe810f34e440     icmp_wqinput/6 icmp_wqinput
 0       90 3   5       200   fffffe810f34e860     icmp_wqinput/5 icmp_wqinput
 0       89 3   4       200   fffffe810f337000     icmp_wqinput/4 icmp_wqinput
 0       88 3   3       200   fffffe810f337420     icmp_wqinput/3 icmp_wqinput
 0       87 3   2       200   fffffe810f337840     icmp_wqinput/2 icmp_wqinput
 0       86 3   1       200   fffffe810f33d360     icmp_wqinput/1 icmp_wqinput
 0       85 3   0       200   fffffe810f33d780     icmp_wqinput/0 icmp_wqinput
 0       84 3   6       200   fffffe810f33dba0           rt_timer rt_timer
 0       83 3   7       200   fffffe810f1fc2c0        vmem_rehash vmem_rehash
 0       82 3   3       200   fffffe810f2d7340          coretemp3 coretemp3
 0       81 3   7       200   fffffe810f2d7760          coretemp2 coretemp2
 0       80 3   1       200   fffffe810f2d7b80          coretemp1 coretemp1
 0       79 3   6       200   fffffe810f2e0320          coretemp0 coretemp0
 0       78 3   4       200   fffffe810f2e0740               mfi0 mfi0
 0       69 3   0       200   fffffe810f1fc6e0            atabus1 atath
 0       68 3   0       200   fffffe810f1fcb00            atabus0 atath
 0       66 3   0       200   fffffe810ed326c0           scsibus0 sccomp
 0       65 3   5       200   fffffe810ed32ae0         usbtask-dr usbtsk
 0       64 3   0       200   fffffe810eca9280         usbtask-hc usbtsk
 0       63 3   7       200   fffffe810eca96a0               bnx3 bnx3
 0       62 3   1       200   fffffe810eca9ac0               bnx2 bnx2
 0       61 3   5       200   fffffe810eb60260               bnx1 bnx1
 0       60 3   4       200   fffffe810eb60680               bnx0 bnx0
 0       59 3   5       200   fffffe810eb60aa0               ipmi ipmipoll
 0       58 3   7       200   fffffe810ea73240            xcall/7 xcall
 0       57 1   7       200   fffffe810ea73660          softser/7
 0       56 1   7       200   fffffe810ea73a80          softclk/7
 0       55 1   7       200   fffffe810ea54220          softbio/7
 0       54 1   7       200   fffffe810ea54640          softnet/7
 0       53 1   7       201   fffffe810ea54a60             idle/7
 0       52 3   6       200   fffffe810ea3d200            xcall/6 xcall
 0       51 1   6       200   fffffe810ea3d620          softser/6
 0       50 1   6       200   fffffe810ea3da40          softclk/6
 0       49 1   6       200   fffffe810ea161e0          softbio/6
 0       48 1   6       200   fffffe810ea16600          softnet/6
 0    :  47 7   6       201   fffffe810ea16a20             idle/6
 0       46 3   5       200   fffffe810e9d71c0            xcall/5 xcall
 0       45 1   5       200   fffffe810e9d75e0          softser/5
 0       44 1   5       200   fffffe810e9d7a00          softclk/5
 0       43 1   5       200   fffffe810e9d01a0          softbio/5
 0       42 1   5       200   fffffe810e9d05c0          softnet/5
 0    :  41 7   5       201   fffffe810e9d09e0             idle/5
 0       40 3   4       200   fffffe810e9a1180            xcall/4 xcall
 0       39 1   4       200   fffffe810e9a15a0          softser/4
 0       38 1   4       200   fffffe810e9a19c0          softclk/4
 0       37 1   4       200   fffffe810e97a160          softbio/4
 0       36 1   4       200   fffffe810e97a580          softnet/4
 0    :  35 7   4       201   fffffe810e97a9a0             idle/4
 0       34 3   3       200   fffffe810e943140            xcall/3 xcall
 0       33 1   3       200   fffffe810e943560          softser/3
 0       32 1   3       200   fffffe810e943980          softclk/3
 0       31 1   3       200   fffffe810e934120          softbio/3
 0       30 1   3       200   fffffe810e934540          softnet/3
 0    :  29 7   3       201   fffffe810e934960             idle/3
 0       28 3   2       200   fffffe810e90d100            xcall/2 xcall
 0       27 1   2       200   fffffe810e90d520          softser/2
 0       26 1   2       200   fffffe810e90d940          softclk/2
 0       25 1   2       200   fffffe810e8f60e0          softbio/2
 0       24 1   2       200   fffffe810e8f6500          softnet/2
 0    :  23 7   2       201   fffffe810e8f6920             idle/2
 0       22 3   1       200   fffffe810e8c50c0            xcall/1 xcall
 0       21 1   1       200   fffffe810e8c54e0          softser/1
 0       20 1   1       200   fffffe810e8c5900          softclk/1
 0       19 1   1       200   fffffe810e8a90a0          softbio/1
 0       18 1   1       200   fffffe810e8a94c0          softnet/1
 0    :  17 7   1       201   fffffe810e8a98e0             idle/1
 0       16 3   5       200   fffffe822de9e080           lnxsyswq lnxsyswq
 0       15 3   5       200   fffffe822de9e4a0             sysmon smtaskq
 0       14 3   2       200   fffffe822de9e8c0         pmfsuspend pmfsuspend
 0       13 3   4       200   fffffe822e2c5060           pmfevent pmfevent
 0       12 3   0       200   fffffe822e2c5480         sopendfree sopendfr
 0       11 3   6       200   fffffe822e2c58a0           nfssilly nfssilly
 0       10 3   3       200   fffffe822f6dc040            cachegc cachegc
 0        9 3   3       200   fffffe822f6dc460             vdrain vdrain
 0        8 3   3       200   fffffe822f6dc880          modunload mod_unld
 0        7 3   0       200   fffffe822f6fd020            xcall/0 xcall
 0    :   6 7   0       200   fffffe822f6fd440          softser/0
 0    :   5 7   0       200   fffffe822f6fd860          softclk/0
 0        4 1   0       200   fffffe822f719000          softbio/0
 0        3 1   0       200   fffffe822f719420          softnet/0
 0        2 1   0       201   fffffe822f719840             idle/0
 0        1 3   2       200   ffffffff81481b20            swapper uvm
 db{0}: trace/t 0t9372
 trace: pid 9372 lid 1 at 0xffff80009f6f78e0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 rw_vector_enter() at netbsd:rw_vector_enter+0x3e8
 genfs_lock() at netbsd:genfs_lock+0x3c
 VOP_LOCK() at netbsd:VOP_LOCK+0x67
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 781169a3e2ca:
 db{0}: trace/t 0t8217
 trace: pid 8217 lid 1 at 0xffff80009f6be8e0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 rw_vector_enter() at netbsd:rw_vector_enter+0x3e8
 genfs_lock() at netbsd:genfs_lock+0x3c
 VOP_LOCK() at netbsd:VOP_LOCK+0x67
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7ac9b2a3e2ca:
 db{0}: trace/t 0t6001
 trace: pid 6001 lid 1 at 0xffff80009eff7950
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 ffs_balloc() at netbsd:ffs_balloc+0x1b06
 ufs_gop_alloc() at netbsd:ufs_gop_alloc+0xa1
 ufs_balloc_range() at netbsd:ufs_balloc_range+0x136
 ffs_write() at netbsd:ffs_write+0x4d7
 VOP_WRITE() at netbsd:VOP_WRITE+0x37
 vn_write() at netbsd:vn_write+0xec
 dofilewrite() at netbsd:dofilewrite+0x97
 sys_write() at netbsd:sys_write+0x5f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 4) ---
 79de3d23e02a:
 db{0}: trace/t 0t5868
 trace: pid 5868 lid 1 at 0xffff80009f0c1a50
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 ffs_update.part.3() at netbsd:ffs_update.part.3+0x292
 ufs_makeinode() at netbsd:ufs_makeinode+0x134
 ufs_create() at netbsd:ufs_create+0x31
 VOP_CREATE() at netbsd:VOP_CREATE+0x3d
 vn_open() at netbsd:vn_open+0x351
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7d9dfe63e2ca:
 db{0}: trace/t 0t11951
 trace: pid 11951 lid 1 at 0xffff80009f7a1c80
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_timedwait_sig() at netbsd:cv_timedwait_sig+0x107
 sbwait() at netbsd:sbwait+0x69
 soreceive() at netbsd:soreceive+0x1233
 dofileread() at netbsd:dofileread+0x8f
 sys_read() at netbsd:sys_read+0x5f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 3) ---
 6fd303c3e22a:
 db{0}: trace/t 0t9862
 trace: pid 9862 lid 1 at 0xffff80009f772b60
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait_sig() at netbsd:cv_wait_sig+0xfe
 do_sys_waitid() at netbsd:do_sys_waitid+0x44e
 do_sys_wait() at netbsd:do_sys_wait+0x8f
 sys___wait450() at netbsd:sys___wait450+0x42
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 449) ---
 7cc8fa83e06a:
 db{0}: trace/t 0t10228
 trace: pid 10228 lid 2 at 0xffff80009f71de40
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 417) ---
 767a01c3e10a:
 db{0}: trace/t 0t7123
 trace: pid 7123 lid 1 at 0xffff80009f5d9c10
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 VOP_BWRITE() at netbsd:VOP_BWRITE+0x51
 ufs_dirremove() at netbsd:ufs_dirremove+0xd8
 ufs_remove() at netbsd:ufs_remove+0x142
 VOP_REMOVE() at netbsd:VOP_REMOVE+0x37
 do_sys_unlinkat() at netbsd:do_sys_unlinkat+0x237
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 10) ---
 7973de4fb2aa:
 db{0}: trace/t 0t9662
 trace: pid 9662 lid 1 at 0xffff80009f074b60
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait_sig() at netbsd:cv_wait_sig+0xfe
 do_sys_waitid() at netbsd:do_sys_waitid+0x44e
 do_sys_wait() at netbsd:do_sys_wait+0x8f
 sys___wait450() at netbsd:sys___wait450+0x42
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 449) ---
 72c419a3e06a:
 db{0}: trace/t 0t9681
 trace: pid 9681 lid 1 at 0xffff80009f0fdb60
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait_sig() at netbsd:cv_wait_sig+0xfe
 do_sys_waitid() at netbsd:do_sys_waitid+0x44e
 do_sys_wait() at netbsd:do_sys_wait+0x8f
 sys___wait450() at netbsd:sys___wait450+0x42
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 449) ---
 74946003e06a:
 db{0}: trace/t 0t1019
 trace: pid 1019 lid 1 at 0xffff80009de6e8a0
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 ffs_update.part.3() at netbsd:ffs_update.part.3+0x292
 ffs_truncate() at netbsd:ffs_truncate+0x1046
 ufs_truncate_retry() at netbsd:ufs_truncate_retry+0x79
 ufs_setattr() at netbsd:ufs_setattr+0x44c
 VOP_SETATTR() at netbsd:VOP_SETATTR+0x34
 vn_open() at netbsd:vn_open+0x26e
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 79538a23e2ca:
 db{0}: trace/t 0t383
 trace: pid 383 lid 1 at 0x1
 sleepq_block() at netbsd:sleepq_block+0x110
 fatal page fault in supervisor mode
 trap type 6 code 0 rip 0xffffffff80238eaa cs 0x8 rflags 0x10246 cr2 0x9 ilevel 0x8 rsp 0xffff80008f69afe0
 curlwp 0xfffffe822f6fd440 pid 0.6 lowest kstack 0xffff80008f6972c0
 kernel: page fault trap, code=0
 Faulted in DDB; continuing...
 db{0}: c
 fatal breakpoint trap in supervisor mode
 trap type 1 code 0 rip 0xffffffff8021dbb5 cs 0x8 rflags 0x202 cr2 0x9 ilevel 0x8 rsp 0xffff80008f69bcf8
 curlwp 0xfffffe822f6fd440 pid 0.6 lowest kstack 0xffff80008f6972c0
 Stopped in pid 0.6 (system) at  netbsd:breakpoint+0x5:  leave
 db{0}: ps
 PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
 5868     1 3   5         0   fffffe8215553960           postdrop biowait
 11951    1 3   5        80   fffffe81da398080           sendmail netio
 6001     1 3   7         0   fffffe82180044a0             vmstat biowait
 9372     1 3   2         0   fffffe81f0125420               tail tstile
 8217     1 3   6         0   fffffe81efc4d320               tail tstile
 9862     1 3   3        80   fffffe81d8509060               cron wait
 4632     1 3   0        80   fffffe822b3bb080                ssh select
 4405     2 3   4        80   fffffe8224c45b40             expect select
 4405     1 3   0        80   fffffe821f14e540             expect parked
 4495     1 3   0        80   fffffe8226a1e8a0                 sh wait
 9356     1 3   5        80   fffffe821b5d0200               perl pipe_rd
 6548     1 3   5        80   fffffe8221863a60                 sh wait
 10228:   2 7   0         0   fffffe81d9f12440             expect
 10228    1 3   7        80   fffffe820e8f9160             expect parked
 7679     1 3   2        80   fffffe822c143460                 sh wait
 12514    1 3   2        80   fffffe821daaa100               perl pipe_rd
 10055    1 3   2        80   fffffe82140c9140                 sh wait
 11571    1 3   0        80   fffffe8221215460                ssh select
 7806     1 3   4        80   fffffe82205b9280                 sh wait
 9266     1 3   0        80   fffffe8221215880                ssh select
 10356    1 3   5        80   fffffe8229b8e760                 sh wait
 7934     1 3   0        80   fffffe8225e1f6e0                ssh select
 10592    1 3   1        80   fffffe82170dc500                 sh wait
 9681     1 3   2        80   fffffe820e8f9580               perl wait
 9305     1 3   1        80   fffffe8219f67b60                 sh wait
 9908     1 3   0        80   fffffe8225fad840                ssh select
 8779     1 3   0        80   fffffe82205b96a0                 sh wait
 6856     1 3   0        80   fffffe8215553120                ssh select
 8803     1 3   1        80   fffffe81eb91a620                 sh wait
 8994     1 3   0        80   fffffe81d9f12860                ssh select
 10470    1 3   6        80   fffffe8225fad420                 sh wait
 9662     1 3   6        80   fffffe821a15d8e0               perl wait
 5076     1 3   5        80   fffffe82140c9980                 sh wait
 7123     1 3   7         0   fffffe81f997ca80               perl biowait
 8956     1 3   5        80   fffffe8225f1c920                 sh wait
 9975     1 3   0        80   fffffe81f0b356e0                ssh select
 5270     1 3   2        80   fffffe822b3bb4a0                 sh wait
 8794     1 3   0        80   fffffe822b9d6760                ssh select
 5168     1 3   2        80   fffffe8215553540                 sh wait
 8565     1 3   0        80   fffffe821951e020                ssh select
 8522     1 3   0        80   fffffe822a8f4100                 sh wait
 8946     1 3   0        80   fffffe820dddda20                ssh select
 8677     1 3   5        80   fffffe822d94a560                 sh wait
 6875     1 3   0        80   fffffe821951e440                ssh select
 8644     1 3   7        80   fffffe821d24b660                 sh wait
 7694     1 3   0        80   fffffe82251dfb20                ssh select
 7939     1 3   3        80   fffffe820354a5a0                 sh wait
 5953     1 3   5        80   fffffe81f997c240                ssh select
 5952     1 3   6        80   fffffe8224c45720                 sh wait
 7773     1 3   0        80   fffffe822140b680                ssh select
 6643     1 3   7        80   fffffe822c5ab680                 sh wait
 6757     1 3   0        80   fffffe81f0125840                ssh select
 8943     1 3   4        80   fffffe81fc3e5640                 sh wait
 7329     1 3   0        80   fffffe822d946780                ssh select
 6222     1 3   0        80   fffffe820af7d1a0                 sh wait
 6930     1 3   0        80   fffffe821a2012a0                ssh select
 6708     1 3   6        80   fffffe821a15d0a0                 sh wait
 5857     1 3   6        80   fffffe8223ee1320                ssh select
 7463     1 3   2        80   fffffe821d4af220                 sh wait
 5817     1 3   0        80   fffffe8222cfd540                ssh select
 6855     1 3   5        80   fffffe8225a53780                 sh wait
 7144     1 3   0        80   fffffe821b5d0620                ssh select
 7222     1 3   5        80   fffffe8221863640                 sh wait
 6589     1 3   0        80   fffffe820af7d5c0                ssh select
 7157     1 3   3        80   fffffe822d94a140                 sh wait
 6877     1 3   2        80   fffffe82039c65e0                ssh select
 6057     1 3   7        80   fffffe821d4af640                 sh wait
 6392     1 3   5        80   fffffe822d71a6e0                ssh select
 4924     1 3   4        80   fffffe8222a64100                 sh wait
 5490     1 3   0        80   fffffe8218a57000                ssh select
 5061     1 3   6        80   fffffe81fbd2daa0                 sh wait
 6508     1 3   0        80   fffffe82251df2e0                ssh select
 4832     1 3   6        80   fffffe81fbd2d680                 sh wait
 5990     1 3   0        80   fffffe8225e1f2c0                ssh select
 5243     1 3   2        80   fffffe820dddd600                 sh wait
 3871     1 3   5        80   fffffe8221476b20                par pause
 2696     1 3   5        80   fffffe822aad3900                 sh wait
 4142     1 3   0        80   fffffe821a62c060                 sh wait
 4763     1 3   0        80   fffffe82299ed1a0                 sh wait
 5220     1 3   0        80   fffffe8222a64520                 sh wait
 4858     1 3   3        80   fffffe821daba920               cron pipe_rd
 3551     1 3   0        80   fffffe821de8b9e0                ssh select
 4159     1 3   5        80   fffffe8226fc48e0                 sh wait
 5091     1 3   0        80   fffffe8221863220                ssh select
 3848     1 3   0        80   fffffe822c143880                 sh wait
 4429     1 3   0        80   fffffe82140c9560                ssh select
 3178     1 3   0        80   fffffe81eb91aa40                 sh wait
 4329     1 3   0        80   fffffe820354a180                ssh select
 4193     1 3   1        80   fffffe820dddd1e0                 sh wait
 3408     1 3   0        80   fffffe82251df700                ssh select
 4951     1 3   0        80   fffffe8221ea5660               bash wait
 4422     1 3   5        80   fffffe820af7d9e0               sshd select
 3919     1 3   0        80   fffffe82299aa6a0               sshd select
 3312     1 3   0        80   fffffe8229b8eb80                ssh select
 3607     1 3   7        80   fffffe8225a53ba0               bash wait
 4021     1 3   0        80   fffffe821e8b72e0               sshd select
 3876     1 3   0        80   fffffe821daba0e0               sshd select
 3633     1 3   1        80   fffffe822aad34e0                ssh select
 3679     1 3   7        80   fffffe822a8f4520               bash wait
 3250     1 3   0        80   fffffe8224c45300               sshd select
 3676     1 3   0        80   fffffe821ffb8200               sshd select
 3306     1 3   0        80   fffffe82253fc020                ssh select
 3556     1 3   2        80   fffffe821951e860                 sh wait
 2753     1 3   0        80   fffffe8221215040                ssh select
 2752     1 3   0        80   fffffe821a201ae0               bash wait
 3789     1 3   0        80   fffffe8228146640               sshd select
 2376     1 3   5        80   fffffe820e8f99a0               sshd select
 1466     1 3   7        80   fffffe8221476700               bash wait
 1782     1 3   0        80   fffffe821daba500               tcsh ttyraw
 2030     1 3   2        80   fffffe821ffb8620               sshd select
 1515     1 3   5        80   fffffe822c5abaa0               sshd select
 1238     1 3   0        80   fffffe821d4afa60                ssh select
 853      1 3   3        80   fffffe822d772580                 sh wait
 1740     1 3   0        80   fffffe821e6ec1e0                ssh select
 459      1 3   7        80   fffffe821e6ec600                 sh wait
 458      1 3   0        80   fffffe821daaa940                ssh select
 457      1 3   1        80   fffffe821e6eca20                 sh wait
 1222     1 3   0        80   fffffe821dd2b1c0                ssh select
 1093     1 3   0        80   fffffe821dd2b5e0                 sh wait
 1092     1 3   0        80   fffffe821dd2ba00                ssh select
 835      1 3   0        80   fffffe821de8b1a0                 sh wait
 1212     1 3   0        80   fffffe821d19a9c0                ssh select
 1467     1 3   0        80   fffffe821e0db160          conserver nanoslp
 1082     1 3   5        80   fffffe821e0db580                 sh wait
 1081     1 3   0        80   fffffe821e0db9a0                ssh select
 824      1 3   5        80   fffffe821ef36140                 sh wait
 1079     1 3   0        80   fffffe821ef36560                ssh select
 1206     1 3   5        80   fffffe821ef36980                 sh wait
 1203     1 3   6        80   fffffe821f14e960          conserver select
 1324     1 3   0        80   fffffe821df920c0                ssh select
 299      1 3   0        80   fffffe821df924e0                 sh wait
 298      1 3   0        80   fffffe821df92900                ssh select
 297      1 3   0        80   fffffe821e2910a0                 sh wait
 296      1 3   0        80   fffffe821e2914c0                ssh select
 1703     1 3   0        80   fffffe821e2918e0                 sh wait
 1190     1 3   0        80   fffffe821e442080                ssh select
 1061     1 3   0        80   fffffe821e4424a0                 sh wait
 1188     1 3   0        80   fffffe821e4428c0                ssh select
 1187     1 3   0        80   fffffe821e6e9060                 sh wait
 674      1 3   0        80   fffffe821e6e9480                ssh select
 289      1 3   0        80   fffffe821e6e98a0                 sh wait
 288      1 3   0        80   fffffe821ff14040                ssh select
 927      1 3   0        80   fffffe821ff14460                 sh wait
 798      1 3   0        80   fffffe821ff14880                ssh select
 1181     1 3   0        80   fffffe822129d020                 sh wait
 1052     1 3   0        80   fffffe822129d440                ssh select
 1051     1 3   0        80   fffffe822dafd120                 sh wait
 1178     1 3   0        80   fffffe821f670280                ssh select
 1177     1 3   0        80   fffffe82283bd680                 sh wait
 1176     1 3   0        80   fffffe8225e1fb00                ssh select
 1431     1 3   7        80   fffffe822129d860                 sh wait
 790      1 3   0        80   fffffe821edea000          conserver select
 1173     1 3   0        80   fffffe821edea420          conserver nanoslp
 1044     1 3   0        80   fffffe82299aaac0                ssh select
 402      1 3   7        80   fffffe822140b260                 sh wait
 400      1 3   0        80   fffffe82214d7360          conserver select
 1293     1 3   0        80   fffffe822d1cc5a0                ssh select
 1292     1 3   0        80   fffffe821f670ac0                 sh wait
 267      1 3   0        80   fffffe82214d7780                ssh select
 266      1 3   4        80   fffffe82214d7ba0                 sh wait
 1415     1 3   5        80   fffffe821f5f2760          conserver select
 1030     1 3   0        80   fffffe821f5f2b80                ssh select
 1157     1 3   3        80   fffffe82299ed9e0                 sh wait
 1156     1 3   0        80   fffffe82299ed5c0                ssh select
 1155     1 3   0        80   fffffe8229e40ae0                 sh wait
 386      1 3   0        80   fffffe821f9a1320                ssh select
 385      1 3   7        80   fffffe821f9a1740                 sh wait
 384      1 3   0        80   fffffe821f9a1b60                ssh select
 447      1 3   7        80   fffffe82211cb300                 sh wait
 830      1 3   0        80   fffffe82211cb720                ssh select
 1021     1 3   7        80   fffffe82299aa280                 sh wait
 888      1 3   0        80   fffffe82211cbb40                ssh select
 1143     1 3   1        80   fffffe82214762e0                 sh wait
 176      1 3   0        80   fffffe8220dab2c0                ssh select
 815      1 3   0        80   fffffe8220dab6e0                 sh wait
 878      1 3   0        80   fffffe8220dabb00                ssh select
 1069     1 3   0        80   fffffe8220f432a0                 sh wait
 1196     1 3   7        80   fffffe8220f436c0                ssh select
 363      1 3   5        80   fffffe8220f43ae0                 sh wait
 361      1 3   0        80   fffffe821f6706a0          conserver select
 1115     1 3   0        80   fffffe8221cb31e0                ssh select
 986      1 3   5        80   fffffe8221cb3600                 sh wait
 985      1 3   0        80   fffffe8221cb3a20                ssh select
 984      1 3   5        80   fffffe822261e1c0                 sh wait
 1239     1 3   0        80   fffffe822261e5e0                ssh select
 854      1 3   5        80   fffffe822261ea00                 sh wait
 789      1 3   0        80   fffffe822a8ba1a0                ssh select
 1108     1 3   5        80   fffffe822a8ba5c0                 sh wait
 851      1 3   0        80   fffffe822a8ba9e0                ssh select
 210      1 3   5        80   fffffe8220918180                 sh wait
 913      1 3   0        80   fffffe82209185a0                ssh select
 208      1 3   5        80   fffffe82209189c0                 sh wait
 783      1 3   0        80   fffffe8222d0a160                ssh select
 974      1 3   0        80   fffffe8222d0a580                 sh wait
 1165     1 3   0        80   fffffe8222d0a9a0                ssh select
 1356     1 3   0        80   fffffe822287f140                 sh wait
 139      1 3   0        80   fffffe822287f560                ssh select
 138      1 3   0        80   fffffe822287f980                 sh wait
 137      1 3   5        80   fffffe8222cfd120          conserver select
 168      1 3   1        80   fffffe822dafd960          conserver select
 1127     1 3   5        80   fffffe8228f78620          conserver select
 933      1 3   5        80   fffffe822dda2920              getty ttyraw
 1031     1 3   0        80   fffffe822dd49100          conserver select
 797      1 3   5        80   fffffe8225a53360          conserver select
 829      1 3   3        80   fffffe82283bd260               cron nanoslp
 1019     1 3   3         0   fffffe822bcd0a00          python2.7 biowait
 112      1 3   0        80   fffffe8229b8e340              inetd kqueue
 772      1 3   0        80   fffffe8223ee1740              nginx kqueue
 803      1 3   0        80   fffffe8223ee1b60              nginx pause
 773      1 3   5        80   fffffe8229e406c0               qmgr kqueue
 951      1 3   5        80   fffffe8228146a60             pickup kqueue
 794      1 3   5        80   fffffe8229e402a0             master kqueue
 696      1 3   0        80   fffffe822d1cc180               sshd select
 725      1 3   3        80   fffffe822d772160           tac_plus select
 686      1 3   6        80   fffffe822806d240             powerd kqueue
 659      1 3   0        80   fffffe822806d660               ntpd pause
 606      4 3   6        80   fffffe822806da80              dhcpd parked
 606      3 3   6        80   fffffe8228146220              dhcpd kqueue
 606      2 3   6        80   fffffe8228f78200              dhcpd parked
 606      1 3   0        80   fffffe822986b1e0              dhcpd parked
 383      1 2   7         0   fffffe8228f78a40            syslogd
 1        1 3   3        80   fffffe810f45a0a0               init wait
 0      123 3   3       200   fffffe822986ba20            npfgc-0 xchicv
 0      122 3   5       200   fffffe822d1cc9c0              ipmi0 ipmi0
 0      121 3   0       200   fffffe822dda2500            physiod physiod
 0      120 3   0       200   fffffe822dd49520           aiodoned aiodoned
 0      119 3   0       200   fffffe822dd49940            ioflush syncer
 0      118 3   0       200   fffffe822dda20e0           pgdaemon pgdaemon
 0      115 3   0       280   fffffe810f4260c0              spkr1 bellcv
 0      114 3   4       280   fffffe810f4264e0           audiomix play
 0      113 3   4       280   fffffe810ed322a0           audiorec record
 0      112 3   5       200   fffffe810f2c9720          atapibus0 sccomp
 0      110 3   0       200   fffffe810f2ba700               usb2 usbevt
 0      109 3   0       200   fffffe810f2ba2e0               usb1 usbevt
 0      108 3   0       200   fffffe810f2bab20               usb3 usbevt
 0      107 3   0       200   fffffe810f2c9300               usb0 usbevt
 0      106 3   5       200   fffffe810f2e0b60               usb4 usbevt
 0      105 3   5       200   fffffe810f426900               usb5 usbevt
 0      104 3   0       200   fffffe810f45a4c0            rt_free rt_free
 0      103 3   0       200   fffffe810f45a8e0              unpgc unpgc
 0      102 3   3       200   fffffe810f463080    key_timehandler key_timehandler
 
 0      101 3   7       200   fffffe810f4634a0    icmp6_wqinput/7 icmp6_wqinput
 0      100 3   6       200   fffffe810f4638c0    icmp6_wqinput/6 icmp6_wqinput
 0       99 3   5       200   fffffe810f3ec060    icmp6_wqinput/5 icmp6_wqinput
 0       98 3   4       200   fffffe810f3ec480    icmp6_wqinput/4 icmp6_wqinput
 0       97 3   3       200   fffffe810f3ec8a0    icmp6_wqinput/3 icmp6_wqinput
 0       96 3   2       200   fffffe810f38d040    icmp6_wqinput/2 icmp6_wqinput
 0       95 3   1       200   fffffe810f38d460    icmp6_wqinput/1 icmp6_wqinput
 0       94 3   0       200   fffffe810f2c9b40    icmp6_wqinput/0 icmp6_wqinput
 0       93 3   0       200   fffffe810f38d880          nd6_timer nd6_timer
 0       92 3   7       200   fffffe810f34e020     icmp_wqinput/7 icmp_wqinput
 0       91 3   6       200   fffffe810f34e440     icmp_wqinput/6 icmp_wqinput
 0       90 3   5       200   fffffe810f34e860     icmp_wqinput/5 icmp_wqinput
 0       89 3   4       200   fffffe810f337000     icmp_wqinput/4 icmp_wqinput
 0       88 3   3       200   fffffe810f337420     icmp_wqinput/3 icmp_wqinput
 0       87 3   2       200   fffffe810f337840     icmp_wqinput/2 icmp_wqinput
 0       86 3   1       200   fffffe810f33d360     icmp_wqinput/1 icmp_wqinput
 0       85 3   0       200   fffffe810f33d780     icmp_wqinput/0 icmp_wqinput
 0       84 3   6       200   fffffe810f33dba0           rt_timer rt_timer
 0       83 3   7       200   fffffe810f1fc2c0        vmem_rehash vmem_rehash
 0       82 3   3       200   fffffe810f2d7340          coretemp3 coretemp3
 0       81 3   7       200   fffffe810f2d7760          coretemp2 coretemp2
 0       80 3   1       200   fffffe810f2d7b80          coretemp1 coretemp1
 0       79 3   6       200   fffffe810f2e0320          coretemp0 coretemp0
 0       78 3   4       200   fffffe810f2e0740               mfi0 mfi0
 0       69 3   0       200   fffffe810f1fc6e0            atabus1 atath
 0       68 3   0       200   fffffe810f1fcb00            atabus0 atath
 0       66 3   0       200   fffffe810ed326c0           scsibus0 sccomp
 0       65 3   5       200   fffffe810ed32ae0         usbtask-dr usbtsk
 0       64 3   0       200   fffffe810eca9280         usbtask-hc usbtsk
 0       63 3   7       200   fffffe810eca96a0               bnx3 bnx3
 0       62 3   1       200   fffffe810eca9ac0               bnx2 bnx2
 0       61 3   5       200   fffffe810eb60260               bnx1 bnx1
 0       60 3   4       200   fffffe810eb60680               bnx0 bnx0
 0       59 3   1       200   fffffe810eb60aa0               ipmi ipmipoll
 0       58 3   7       200   fffffe810ea73240            xcall/7 xcall
 0       57 1   7       200   fffffe810ea73660          softser/7
 0       56 1   7       200   fffffe810ea73a80          softclk/7
 0       55 1   7       200   fffffe810ea54220          softbio/7
 0       54 1   7       200   fffffe810ea54640          softnet/7
 0    :  53 7   7       201   fffffe810ea54a60             idle/7
 0       52 3   6       200   fffffe810ea3d200            xcall/6 xcall
 0       51 1   6       200   fffffe810ea3d620          softser/6
 0       50 1   6       200   fffffe810ea3da40          softclk/6
 0       49 1   6       200   fffffe810ea161e0          softbio/6
 0       48 1   6       200   fffffe810ea16600          softnet/6
 0    :  47 7   6       201   fffffe810ea16a20             idle/6
 0       46 3   5       200   fffffe810e9d71c0            xcall/5 xcall
 0       45 1   5       200   fffffe810e9d75e0          softser/5
 0       44 1   5       200   fffffe810e9d7a00          softclk/5
 0       43 1   5       200   fffffe810e9d01a0          softbio/5
 0       42 1   5       200   fffffe810e9d05c0          softnet/5
 0    :  41 7   5       201   fffffe810e9d09e0             idle/5
 0       40 3   4       200   fffffe810e9a1180            xcall/4 xcall
 0       39 1   4       200   fffffe810e9a15a0          softser/4
 0       38 1   4       200   fffffe810e9a19c0          softclk/4
 0       37 1   4       200   fffffe810e97a160          softbio/4
 0       36 1   4       200   fffffe810e97a580          softnet/4
 0    :  35 7   4       201   fffffe810e97a9a0             idle/4
 0       34 3   3       200   fffffe810e943140            xcall/3 xcall
 0       33 1   3       200   fffffe810e943560          softser/3
 0       32 1   3       200   fffffe810e943980          softclk/3
 0       31 1   3       200   fffffe810e934120          softbio/3
 0       30 1   3       200   fffffe810e934540          softnet/3
 0    :  29 7   3       201   fffffe810e934960             idle/3
 0       28 3   2       200   fffffe810e90d100            xcall/2 xcall
 0       27 1   2       200   fffffe810e90d520          softser/2
 0       26 1   2       200   fffffe810e90d940          softclk/2
 0       25 1   2       200   fffffe810e8f60e0          softbio/2
 0       24 1   2       200   fffffe810e8f6500          softnet/2
 0    :  23 7   2       201   fffffe810e8f6920             idle/2
 0       22 3   1       200   fffffe810e8c50c0            xcall/1 xcall
 0       21 1   1       200   fffffe810e8c54e0          softser/1
 0       20 1   1       200   fffffe810e8c5900          softclk/1
 0       19 1   1       200   fffffe810e8a90a0          softbio/1
 0       18 1   1       200   fffffe810e8a94c0          softnet/1
 0    :  17 7   1       201   fffffe810e8a98e0             idle/1
 0       16 3   5       200   fffffe822de9e080           lnxsyswq lnxsyswq
 0       15 3   5       200   fffffe822de9e4a0             sysmon smtaskq
 0       14 3   2       200   fffffe822de9e8c0         pmfsuspend pmfsuspend
 0       13 3   4       200   fffffe822e2c5060           pmfevent pmfevent
 0       12 3   0       200   fffffe822e2c5480         sopendfree sopendfr
 0       11 3   6       200   fffffe822e2c58a0           nfssilly nfssilly
 0       10 3   3       200   fffffe822f6dc040            cachegc cachegc
 0        9 3   3       200   fffffe822f6dc460             vdrain vdrain
 0        8 3   3       200   fffffe822f6dc880          modunload mod_unld
 0        7 3   0       200   fffffe822f6fd020            xcall/0 xcall
 0    :   6 7   0       200   fffffe822f6fd440          softser/0
 0    :   5 7   0       200   fffffe822f6fd860          softclk/0
 0        4 1   0       200   fffffe822f719000          softbio/0
 0        3 1   0       200   fffffe822f719420          softnet/0
 0        2 1   0       201   fffffe822f719840             idle/0
 0        1 3   2       200   ffffffff81481b20            swapper uvm
 db{0}: trace/t 0t5868
 trace: pid 5868 lid 1 at 0xffff80009f0c1a50
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 ffs_update.part.3() at netbsd:ffs_update.part.3+0x292
 ufs_makeinode() at netbsd:ufs_makeinode+0x134
 ufs_create() at netbsd:ufs_create+0x31
 VOP_CREATE() at netbsd:VOP_CREATE+0x3d
 vn_open() at netbsd:vn_open+0x351
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7d9dfe63e2ca:
 db{0}: trace/t 0t6001
 trace: pid 6001 lid 1 at 0xffff80009eff7950
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 ffs_balloc() at netbsd:ffs_balloc+0x1b06
 ufs_gop_alloc() at netbsd:ufs_gop_alloc+0xa1
 ufs_balloc_range() at netbsd:ufs_balloc_range+0x136
 ffs_write() at netbsd:ffs_write+0x4d7
 VOP_WRITE() at netbsd:VOP_WRITE+0x37
 vn_write() at netbsd:vn_write+0xec
 dofilewrite() at netbsd:dofilewrite+0x97
 sys_write() at netbsd:sys_write+0x5f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 4) ---
 79de3d23e02a:
 db{0}: trace/t 0t9372
 trace: pid 9372 lid 1 at 0xffff80009f6f78e0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 rw_vector_enter() at netbsd:rw_vector_enter+0x3e8
 genfs_lock() at netbsd:genfs_lock+0x3c
 VOP_LOCK() at netbsd:VOP_LOCK+0x67
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 781169a3e2ca:
 db{0}: trace/t 0t8217
 trace: pid 8217 lid 1 at 0xffff80009f6be8e0
 sleepq_block() at netbsd:sleepq_block+0x97
 turnstile_block() at netbsd:turnstile_block+0x35c
 rw_vector_enter() at netbsd:rw_vector_enter+0x3e8
 genfs_lock() at netbsd:genfs_lock+0x3c
 VOP_LOCK() at netbsd:VOP_LOCK+0x67
 vn_lock() at netbsd:vn_lock+0x90
 namei_tryemulroot() at netbsd:namei_tryemulroot+0x193
 namei() at netbsd:namei+0x29
 vn_open() at netbsd:vn_open+0x92
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 7ac9b2a3e2ca:
 db{0}: trace/t 0t10228
 trace: pid 10228 lid 2 at 0xffff80009f71de40
 selcommon.part.2() at netbsd:selcommon.part.2+0x18b
 sys___select50() at netbsd:sys___select50+0x6f
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 417) ---
 767a01c3e10a:
 db{0}: trace/t 0t7123
 trace: pid 7123 lid 1 at 0xffff80009f5d9c10
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 VOP_BWRITE() at netbsd:VOP_BWRITE+0x51
 ufs_dirremove() at netbsd:ufs_dirremove+0xd8
 ufs_remove() at netbsd:ufs_remove+0x142
 VOP_REMOVE() at netbsd:VOP_REMOVE+0x37
 do_sys_unlinkat() at netbsd:do_sys_unlinkat+0x237
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 10) ---
 7973de4fb2aa:
 db{0}: trace/t 0t1019
 trace: pid 1019 lid 1 at 0xffff80009de6e8a0
 sleepq_block() at netbsd:sleepq_block+0x97
 cv_wait() at netbsd:cv_wait+0xfb
 biowait() at netbsd:biowait+0x5d
 bwrite() at netbsd:bwrite+0x153
 ffs_update.part.3() at netbsd:ffs_update.part.3+0x292
 ffs_truncate() at netbsd:ffs_truncate+0x1046
 ufs_truncate_retry() at netbsd:ufs_truncate_retry+0x79
 ufs_setattr() at netbsd:ufs_setattr+0x44c
 VOP_SETATTR() at netbsd:VOP_SETATTR+0x34
 vn_open() at netbsd:vn_open+0x26e
 do_open() at netbsd:do_open+0x112
 do_sys_openat() at netbsd:do_sys_openat+0x68
 sys_open() at netbsd:sys_open+0x24
 syscall() at netbsd:syscall+0x1d8
 --- syscall (number 5) ---
 79538a23e2ca:
 db{0}: call cpu_reset
 


Home | Main Index | Thread Index | Old Index