NetBSD-Bugs archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
kern/54036: system hangs in RAM starvation
>Number: 54036
>Category: kern
>Synopsis: system hangs in RAM starvation
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: kern-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sun Mar 03 07:20:00 +0000 2019
>Originator: S.P.Zeidler
>Release: NetBSD 8.0_STABLE
>Organization:
The NetBSD Foundation
>Environment:
System: NetBSD franklin.NetBSD.org 8.0_STABLE NetBSD 8.0_STABLE (FRANKLIN) #5: Wed Jan 30 06:56:54 UTC 2019 spz%franklin.NetBSD.org@localhost:/home/netbsd/8/amd64/obj/sys/arch/amd64/compile/FRANKLIN amd64
Architecture: x86_64
Machine: amd64
>Description:
system pings, no other interaction except breaking into ddb is possible.
ddb says:
db{0}> ps /l
PID LID S CPU FLAGS STRUCT LWP * NAME WAIT
27876 1 3 11 0 fffffe8a57e246a0 cron flt_pmfail2
16220 1 3 2 0 fffffe8a57e24ac0 sshd flt_pmfail2
4575 1 3 3 0 fffffe8774c13460 sshd flt_pmfail2
16088 1 3 7 0 fffffe8774c13040 cron flt_noram5
17331 1 3 9 0 fffffe8a482d89c0 cron flt_pmfail2
1228 1 3 5 0 fffffe8bc7061320 xymonlaunch flt_noram5
24078 1 3 4 0 fffffe81900b7060 cron uvn_fp1
11898 1 3 7 0 fffffe8bc7061740 sshd flt_noram5
28813 1 3 12 0 fffffe818692e0c0 cron uvn_fp1
347 1 3 8 0 fffffe8535c4e9e0 sshd flt_pmfail2
27243 1 3 6 0 fffffe8157ee10e0 cron wait
9020 1 3 6 0 fffffe81f47498e0 sshd flt_pmfail2
15296 1 3 12 0 fffffe8b845bf940 cron wait
25198 1 3 11 0 fffffe8a20f806c0 sh flt_noram5
10884 1 3 12 0 fffffe8b845bf520 cron wait
21241 1 3 0 80 fffffe896eaa79e0 awk pipe_rd
14535 1 3 0 0 fffffe82988a8080 envstat ciss_cmd
12143 1 3 0 80 fffffe82988a88c0 sh wait
18295 1 3 0 80 fffffe8912aeaaa0 sh pipe_rd
4373 1 3 4 1000000 fffffe86ddacf5e0 pickup plpg
27430 1 3 2 0 fffffe8c30898ae0 gpg uvn_fp1
9025 1 3 13 0 fffffe826fd235e0 gpg uvn_fp1
16139 1 3 15 0 fffffe8be6d86320 ssh uvn_fp1
3542 1 3 2 80 fffffe83add4c680 ttcp netio
22199 1 3 9 80 fffffe8962fcb620 ttcp netio
10428 1 3 15 80 fffffe82127fd520 dump pause
23778 1 3 15 80 fffffe82127fd100 dump pause
23842 1 3 0 80 fffffe81585f49a0 dump pipe_wr
3427 1 3 0 80 fffffe8c30134660 dump netio
329 1 3 10 80 fffffe857b7470e0 gpg pipe_wr
8239 1 3 11 80 fffffe857b747920 gpg pipe_wr
28917 1 3 6 80 fffffe83add4caa0 gzip pipe_wr
9418 1 3 4 80 fffffe83b1c78480 dump wait
22418 1 3 13 80 fffffe8c2fd79340 sh wait
23497 1 3 4 80 fffffe85f60aa680 sh wait
9193 1 3 3 80 fffffe896eaa75c0 sshd select
17220 1 3 0 80 fffffe8259a97320 sshd select
20551 1 3 10 0 fffffe8a482d85a0 ssh uvn_fp1
1765 1 3 0 0 fffffe86a2e09040 ssh uvn_fp1
6785 1 3 2 80 fffffe86ddacfa00 sh wait
8154 1 3 2 80 fffffe83add4c260 sh wait
27712 1 3 4 80 fffffe889cd99a40 sh wait
26768 1 3 1 0 fffffe815865a9e0 ssh uvn_fp1
2474 1 3 4 80 fffffe826fd231c0 sh wait
11331 1 3 3 80 fffffe8c309dd200 sh wait
1957 1 3 14 80 fffffe8a20f802a0 sh wait
15117 1 3 10 80 fffffe826fd23a00 sh wait
9881 1 3 9 80 fffffe864607e5c0 sh wait
22600 1 3 8 80 fffffe8535c4e1a0 sh wait
8665 1 3 5 80 fffffe8774c13880 sh wait
29979 1 3 0 80 fffffe857b747500 cron pipe_rd
20720 1 3 12 80 fffffe8b845bf100 tcsh ttyraw
15034 1 3 12 80 fffffe8a57e24280 sh ttyraw
23287 1 3 0 80 fffffe889cd99200 su wait
13904 1 3 0 80 fffffe8a482d8180 sh wait
787 1 3 12 80 fffffe8bc7061b60 sh ttyraw
2839 1 3 0 80 fffffe8835e42340 su wait
29221 1 3 1 80 fffffe8835e42760 sh wait
20340 1 3 15 80 fffffe8674cd6880 sh ttyraw
18045 1 3 5 80 fffffe8962fcb200 su wait
3359 1 3 4 80 fffffe86a2e09460 tcsh ttyraw
15116 1 3 5 80 fffffe86a2e09880 tcsh pause
15173 1 3 10 80 fffffe88b6d72440 tcsh ttyraw
17363 1 3 14 80 fffffe8912aea680 tcsh pause
13194 1 3 11 80 fffffe88b23dfa60 tcsh ttyraw
12756 1 3 3 80 fffffe889cd99620 tcsh pause
1155 1 3 0 80 fffffe8c2fd79760 screen-4.6.2 select
344 1 3 0 0 fffffe81900b78a0 getty uvn_fp1
591 1 3 11 0 fffffe82988a84a0 cron flt_noram5
605 1 3 0 80 fffffe8c30134240 inetd kqueue
785 1 3 5 1000000 fffffe8c308982a0 qmgr plpg
876 1 3 2 1000000 fffffe8311dc41c0 master socket
557 1 3 3 0 fffffe8912aea260 sshd flt_noram5
462 1 3 0 80 fffffe8912a7a640 powerd kqueue
502 1 3 10 1000000 fffffe8c30134a80 ntpd uvn_fp1
452 1 3 2 80 fffffe8912a7a220 xymonlaunch nanoslp
436 1 3 3 80 fffffe8c309dda40 blacklistd select
286 19 3 13 80 fffffe8311dc45e0 named kqueue
286 18 3 10 80 fffffe8311dc4a00 named parked
286 17 3 9 80 fffffe818207d1a0 named parked
286 16 3 15 0 fffffe818207d5c0 named plpg
286 15 3 10 80 fffffe818207d9e0 named parked
286 14 3 14 80 fffffe818fba7180 named parked
286 13 3 0 80 fffffe818fba75a0 named parked
286 12 3 4 80 fffffe818fba79c0 named parked
286 11 3 12 80 fffffe8c33dff160 named parked
286 10 3 8 0 fffffe8c33dff580 named uvn_fp1
286 9 3 3 80 fffffe8c33dff9a0 named parked
286 8 3 14 80 fffffe8b0ecf1140 named parked
286 7 3 5 80 fffffe8b0ecf1560 named parked
286 6 3 3 80 fffffe8b0ecf1980 named parked
286 5 3 2 80 fffffe81f47490a0 named parked
286 4 3 6 80 fffffe818692e4e0 named parked
286 3 3 11 80 fffffe818692e900 named parked
286 2 3 13 80 fffffe81f47494c0 named parked
286 1 3 0 80 fffffe815861b180 named sigwait
255 1 3 14 0 fffffe8191996120 syslogd uvn_fp1
231 1 3 11 80 fffffe81820cd920 dhcpcd wait
1 1 3 7 80 fffffe815867e000 init wait
0 186 3 9 200 fffffe8191996540 ipmi0 ipmi0
0 185 3 15 200 fffffe81820cd0e0 agrmon agrmon
0 184 3 0 200 fffffe81587ec020 physiod physiod
0 183 3 0 200 fffffe8158a29040 aiodoned aiodoned
0 182 3 0 200 fffffe8158a29460 ioflush syncer
0 181 3 0 200 fffffe8158a29880 pgdaemon livelock
0 178 3 13 200 fffffe815861b9c0 usb0 usbevt
0 177 3 3 200 fffffe81585f4160 usb2 usbevt
0 176 3 14 200 fffffe815861b5a0 usb1 usbevt
0 175 3 11 200 fffffe81587ec440 usb3 usbevt
0 174 3 7 200 fffffe81587ec860 npfgc-0 npfgccv
0 173 3 15 200 fffffe815867e420 rt_free rt_free
0 172 3 8 200 fffffe815867e840 unpgc unpgc
0 171 3 11 200 fffffe81585f4580 key_timehandler key_timehandler
0 170 3 15 200 fffffe81587b6360 icmp6_wqinput/1 icmp6_wqinput
0 169 3 14 200 fffffe81587b6780 icmp6_wqinput/1 icmp6_wqinput
0 168 3 13 200 fffffe81587b6ba0 icmp6_wqinput/1 icmp6_wqinput
0 167 3 12 200 fffffe81587a7340 icmp6_wqinput/1 icmp6_wqinput
0 166 3 11 200 fffffe81587a7760 icmp6_wqinput/1 icmp6_wqinput
0 165 3 10 200 fffffe81587a7b80 icmp6_wqinput/1 icmp6_wqinput
0 164 3 9 200 fffffe81587a0320 icmp6_wqinput/9 icmp6_wqinput
0 163 3 8 200 fffffe81587a0740 icmp6_wqinput/8 icmp6_wqinput
0 162 3 7 200 fffffe81587a0b60 icmp6_wqinput/7 icmp6_wqinput
0 161 3 6 200 fffffe8158791300 icmp6_wqinput/6 icmp6_wqinput
0 160 3 5 200 fffffe8158791720 icmp6_wqinput/5 icmp6_wqinput
0 159 3 4 200 fffffe8158791b40 icmp6_wqinput/4 icmp6_wqinput
0 158 3 3 200 fffffe81587c22e0 icmp6_wqinput/3 icmp6_wqinput
0 157 3 2 200 fffffe81587c2700 icmp6_wqinput/2 icmp6_wqinput
0 156 3 1 200 fffffe81587c2b20 icmp6_wqinput/1 icmp6_wqinput
0 155 3 0 200 fffffe815872b2c0 icmp6_wqinput/0 icmp6_wqinput
0 154 3 0 200 fffffe815872b6e0 nd6_timer nd6_timer
0 153 3 15 200 fffffe815872bb00 icmp_wqinput/15 icmp_wqinput
0 152 3 14 200 fffffe81586f42a0 icmp_wqinput/14 icmp_wqinput
0 151 3 13 200 fffffe81586f46c0 icmp_wqinput/13 icmp_wqinput
0 150 3 12 200 fffffe81586f4ae0 icmp_wqinput/12 icmp_wqinput
0 149 3 11 200 fffffe81586dd280 icmp_wqinput/11 icmp_wqinput
0 148 3 10 200 fffffe81586dd6a0 icmp_wqinput/10 icmp_wqinput
0 147 3 9 200 fffffe81586ddac0 icmp_wqinput/9 icmp_wqinput
0 146 3 8 200 fffffe81586de260 icmp_wqinput/8 icmp_wqinput
0 145 3 7 200 fffffe81586de680 icmp_wqinput/7 icmp_wqinput
0 144 3 6 200 fffffe81586deaa0 icmp_wqinput/6 icmp_wqinput
0 143 3 5 200 fffffe81586cf240 icmp_wqinput/5 icmp_wqinput
0 142 3 4 200 fffffe81586cf660 icmp_wqinput/4 icmp_wqinput
0 141 3 3 200 fffffe81586cfa80 icmp_wqinput/3 icmp_wqinput
0 140 3 2 200 fffffe81586c0220 icmp_wqinput/2 icmp_wqinput
0 139 3 1 200 fffffe81586c0640 icmp_wqinput/1 icmp_wqinput
0 138 3 0 200 fffffe81586c0a60 icmp_wqinput/0 icmp_wqinput
0 137 3 14 200 fffffe8158657200 rt_timer rt_timer
0 136 3 1 200 fffffe815865a5c0 vmem_rehash vmem_rehash
0 135 3 3 200 fffffe8158657620 coretemp7 coretemp7
0 134 3 10 200 fffffe8158657a40 coretemp6 coretemp6
0 133 3 8 200 fffffe81586301e0 coretemp5 coretemp5
0 132 3 7 200 fffffe8158630600 coretemp4 coretemp4
0 131 3 15 200 fffffe8158630a20 coretemp3 coretemp3
0 130 3 9 200 fffffe81586311c0 coretemp2 coretemp2
0 129 3 10 200 fffffe81586315e0 coretemp1 coretemp1
0 128 3 13 200 fffffe8158631a00 coretemp0 coretemp0
0 127 3 9 200 fffffe815865a1a0 ciss0 ciss0
0 118 3 14 200 fffffe8158507140 iic0 iicintr
0 117 3 15 200 fffffe8158507560 atabus5 atath
0 116 3 11 200 fffffe8158507980 atabus4 atath
0 115 3 0 200 fffffe81584c4120 atabus3 atath
0 114 3 12 200 fffffe81584c4540 atabus2 atath
0 113 3 0 200 fffffe81584c4960 atabus1 atath
0 112 3 11 200 fffffe8158427100 atabus0 atath
0 111 3 14 200 fffffe8158427520 usbtask-dr usbtsk
0 110 3 11 200 fffffe8158427940 usbtask-hc usbtsk
0 108 3 11 200 fffffe8157ee1500 scsibus0 sccomp
0 107 3 1 200 fffffe8157ee1920 ipmi ipmipoll
0 106 3 15 200 fffffe8157dd90c0 xcall/15 xcall
0 105 1 15 200 fffffe8157dd94e0 softser/15
0 104 1 15 200 fffffe8157dd9900 softclk/15
0 103 1 15 200 fffffe8157dfa0a0 softbio/15
0 102 1 15 200 fffffe8157dfa4c0 softnet/15
0 > 101 7 15 201 fffffe8157dfa8e0 idle/15
0 100 3 14 200 fffffe8157d89080 xcall/14 xcall
0 99 1 14 200 fffffe8157d894a0 softser/14
0 98 1 14 200 fffffe8157d898c0 softclk/14
0 97 1 14 200 fffffe8157daa060 softbio/14
0 96 1 14 200 fffffe8157daa480 softnet/14
0 > 95 7 14 201 fffffe8157daa8a0 idle/14
0 94 3 13 200 fffffe8157d31040 xcall/13 xcall
0 93 1 13 200 fffffe8157d31460 softser/13
0 92 1 13 200 fffffe8157d31880 softclk/13
0 91 1 13 200 fffffe8157d4a020 softbio/13
0 90 1 13 200 fffffe8157d4a440 softnet/13
0 > 89 7 13 201 fffffe8157d4a860 idle/13
0 88 3 12 200 fffffe8157cd9000 xcall/12 xcall
0 87 1 12 200 fffffe8157cd9420 softser/12
0 86 1 12 200 fffffe8157cd9840 softclk/12
0 85 1 12 200 fffffe8157cf2360 softbio/12
0 84 1 12 200 fffffe8157cf2780 softnet/12
0 > 83 7 12 201 fffffe8157cf2ba0 idle/12
0 82 3 11 200 fffffe8157c81340 xcall/11 xcall
0 81 1 11 200 fffffe8157c81760 softser/11
0 80 1 11 200 fffffe8157c81b80 softclk/11
0 79 1 11 200 fffffe8157c9a320 softbio/11
0 78 1 11 200 fffffe8157c9a740 softnet/11
0 > 77 7 11 201 fffffe8157c9ab60 idle/11
0 76 3 10 200 fffffe8157c31300 xcall/10 xcall
0 75 1 10 200 fffffe8157c31720 softser/10
0 74 1 10 200 fffffe8157c31b40 softclk/10
0 73 1 10 200 fffffe8157c422e0 softbio/10
0 72 1 10 200 fffffe8157c42700 softnet/10
0 > 71 7 10 201 fffffe8157c42b20 idle/10
0 70 3 9 200 fffffe8157bd92c0 xcall/9 xcall
0 69 1 9 200 fffffe8157bd96e0 softser/9
0 68 1 9 200 fffffe8157bd9b00 softclk/9
0 67 1 9 200 fffffe8157bea2a0 softbio/9
0 66 1 9 200 fffffe8157bea6c0 softnet/9
0 > 65 7 9 201 fffffe8157beaae0 idle/9
0 64 3 8 200 fffffe8157b81280 xcall/8 xcall
0 63 1 8 200 fffffe8157b816a0 softser/8
0 62 1 8 200 fffffe8157b81ac0 softclk/8
0 61 1 8 200 fffffe8157b92260 softbio/8
0 60 1 8 200 fffffe8157b92680 softnet/8
0 > 59 7 8 201 fffffe8157b92aa0 idle/8
0 58 3 7 200 fffffe8157b31240 xcall/7 xcall
0 57 1 7 200 fffffe8157b31660 softser/7
0 56 1 7 200 fffffe8157b31a80 softclk/7
0 55 1 7 200 fffffe8157b3a220 softbio/7
0 54 1 7 200 fffffe8157b3a640 softnet/7
0 > 53 7 7 201 fffffe8157b3aa60 idle/7
0 52 3 6 200 fffffe8157ad9200 xcall/6 xcall
0 51 1 6 200 fffffe8157ad9620 softser/6
0 50 1 6 200 fffffe8157ad9a40 softclk/6
0 49 1 6 200 fffffe8157ae21e0 softbio/6
0 48 1 6 200 fffffe8157ae2600 softnet/6
0 > 47 7 6 201 fffffe8157ae2a20 idle/6
0 46 3 5 200 fffffe8157a891c0 xcall/5 xcall
0 45 1 5 200 fffffe8157a895e0 softser/5
0 44 1 5 200 fffffe8157a89a00 softclk/5
0 43 1 5 200 fffffe8157a821a0 softbio/5
0 42 1 5 200 fffffe8157a825c0 softnet/5
0 > 41 7 5 201 fffffe8157a829e0 idle/5
0 40 3 4 200 fffffe8157a31180 xcall/4 xcall
0 39 1 4 200 fffffe8157a315a0 softser/4
0 38 1 4 200 fffffe8157a319c0 softclk/4
0 37 1 4 200 fffffe8157a2a160 softbio/4
0 36 1 4 200 fffffe8157a2a580 softnet/4
0 > 35 7 4 201 fffffe8157a2a9a0 idle/4
0 34 3 3 200 fffffe81579d9140 xcall/3 xcall
0 33 1 3 200 fffffe81579d9560 softser/3
0 32 1 3 200 fffffe81579d9980 softclk/3
0 31 1 3 200 fffffe81579d2120 softbio/3
0 30 1 3 200 fffffe81579d2540 softnet/3
0 > 29 7 3 201 fffffe81579d2960 idle/3
0 28 3 2 200 fffffe8157989100 xcall/2 xcall
0 27 1 2 200 fffffe8157989520 softser/2
0 26 1 2 200 fffffe8157989940 softclk/2
0 25 1 2 200 fffffe815797a0e0 softbio/2
0 24 1 2 200 fffffe815797a500 softnet/2
0 > 23 7 2 201 fffffe815797a920 idle/2
0 22 3 1 200 fffffe815793a0c0 xcall/1 xcall
0 21 1 1 200 fffffe815793a4e0 softser/1
0 20 1 1 200 fffffe815793a900 softclk/1
0 19 1 1 200 fffffe81579170a0 softbio/1
0 18 1 1 200 fffffe81579174c0 softnet/1
0 > 17 7 1 201 fffffe81579178e0 idle/1
0 16 3 0 200 fffffe8c345a9080 lnxsyswq lnxsyswq
0 15 3 11 200 fffffe8c345a94a0 sysmon smtaskq
0 14 3 15 200 fffffe8c345a98c0 pmfsuspend pmfsuspend
0 13 3 6 200 fffffe8c365bc060 pmfevent pmfevent
0 12 3 0 200 fffffe8c365bc480 sopendfree sopendfr
0 11 3 10 200 fffffe8c365bc8a0 nfssilly nfssilly
0 10 3 11 200 fffffe8c3c5e1040 cachegc cachegc
0 9 3 12 200 fffffe8c3c5e1460 vdrain vdrain
0 8 3 6 200 fffffe8c3c5e1880 modunload mod_unld
0 7 3 0 200 fffffe8c3c5f0020 xcall/0 xcall
0 6 1 0 200 fffffe8c3c5f0440 softser/0
0 5 3 0 200 fffffe8c3c5f0860 softclk/0 tstile
0 4 1 0 200 fffffe8c3c615000 softbio/0
0 3 1 0 200 fffffe8c3c615420 softnet/0
0 > 2 7 0 201 fffffe8c3c615840 idle/0
0 1 3 3 200 ffffffff80e38080 swapper uvm
db{0}> show uvmexp
Current UVM status:
pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12, ncolors=8
12217743 VM pages: 7544421 active, 3629521 inactive, 2672 wired, 1 free
pages 39682 anon, 11130252 file, 6680 exec
freemin=4096, free-target=5461, wired-max=4072581
resv-pg=1, resv-kernel=80, zeropages=1
cpu0:
faults=647931110, traps=609904544, intrs=96975159, ctxswitch=512461218
softint=1180102923, syscalls=2346237049
cpu1:
faults=131085353, traps=124058789, intrs=164484458, ctxswitch=98155358
softint=170520477, syscalls=428540379
cpu2:
faults=120999314, traps=112827570, intrs=224832334, ctxswitch=91712328
softint=225182426, syscalls=378134812
cpu3:
faults=111384531, traps=105768382, intrs=203587703, ctxswitch=93342004
softint=211663798, syscalls=409371888
cpu4:
faults=75191805, traps=69246271, intrs=160494015, ctxswitch=91151425
softint=164564259, syscalls=322404754
cpu5:
faults=67474771, traps=62010105, intrs=150783651, ctxswitch=69549006
softint=152817126, syscalls=215347786
cpu6:
faults=77127954, traps=69514051, intrs=148937064, ctxswitch=81260149
softint=151892079, syscalls=272830083
cpu7:
faults=226844871, traps=212338728, intrs=1140172416, ctxswitch=175318867
softint=1157899935, syscalls=1092653080
cpu8:
faults=175467910, traps=169740548, intrs=91715496, ctxswitch=110416802
softint=96618000, syscalls=436311492
cpu9:
faults=70160398, traps=65200653, intrs=86833801, ctxswitch=67012059
softint=86492803, syscalls=223763354
cpu10:
faults=111846244, traps=105518664, intrs=0, ctxswitch=99360386
softint=6658100, syscalls=307514637
cpu11:
faults=81761425, traps=77033721, intrs=0, ctxswitch=73061061
softint=5925661, syscalls=239964691
cpu12:
faults=49407258, traps=45291649, intrs=0, ctxswitch=61287665
softint=12817669, syscalls=191063974
cpu13:
faults=69727014, traps=65286090, intrs=0, ctxswitch=66023740
softint=3489686, syscalls=210936957
cpu14:
faults=112615876, traps=106161667, intrs=0, ctxswitch=97285330
softint=6223289, syscalls=371130546
cpu15:
faults=162573524, traps=145509885, intrs=0, ctxswitch=160550401
softint=22321112, syscalls=490065840
fault counts:
noram=112, noanon=0, pgwait=0, pgrele=0
ok relocks(total)=1552205(1552213), anget(retrys)=190385683(0), amapcopy=90285755
neighbor anon/obj pg=38721197/1231386720, gets(lock/unlock)=356622225/1552211
cases: anon=177038911, anoncow=13346641, obj=259576916, prcopy=97054332, przero=691345119
daemon and swap counts:
woke=65889, revs=65882, scans=1442191948, obscans=1328029685, anscans=0
busy=14855, freed=1328029685, reactivate=1203069, deactivate=1465300648
pageouts=0, pending=0, nswget=0
nswapdev=1, swpgavail=12580679
swpages=12580679, swpginuse=0, swpgonly=0, paging=0
>How-To-Repeat:
The system hung while performing the monthly full backups.
I'll know by tomorrow if that is systematic.
>Fix:
Home |
Main Index |
Thread Index |
Old Index