tech-kern archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: High kernel time, page scan rate & reclaims?
For the archives, since I just got annoyed again by the behaviour (I'm
running netbsd-9), this was likely fixed in:
PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour
in -current, and will be in netbsd-10.
On Wed, Jun 27, 2018 at 09:32:27PM +1000, Paul Ripke wrote:
> I recently bumped my generally idle home server from 4GiB to 16GiB RAM,
> and have noticed a strange behaviour occasionally. While running:
> find /b -name \*gz -printx | nice xargs gzip -vt
> from a ~30MiB/s USB drive to verify some archives, I see a sudden huge
> burst of page scanning, page reclaims, and high kernel time (x86_64,
> 8 core machine). During this time, performance obviously suffers.
>
> vmstat 1:
> procs memory page disks faults cpu
> r b avm fre flt re pi po fr sr w0 w1 in sy cs us sy id
> 0 1 11656768 41772 2569 0 0 0 9277 9515 3 3 1665 2025 1655 1 11 88
> 0 1 11688900 12656 4017 0 0 0 753 821 213 213 1811 3166 2670 1 4 95
> 1 0 11675024 28172 2551 0 0 0 8972 9227 0 0 1562 2627 1504 2 11 87
> 0 1 11680508 42028 2649 0 0 0 8773 8909 0 0 1225 2217 1559 3 11 86
> 0 2 11716988 11816 4744 0 0 0 421 472 13 13 2506 2500 2637 1 2 97
> 0 1 11702644 32312 2034 14325 0 0 9179 548630 42 67 960 2259 1542 0 12 88
> 0 2 11696984 43672 2880 0 0 0 8602 8705 0 0 1814 2059 1647 0 4 96
> 0 2 11728608 12028 3970 0 0 0 0 0 2 4 1615 4365 2328 1 0 99
> 0 1 11742108 24492 2705 0 0 0 8521 8607 57 53 1739 1944 1994 7 5 88
> 0 1 11750532 38572 2769 0 0 0 9053 9241 0 0 1077 2182 1708 4 7 89
> 0 1 12677464 26356 3768 4476 0 0 4466 228811 14 14 2119 2836 2271 8 5 87
> 2 0 11763924 26672 2312 9130 0 0 4683 320835 181 181 991 2818 1771 0 6 94
> 0 1 11785516 39212 2761 2 0 0 8667 8782 1 1 1639 2767 1659 0 4 96
> 1 1 12029352 39784 3961 333 0 0 8079 61311 0 0 1520 2900 2162 1 2 97
> 0 1 11830136 23448 2096 3914 0 0 105 487391 0 0 1385 2128 1413 0 9 91
> 1 1 11846328 11936 2025 2844 0 0 1151 541475 0 0 830 2104 1350 1 6 93
> 0 3 11945764 10796 138 2693 0 0 7 1080642 0 0 803 1710 590 0 12 88
> 0 2 12377108 10388 54 2178 0 0 4 1212719 0 3 72 3561 700 0 13 87
> 4 0 13558624 10324 41 2428 0 0 21 1375681 12 10 794 2644 619 0 40 60
> 2 1 14009652 9788 185 1549 0 0 5 1192572 0 0 117 3187 712 0 31 69
> 3 0 14009796 9772 7 1882 0 0 0 1080436 22 19 969 2361 1084 0 35 65
> 5 0 14009952 9744 20 5581 0 0 0 1079241 45 45 344 2113 929 0 39 60
> 4 0 14010280 9568 24 5330 0 0 0 1060952 1 2 682 2600 555 0 59 41
> 0 3 14010312 9580 9 1299 0 0 6 1080506 1 1 273 3402 816 11 50 38
> 3 1 14010284 9472 19 1745 0 0 8 1077594 25 25 698 1659 626 0 23 77
> 1 2 14010420 9464 13 1609 0 0 0 1080089 0 0 42 2898 560 0 19 81
> 0 2 14010436 9316 23 1838 0 0 0 1061402 0 0 728 3458 851 0 13 87
> 0 2 14010532 9292 14 2299 0 0 2 1080078 0 1 113 1884 576 0 15 85
> 3 2 14010556 9288 7 1454 0 0 8 1080671 12 16 973 2995 1074 0 38 62
> 5 1 14010592 9284 10 1808 0 0 8 1080241 7 10 231 1794 556 0 61 39
> 0 3 14010656 9076 23 1713 0 0 4 1080539 36 37 874 2521 1087 0 35 65
> 2 1 14011328 8164 176 1971 0 0 8 1080458 28 24 257 2938 902 0 38 62
> 0 4 14011444 8160 7 3074 0 0 1 1080940 0 0 839 1681 471 0 57 43
> 5 0 14011876 7660 164 2756 0 0 23 1080608 2 4 709 1621 656 0 25 75
> 0 3 14012504 6844 205 2898 0 0 0 1080425 1 5 237 1623 582 0 49 51
> 5 1 13794208 6892 15 4542 0 0 20 1081070 2 0 772 5039 1196 0 46 54
> 13 3 11868880 6784 22 5656 0 0 28 1625980 1 0 255 1343 413 0 77 23
> 5 1 11941952 6760 15 3600 0 0 11 1076077 7 8 886 2977 586 0 68 32
> 2 3 12062924 3120 959 2393 0 0 8 1133045 3 3 768 6350 1584 0 38 61
> 4 4 12187780 2756 59 3661 0 0 9 1112234 17 23 1357 7822 2072 1 62 37
> 6 1 11996460 2556 59 10342 0 0 4 1034585 94 55 615 3225 1266 0 68 31
> 23 1 11929016 3164 36 7641 0 0 26 1054593 43 61 1201 2863 839 0 100 0
> 6 3 11910632 3444 8 7435 0 0 16 1052934 16 17 594 4200 1171 0 78 21
> 7 0 12163800 4088 29 2777 0 0 17 1080661 0 2 370 1463 396 0 69 31
> 10 4 13160972 4588 17 2104 0 0 11 1081658 0 2 472 2303 484 1 67 32
> 8 1 13740000 5232 12 1600 0 0 52 1081632 0 2 885 1059 327 0 84 16
> 11 3 14013912 5684 8 1371 0 0 0 1042740 0 1 410 1557 416 4 73 23
> 11 0 11929820 6020 16 1432 0 0 2 1042952 0 1 530 1431 361 0 82 18
> 6 0 12630784 6204 75 2257 0 0 3 1042260 3 27 285 1834 612 0 70 30
> 6 2 12712484 6124 35 5872 0 0 26 1060289 1 1 114 2246 362 0 91 9
> 4 2 13605572 6212 562 3721 0 0 5 1079855 60 48 745 2298 1100 0 62 38
> 1 7 14013352 6224 31 2298 0 0 25 1078799 29 27 496 1795 798 0 68 32
> 5 3 14013716 5660 89 7506 0 0 12 1080311 16 26 937 2892 685 0 57 43
> 1 5 14014188 5016 160 2367 0 0 17 1079323 8 7 705 2063 691 0 18 82
> 0 6 13716768 4560 108 2282 0 0 17 1080616 3 7 674 2123 740 0 14 85
> 0 5 12962172 4228 164 2119 0 0 38 1080527 3 3 186 1944 632 0 14 86
> 3 2 12320244 4240 14 2461 0 0 12 1058865 0 2 394 2152 518 0 39 61
> 9 0 12563024 4180 13 1552 0 0 0 1079018 0 1 357 3519 926 0 76 23
> 0 5 12339224 1784 741 2864 0 0 25 1079423 3 8 713 4092 1077 0 15 85
> 3 4 11924212 1236 423 2381 0 0 61 1099422 19 22 446 3158 1004 0 15 85
> 2 2 12027760 4060 58 2486 0 0 4 1063631 4 0 585 2315 455 0 62 38
> 1 2 12374168 3436 354 2274 0 0 62 1080451 17 16 306 2367 912 0 46 54
> 2 1 12949216 3492 24 2507 0 0 58 1080756 29 24 369 2315 799 0 50 49
> 5 0 9376604 43776 820 2125 0 0 10288 1091040 72 54 729 2378 1252 0 46 54
> 0 1 9408724 12592 4348 0 0 0 0 0 82 106 2300 3196 2863 1 3 96
> 1 1 9395364 27804 3254 0 0 0 10268 10710 2 1 1770 6378 2710 5 11 84
> 1 1 9382756 39492 3208 0 0 0 9345 9447 0 0 2687 9029 3538 6 8 86
> 1 0 9414960 22432 4025 0 0 0 3770 3821 0 0 2265 4224 2674 1 1 98
>
> vmstat -s:
> 4096 bytes per page
> 8 page colors
> 4056270 pages managed
> 10321 pages free
> 2767312 pages active
> 747242 pages inactive
> 0 pages paging
> 3688 pages wired
> 7960 zero pages
> 1 reserve pagedaemon pages
> 40 reserve kernel pages
> 354538 anonymous pages
> 2971292 cached file pages
> 192524 cached executable pages
> 2048 minimum free pages
> 2730 target free pages
> 1352090 maximum wired pages
> 1 swap devices
> 2621439 swap pages
> 0 swap pages in use
> 0 swap allocations
> 949887396 total faults taken
> 774924866 traps
> 288358214 device interrupts
> 922225669 CPU context switches
> 546785719 software interrupts
> 2867987678 system calls
> 0 pagein requests
> 0 pageout requests
> 0 pages swapped in
> 0 pages swapped out
> 1604751 forks total
> 1252330 forks blocked parent
> 1252330 forks shared address space with parent
> 203775339 pagealloc zero wanted and avail
> 306860999 pagealloc zero wanted and not avail
> 58583 aborts of idle page zeroing
> 728854095 pagealloc desired color avail
> 78479879 pagealloc desired color not avail
> 358153947 pagealloc local cpu avail
> 449180027 pagealloc local cpu not avail
> 43 faults with no memory
> 0 faults with no anons
> 0 faults had to wait on pages
> 0 faults found released page
> 354190 faults relock (352928 ok)
> 202394428 anon page faults
> 0 anon retry faults
> 42172180 amap copy faults
> 23031397 neighbour anon page faults
> 663678460 neighbour object page faults
> 204244025 locked pager get faults
> 354188 unlocked pager get faults
> 191281904 anon faults
> 11118481 anon copy on write faults
> 163305028 object faults
> 40964361 promote copy faults
> 475034438 promote zero fill faults
> 25320 times daemon wokeup
> 25363 revolutions of the clock hand
> 215576033 pages freed by daemon
> 894484695 pages scanned by daemon
> 0 anonymous pages scanned by daemon
> 215576033 object pages scanned by daemon
> 22678911 pages reactivated
> 2004 pages found busy by daemon
> 0 total pending pageouts
> 909312286 pages deactivated
> 2183398810 total name lookups
> 1927927827 good hits
> 106350213 negative hits
> 2457145 bad hits
> 6724600 false hits
> 132111755 miss
> 7827270 too long
> 63323906 pass2 hits
> 64765455 2passes
> cache hits (88% pos + 4% neg) system 2% per-process
> deletions 0%, falsehits 0%, toolong 0%
>
> The only tunings I have are from way back:
>
> vm.anonmin=30 # default 10
> vm.filemax=30 # default 50
>
> Kernel is:
> NetBSD slave 8.0_RC1 NetBSD 8.0_RC1 (SLAVE) #0: Mon Apr 30 14:24:07 AEST 2018 stix@slave:/home/netbsd/netbsd-8/obj.amd64/home/netbsd/netbsd-8/src/sys/arch/amd64/compile/SLAVE amd64
>
> --
> Paul Ripke
> "Great minds discuss ideas, average minds discuss events, small minds
> discuss people."
> -- Disputed: Often attributed to Eleanor Roosevelt. 1948.
--
Paul Ripke
"Great minds discuss ideas, average minds discuss events, small minds
discuss people."
-- Disputed: Often attributed to Eleanor Roosevelt. 1948.
Home |
Main Index |
Thread Index |
Old Index