High vm scan rate and dropped keystrokes thru X?

[Paul Ripke <stix%stix.id.au@localhost>]

NetBSD 9.2, amd64, 16GiB RAM, quad core + hyperthreading.

I've repeatedly noticed an issue where a large amount of disk reads can
result in lost keystrokes, jerky mouse behaviour and other weirdness.

On this occasion, I was trying to "zstd -vt" a 400GiB backup archive
from an ffsv2 USB attached HDD. Normally, it hums along at 30MiB/s, and
the system is perfectly capable of other tasks. But occasionally (maybe
even once or twice a minute), the system partially wedges, drops
keystrokes (logged in via X), jerky mouse, and largely unresponsive.

"vmstat 1" during these events shows climbing runqueue, falling free
memory, high reclaim rate, very high scan rate, and 8 CPUs worth of
system time - and I hear the BIOS spinning up the CPU fan.

 procs    memory      page                       disks   faults      cpu
 r b      avm    fre  flt  re  pi   po   fr   sr w0 w1   in   sy  cs us sy id
 0 2 12214336  86564 4043   0   0    0    0    0 66 66 2415 9142 4588 0  3 97
 0 2 12246244  54100 4171   0   0    0    0    0  1  0 2040 16832 4405 1 1 98
 4 2 12277980  21652 4103   0   0    0    0    0 13  6 2075 13280 4222 1 3 96
 1 2 12264920  36772 3593 730   0    0 10950 44492 0  0 1934 9351 3595 1  3 96
 1 2 12297212   8880 4043   0   0    0 1080 1301  0  0 1994 8011 3396 0  1 99
 2 2 12275644  26016 3612 3942  0    0 11516 52959 7  7 2011 8851 3658 1  2 97
 1 2 12264288  37536 4370 2238  1    0 10020 44666 1  0 1975 11566 3643 1 3 96
 1 2 12296692  34392 4182   0   0    0 7347 7534 29 14 2333 16088 4706 1 2 97
 1 3 14096876  43680 1895 40019 0    0 6029 697167 10 10 1192 9816 2831 1 12 87
 3 2 12283292  18412 3240   0   0    0    0    0 54 54 2004 8961 3589 1  5 94
 3 2 12260528  42140 2571 11816 0    0 11055 233685 0 0 1559 7628 2929 0 11 88
 0 2 12292320  14028 4106   0   1    0  893  928  1  0 2134 11051 3810 1 1 98
 1 2 12277084  26908 2753 4339  0    0 8147 231620 19 8 1567 17264 3798 1 14 85
 9 1 12274492  28764 2441 22971 0    0 5247 1078197 10 13 1792 15713 4132 1 16 83
 3 1 16027172  11588 2250 18831 0    0  242 1078430 195 82 1665 7999 4199 1 17 82
 5 2 13761124  11968  161 6880  0    0  406 1078429 39 33 734 6303 1888 1 53 46
 14 3 12292180 10920  276 17300 0    0   17 1078429 10 127 580 9782 1833 0 76 23
 0 3 13678700   8304  543 13567 0    0   21 1424324 30 13 713 6863 1786 1 43 56
 5 2 16608812   8212   44 17185 0    0   39 1811100 7 4  553 5491 1479 1 18 82
 7 1 15149532   8196   22 5954  0    0   11 1078667 35 33 605 4981 1571 0 62 38
 6 2 12453392   8060    9 9646  0    0    1 1078680 100 106 559 5284 1446 0 60 40
 3 2 13072560   8084   23 8158  0    0   13 1273217 1 1  433 6344 1389 0 79 21
 8 1 16595948   8060    9 15398 1    0    0 1959533 1 1  507 4450 1207 0 44 56
 3 3 15631436   7888   43 9199  0    0   47 1082027 4 2  339 4153 1121 0 61 39
 4 3 14373172   7940   10 9538  0    0   15 1078707 0 0  408 4972 1172 0 60 40
 1 6 12424060   7000  206 9925  0    0    8 1078463 20 24 628 7176 1881 0 47 53
 8 4 14632192   5888  413 13300 0    0   43 1662082 13 8 427 8161 1813 1 67 33
 8 2 16117916   6468  405 17990 0    0   53 1450183 3 1  523 10788 1787 0 69 31
 9 4 16260308   7500   31 10739 0    0   30 1201978 17 15 541 4683 1284 1 62 38
 10 4 14928620  9044   27 7773  0    0   14 1078812 5 4  384 4760 1284 1 83 16
 7 3 13699356  10464   30 6783  0    0   17 1079299 3 2  392 5252 1109 0 82 18
 1 3 13023464   9600  651 9757  0    0   14 1260070 75 61 920 9891 2786 0 35 64
 0 6 16074608   9512   42 13714 0    0   32 1841906 2 11 463 5624 1379 0 37 63
 13 1 16224928  9676   33 10763 0    0   88 1214395 8 12 512 4572 1360 0 32 67
 28 3 14692152  9060   28 4717  0    0    0 1079161 1 0  524 3802 967  0 66 34
 3 9 12545240   8108   31 12217 0    0    5 1079165 4 2  637 6603 1341 0 44 55
 12 3 13690532  7692   23 12431 0    0    8 1427075 48 48 571 4545 1474 0 55 45
 10 3 15619384  7432   15 8081  0    0   10 1561463 4 2  400 4818 1289 0 66 34
 25 2 16226876  7300    5 10344 0    0    0 1231134 0 0  425 4729 1140 0 98  2
 14 2 16615924  7236    5 7563  0    0    1 1176532 6 6  452 4660 1232 0 99  1
 38 2 15239700  7168    8 8346  0    0    0 1079291 8 7  625 4333 1127 0 99  1
 0 2 12275088  26032 3857   0   1    0 11140 11261 95 89 2127 131045 5501 3 40 58
 1 2 12265968  37412 3728   0   0    0 10115 10523 17 15 1770 16068 3847 1 4 95
 1 2 12259936  43424 3906   0   0    0 8966 9136  9  9 1864 12628 3535 1 2 97
 0 2 12292244  11120 4063   0   0    0    0    0  3  1 2087 9112 3538 1  2 97
 0 2 12287804  15964 4414   0   0    0 8524 8609  8  2 1956 8249 3440 1  3 96
 1 1 12282612  20756 3674   0   0    0 8529 8614 42 29 2126 11998 4235 1 4 95
 1 1 12265308  37156 3602   0   0    0 11299 11826 0  0 1831 7291 3287 0  3 97
 0 2 12280260  43680 3784   0   0    0 9141 9345  7  7 1941 11775 3904 0 2 97
 1 2 12290916  12452 3905   0   0    0    0    0  0  0 1933 5999 3066 0  2 98

I'm wondering if my tweaked vm sysctl's might be to blame?

vm.anonmin=30
vm.filemax=20

But they're not a huge departure from default.

-- 
Paul Ripke
"Great minds discuss ideas, average minds discuss events, small minds
 discuss people."
-- Disputed: Often attributed to Eleanor Roosevelt. 1948.