Re: High vm scan rate and dropped keystrokes thru X?

[Paul Ripke <stix%stix.id.au@localhost>]

On Mon, Jul 26, 2021 at 05:53:19PM +1200, Lloyd Parkes wrote:
> It has been a very long time since I had to look at UVM stuff, but luckily
> past me post to
> https://mail-index.netbsd.org/tech-repository/2010/02/01/msg000364.html.
> Well done past me.
> 
> Copying from that post, I was using
>   vm.anonmin = 10
>   vm.filemin = 5
>   vm.execmin = 5
>   vm.anonmax = 90
>   vm.filemax = 10
>   vm.execmax = 30
> 
> 
> On 25/07/21 5:37 pm, Paul Ripke wrote:
> > NetBSD 9.2, amd64, 16GiB RAM, quad core + hyperthreading.
> 
> Sounds normal enough.
> 
> >   procs    memory      page                       disks   faults      cpu
> >   r b      avm    fre  flt  re  pi   po   fr   sr w0 w1   in   sy  cs us sy id
> >   0 2 12214336  86564 4043   0   0    0    0    0 66 66 2415 9142 4588 0  3 97
> 
> That's 12GB of RAM in use and 86MB of RAM free. Sounds pretty awful to me.

Sounds normal to me - I don't expect to see any free RAM unless I've just
- exited a large process
- deleted a large file with large cache footprint
- released a large chunk of RAM by other means (mmap, madvise, semctl, etc).

> What does top or vmstat -s say about pages active/inactive and
> anonymous/cachdd file/cached executable pages. This might give you a hint
> about where all your memory has gone and what it is being used for.

A big chunk of it is in file cache, which is unsurprising when reading
thru a 400GiB file...

From top, around the time things go south - note that these firefox processes
aren't actually that busy, their percentages are normally <2%, but the
percentages spike up during periods of high scan rate. I'm pretty sure this
is just a monitoring artifact.

load averages:  3.08,  2.79,  1.87;               up 41+10:40:47
130 processes: 1 runnable, 124 sleeping, 1 stopped, 4 on CPU
CPU states:  0.2% user,  0.0% nice, 36.8% system,  1.9% interrupt, 60.9% idle
Memory: 8545M Act, 3257M Inact, 441M Wired, 446M Exec, 2842M File, 8120K Free
Swap: 10G Total, 3074M Used, 7166M Free

  PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
    0 root     126    0     0K   41M CPU/7     25.7H 53.27% 53.27% [system]
 5629 stix      43    0  3512M  641M parked/4  58:59 29.69% 29.69% firefox
10881 stix      43    0  3806M  920M parked/5 304:50 14.55% 14.55% firefox
12200 stix      43    0  3276M  555M parked/2 921:49 12.99% 12.99% firefox
 6583 stix     223    0    33M 6604K CPU/2      0:59 10.74% 10.74% zstd
19767 stix      41    0  3708M  906M CPU/6    274:46  6.05%  6.05% firefox
 1110 root      85    0   231M   29M select/1 313:02  4.49%  4.49% X
 4227 stix      85    0    60M   32M ttyraw/0  26.6H  4.05%  4.05% systat
28842 stix      85    0  2791M 1211M psem/5   700:48  1.17%  1.17% java
  981 stix      85    0    78M 4952K select/6  23:28  0.88%  0.88% xterm

Looking at 'vmstat -s' around the time of badness, I don't see anything
obvious standing out. Apart from apparently we have a bug causing several
counters to either be negative or up around int64_max...

     4096 bytes per page
        8 page colors
  4055762 pages managed
     2485 pages free
  2031945 pages active                                       
   991997 pages inactive                        
        0 pages paging
   112918 pages wired
     1999 zero pages
        1 reserve pagedaemon pages
       40 reserve kernel pages             
   118829 boot kernel pages                     
   873053 kernel pool pages                                                                                                              
  2280040 anonymous pages                                                                                                                
   742869 cached file pages                
   114192 cached executable pages               
     2048 minimum free pages                                                                                                             
     2730 target free pages                                                                                                              
  1351920 maximum wired pages                    
        1 swap devices                                                                                                                   
  2621439 swap pages                           
   786650 swap pages in use                           
  6124373 swap allocations                     
12709593653 total faults taken
11622482264 traps                                                                                                                        
1587775504 device interrupts                                                                                                             
10242360957 CPU context switches
3331649724 software interrupts
51409703491 system calls
  6103141 pagein requests
  1446005 pageout requests
        0 pages swapped in
 11581719 pages swapped out
 19154795 forks total
  8291070 forks blocked parent
  8291070 forks shared address space with parent
18446744072435606437 pagealloc zero wanted and avail
836249173 pagealloc zero wanted and not avail
  2837515 aborts of idle page zeroing
1642113718 pagealloc desired color avail
444528218 pagealloc desired color not avail
18446744071642829630 pagealloc local cpu avail
18446744073567948242 pagealloc local cpu not avail
  3061387 faults with no memory
        0 faults with no anons
   187195 faults had to wait on pages
        0 faults found released page
321394271 faults relock (321099659 ok)
2059885213 anon page faults
  6613287 anon retry faults
1013596840 amap copy faults
947559669 neighbour anon page faults
18446744073644761375 neighbour object page faults
18446744072139427829 locked pager get faults
314680200 unlocked pager get faults
1712939778 anon faults
347072427 anon copy on write faults
2045952292 object faults
679029853 promote copy faults
18446744072655521116 promote zero fill faults
  1910375 times daemon wokeup
   177319 revolutions of the clock hand
1037588023 pages freed by daemon
18446744072483148432 pages scanned by daemon
 12249099 anonymous pages scanned by daemon
1026614387 object pages scanned by daemon
511505458 pages reactivated
   117779 pages found busy by daemon
 10135642 total pending pageouts
18446744072509618517 pages deactivated
10156800349 total name lookups
9015302910 good hits
742050237 negative hits
 11162879 bad hits
 17016266 false hits
306084096 miss
 65183961 too long
175090846 pass2 hits
183383171 2passes
          cache hits (88% pos + 7% neg) system 1% per-process
          deletions 0%, falsehits 0%, toolong 0%

-- 
Paul Ripke
"Great minds discuss ideas, average minds discuss events, small minds
 discuss people."
-- Disputed: Often attributed to Eleanor Roosevelt. 1948.