Re: High system times on Scaleway Cloud server

[Alexander Nasonov <alnsn%yandex.ru@localhost>]

Alexander Nasonov wrote:
> System times went up to 90-99% across all cores and I quickly reran
> the command:
> 
> # vmstat -e|awk 'NR==1||NF<3||$(NF-1)>1000'
> interrupt                                         total  rate type
> pmap pages remapped                             6208493  1060 misc
> pmap pages mapped (including remapped)         94106306 16078 misc   <-
> pmap managed pages mapped                      93952303 16051 misc   <-
> pmap user pages mapped                         92805605 15856 misc   <-
> pmap pmap_protect managed pages                 7099051  1212 misc
> pmap pmap_protect non-exists pages             22854070  3904 misc   <-
> pmap pv_entry free and unlink                  92333957 15775 misc   <-
> pmap pv_entry allocate and link                92464763 15797 misc   <-
> 
> Marked entries have about 2 times higher rates compared to the
> previous measurement.

System time is also quite high on Pinebook but not as extreme.
It's often at %50-70% when building NetBSD with -j4. There are
many differences, though.  Pinebook only has 4 cores, it doesn't
run inside a VM and disks aren't encrypted.

# vmstat -e|awk 'NR==1||NF<3||$(NF-1)>1000'
interrupt                                         total  rate type
pmap pages remapped                             1618281  2910 misc
pmap pages mapped (including remapped)         22547767 40553 misc
pmap managed pages mapped                      22456977 40390 misc
pmap user mapping changed                       1209363  2175 misc
pmap user pages mapped                         22309098 40124 misc
pmap pmap_protect managed pages                 1707750  3071 misc
pmap pmap_protect non-exists pages              3365647  6053 misc
pmap pmap_deactivate call                       1558572  2803 misc
pmap pmap_activate call                         1558230  2802 misc
pmap pv_entry free and unlink                  21999341 39567 misc
pmap pv_entry allocate and link                22047600 39653 misc
pmap page table page free (uvm_pagefree)         725050  1304 misc
pmap page table page allocate (uvm_pagealloc)    726238  1306 misc
cpu1 runqueue stay                               713871  1283 misc
cpu2 runqueue stay                               641845  1154 misc
cpu3 runqueue stay                               711817  1280 misc
armgic (cpu1) irq 94                             612847  1102 intr
armgic (cpu2) irq 94                             599036  1077 intr
armgic (cpu3) irq 94                             563064  1012 intr


Some pmap entries are at ~40k rate:

# vmstat -e|awk 'NR==1||NF<3||$(NF-1)>10000'
interrupt                                         total  rate type
pmap pages mapped (including remapped)         50992312 42352 misc
pmap managed pages mapped                      50830520 42218 misc
pmap user pages mapped                         50504733 41947 misc
pmap pv_entry free and unlink                  50022954 41547 misc
pmap pv_entry allocate and link                50062730 41580 misc


tprof doesn't work on Pinebook.

-- 
Alex

load averages:  3.99,  3.54,  2.90;               up 0+00:29:30       12:08:35
71 processes: 3 runnable, 64 sleeping, 4 on CPU
CPU0 states: 22.7% user,  0.0% nice, 72.6% system,  2.5% interrupt,  2.2% idle
CPU1 states: 26.8% user,  0.0% nice, 71.3% system,  0.6% interrupt,  1.2% idle
CPU2 states: 24.8% user,  0.0% nice, 73.8% system,  0.8% interrupt,  0.6% idle
CPU3 states: 29.7% user,  0.0% nice, 69.1% system,  0.2% interrupt,  1.0% idle
Memory: 425M Act, 1324K Inact, 15M Wired, 40M Exec, 276M File, 1363M Free
Swap: 128M Total, 128M Free
Seconds to delay:
  PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
  938 alnsn     78    0    21M   16M kqueue/2   2:46 20.17% 20.17% tmux
    0 root       0    0     0K   87M CPU/0      2:29  0.00%  8.11% [system]
 5334 alnsn     85    0  6516K 2284K select/3   0:14  6.05%  6.05% nbmake
14149 alnsn     27    0    37M   21M RUN/1      0:00 48.00%  2.34% cc1
  370 tor       85    0    46M   38M kqueue/0   0:36  1.90%  1.90% tor
24710 alnsn     78    0  6736K 1832K wait/3     0:00  1.44%  1.37% sh
10316 alnsn     27    0    35M   18M CPU/2      0:00 23.00%  1.12% cc1
 4898 alnsn     78    0  8596K 2472K select/0   0:00  1.27%  0.78% nbmake
 2520 alnsn     85    0    11M 2232K select/3   0:00  0.56%  0.34% nbmake
 6099 alnsn     85    0    12M 2264K select/3   0:02  0.29%  0.29% nbmake
15260 alnsn     27    0    28M   13M RUN/3      0:00  6.00%  0.29% cc1
 1048 alnsn     85    0  6396K 3652K select/3   0:00  0.24%  0.24% nbmake
 5785 alnsn     85    0  7336K 1856K select/3   0:00  0.25%  0.24% nbmake
 5097 alnsn     85    0  8780K 2284K select/3   0:02  0.10%  0.10% nbmake
24928 alnsn     85    0    12M 2276K select/3   0:00  0.10%  0.10% nbmake