NetBSD-Bugs archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: port-amd64/54988: possible memory leaks/swap problems



The following reply was made to PR port-amd64/54988; it has been noted by GNATS.

From: mlh%goathill.org@localhost (MLH)
To: gnats-bugs%netbsd.org@localhost
Cc: ad%netbsd.org@localhost, gnats-admin%netbsd.org@localhost, netbsd-bugs%netbsd.org@localhost, 
 mlh%goathill.org@localhost
Subject: Re: port-amd64/54988: possible memory leaks/swap problems
Date: Tue, 24 Mar 2020 17:16:30 -0400 (EDT)

 MLH wrote:
 >  >  > So the system is down to less than 20M of phymem and never recovers
 >  >  > much more than maybe 10M of it. Have to reboot to reclaim a workable
 >  >  > system, so I have to reboot 2-3 times a day to do much of any work.
 >  >  >
 >  >  what does vmstat -mvW show when memory gets low the whole output?
 >  >  
 >  >  I'm interested especially in the kva pools.
 >  >  
 >  >  Lars
 >  
 >  Ok. I will check that next time. As of NetBSD 9.99.51 Sun Mar 22,
 
 Two operations, build a kernel then build a -current distribution
 but didn't finish the second. It ground to a halt so I stopped it.
 
 Took three sets of vmstats before building the kernel, before
 building the distribution, and after stopping it and rebooting.
 
 Before building the kernel
 $ vmstat
  procs    memory      page                       disks   faults      cpu
  r b      avm    fre  flt  re  pi   po   fr   sr w0 w1   in   sy  cs us sy id
  2 1   823440 587248  348   0   0    0   34   46  2 20  127 2511 574  0  0 100
 $ vmstat -mvW 
 Memory resource pool statistics
 Name               Size    Requests    Fail    Releases    InUse    Avail      Pgreq      Pgrel   Npage PageSz  Hiwat Minpg  Maxpg Idle   Flags   Util
 ah_tdb_crypto       192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 aio_jobs_pool       136           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 aio_lio_pool         48           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 amappl               88       10306       0        6799     3507     4278        175          2     173   4096    175     0    inf    0 0x10040  43.6%
 anonpl               40      246829       0      193244    53585   104985       1777        207    1570   4096   1777     0    inf    1 0x01040  33.3%
 ataspl              160     2501711       0     2501711        0       50          4          2       2   4096      3     0    inf    2 0x10040   0.0%
 biopl               304       29139       0       28899      240        7       1558       1539      19   4096   1558     0    inf    0 0x10040  93.8%
 brtpl                64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 buf16k            16392       11283       0       10529      754        2       3208       2956     252  65536   1505     1      1    0 0x10040  74.8%
 buf1k              1032          59       0          59        0        3         19         18       1   4096     19     1      1    1 0x10040   0.0%
 buf2k              2056       36539       0       35314     1225        1      36540      35314    1226   4096  17763     1      1    1 0x10040  50.2%
 buf32k            32776       38146       0       28418     9728        1      38147      28418    9729  65536   9729     1      1    1 0x10040  50.0%
 buf4k              4096      137128       0       62989    74139        1     137129      62989   74140   4096  88754     1      1    1 0x10000 100.0%
 buf512b             520        1186       0        1185        1        6        138        137       1   4096    137     1      1    0 0x10040  12.7%
 buf64k            65536         240       0         232        8        1        241        232       9  65536    241     1      1    1 0x10000  88.9%
 buf8k              8200        1947       0        1483      464        5        170        103      67  65536     80     1      1    0 0x10040  86.7%
 bufpl               304       95792       0        7819    87973     7096       7369         56    7313   4096   7369     0    inf   53 0x10040  89.3%
 carp                 32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 carp6                32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 ccdbuf              336           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cd9660nopl          208           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptdesc           128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptkop            384           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptop             320           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 csepl               232           0       0           0        0      323         19          0      19   4096     19    19    inf   19 0x10040   0.0%
 cwdi                 64         247       0         157       90       99          3          0       3   4096      3     0    inf    0 0x10040  46.9%
 dbregs              144           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 dirhepl              48           0       0           0        0        0          0          0       0   4096      0     0    313    0 0x00040   ---
 dirhpl              304           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 efsinopl            248           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ehcixfer            384           7       0           3        4        6          1          0       1   4096      1     0    inf    0 0x10040  37.5%
 ehcixfer            384          18       0          14        4        6          2          1       1   4096      2     0    inf    0 0x10040  37.5%
 esp_tdb_crypto      128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 execargs         262144        2335       0        2335        0        2          8          6       2 262144      4     0     16    2 0x10c00   0.0%
 ext2fsinopl         256           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 extent               48           9       0           8        1       83          1          0       1   4096      1     0    inf    0 0x00040   1.2%
 fcrpl               184          31       0          30        1       83          4          0       4   4096      4     4    inf    3 0x10040   1.1%
 fdfile               64        3261       0        1048     2213      496         43          0      43   4096     43     0    inf    0 0x11040  80.4%
 ffsdino1            136           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ffsdino2            264      251731       0       42894   208837       38      14321        396   13925   4096  14320     0    inf    2 0x10040  96.7%
 ffsino              256      251571       0       42734   208837       43      13427        372   13055   4096  13425     0    inf    2 0x10000 100.0%
 file                128        2201       0         624     1577      345         64          2      62   4096     64     0    inf    0 0x10040  79.5%
 filedesc            832         243       0         153       90       38         38          6      32   4096     38     0    inf    0 0x10040  57.1%
 icmp                 32           2       0           2        0      126          1          0       1   4096      1     0    inf    1 0x00040   0.0%
 icmp6                32           6       0           6        0        0          2          2       0   4096      1     0    inf    0 0x00040   ---
 igmppl               40           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 in6pcbpl            280        3247       0        3234       13        1          3          2       1   4096      3     0    inf    0 0x10040  88.9%
 inmltpl              56           2       0           0        2       70          1          0       1   4096      1     0    inf    0 0x00040   2.7%
 inpcbpl             240        3215       0        3192       23        9          9          7       2   4096      9     0    inf    0 0x10040  67.4%
 ipcomp_tdb_cryp     128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ipfrenpl             64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 kcpuset              64         377       0         175      202      176          6          0       6   4096      6     0    inf    0 0x10040  52.6%
 kcredpl             192         573       0         177      396      129         25          0      25   4096     25     0    inf    0 0x10040  74.2%
 kmem-00008            8           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10400   ---
 kmem-00016           16       10441       0        6306     4135     5849         40          1      39   4096     40     0    inf    0 0x10400  41.4%
 kmem-00032           32        8014       0        4258     3756     3924         60          0      60   4096     60     0    inf    0 0x10400  48.9%
 kmem-00064          128       12205       0        7821     4384     2343        377        160     217   4096    377     0    inf    1 0x10040  63.1%
 kmem-00128          192        9384       0        2341     7043      895        379          1     378   4096    379     0    inf    0 0x10040  87.3%
 kmem-00192          256     4088775       0         182  4088593       15     255538          0  255538   4096 255538     0    inf    0 0x10000 100.0%
 kmem-00256          320         958       0         423      535      209         70          8      62   4096     70     0    inf    0 0x10040  67.4%
 kmem-00320          384        1160       0         346      814      206        107          5     102   4096    107     0    inf    0 0x10040  74.8%
 kmem-00384          448         519       0         134      385       65         50          0      50   4096     50     0    inf    0 0x10040  84.2%
 kmem-00448          512         369       0         149      220       68         39          3      36   4096     39     0    inf    0 0x10000  76.4%
 kmem-00512          576         279       0         119      160       50         31          1      30   4096     31     0    inf    0 0x10040  75.0%
 kmem-00768          832        1034       0         367      667      121        240         43     197   4096    240     0    inf    0 0x10040  68.8%
 kmem-01024         1088        2493       0        1499      994      233        700        291     409   4096    700     0    inf    0 0x10040  64.6%
 kmem-02048         2112        1202       0         774      428        0       1192        764     428   4096   1129     0    inf    0 0x10040  51.6%
 kmem-04096         4096         256       0         102      154        0        250         96     154   4096    193     0    inf    0 0x10000 100.0%
 ksiginfo            136         207       0         147       60       27          5          2       3   4096      3     0    inf    0 0x10040  66.4%
 ktrace              128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 kva-12288         12288         115       0          70       45       18          4          1       3 262144      4     0    inf    0 0x10e00  70.3%
 kva-16384         16384          81       0          59       22       10          4          2       2 262144      4     0    inf    0 0x10e00  68.8%
 kva-20480         20480          83       0          55       28        8          7          4       3 262144      7     0    inf    0 0x10e00  72.9%
 kva-24576         24576          30       0          19       11        9          4          2       2 262144      3     0    inf    0 0x10e00  51.6%
 kva-28672         28672          13       0          11        2        7          3          2       1 262144      2     0    inf    0 0x10e00  21.9%
 kva-32768         32768           8       0           6        2        6          1          0       1 262144      1     0    inf    0 0x10e00  25.0%
 kva-36864         36864          17       0           6       11        3          3          1       2 262144      3     0    inf    0 0x10e00  77.3%
 kva-4096           4096           0       0           0        0        0          0          0       0 262144      0     0    inf    0 0x10e00   ---
 kva-40960         40960           3       0           2        1        5          1          0       1 262144      1     0    inf    0 0x10e00  15.6%
 kva-49152         49152           5       0           4        1        4          1          0       1 262144      1     0    inf    0 0x10e00  18.8%
 kva-65536         65536           4       0           4        0        0          1          1       0 262144      1     0    inf    0 0x10e00   ---
 kva-8192           8192         105       0          58       47       17          3          1       2 262144      3     0    inf    0 0x10e00  73.4%
 l2cap_pdu            56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 l2cap_req           128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsdinopl           264           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsinoextpl         200           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsinopl            224           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfslbnpool           32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 llentrypl           280           1       0           0        1       13          1          0       1   4096      1     0    inf    0 0x10040   6.8%
 lockf               120         101       0          82       19       14          3          2       1   4096      3     0    inf    0 0x10040  55.7%
 lwppl              1088         306       0         127      179       49         97         21      76   4096     97     0    inf    6 0x10040  62.6%
 mbpl                520         806       0         415      391      162         83          4      79   4096     83     3    inf    2 0x10040  62.8%
 mclpl              2112         598       0         280      318        8        473        147     326   4096    426     8 130541    8 0x10040  50.3%
 mqmsgpl            1088           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 msdosfhpl            56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 msdosnopl           208          63       0          63        0        0          4          4       0   4096      4     0    inf    0 0x10040   ---
 mutex                64      235184       0       40771   194413    18905       3416         30    3386   4096   3416     0    inf   57 0x10040  89.7%
 nchentry            192      228451       0       31717   196734    12174       9948          0    9948   4096   9948     0    inf    0 0x10040  92.7%
 nfsnodepl           280           3       0           1        2       12          1          0       1   4096      1     0    inf    0 0x10040  13.7%
 nfsreqcachepl       104           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 nfsrvdescpl         256           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 nfsvapl             184           3       0           1        2       19          1          0       1   4096      1     0    inf    0 0x10040   9.0%
 npfcn4pl            144           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npfcn6pl            192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npfnatpl             96           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npftblpl             48           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 pcache             2752          86       0           4       82        0         82          0      82   4096     82     0    inf    0 0x10040  67.2%
 pcachecpu            64         267       0           0      267       48          5          0       5   4096      5     0    inf    0 0x10040  83.4%
 pcglarge           1088        6709       0        6306      403        2       1643       1508     135   4096   1516     0    inf    0 0x10040  79.3%
 pcgnormal           320       50648       0       42378     8270     2698       1462        548     914   4096   1380     0    inf    0 0x10040  70.7%
 pdict128            192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 pdict16              80         301       0         252       49        1          2          1       1   4096      2     0    inf    0 0x10040  95.7%
 pdict32              96          15       0           2       13       29          1          0       1   4096      1     0    inf    0 0x10040  30.5%
 pdppl              4096         181       0          87       94        0        179         85      94   4096    167     0    inf    0 0x10000 100.0%
 pewpl                32           0       0           0        0      126          1          0       1   4096      1     1      1    1 0x00040   0.0%
 phpool-1024         184           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-128           72          60       0           0       60       52          2          0       2   4096      2     0    inf    0 0x10040  52.7%
 phpool-2048         312           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 phpool-256           88          40       0           1       39        6          1          0       1   4096      1     0    inf    0 0x10040  83.8%
 phpool-4096         568           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-512          120           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-64            64      407275       0       63981   343294       56       5493         43    5450   4096   5450     0    inf    0 0x10040  98.4%
 phpool-64            64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 piperd              320         163       0          82       81       63         12          0      12   4096     12     0    inf    0 0x10040  52.7%
 pipewr              320         186       0         108       78       78         14          1      13   4096     14     0    inf    0 0x10040  46.9%
 plimitpl            240          61       0          39       22       26          4          1       3   4096      4     0    inf    0 0x10040  43.0%
 pmappl              512         181       0          87       94       74         21          0      21   4096     21     0    inf    0 0x10000  56.0%
 pnbufpl            1032         189       0         142       47        1         37         21      16   4096     30     0    inf    0 0x10040  74.0%
 procpl              896         127       0          39       88       28         30          1      29   4096     30     0    inf    0 0x10040  66.4%
 proparay             56         130       0          13      117       27          2          0       2   4096      2     0    inf    0 0x00040  80.0%
 propdata             48           1       0           0        1       83          1          0       1   4096      1     0    inf    0 0x00040   1.2%
 propdict             56         507       0         156      351        9          6          1       5   4096      6     0    inf    0 0x00040  96.0%
 propnmbr             64          47       0          11       36       27          1          0       1   4096      1     0    inf    0 0x10040  56.2%
 propstng             48         860       0         290      570       18          8          1       7   4096      8     0    inf    0 0x00040  95.4%
 pstatspl            456         128       0          40       88       32         15          0      15   4096     15     0    inf    0 0x10040  65.3%
 ptimerpl            328         101       0          84       17        7          4          2       2   4096      4     0    inf    0 0x10040  68.1%
 ptimerspl           312         101       0          84       17        9          3          1       2   4096      3     0    inf    0 0x10000  64.7%
 puffpnpl            248           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 puffprkl            120           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 puffvapl            184           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 pvpl                128      213753       0      110985   102768    21232       6789       2789    4000   4096   6789     0    inf    0 0x11040  80.3%
 ractx                40      139076       0       20272   118804      174       1178          0    1178   4096   1178     0    inf    1 0x00040  98.5%
 radixnode           192       85018       0       43075    41943     7092       3340       1005    2335   4096   3340     0    inf    1 0x11040  84.2%
 raidpsspl           200           0       0           0        0       20          1          0       1   4096      1     1      2    1 0x10040   0.0%
 rf_alloclist_pl     264     1962391       0     1962390        1      104         13          6       7   4096      7     5     18    6 0x10040   0.9%
 rf_asm_pl           496     1962390       0     1962390        0       96         24         12      12   4096     12     8     24   12 0x10040   0.0%
 rf_asmhdr_pl         32      915977       0      915977        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_asmhle_pl         24           0       0           0        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_callbackfpl       32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_callbackvpl       32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_dagh_pl          136     1962390       0     1962390        0       87          6          3       3   4096      3     2      5    3 0x10040   0.0%
 rf_daglist_pl       336     1962390       0     1962390        0       72         14          8       6   4096      6     3     11    6 0x10040   0.0%
 rf_dagnode_pl       680     8187823       0     8187823        0      252        107         65      42   4096     47    22     86   42 0x10000   0.0%
 rf_dagpcache_pl     720           0       0           0        0       10          2          0       2   4096      2     2     26    2 0x10040   0.0%
 rf_dqd_pl           208     2300653       0     2300653        0      114         12          6       6   4096      7     4     14    6 0x10040   0.0%
 rf_fss_pl            48           0       0           0        0       84          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_funclist_pl       24     1962390       0     1962390        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_mcpair_pl         48           0       0           0        0       84          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_pda_pl            64     3924780       0     3924780        0      126          2          0       2   4096      2     2      4    2 0x10040   0.0%
 rf_rad_pl           488      915977       0      915977        0       40          6          1       5   4096      5     4     16    5 0x10040   0.0%
 rf_reconbuffer_     112           0       0           0        0       36          1          0       1   4096      1     1      2    1 0x10040   0.0%
 rf_revent_pl         32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_stripelock_p      56     1962390       0     1962390        0       72          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_vfple_pl          24           0       0           0        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_vple_pl           24          30       0           0       30      138          1          0       1   4096      1     1      2    0 0x00040  17.6%
 rfcomm_credit        32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 rndctx               24          29       0          20        9      159          1          0       1   4096      1     0    inf    0 0x00040   5.3%
 rndsample           544          65       0          40       25       10          6          1       5   4096      6     0    586    0 0x10040  66.4%
 rndtemp             520           5       0           4        1        6          2          1       1   4096      1     0    inf    0 0x10040  12.7%
 rtentpl             328          28       0           2       26       10          3          0       3   4096      3     0    inf    0 0x10040  69.4%
 rttmrpl              72           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 rwlock               64      257353       0       45032   212321     7045       3519         37    3482   4096   3519     0    inf    0 0x10040  95.3%
 sackholepl           40          17       0          17        0      101          2          1       1   4096      1     0    inf    1 0x00040   0.0%
 scxspl              264       29890       0       29890        0       30          2          0       2   4096      2     2    inf    2 0x10040   0.0%
 sigacts            3096         180       0          90       90        0        178         88      90   4096    170     0    inf    0 0x10040  75.6%
 smbfsnopl           176           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 smbrqpl             296           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 smbt2pl             232           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 socket              616         492       0         157      335       43         75         12      63   4096     75     0    inf    0 0x10040  80.0%
 swp vnd             312           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 swp vnx              40           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 synpl               320          29       0          29        0        0          1          1       0   4096      1     0    inf    0 0x10040   ---
 tcpcbpl             840         829       0         815       14        2         33         29       4   4096     33     0    inf    0 0x10040  71.8%
 tcpipqepl            64         508       0         508        0        0          1          1       0   4096      1     0    inf    0 0x10040   ---
 thplthrd             80           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 tmpfs_dirent         56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 tmpfs_node          224           1       0           0        1       17          1          0       1   4096      1     0    inf    0 0x10040   5.5%
 tstile              128         298       0         116      182      128         10          0      10   4096     10     0    inf    0 0x10040  56.9%
 uaoeltpl            104           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 uarea             20480         208       0         116       92        0        206        114      92  20480    200     0    inf    0 0x10c00 100.0%
 uareasys          20480          93       0           3       90        0         93          3      90  20480     93     0    inf    0 0x10c00 100.0%
 ufsdir              272           9       0           7        2       13          2          1       1   4096      1     0    inf    0 0x10000  13.3%
 ufsdq                88           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 vcachepl            640      267595       0       58727   208868     1444      35804        752   35052   4096  35804     0    inf  240 0x10040  93.1%
 vmembt               64       23267       0        8983    14284     2411        265          0     265   4096    265     0    inf    0 0x10040  84.2%
 vmmpepl             192       23808       0       12766    11042    11113       1107         52    1055   4096   1107     0    inf    0 0x10040  49.1%
 vmsppl              360         176       0          86       90       64         16          2      14   4096     16     0    inf    0 0x10040  56.5%
 wapbldealloc         40        1022       0        1022        0      101          5          4       1   4096      4     0    inf    1 0x00040   0.0%
 wapblentrypl         48        3749       0        3749        0       84          1          0       1   4096      1     0    inf    1 0x00040   0.0%
 wapblinopl           40      140898       0      140845       53      149          2          0       2   4096      2     0    inf    0 0x00040  25.9%
 Totals                     37908998       0    31685278  6223720   221228     589460     141572  447888
 
 In use 2041819K, total allocated 2400228K; utilization 85.1%
 -------
 before building the distribution
 $ vmstat
  procs    memory      page                       disks   faults      cpu
  r b      avm    fre  flt  re  pi   po   fr   sr w0 w1   in   sy  cs us sy id
  1 0   494576 781856  867   1   0    2   44   62  3 23  143 2886 674  1  0 99
 $ vmstat -mvW > no2
 Memory resource pool statistics
 Name               Size    Requests    Fail    Releases    InUse    Avail      Pgreq      Pgrel   Npage PageSz  Hiwat Minpg  Maxpg Idle   Flags   Util
 ah_tdb_crypto       192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 aio_jobs_pool       136           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 aio_lio_pool         48           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 amappl               88       13244       0        6900     6344     1441        175          2     173   4096    175     0    inf    0 0x10040  78.8%
 anonpl               40      555865       0      202748   353117       80       3705        208    3497   4096   3497     0    inf    0 0x01040  98.6%
 ataspl              160     3840941       0     3840941        0       75          5          2       3   4096      3     0    inf    3 0x10040   0.0%
 biopl               304       49975       0       43487     6488     8111       2662       1539    1123   4096   1558     0    inf  623 0x10040  42.9%
 brtpl                64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 buf16k            16392       11747       0       11033      714        3       3236       2997     239  65536   1505     1      1    1 0x10040  74.7%
 buf1k              1032          59       0          59        0        3         19         18       1   4096     19     1      1    1 0x10040   0.0%
 buf2k              2056       36566       0       36299      267        1      36567      36299     268   4096  17763     1      1    1 0x10040  50.0%
 buf32k            32776       42414       0       34726     7688        1      42415      34726    7689  65536   9818     1      1    1 0x10040  50.0%
 buf4k              4096      147134       0       71769    75365        1     147135      71769   75366   4096  88754     1      1    1 0x10000 100.0%
 buf512b             520        1252       0        1250        2        5        138        137       1   4096    137     1      1    0 0x10040  25.4%
 buf64k            65536         240       0         232        8        1        241        232       9  65536    241     1      1    1 0x10000  88.9%
 buf8k              8200        2079       0        1917      162      104        171        133      38  65536     80     1      1    0 0x10040  53.3%
 bufpl               304       95954       0        7819    88135     6245       7369        109    7260   4096   7369     0    inf    0 0x10040  90.1%
 carp                 32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 carp6                32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 ccdbuf              336           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cd9660nopl          208           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptdesc           128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptkop            384           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptop             320           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 csepl               232           0       0           0        0      323         19          0      19   4096     19    19    inf   19 0x10040   0.0%
 cwdi                 64         363       0         259      104      148          4          0       4   4096      4     0    inf    1 0x10040  40.6%
 dbregs              144           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 dirhepl              48           0       0           0        0        0          0          0       0   4096      0     0    313    0 0x00040   ---
 dirhpl              304           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 efsinopl            248           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ehcixfer            384           7       0           3        4        6          1          0       1   4096      1     0    inf    0 0x10040  37.5%
 ehcixfer            384          18       0          14        4        6          2          1       1   4096      2     0    inf    0 0x10040  37.5%
 esp_tdb_crypto      128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 execargs         262144       24495       0       24495        0        1         10          9       1 262144      4     0     16    1 0x10c00   0.0%
 ext2fsinopl         256           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 extent               48           9       0           8        1       83          1          0       1   4096      1     0    inf    0 0x00040   1.2%
 fcrpl               184          31       0          30        1       83          4          0       4   4096      4     4    inf    3 0x10040   1.1%
 fdfile               64        3623       0        1353     2270      439         43          0      43   4096     43     0    inf    0 0x11040  82.5%
 ffsdino1            136           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ffsdino2            264      251767       0       76625   175142    33733      14321        396   13925   4096  14320     0    inf   16 0x10040  81.1%
 ffsino              256      251607       0       76465   175142    33738      13427        372   13055   4096  13425     0    inf   14 0x10000  83.8%
 file                128        2408       0         684     1724      198         64          2      62   4096     64     0    inf    0 0x10040  86.9%
 filedesc            832         359       0         255      104       92         55          6      49   4096     49     0    inf   13 0x10040  43.1%
 icmp                 32           2       0           2        0        0          1          1       0   4096      1     0    inf    0 0x00040   ---
 icmp6                32           6       0           6        0        0          2          2       0   4096      1     0    inf    0 0x00040   ---
 igmppl               40           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 in6pcbpl            280        3706       0        3693       13        1          3          2       1   4096      3     0    inf    0 0x10040  88.9%
 inmltpl              56           2       0           0        2       70          1          0       1   4096      1     0    inf    0 0x00040   2.7%
 inpcbpl             240        3607       0        3584       23        9          9          7       2   4096      9     0    inf    0 0x10040  67.4%
 ipcomp_tdb_cryp     128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ipfrenpl             64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 kcpuset              64         575       0         175      400       41          7          0       7   4096      7     0    inf    0 0x10040  89.3%
 kcredpl             192         601       0         200      401      124         25          0      25   4096     25     0    inf    1 0x10040  75.2%
 kmem-00008            8           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10400   ---
 kmem-00016           16       12577       0        6527     6050     3934         40          1      39   4096     40     0    inf    0 0x10400  60.6%
 kmem-00032           32        9464       0        4553     4911     2769         60          0      60   4096     60     0    inf    0 0x10400  63.9%
 kmem-00064          128       13384       0        8163     5221     1475        377        161     216   4096    377     0    inf    0 0x10040  75.5%
 kmem-00128          192       11378       0        3200     8178       12        391          1     390   4096    390     0    inf    0 0x10040  98.3%
 kmem-00192          256     6067336       1         222  6067114        6     379195          0  379195   4096 379195     0    inf    0 0x10000 100.0%
 kmem-00256          320        1364       0         567      797        7         75          8      67   4096     70     0    inf    0 0x10040  92.9%
 kmem-00320          384        1628       0         479     1149        1        120          5     115   4096    115     0    inf    0 0x10040  93.7%
 kmem-00384          448         900       0         228      672        3         75          0      75   4096     75     0    inf    0 0x10040  98.0%
 kmem-00448          512         696       0         244      452        4         60          3      57   4096     57     0    inf    0 0x10000  99.1%
 kmem-00512          576         589       0         202      387        5         57          1      56   4096     56     0    inf    0 0x10040  97.2%
 kmem-00768          832        1727       0         471     1256        0        357         43     314   4096    314     0    inf    0 0x10040  81.2%
 kmem-01024         1088        4666       0        1654     3012        0       1295        291    1004   4096   1004     0    inf    0 0x10040  79.7%
 kmem-02048         2112        2499       0         815     1684        0       2448        764    1684   4096   1684     0    inf    0 0x10040  51.6%
 kmem-04096         4096         476       0         187      289        0        385         96     289   4096    289     0    inf    0 0x10000 100.0%
 ksiginfo            136         326       0         238       88       28          6          2       4   4096      4     0    inf    0 0x10040  73.0%
 ktrace              128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 kva-12288         12288         270       0         165      105       21          7          1       6 262144      6     0    inf    1 0x10e00  82.0%
 kva-16384         16384         215       0         147       68       28          8          2       6 262144      6     0    inf    1 0x10e00  70.8%
 kva-20480         20480         228       0         143       85       35         14          4      10 262144     10     0    inf    2 0x10e00  66.4%
 kva-24576         24576         111       0          41       70        0          9          2       7 262144      7     0    inf    0 0x10e00  93.8%
 kva-28672         28672          30       0          28        2       16          4          2       2 262144      2     0    inf    1 0x10e00  10.9%
 kva-32768         32768          20       0          18        2       14          2          0       2 262144      2     0    inf    1 0x10e00  12.5%
 kva-36864         36864          19       0           8       11        3          3          1       2 262144      3     0    inf    0 0x10e00  77.3%
 kva-4096           4096           0       0           0        0        0          0          0       0 262144      0     0    inf    0 0x10e00   ---
 kva-40960         40960           4       0           3        1        5          1          0       1 262144      1     0    inf    0 0x10e00  15.6%
 kva-49152         49152           5       0           5        0        5          1          0       1 262144      1     0    inf    1 0x10e00   0.0%
 kva-65536         65536           4       0           4        0        0          1          1       0 262144      1     0    inf    0 0x10e00   ---
 kva-8192           8192         260       0         143      117       11          5          1       4 262144      4     0    inf    0 0x10e00  91.4%
 l2cap_pdu            56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 l2cap_req           128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsdinopl           264           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsinoextpl         200           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsinopl            224           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfslbnpool           32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 llentrypl           280           1       0           0        1       13          1          0       1   4096      1     0    inf    0 0x10040   6.8%
 lockf               120         111       0          96       15       18          3          2       1   4096      3     0    inf    0 0x10040  43.9%
 lwppl              1088         417       0         223      194       91        116         21      95   4096     97     0    inf   18 0x10040  54.2%
 mbpl                520         992       0         586      406      133         83          6      77   4096     83     3    inf    3 0x10040  66.9%
 mclpl              2112         772       0         402      370       18        541        153     388   4096    426     8 130541   18 0x10040  49.2%
 mqmsgpl            1088           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 msdosfhpl            56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 msdosnopl           208          63       0          63        0        0          4          4       0   4096      4     0    inf    0 0x10040   ---
 mutex                64      250040       0       74478   175562    34165       3416         87    3329   4096   3416     0    inf    1 0x10040  82.4%
 nchentry            192      240642       0       31879   208763      166       9949          0    9949   4096   9949     0    inf    0 0x10040  98.4%
 nfsnodepl           280           3       0           1        2       12          1          0       1   4096      1     0    inf    0 0x10040  13.7%
 nfsreqcachepl       104           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 nfsrvdescpl         256           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 nfsvapl             184           3       0           1        2       19          1          0       1   4096      1     0    inf    0 0x10040   9.0%
 npfcn4pl            144           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npfcn6pl            192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npfnatpl             96           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npftblpl             48           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 pcache             2752          86       0           4       82        0         82          0      82   4096     82     0    inf    0 0x10040  67.2%
 pcachecpu            64         267       0           0      267       48          5          0       5   4096      5     0    inf    0 0x10040  83.4%
 pcglarge           1088       14696       0        9686     5010        0       3178       1508    1670   4096   1670     0    inf    0 0x10040  79.7%
 pcgnormal           320       99658       0       97237     2421     2943       4687       4240     447   4096   4139     0    inf  245 0x10040  42.3%
 pdict128            192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 pdict16              80         301       0         252       49        1          2          1       1   4096      2     0    inf    0 0x10040  95.7%
 pdict32              96          15       0           2       13       29          1          0       1   4096      1     0    inf    0 0x10040  30.5%
 pdppl              4096         305       0         203      102       91        278         85     193   4096    193     0    inf   91 0x10000  52.8%
 pewpl                32           0       0           0        0      126          1          0       1   4096      1     1      1    1 0x00040   0.0%
 phpool-1024         184           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-128           72          60       0           0       60       52          2          0       2   4096      2     0    inf    0 0x10040  52.7%
 phpool-2048         312           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 phpool-256           88          40       0           1       39        6          1          0       1   4096      1     0    inf    0 0x10040  83.8%
 phpool-4096         568           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-512          120           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-64            64      541370       0       72809   468561       33       7481         43    7438   4096   7438     0    inf    0 0x10040  98.4%
 phpool-64            64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 piperd              320         272       0         210       62       94         13          0      13   4096     13     0    inf    2 0x10040  37.3%
 pipewr              320         296       0         235       61       95         14          1      13   4096     14     0    inf    0 0x10040  36.7%
 plimitpl            240         147       0         132       15       97          8          1       7   4096      7     0    inf    4 0x10040  12.6%
 pmappl              512         305       0         203      102       98         25          0      25   4096     25     0    inf    2 0x10000  51.0%
 pnbufpl            1032         310       0         283       27       66         52         21      31   4096     31     0    inf   22 0x10040  21.9%
 procpl              896         236       0         141       95       97         49          1      48   4096     48     0    inf   14 0x10040  43.3%
 proparay             56         130       0          13      117       27          2          0       2   4096      2     0    inf    0 0x00040  80.0%
 propdata             48           1       0           0        1       83          1          0       1   4096      1     0    inf    0 0x00040   1.2%
 propdict             56         507       0         156      351        9          6          1       5   4096      6     0    inf    0 0x00040  96.0%
 propnmbr             64          47       0          11       36       27          1          0       1   4096      1     0    inf    0 0x10040  56.2%
 propstng             48         860       0         290      570       18          8          1       7   4096      8     0    inf    0 0x00040  95.4%
 pstatspl            456         237       0         142       95       97         24          0      24   4096     24     0    inf    6 0x10040  44.1%
 ptimerpl            328         113       0          92       21        3          4          2       2   4096      4     0    inf    0 0x10040  84.1%
 ptimerspl           312         113       0          92       21        5          3          1       2   4096      3     0    inf    0 0x10000  80.0%
 puffpnpl            248           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 puffprkl            120           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 puffvapl            184           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 pvpl                128      254945       0      223567    31378   107254       7261       2789    4472   4096   6789     0    inf  314 0x11040  21.9%
 ractx                40      147873       0       66718    81155    16714       1178        209     969   4096   1178     0    inf    0 0x00040  81.8%
 radixnode           192       88133       0       68192    19941    26826       3340       1113    2227   4096   3340     0    inf   54 0x11040  42.0%
 raidpsspl           200           0       0           0        0       20          1          0       1   4096      1     1      2    1 0x10040   0.0%
 rf_alloclist_pl     264     2725865       0     2725864        1      104         17         10       7   4096      7     5     18    6 0x10040   0.9%
 rf_asm_pl           496     2725864       0     2725864        0       96         32         20      12   4096     12     8     24   12 0x10040   0.0%
 rf_asmhdr_pl         32     1326198       0     1326198        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_asmhle_pl         24           0       0           0        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_callbackfpl       32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_callbackvpl       32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_dagh_pl          136     2725864       0     2725864        0       87          8          5       3   4096      3     2      5    3 0x10040   0.0%
 rf_daglist_pl       336     2725864       0     2725864        0       72         17         11       6   4096      6     3     11    6 0x10040   0.0%
 rf_dagnode_pl       680    11740276       0    11740276        0      282        157        110      47   4096     47    22     86   47 0x10000   0.0%
 rf_dagpcache_pl     720           0       0           0        0       10          2          0       2   4096      2     2     26    2 0x10040   0.0%
 rf_dqd_pl           208     3562684       0     3562684        0      133         19         12       7   4096      7     4     14    7 0x10040   0.0%
 rf_fss_pl            48           0       0           0        0       84          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_funclist_pl       24     2725864       0     2725864        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_mcpair_pl         48           0       0           0        0       84          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_pda_pl            64     5451728       0     5451728        0      126          2          0       2   4096      2     2      4    2 0x10040   0.0%
 rf_rad_pl           488     1326198       0     1326198        0       40          7          2       5   4096      5     4     16    5 0x10040   0.0%
 rf_reconbuffer_     112           0       0           0        0       36          1          0       1   4096      1     1      2    1 0x10040   0.0%
 rf_revent_pl         32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_stripelock_p      56     2725864       0     2725864        0       72          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_vfple_pl          24           0       0           0        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_vple_pl           24          30       0           0       30      138          1          0       1   4096      1     1      2    0 0x00040  17.6%
 rfcomm_credit        32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 rndctx               24          32       0          24        8      160          1          0       1   4096      1     0    inf    0 0x00040   4.7%
 rndsample           544          83       0          55       28        7          6          1       5   4096      6     0    586    0 0x10040  74.4%
 rndtemp             520           8       0           8        0        7          2          1       1   4096      1     0    inf    1 0x10040   0.0%
 rtentpl             328          28       0           2       26       10          3          0       3   4096      3     0    inf    0 0x10040  69.4%
 rttmrpl              72           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 rwlock               64      260546       0       78348   182198    37168       3519         37    3482   4096   3519     0    inf    0 0x10040  81.8%
 sackholepl           40          23       0          23        0        0          2          2       0   4096      1     0    inf    0 0x00040   ---
 scxspl              264       29890       0       29890        0       30          2          0       2   4096      2     2    inf    2 0x10040   0.0%
 sigacts            3096         352       0         181      171       24        283         88     195   4096    195     0    inf   24 0x10040  66.3%
 smbfsnopl           176           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 smbrqpl             296           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 smbt2pl             232           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 socket              616         514       0         177      337       41         75         12      63   4096     75     0    inf    0 0x10040  80.4%
 swp vnd             312           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 swp vnx              40           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 synpl               320          29       0          29        0        0          1          1       0   4096      1     0    inf    0 0x10040   ---
 tcpcbpl             840         840       0         826       14        2         33         29       4   4096     33     0    inf    0 0x10040  71.8%
 tcpipqepl            64         514       0         514        0        0          2          2       0   4096      1     0    inf    0 0x10040   ---
 thplthrd             80           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 tmpfs_dirent         56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 tmpfs_node          224           1       0           0        1       17          1          0       1   4096      1     0    inf    0 0x10040   5.5%
 tstile              128         463       0         198      265       45         10          0      10   4096     10     0    inf    0 0x10040  82.8%
 uaoeltpl            104          79       0          36       43       33          2          0       2   4096      2     0    inf    0 0x10040  54.6%
 uarea             20480         368       0         198      170       20        304        114     190  20480    200     0    inf   20 0x10c00  89.5%
 uareasys          20480          93       0           3       90        0         93          3      90  20480     93     0    inf    0 0x10c00 100.0%
 ufsdir              272          15       0          11        4       11          2          1       1   4096      1     0    inf    0 0x10000  26.6%
 ufsdq                88           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 vcachepl            640      267835       0       92613   175222    33692      35804        985   34819   4096  35804     0    inf  177 0x10040  78.6%
 vmembt               64       37812       0       16940    20872     5084        412          0     412   4096    412     0    inf    0 0x10040  79.2%
 vmmpepl             192       37948       0       18322    19626     2529       1107         52    1055   4096   1107     0    inf    3 0x10040  87.2%
 vmsppl              360         340       0         174      166       32         20          2      18   4096     18     0    inf    0 0x10040  81.1%
 wapbldealloc         40       55995       0       55995        0      101         10          9       1   4096      6     0    inf    1 0x00040   0.0%
 wapblentrypl         48        4389       0        4389        0       84          1          0       1   4096      1     0    inf    1 0x00040   0.0%
 wapblinopl           40      908209       0      908156       53      149          2          0       2   4096      2     0    inf    0 0x00040  25.9%
 Totals                     54483904       1    46093266  8390638   364760     740626     162158  578468
 
 In use 2453135K, total allocated 2800216K; utilization 87.6%
 -------
 After stopping the distribution build
 
 $ vmstat
  procs    memory      page                       disks   faults      cpu
  r b      avm    fre  flt  re  pi   po   fr   sr w0 w1   in   sy  cs us sy id
  0 4  1114236   6520 3327  57  36   41  139  290 17 51  278 5451 1106 2  1 97
 $ vmstat -mvW
 Memory resource pool statistics
 Name               Size    Requests    Fail    Releases    InUse    Avail      Pgreq      Pgrel   Npage PageSz  Hiwat Minpg  Maxpg Idle   Flags   Util
 ah_tdb_crypto       192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 aio_jobs_pool       136           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 aio_lio_pool         48           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 amappl               88       21768       0       16613     5155     3125        210         26     184   4096    208     0    inf    0 0x10040  60.2%
 anonpl               40     1841955       0     1561284   280671        8       9372       6593    2779   4096   6821     0    inf    0 0x01040  98.6%
 ataspl              160    11443306       0    11443305        1       49         16         14       2   4096      3     0    inf    1 0x10040   2.0%
 biopl               304       76780       0       76638      142        1       3377       3366      11   4096   1558     0    inf    0 0x10040  95.8%
 brtpl                64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 buf16k            16392       12701       0       12556      145       92       3285       3206      79  65536   1505     1      1    0 0x10040  45.9%
 buf1k              1032          59       0          59        0        3         19         18       1   4096     19     1      1    1 0x10040   0.0%
 buf2k              2056       36687       0       36639       48        1      36688      36639      49   4096  17763     1      1    1 0x10040  49.2%
 buf32k            32776       46547       0       44012     2535        1      46548      44012    2536  65536   9993     1      1    1 0x10040  50.0%
 buf4k              4096      150428       0      140934     9494        1     150429     140934    9495   4096  88754     1      1    1 0x10000 100.0%
 buf512b             520        1570       0        1569        1        6        138        137       1   4096    137     1      1    0 0x10040  12.7%
 buf64k            65536         240       0         232        8        1        241        232       9  65536    241     1      1    1 0x10000  88.9%
 buf8k              8200        2233       0        2154       79      110        171        144      27  65536     80     1      1    0 0x10040  36.6%
 bufpl               304       97802       0       85412    12390    35372       7369       3695    3674   4096   7369     0    inf  183 0x10040  25.0%
 carp                 32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 carp6                32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 ccdbuf              336           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cd9660nopl          208           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptdesc           128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptkop            384           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 cryptop             320           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 csepl               232           0       0           0        0      323         19          0      19   4096     19    19    inf   19 0x10040   0.0%
 cwdi                 64        1021       0         918      103       86          5          2       3   4096      4     0    inf    0 0x10040  53.6%
 dbregs              144           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 dirhepl              48           0       0           0        0        0          0          0       0   4096      0     0    313    0 0x00040   ---
 dirhpl              304           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 efsinopl            248           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ehcixfer            384           7       0           3        4        6          1          0       1   4096      1     0    inf    0 0x10040  37.5%
 ehcixfer            384          23       0          19        4        6          2          1       1   4096      2     0    inf    0 0x10040  37.5%
 esp_tdb_crypto      128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 execargs         262144      432267       0      432267        0        1         46         45       1 262144      4     0     16    1 0x10c00   0.0%
 ext2fsinopl         256           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 extent               48           9       0           8        1       83          1          0       1   4096      1     0    inf    0 0x00040   1.2%
 fcrpl               184          31       0          30        1       83          4          0       4   4096      4     4    inf    3 0x10040   1.1%
 fdfile               64        6281       0        4099     2182      527         43          0      43   4096     43     0    inf    0 0x11040  79.3%
 ffsdino1            136           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ffsdino2            264      320936       0      115170   205766     4204      14396        398   13998   4096  14320     0    inf    0 0x10040  94.7%
 ffsino              256      316048       0      110282   205766     4234      13497        372   13125   4096  13425     0    inf    0 0x10000  98.0%
 file                128        3840       0        2223     1617      305         65          3      62   4096     64     0    inf    0 0x10040  81.5%
 filedesc            832        1025       0         922      103       49         74         36      38   4096     53     0    inf    0 0x10040  55.1%
 icmp                 32           2       0           2        0        0          1          1       0   4096      1     0    inf    0 0x00040   ---
 icmp6                32           8       0           8        0        0          3          3       0   4096      1     0    inf    0 0x00040   ---
 igmppl               40           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 in6pcbpl            280        4328       0        4315       13        1          3          2       1   4096      3     0    inf    0 0x10040  88.9%
 inmltpl              56           2       0           0        2       70          1          0       1   4096      1     0    inf    0 0x00040   2.7%
 inpcbpl             240        4128       0        4106       22       10          9          7       2   4096      9     0    inf    0 0x10040  64.5%
 ipcomp_tdb_cryp     128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 ipfrenpl             64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 kcpuset              64        1033       0         813      220      221          8          1       7   4096      8     0    inf    0 0x10040  49.1%
 kcredpl             192         868       0         484      384      120         25          1      24   4096     25     0    inf    0 0x10040  75.0%
 kmem-00008            8           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10400   ---
 kmem-00016           16       20492       0       15691     4801     5183         40          1      39   4096     40     0    inf    0 0x10400  48.1%
 kmem-00032           32       17486       0       13197     4289     3391         60          0      60   4096     60     0    inf    0 0x10400  55.8%
 kmem-00064          128       22091       0       17137     4954     1649        377        164     213   4096    377     0    inf    0 0x10040  72.7%
 kmem-00128          192       31475       0       23722     7753      836        417          8     409   4096    414     0    inf    0 0x10040  88.9%
 kmem-00192          256    13022181      10        3403 13018778        6     813674          0  813674   4096 813674     0    inf    0 0x10000 100.0%
 kmem-00256          320        3300       0        2702      598      194        120         54      66   4096    111     0    inf    0 0x10040  70.8%
 kmem-00320          384        4077       0        3206      871      229        191         81     110   4096    183     0    inf    0 0x10040  74.2%
 kmem-00384          448        4621       0        4211      410       94        159        103      56   4096    142     0    inf    0 0x10040  80.1%
 kmem-00448          512        4278       0        4002      276       28        166        128      38   4096    132     0    inf    0 0x10000  90.8%
 kmem-00512          576        1910       0        1747      163       82        137        102      35   4096    118     0    inf    0 0x10040  65.5%
 kmem-00768          832        5234       0        4432      802       34        803        594     209   4096    649     0    inf    0 0x10040  77.9%
 kmem-01024         1088       17119       0       14256     2863       23       3050       2088     962   4096   1665     0    inf    7 0x10040  79.1%
 kmem-02048         2112        7454       0        6099     1355        0       5321       3966    1355   4096   3120     0    inf    0 0x10040  51.6%
 kmem-04096         4096        1144       0        1042      102        1        568        465     103   4096    335     0    inf    1 0x10000  99.0%
 ksiginfo            136        1906       0        1873       33       25         11          9       2   4096      4     0    inf    0 0x10040  54.8%
 ktrace              128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 kva-12288         12288         461       0         443       18       45          8          5       3 262144      6     0    inf    0 0x10e00  28.1%
 kva-16384         16384         400       0         396        4       28          9          7       2 262144      6     0    inf    0 0x10e00  12.5%
 kva-20480         20480         393       0         370       23       13         16         13       3 262144     10     0    inf    0 0x10e00  59.9%
 kva-24576         24576         249       0         241        8        2         12         11       1 262144     10     0    inf    0 0x10e00  75.0%
 kva-28672         28672          73       0          73        0        0          9          9       0 262144      5     0    inf    0 0x10e00   ---
 kva-32768         32768          69       0          68        1        7          3          2       1 262144      2     0    inf    0 0x10e00  12.5%
 kva-36864         36864          23       0          12       11        3          3          1       2 262144      3     0    inf    0 0x10e00  77.3%
 kva-4096           4096           0       0           0        0        0          0          0       0 262144      0     0    inf    0 0x10e00   ---
 kva-40960         40960           5       0           4        1        5          1          0       1 262144      1     0    inf    0 0x10e00  15.6%
 kva-49152         49152           5       0           5        0        0          1          1       0 262144      1     0    inf    0 0x10e00   ---
 kva-65536         65536           4       0           4        0        0          1          1       0 262144      1     0    inf    0 0x10e00   ---
 kva-8192           8192         408       0         374       34       30          5          3       2 262144      4     0    inf    0 0x10e00  53.1%
 l2cap_pdu            56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 l2cap_req           128           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsdinopl           264           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsinoextpl         200           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfsinopl            224           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 lfslbnpool           32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 llentrypl           280           1       0           0        1       13          1          0       1   4096      1     0    inf    0 0x10040   6.8%
 lockf               120         300       0         288       12       21          5          4       1   4096      3     0    inf    0 0x10040  35.2%
 lwppl              1088         909       0         713      196       29        130         55      75   4096     98     0    inf    0 0x10040  69.4%
 mbpl                520        3781       0        3445      336       35         93         40      53   4096     83     3    inf    1 0x10040  80.5%
 mclpl              2112        3267       0        2959      308       16       1030        706     324   4096    426     8 130541   16 0x10040  49.0%
 mqmsgpl            1088           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 msdosfhpl            56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 msdosnopl           208          63       0          63        0        0          4          4       0   4096      4     0    inf    0 0x10040   ---
 mutex                64      312788       0      106590   206198     4285       3428         87    3341   4096   3416     0    inf    0 0x10040  96.4%
 nchentry            192      241413       0       32835   208578      351       9949          0    9949   4096   9949     0    inf    0 0x10040  98.3%
 nfsnodepl           280           3       0           1        2       12          1          0       1   4096      1     0    inf    0 0x10040  13.7%
 nfsreqcachepl       104           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 nfsrvdescpl         256           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 nfsvapl             184           3       0           1        2       19          1          0       1   4096      1     0    inf    0 0x10040   9.0%
 npfcn4pl            144           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npfcn6pl            192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npfnatpl             96           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 npftblpl             48           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 pcache             2752          86       0           4       82        0         82          0      82   4096     82     0    inf    0 0x10040  67.2%
 pcachecpu            64         267       0           0      267       48          5          0       5   4096      5     0    inf    0 0x10040  83.4%
 pcglarge           1088       40125       0       40108       17       25       8394       8380      14   4096   3420     0    inf    8 0x10040  32.3%
 pcgnormal           320      156663       0      156468      195       45       6339       6319      20   4096   4139     0    inf    0 0x10040  76.2%
 pdict128            192           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 pdict16              80         321       0         272       49        1          3          2       1   4096      2     0    inf    0 0x10040  95.7%
 pdict32              96          15       0           2       13       29          1          0       1   4096      1     0    inf    0 0x10040  30.5%
 pdppl              4096         950       0         847      103        0        478        375     103   4096    211     0    inf    0 0x10000 100.0%
 pewpl                32           0       0           0        0      126          1          0       1   4096      1     1      1    1 0x00040   0.0%
 phpool-1024         184           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-128           72          60       0           0       60       52          2          0       2   4096      2     0    inf    0 0x10040  52.7%
 phpool-2048         312           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 phpool-256           88          40       0           1       39        6          1          0       1   4096      1     0    inf    0 0x10040  83.8%
 phpool-4096         568           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-512          120           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 phpool-64            64      980595       0      143772   836823        6      13326         43   13283   4096  13283     0    inf    0 0x10040  98.4%
 phpool-64            64           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 piperd              320         665       0         600       65       79         15          3      12   4096     15     0    inf    0 0x10040  42.3%
 pipewr              320         755       0         692       63       93         16          3      13   4096     15     0    inf    0 0x10040  37.9%
 plimitpl            240         542       0         524       18       30         12          9       3   4096      7     0    inf    0 0x10040  35.2%
 pmappl              512         950       0         847      103       89         33          9      24   4096     27     0    inf    0 0x10000  53.6%
 pnbufpl            1032        1384       0        1372       12        0        105        101       4   4096     33     0    inf    0 0x10040  75.6%
 procpl              896         715       0         613      102       26         60         28      32   4096     50     0    inf    0 0x10040  69.7%
 proparay             56         134       0          17      117       27          2          0       2   4096      2     0    inf    0 0x00040  80.0%
 propdata             48           1       0           0        1       83          1          0       1   4096      1     0    inf    0 0x00040   1.2%
 propdict             56         523       0         172      351        9          6          1       5   4096      6     0    inf    0 0x00040  96.0%
 propnmbr             64          49       0          13       36       27          1          0       1   4096      1     0    inf    0 0x10040  56.2%
 propstng             48         892       0         322      570       18          8          1       7   4096      8     0    inf    0 0x00040  95.4%
 pstatspl            456         720       0         618      102       42         29         11      18   4096     25     0    inf    0 0x10040  63.1%
 ptimerpl            328         117       0          96       21        3          4          2       2   4096      4     0    inf    0 0x10040  84.1%
 ptimerspl           312         117       0          96       21        5          3          1       2   4096      3     0    inf    0 0x10000  80.0%
 puffpnpl            248           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 puffprkl            120           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 puffvapl            184           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 pvpl                128      300499       0      288687    11812    69005       7261       4654    2607   4096   6789     0    inf  329 0x11040  14.2%
 ractx                40      230136       0      138235    91901        9       1478        568     910   4096   1178     0    inf    0 0x00040  98.6%
 radixnode           192      145216       0      140681     4535    18565       3340       2240    1100   4096   3340     0    inf   25 0x11040  19.3%
 raidpsspl           200           0       0           0        0       20          1          0       1   4096      1     1      2    1 0x10040   0.0%
 rf_alloclist_pl     264     5193794       0     5193793        1       74         68         63       5   4096      7     5     18    4 0x10040   1.3%
 rf_asm_pl           496     5193793       0     5193793        0       88        161        150      11   4096     12     8     24   11 0x10040   0.0%
 rf_asmhdr_pl         32     2170796       0     2170796        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_asmhle_pl         24           0       0           0        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_callbackfpl       32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_callbackvpl       32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_dagh_pl          136     5193793       0     5193793        0       58         17         15       2   4096      3     2      5    2 0x10040   0.0%
 rf_daglist_pl       336     5193793       0     5193793        0       60        108        103       5   4096      6     3     11    5 0x10040   0.0%
 rf_dagnode_pl       680    22834855       0    22834855        0      222        872        835      37   4096     47    22     86   37 0x10000   0.0%
 rf_dagpcache_pl     720           0       0           0        0       10          2          0       2   4096      2     2     26    2 0x10040   0.0%
 rf_dqd_pl           208     7253476       0     7253476        0      114        116        110       6   4096      7     4     14    6 0x10040   0.0%
 rf_fss_pl            48           0       0           0        0       84          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_funclist_pl       24     5193793       0     5193793        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_mcpair_pl         48           0       0           0        0       84          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_pda_pl            64    10387586       0    10387586        0      126          2          0       2   4096      2     2      4    2 0x10040   0.0%
 rf_rad_pl           488     2170796       0     2170796        0       40         44         39       5   4096      5     4     16    5 0x10040   0.0%
 rf_reconbuffer_     112           0       0           0        0       36          1          0       1   4096      1     1      2    1 0x10040   0.0%
 rf_revent_pl         32           0       0           0        0      126          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_stripelock_p      56     5193793       0     5193793        0       72          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_vfple_pl          24           0       0           0        0      168          1          0       1   4096      1     1      2    1 0x00040   0.0%
 rf_vple_pl           24          30       0           0       30      138          1          0       1   4096      1     1      2    0 0x00040  17.6%
 rfcomm_credit        32           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 rndctx               24          37       0          29        8      160          1          0       1   4096      1     0    inf    0 0x00040   4.7%
 rndsample           544         206       0         187       19       16          6          1       5   4096      6     0    586    0 0x10040  50.5%
 rndtemp             520          13       0          13        0        0          3          3       0   4096      1     0    inf    0 0x10040   ---
 rtentpl             328          28       0           2       26       10          3          0       3   4096      3     0    inf    0 0x10040  69.4%
 rttmrpl              72           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 rwlock               64      351347       0      140127   211220     8146       3519         37    3482   4096   3519     0    inf    0 0x10040  94.8%
 sackholepl           40          23       0          23        0        0          2          2       0   4096      1     0    inf    0 0x00040   ---
 scxspl              264       29890       0       29890        0       30          2          0       2   4096      2     2    inf    2 0x10040   0.0%
 sigacts            3096         952       0         849      103        0        426        323     103   4096    210     0    inf    0 0x10040  75.6%
 smbfsnopl           176           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 smbrqpl             296           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 smbt2pl             232           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 socket              616         740       0         414      326       52         75         12      63   4096     75     0    inf    0 0x10040  77.8%
 swp vnd             312           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10000   ---
 swp vnx              40           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 synpl               320          29       0          29        0        0          1          1       0   4096      1     0    inf    0 0x10040   ---
 tcpcbpl             840         848       0         835       13        3         33         29       4   4096     33     0    inf    0 0x10040  66.7%
 tcpipqepl            64         515       0         515        0        0          3          3       0   4096      1     0    inf    0 0x10040   ---
 thplthrd             80           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 tmpfs_dirent         56           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x00040   ---
 tmpfs_node          224           1       0           0        1       17          1          0       1   4096      1     0    inf    0 0x10040   5.5%
 tstile              128         951       0         755      196      114         10          0      10   4096     10     0    inf    0 0x10040  61.2%
 uaoeltpl            104         226       0         179       47       29          3          1       2   4096      3     0    inf    0 0x10040  59.7%
 uarea             20480         850       0         744      106        0        415        309     106  20480    202     0    inf    0 0x10c00 100.0%
 uareasys          20480          93       0           3       90        0         93          3      90  20480     93     0    inf    0 0x10c00 100.0%
 ufsdir              272          70       0          69        1       14         12         11       1   4096      1     0    inf    0 0x10000   6.6%
 ufsdq                88           0       0           0        0        0          0          0       0   4096      0     0    inf    0 0x10040   ---
 vcachepl            640      345989       0      140179   205810     4130      35991       1001   34990   4096  35804     0    inf    0 0x10040  91.9%
 vmembt               64       62704       0       34209    28495      926        467          0     467   4096    467     0    inf    0 0x10040  95.3%
 vmmpepl             192       62637       0       48291    14346     8229       1250        175    1075   4096   1194     0    inf    0 0x10040  62.6%
 vmsppl              360         941       0         838      103       84         23          6      17   4096     20     0    inf    0 0x10040  53.3%
 wapbldealloc         40       70640       0       70640        0        0         16         16       0   4096      6     0    inf    0 0x00040   ---
 wapblentrypl         48        6985       0        6985        0       84          4          3       1   4096      1     0    inf    1 0x00040   0.0%
 wapblinopl           40     2332333       0     2332276       57      145          2          0       2   4096      2     0    inf    0 0x00040  27.8%
 Totals                    109665202      10    94066088 15599114   178659    1210640     274601  936039
 
 In use 3770733K, total allocated 3908712K; utilization 96.5%
 
 



Home | Main Index | Thread Index | Old Index