Subject: vmstat -m out of controll
To: None <tech-kern@NetBSD.org>
From: Martin Husemann <martin@duskware.de>
List: tech-kern
Date: 12/06/2004 10:12:56
[Sending this to tech-kern instead of tech-userlevel since this is more a
"how to interpret kernel data" question]
I recently had a machine (Sparc Station 2, sun4c, 64 MB memory) panic since
it ran out of kernel memory. So I enabled KMEMSTATS and had it run with
this for nearly a week now. Looking at the output of vmstat -m made me
slightly nervous:
In use 58305K, total allocated 1956K; utilization 2980.8%
If I do the math myself, I end up with 2.4 MB kernel memory in the malloc
part, and 489 pages in pools, this machine has 4k pages, so that should
be something around 4.3 MB kernel memory in use total.
Am I right and vmstat is wrong? If not, I'd expect my machine to crash again
soon ;-)
Martin
Complete vmstat -m output:
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 253 259 40616 1280 0
32 2429 3459 1971917 640 188
64 1014 138 163937 320 0
128 231 25 5074763 160 0
256 207 17 11374 80 0
512 2955 3693 2531328 40 12240
1024 103 1 3193 20 0
2048 10 8 3299 10 0
4096 14 0 31 5 0
8192 6 0 7 5 0
16384 6 0 6 5 0
32768 3 0 3 5 0
65536 5 0 5 5 0
Memory usage type by bucket size
Size Type(s)
16 VM swap, UVM amap, soname, proc, UFS mount, routetbl, temp, pcb,
devbuf, exec, kernfs mount
32 VM swap, UVM amap, ether_multi, ifaddr, sysctldata, fragtbl, routetbl,
temp, pcb, devbuf, NFS req
64 VM swap, UVM amap, vnodes, lockf, prop, session, routetbl, temp, pcb,
devbuf, NFS diroff, NFS req
128 VM swap, UVM amap, ifaddr, ttys, prop, sysctlnode, cred, routetbl,
temp, DMA map, devbuf, kevent
256 UVM amap, ifaddr, vnodes, subproc, proc, routetbl, temp, devbuf, exec,
kevent, newblk, NFS mount, NFS diroff
512 VM map, UVM amap, mount, subproc, proc, temp, pcb, devbuf, exec,
ioctlops, NFS mount
1024 UVM aobj, UVM amap, ttys, sysctlnode, temp, devbuf, exec, NFS mount,
NQNFS Lease
2048 UVM amap, ifaddr, namecache, sysctlnode, in_multi, temp, devbuf, exec
4096 UVM amap, sysctlnode, UFS mount, temp, devbuf, pagedep
8192 sysctlnode, temp
16384 namecache, sysctlnode, UFS mount, inodedep, NFS node
32768 temp
65536 temp, devbuf
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
VM swap 5 1K 1K 3687K 5 0 0 16,32,64,128
VM map 3 2K 2K 3687K 3 0 0 512
UVM aobj 1 1K 1K 3687K 1 0 0 1024
UVM amap 426 50K 53K 3687K 109639 0 0 16,32,64,128,256,512,1024,2048,4096
ether_multi 1 1K 1K 3687K 1 0 0 32
ifaddr 24 5K 5K 3687K 25 0 0 32,128,256,2048
mount 3 2K 2K 3687K 3 0 0 512
vnodes 795 53K 53K 3687K 840 0 0 64,256
lockf 1 1K 1K 3687K 246 0 0 64
namecache 3 34K 34K 3687K 3 0 0 2048,16384
soname 1 1K 1K 3687K 1239 0 0 16
ttys 91 59K 59K 3687K 91 0 0 128,1024
prop 2 1K 1K 3687K 2 0 0 64,128
sysctldata 0 0K 1K 3687K 1 0 0 32
sysctlnode 34 57K 58K 3687K 47 0 0 128,1024,2048,4096,8192,16384
cred 39 5K 5K 3687K 2938 0 0 128
subproc 8 3K 5K 3687K 727 0 0 256,512
session 11 1K 1K 3687K 624 0 0 64
proc 4 1K 2K 3687K 31 0 0 16,256,512
in_multi 1 2K 2K 3687K 1 0 0 2048
UFS mount 4 25K 25K 3687K 4 0 0 16,4096,16384
fragtbl 0 0K 1K 3687K 835 0 0 32
routetbl 34 2K 2K 3687K 95 0 0 16,32,64,128,256
temp 5405 1737K 3687K 3687K 9565594 0 0 16,32,64,128,256,512,1024,2048,4096,8192,32768,65536
pcb 16 4K 4K 3687K 1282 0 0 16,32,64,512
DMA map 4 1K 1K 3687K 4 0 0 128
devbuf 129 287K 287K 3687K 208 0 0 16,32,64,128,256,512,1024,2048,4096,65536
exec 1 1K 5K 3687K 6131 0 0 16,256,512,1024,2048
kevent 2 1K 1K 3687K 2 0 0 128,256
ioctlops 0 0K 1K 3687K 5 0 0 512
newblk 1 1K 1K 3687K 1 0 0 256
inodedep 1 16K 16K 3687K 1 0 0 16384
pagedep 1 4K 4K 3687K 1 0 0 4096
NFS mount 1 1K 2K 3687K 3 0 0 256,512,1024
NFS diroff 165 38K 38K 3687K 1141 0 0 64,256
NFS req 0 0K 1K 3687K 108728 0 0 32,64
NQNFS Lease 1 1K 1K 3687K 1 0 0 1024
NFS node 1 16K 16K 3687K 1 0 0 16384
kernfs mount 1 1K 1K 3687K 1 0 0 16
Memory totals: In Use Free Requests
2400K 1992K 9800505
Memory resource pool statistics
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
phpool 40 4056 0 0 0 0 1 0 0 inf 68
pcgpool 144 4056 0 0 0 0 1 0 0 inf 10
vmsppl 344 4056 0 0 0 0 2 0 0 inf 3855
vmmpepl 96 4056 0 0 0 0 12 0 0 inf 96865
vmmpekpl 96 4056 0 0 0 0 1 0 0 inf 2638
uaoeltpl 88 4056 0 0 0 0 0 0 0 inf 0
aobjpl 80 4056 0 0 0 0 0 0 0 inf 0
pvtable 16 4056 0 0 0 0 13 0 3 inf 573557
pmappl 2984 4056 0 0 0 0 22 0 0 inf 22
amappl 64 4056 0 0 0 0 4 0 0 inf 34049
extent 24 4056 0 0 0 0 1 0 0 inf 2
bufpl 152 4056 0 0 0 0 8 0 0 inf 2215
kqueuepl 320 4056 0 0 0 0 1 0 0 inf 1
knotepl 80 4056 0 0 0 0 1 0 0 inf 5
sockpl 200 4056 0 0 0 0 1 0 0 inf 1464
ttypl 288 4056 0 0 0 0 2 0 0 inf 18
rndsample 528 4056 0 0 0 0 1 0 0 inf 2
procpl 448 4056 0 0 0 0 4 0 0 inf 3864
lwppl 136 4056 0 0 0 0 2 0 0 inf 3864
lwpucpl 448 4056 0 0 0 0 0 0 0 inf 0
pgrppl 16 4056 0 0 0 0 1 0 0 inf 626
pcredpl 24 4056 0 0 0 0 1 0 0 inf 3864
plimitpl 160 4056 0 0 0 0 1 0 0 inf 83
pstatspl 224 4056 0 0 0 0 2 0 0 inf 3864
rusgepl 72 4056 0 0 0 0 1 0 0 inf 3842
raspl 24 4056 0 0 0 0 0 0 0 inf 0
sadatapl 64 4056 0 0 0 0 0 0 0 inf 0
saupcpl 960 4056 0 0 0 0 0 0 0 inf 0
sastackpl 24 4056 0 0 0 0 0 0 0 inf 0
savppl 80 4056 0 0 0 0 0 0 0 inf 0
ptimerpl 128 4056 0 0 0 0 1 0 0 inf 35
filepl 80 4056 0 0 0 0 2 0 0 inf 26340
cwdipl 16 4056 0 0 0 0 1 0 0 inf 3855
fdescpl 280 4056 0 0 0 0 2 0 0 inf 3855
vnodepl 232 4056 0 0 0 0 104 0 0 inf 1933
ncachepl 88 4056 0 0 0 0 39 0 0 inf 2226
sdpcpool 152 4056 0 0 0 0 0 0 0 inf 0
pagedeppl 72 4056 0 0 0 0 0 0 0 inf 0
inodedeppl 88 4056 0 0 0 0 0 0 0 inf 0
newblkpl 40 4056 0 0 0 0 0 0 0 inf 0
bmsafemappl 40 4056 0 0 0 0 0 0 0 inf 0
allocdirectpl 80 4056 0 0 0 0 0 0 0 inf 0
indirdeppl 32 4056 0 0 0 0 0 0 0 inf 0
allocindirpl 72 4056 0 0 0 0 0 0 0 inf 0
freefragpl 40 4056 0 0 0 0 0 0 0 inf 0
freeblkspl 176 4056 0 0 0 0 0 0 0 inf 0
freefilepl 40 4056 0 0 0 0 0 0 0 inf 0
diraddpl 40 4056 0 0 0 0 0 0 0 inf 0
mkdirpl 32 4056 0 0 0 0 0 0 0 inf 0
dirrempl 40 4056 0 0 0 0 0 0 0 inf 0
newdirblkpl 24 4056 0 0 0 0 0 0 0 inf 0
ffsinopl 232 4056 0 0 0 0 0 0 0 inf 0
dino1pl 128 4056 0 0 0 0 0 0 0 inf 0
nfsnodepl 384 4056 0 0 0 0 176 0 0 inf 1925
nfsvapl 112 4056 0 0 0 0 49 0 0 inf 1925
ncr53c9x_ecb 56 4056 0 0 0 0 1 0 1 inf 14
scxspl 152 4056 0 0 0 0 2 0 2 inf 14
shmmp 16 4056 0 0 0 0 0 0 0 inf 0
rtentpl 136 4056 0 0 0 0 1 0 0 inf 29
inmltpl 32 4056 0 0 0 0 1 0 0 inf 7
ipqepl 48 4056 0 0 0 0 1 0 0 inf 1724
rttmrpl 32 4056 0 0 0 0 0 0 0 inf 0
ipflowpl 72 4056 0 0 0 0 0 0 0 inf 0
inpcbpl 112 4056 0 0 0 0 1 0 0 inf 192
tcpcbpl 320 4056 0 0 0 0 1 0 0 inf 25
tcpipqepl 48 4056 0 0 0 0 0 0 0 inf 0
synpl 176 4056 0 0 0 0 1 0 0 inf 9
igmppl 24 4056 0 0 0 0 0 0 0 inf 0
sigapl 2056 4056 0 0 0 0 22 0 0 inf 3855
siginfo 128 4056 0 0 0 0 0 0 0 inf 0
ksiginfo 48 4056 0 0 0 0 0 0 0 inf 0
swp vnx 24 4056 0 0 0 0 0 0 0 inf 0
swp vnd 160 4056 0 0 0 0 0 0 0 inf 0
pipepl 208 4056 0 0 0 0 1 0 0 inf 3252
In use 58305K, total allocated 1956K; utilization 2980.8%