Subject: Re: buffer cache tuning - running out of kernel virtual space
To: None <tech-kern@netbsd.org>
From: Chuck Silvers <chuq@chuq.com>
List: tech-kern
Date: 12/17/2005 20:05:06
--GvXjxJ+pjyke8COw
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
On Sat, Dec 10, 2005 at 04:50:06PM -0800, Chuck Silvers wrote:
> while yamt and I were looking into problems with the project autobuild
> machines, ...
I went back through my mail on this and found that we did manage to get
a dump from this problem once. I've attached the "vmstat -m" output.
basically all of the processes waiting for KVA have a stack trace like this:
(gdb) bt
#0 0xe20f1750 in ?? ()
#1 0xc01c4bd9 in bpendtsleep ()
#2 0xc01d20c0 in pool_get ()
#3 0xc022bc69 in uvm_km_alloc_poolpage_cache ()
#4 0xc01d3d58 in pool_page_alloc_nointr ()
#5 0xc01d3a29 in pool_allocator_alloc ()
#6 0xc01d1fef in pool_get ()
#7 0xc01d354d in pool_cache_get_paddr ()
#8 0xc01fae72 in namei ()
#9 0xc020a803 in vn_open ()
#10 0xc02029d9 in sys_open ()
#11 0xc0252d61 in syscall_plain ()
according to vmstat, there are no elements free in pnbuf_pool, but there's
plenty of memory that could be reclaimed from other pools. so why isn't
it being freed up?
the problem is that when we fail to allocate memory for a pool, we only
try to drain pools using the same allocator. in this case, there is only
one pool using the kernel_map KVA allocator, which has no space available.
we need to recursively drain the other allocators which use the KVA pool
as their back-end.
it would also help to set a drain hook on the buffer cache pools,
so that they can free some of the memory used by free buffers
when there is virtual memory pressure. hopefully this could also
replace the special-purpose buf_drain() hook in the pagedaemon.
-Chuck
--GvXjxJ+pjyke8COw
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=vmstat
Name Size Requests Fail Releases InUse Avail Pgreq Pgrel Npage PageSz Hiwat Minpg Maxpg Idle Flags Util
phpool-0 40 10927 0 985 9942 57 99 0 99 4096 99 0 inf 0 0x040 98.1%
phpool-32 72 1954 0 145 1809 39 33 0 33 4096 33 0 inf 0 0x040 96.4%
pcgpool 140 175 0 8 167 1 6 0 6 4096 6 0 inf 0 0x040 95.1%
pmappl 76 5386090 0 5386039 51 108 4 1 3 4096 3 0 inf 2 0x040 31.5%
pdppl 4096 2272 0 2168 104 23 203 76 127 4096 127 0 inf 23 0x000 81.9%
kvakernel 4096 61648 0 9040 52608 0 1789 145 1644 131072 1644 0 0 0 0x604 100.0%
kvakmem 4096 5275 0 3 5272 8 165 0 165 131072 165 0 0 0 0x600 99.8%
inpcbpl 108 1547 0 1514 33 41 2 0 2 4096 2 0 inf 0 0x040 43.5%
inmltpl 32 2 0 0 2 124 1 0 1 4096 1 0 inf 0 0x040 1.6%
tcpipqepl 44 3000 0 3000 0 92 2 1 1 4096 1 0 inf 1 0x040 0.0%
synpl 172 497 0 497 0 23 2 1 1 4096 1 0 inf 1 0x040 0.0%
sackholepl 20 49 0 49 0 202 2 1 1 4096 1 0 inf 1 0x040 0.0%
tcpcbpl 360 990 0 963 27 17 4 0 4 4096 4 0 inf 0 0x040 59.3%
sdpcpool 124 5018179 0 5018179 0 8288 2275 2016 259 4096 2049 0 inf 259 0x040 0.0%
pagedeppl 68 158975 0 158975 0 10266 220 46 174 4096 174 0 inf 174 0x040 0.0%
inodedeppl 88 4760987 0 4760987 0 123786 2691 0 2691 4096 2691 0 inf 2691 0x040 0.0%
newblkpl 36 5620055 0 5620055 0 112 1 0 1 4096 1 0 inf 1 0x040 0.0%
bmsafemappl 36 23442 0 23442 0 224 3 1 2 4096 2 0 inf 2 0x040 0.0%
allocdirectpl 80 4575688 0 4575688 0 8750 1598 1423 175 4096 1424 0 inf 175 0x040 0.0%
indirdeppl 32 41802 0 41802 0 252 4 2 2 4096 3 0 inf 2 0x040 0.0%
allocindirpl 64 1044367 0 1044367 0 10710 193 23 170 4096 170 0 inf 170 0x040 0.0%
freefragpl 40 260825 0 260825 0 202 4 2 2 4096 3 0 inf 2 0x040 0.0%
freeblkspl 172 2918765 0 2918765 0 111550 8875 4025 4850 4096 4850 0 inf 4850 0x040 0.0%
freefilepl 36 3640832 0 3640832 0 112112 1832 831 1001 4096 1001 0 inf 1001 0x040 0.0%
diraddpl 36 4302327 0 4302327 0 13328 593 474 119 4096 475 0 inf 119 0x040 0.0%
mkdirpl 32 126624 0 126624 0 2520 101 81 20 4096 81 0 inf 20 0x040 0.0%
dirrempl 36 4288585 0 4288585 0 120624 1077 0 1077 4096 1077 0 inf 1077 0x040 0.0%
newdirblkpl 20 7473 0 7473 0 202 2 1 1 4096 1 0 inf 1 0x040 0.0%
ffsinopl 196 7333299 0 7132963 200336 80924 14063 0 14063 4096 14063 0 inf 1364 0x040 68.2%
dino1pl 128 7333299 0 7132963 200336 80927 9073 0 9073 4096 9073 0 inf 816 0x040 69.0%
ufsdirhash 2048 25598 0 24148 1450 414 932 0 932 4096 932 0 inf 182 0x000 77.8%
ufsdirpl 264 4276631 0 4276631 0 15 1 0 1 4096 1 0 inf 1 0x040 0.0%
rndsample 528 124435 0 124432 3 4 1 0 1 4096 1 0 inf 0 0x040 38.7%
filepl 56 227679038 0 227678902 136 224 8 3 5 4096 5 0 inf 3 0x040 37.2%
cwdipl 16 5395249 0 5395197 52 201 1 0 1 4096 1 0 inf 0 0x040 20.3%
fdescpl 284 5395249 0 5395197 52 88 18 8 10 4096 10 0 inf 6 0x040 36.1%
kqueuepl 260 2 0 0 2 13 1 0 1 4096 1 0 inf 0 0x040 12.7%
knotepl 72 15 0 0 15 41 1 0 1 4096 1 0 inf 0 0x040 26.4%
procpl 408 5395260 0 5395198 62 78 25 11 14 4096 14 0 inf 7 0x000 44.1%
lwppl 132 5395264 0 5395198 66 84 8 3 5 4096 5 0 inf 2 0x040 42.5%
pgrppl 16 1024749 0 1024729 20 233 1 0 1 4096 1 0 inf 0 0x040 7.8%
pcredpl 24 5395260 0 5395198 62 107 1 0 1 4096 1 0 inf 0 0x040 36.3%
plimitpl 176 23742 0 23738 4 19 1 0 1 4096 1 0 inf 0 0x040 17.2%
pstatspl 224 5395260 0 5395198 62 82 14 6 8 4096 8 0 inf 4 0x040 42.4%
rusgepl 72 5395198 0 5395198 0 56 1 0 1 4096 1 0 inf 1 0x040 0.0%
sadatapl 40 1 0 0 1 100 1 0 1 4096 1 0 inf 0 0x040 1.0%
saupcpl 1612 13728 0 13726 2 0 1 0 1 4096 1 0 inf 0 0x040 78.7%
sastackpl 24 16 0 0 16 153 1 0 1 4096 1 0 inf 0 0x040 9.4%
savppl 52 1 0 0 1 77 1 0 1 4096 1 0 inf 0 0x040 1.3%
ptimerpl 124 518 0 505 13 19 1 0 1 4096 1 0 inf 0 0x040 39.4%
sessionpl 40 600 0 580 20 81 1 0 1 4096 1 0 inf 0 0x040 19.5%
credpl 84 5173604 0 5173580 24 24 1 0 1 4096 1 0 inf 0 0x040 49.2%
siginfo 128 3066 0 3066 0 31 1 0 1 4096 1 0 inf 1 0x040 0.0%
ksiginfo 44 1 0 0 1 91 1 0 1 4096 1 0 inf 0 0x040 1.1%
pipepl 144 3857782 0 3857756 26 170 14 7 7 4096 8 0 inf 6 0x040 13.1%
ttypl 264 66 0 0 66 9 5 0 5 4096 5 0 inf 0 0x040 85.1%
sockpl 216 2752 0 2703 49 5 3 0 3 4096 3 0 inf 0 0x040 86.1%
ncachepl 84 8910172 0 8729830 180342 95082 5738 0 5738 4096 5738 0 inf 57 0x040 64.5%
lockfpl 52 3529072 0 3529069 3 75 1 0 1 4096 1 0 inf 0 0x040 3.8%
vnodepl 168 3841201 0 3640862 200339 80917 11719 0 11719 4096 11719 0 inf 1088 0x040 70.1%
rtentpl 132 36 0 31 5 25 1 0 1 4096 1 0 inf 0 0x040 16.1%
rttmrpl 32 210 0 210 0 126 1 0 1 4096 1 0 inf 1 0x040 0.0%
amappl 40 81071385 0 81070725 660 3481 41 0 41 4096 41 0 inf 34 0x040 15.7%
vmsppl 224 5386090 0 5386040 50 94 8 0 8 4096 8 0 inf 5 0x040 34.2%
vmmpepl 88 171827095 0 171825804 1291 6069 160 0 160 4096 160 0 inf 131 0x040 17.3%
ataspl 52 7964404 0 7964403 1 77 1 0 1 4096 1 0 inf 0 0x040 1.3%
bufpl 124 15869279 0 15710490 158789 2779 5049 0 5049 4096 5049 0 inf 14 0x040 95.2%
buf1k 1024 31 0 31 0 64 1 0 1 65536 1 1 1 1 0x000 0.0%
buf2k 2048 174819 0 31163 143656 4280 4695 72 4623 65536 4649 1 1 0 0x000 97.1%
buf4k 4096 3637 0 2941 696 24 58 13 45 65536 45 1 1 0 0x000 96.7%
buf8k 8192 1905 56 1604 301 3 67 29 38 65536 40 1 1 0 0x000 99.0%
buf16k 16384 40728 2275 26592 14136 364 4409 784 3625 65536 3678 1 1 1 0x000 97.5%
mbpl 256 2243 0 0 2243 29 142 0 142 4096 142 1 inf 1 0x000 98.7%
mclpl 2048 771 0 0 771 9 390 0 390 4096 390 4 512 4 0x000 98.8%
pnbufpl 1024 12 0 0 12 0 3 0 3 4096 3 0 inf 0 0x000 100.0%
raidpl 132 7963693 0 7963693 0 90 3 0 3 4096 3 2 4 3 0x040 0.0%
sigapl 2052 5395249 0 5395197 52 76 128 0 128 4096 128 0 inf 76 0x040 20.4%
rf_alloclist_pl 168 4239244 0 4239243 1 95 4 0 4 4096 4 3 11 3 0x040 1.0%
rf_asmhdr_pl 16 5114020 0 5114020 0 253 1 0 1 4096 1 1 2 1 0x040 0.0%
rf_asm_pl 288 5373821 0 5373821 0 84 6 0 6 4096 6 5 14 6 0x040 0.0%
rf_pda_pl 40 10747642 0 10747642 0 101 1 0 1 4096 1 1 2 1 0x040 0.0%
rf_vple_pl 8 30 0 0 30 477 1 0 1 4096 1 1 2 0 0x040 5.9%
rf_rad_pl 328 2844864 0 2844864 0 48 4 0 4 4096 4 3 11 4 0x040 0.0%
rf_stripelock_pl 28 3104665 0 3104665 0 144 1 0 1 4096 1 1 2 1 0x040 0.0%
rf_mcpair_pl 12 1134578 0 1134578 0 338 1 0 1 4096 1 1 2 1 0x040 0.0%
rf_dagnode_pl 444 20681422 0 20681422 0 189 21 0 21 4096 21 15 57 21 0x040 0.0%
rf_dagh_pl 76 4239243 0 4239243 0 53 1 0 1 4096 1 1 3 1 0x040 0.0%
rf_daglist_pl 236 3104665 0 3104665 0 51 3 0 3 4096 3 2 8 3 0x040 0.0%
rf_funclist_pl 8 3104665 0 3104665 0 507 1 0 1 4096 1 1 2 1 0x040 0.0%
rf_dqd_pl 116 7963693 0 7963693 0 102 3 0 3 4096 3 2 8 3 0x040 0.0%
extent 20 2 0 0 2 200 1 0 1 4096 1 0 inf 0 0x040 1.0%
--GvXjxJ+pjyke8COw--