NetBSD-Bugs archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
kern/44625: memory exhaustion on NFS server while writing many files
>Number: 44625
>Category: kern
>Synopsis: memory exhaustion on NFS server while writing many files
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: kern-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Wed Feb 23 04:35:00 +0000 2011
>Originator: Takahiro HAYASHI
>Release: NetBSD-current (5.99.40) 2010-11-30 10:50:00 UTC or later
>Organization:
>Environment:
NFS-client:
System: NetBSD halt 5.99.46 NetBSD 5.99.46 (HALT) #0: Tue Feb 22 12:34:31 JST
2011 root@halt:/usr/build2/obj.i386/sys/arch/i386/compile/HALT i386
PhysicalMemory: 4096MB
mount-opts: ruin:/usr/nfs on /usr/nfs type nfs (fsid: 0xb02/0x70b, reads: sync
0 async 0, writes: sync 0 async 0, [nfs: addr=fc00::6c, port=2049, addrlen=28,
sotype=1, proto=0, fhsize=0, flags=0x8240<intr,nfsv3,resvport>, wsize=32768,
rsize=32768, readdirsize=8192, timeo=300, retrans=10, maxgrouplist=16,
readahead=2, leaseterm=0, deadthresh=9])
NFS-server:
System: NetBSD ruin 5.99.46 NetBSD 5.99.46 (MONOLITHIC) #0: Tue Feb 22 12:15:45
JST 2011 root@halt:/usr/build2/obj.i386/sys/arch/i386/compile/MONOLITHIC i386
PhysicalMemory: 512MB
>Description:
The utilization of kernel memory resource pool on NFS
server shown at last line from "vmstat -m" increases up to
most of available memory while writing many files via NFS.
The memory utilization does not decrease even if
copy is stopped and copied files are removed.
mbuf does not seem to leak as far as i see netstat -m.
At last, console of NFS server does not respond,
however, the server responds to ping.
This does not happen when writing a single huge file.
The exported filesystem on server is FFSv2 with WAPBL.
The following is diff between outputs from vmstat -m
on NFS server at a certain time and 22 minutes later.
--- 0 2011-02-22 13:25:45.000000000 +0900
+++ 3 2011-02-22 13:47:13.000000000 +0900
@@ -1,18 +1,18 @@
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
- 16 175 81 33796 1280 0
- 32 735 161 2119895 640 0
- 64 988 36 6963 320 0
- 128 760 8 4969 160 0
- 256 47 17 7607 80 0
+ 16 175 81 71073 1280 0
+ 32 736 160 8101764 640 0
+ 64 988 36 11138 320 0
+ 128 761 7 8265 160 0
+ 256 47 17 14669 80 0
512 34 6 45 40 0
1024 115 1 385 20 0
2048 32 2 1798 10 0
- 4096 154 3 577 5 0
+ 4096 154 3 721 5 0
8192 4 3 9 5 0
16384 3 0 4 5 0
- 32768 1 0 1 5 0
- 65536 11 0 11 5 0
+ 32768 0 0 1 5 0
+ 65536 12 0 12 5 0
131072 1 0 1 5 0
Memory usage type by bucket size
@@ -38,12 +38,12 @@
8192 wapbl, vmem, sysctlnode, temp
16384 vmem, sysctlnode, temp
32768 vmem
- 65536 devbuf
+ 65536 vmem, devbuf
131072 blist
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
- wapbl 5 14K 14K 78644K 674 0 0
32:0,256:1,512:2,4096:1,8192:1
+ wapbl 5 14K 14K 78644K 1522 0 0
32:0,256:1,512:2,4096:1,8192:1
NTFS mount 0 0K 1K 78644K 1 0 0 256:0
MSDOSFS mount 0 0K 1K 78644K 1 0 0 256:0
prop dictionary 67 9K 9K 78644K 84 0 0 128:67
@@ -53,11 +53,11 @@
VM map 2 1K 1K 78644K 2 0 0 512:2
USB 80 11K 11K 78644K 90 0 0
16:10,32:10,64:34,128:15,256:10,1024:0,4096:1
USB device 30 31K 31K 78644K 30 0 0
16:10,128:4,256:1,2048:15
- soname 3 1K 1K 78644K 111 0 0
16:1,32:1,128:1
- packet tags 1 1K 4K 78644K 2109809 0 0 32:1
- sockaddr 62 2K 2K 78644K 201 0 0 16:9,32:53
+ soname 3 1K 1K 78644K 186 0 0
16:1,32:1,128:1
+ packet tags 0 0K 4K 78644K 8084258 0 0 32:0
+ sockaddr 64 2K 2K 78644K 376 0 0 16:9,32:55
tcpcongctl 2 1K 1K 78644K 2 0 0 32:2
- vmem 6 35K 51K 78644K 12 0 0
16:1,64:1,512:0,2048:3,4096:0,8192:0,16384:0,32768:1
+ vmem 6 59K 87K 78644K 13 0 0
16:1,64:1,512:0,2048:3,4096:0,8192:0,16384:0,32768:0,65536:1
blist 2 73K 73K 78644K 2 0 0
32:1,131072:1
NETSMBDEV 1 1K 1K 78644K 1 0 0 16:1
RAIDframe 10 35K 35K 78644K 10 0 0
32:1,2048:1,4096:8
@@ -66,9 +66,9 @@
sysctldata 97 11K 11K 78644K 138 0 0
16:48,64:23,128:16,256:8,512:0,1024:0,2048:2
devbuf 121 750K 750K 78644K 129 0 0
16:13,32:19,64:5,128:29,256:11,512:25,1024:2,4096:6,65536:11
DMA map 305 524K 524K 78644K 305 0 0
64:176,256:1,4096:128
- pcb 9 1K 2K 78644K 128 0 0
16:2,32:0,128:7
- temp 5 38K 62K 78644K 58079 0 0
16:0,32:0,64:0,128:0,256:1,512:0,1024:1,2048:0,4096:1,8192:0,16384:2
- routetbl 37 4K 4K 78644K 69 0 0
16:8,32:6,128:22,256:1
+ pcb 9 1K 2K 78644K 221 0 0
16:2,32:0,128:7
+ temp 5 38K 62K 78644K 114574 0 0
16:0,32:0,64:0,128:0,256:1,512:0,1024:1,2048:0,4096:1,8192:0,16384:2
+ routetbl 38 4K 4K 78644K 70 0 0
16:8,32:6,128:23,256:1
UFS mount 3 7K 7K 78644K 3 0 0
256:1,2048:1,4096:1
Export Host 2 1K 1K 78644K 2 0 0 128:2
ip_moptions 1 1K 1K 78644K 1 0 0 16:1
@@ -87,115 +87,116 @@
acpi 1976 140K 142K 78644K 7594 0 0
16:44,32:616,64:735,128:578,256:2,512:1,1024:0
Memory totals: In Use Free Requests
- 1887K 58K 2177748
+ 1911K 58K 8209885
Memory resource pool statistics
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg
Idle
-amappl 48 252 0 28 3 0 3 3 0 inf 0
-anonpl 16 2839 0 102 11 0 11 11 0 inf 0
-ataspl 56 126338 0 126338 1 0 1 1 0 inf
1
-biopl 176 67 0 33 2 0 2 2 0 inf 0
-buf16k 16384 1616 0 884 259 65 194 206 1 1
1
+amappl 48 309 0 28 4 0 4 4 0 inf 0
+anonpl 16 3046 0 102 12 0 12 12 0 inf 0
+ataspl 56 268759 0 268759 1 0 1 1 0 inf
1
+biopl 176 70 0 33 2 0 2 2 0 inf 0
+buf16k 16384 2927 0 2448 309 150 159 234 1 1 0
buf1k 1024 2 0 2 1 0 1 1 1 1
1
-buf2k 2048 10797 2 2953 253 7 246 246 1 1 0
-buf4k 4096 67 0 45 2 0 2 2 1 1 0
+buf2k 2048 20897 2 17308 288 90 198 270 1 1 0
+buf4k 4096 122 0 98 2 0 2 2 1 1 0
buf512b 512 1 0 1 1 0 1 1 1 1
1
-buf8k 8192 32 0 20 4 2 2 3 1 1 0
-bufpl 176 8848 0 238 375 0 375 375 0 inf 0
-cwdi 64 40 0 23 1 0 1 1 0 inf 0
-execargs 262144 141 0 141 2 1 1 1 0 16
1
+buf8k 8192 44 0 39 5 3 2 3 1 1
1
+bufpl 176 9709 0 238 412 0 412 412 0 inf 0
+cwdi 64 64 0 23 1 0 1 1 0 inf 0
+execargs 262144 221 0 221 2 1 1 1 0 16
1
extent 20 133 0 119 1 0 1 1 0 inf 0
-fdfile 64 189 0 88 3 0 3 3 0 inf 0
-ffsdino2 256 32721 0 14 2045 0 2045 2045 0 inf 0
-ffsino 172 32709 0 5 1422 0 1422 1422 0 inf 0
-file 64 97 0 9 2 0 2 2 0 inf 0
-filedesc 704 38 0 24 8 0 8 8 0 inf
2
-fstrans 20 25 0 3 1 0 1 1 0 inf 0
-in6pcbpl 144 42 0 23 1 0 1 1 0 inf 0
+fdfile 64 287 0 88 4 0 4 4 0 inf 0
+ffsdino2 256 32746 0 14 2046 0 2046 2046 0 inf 0
+ffsino 172 32735 0 5 1424 0 1424 1424 0 inf 0
+file 64 111 0 9 2 0 2 2 0 inf 0
+filedesc 704 64 0 24 8 0 8 8 0 inf 0
+fstrans 20 50 0 3 1 0 1 1 0 inf 0
+in6pcbpl 144 48 0 29 1 0 1 1 0 inf 0
inmltpl 32 2 0 0 1 0 1 1 0 inf 0
-inpcbpl 112 38 0 23 1 0 1 1 0 inf 0
-kcredpl 192 87 0 0 5 0 5 5 0 inf 0
-kmem-100 100 102 0 0 21 0 21 21 0 inf 0
-kmem-1024 1024 77 0 0 77 0 77 77 0 inf 0
-kmem-12 12 196 0 0 5 0 5 5 0 inf 0
-kmem-128 128 44 0 0 11 0 11 11 0 inf 0
-kmem-1280 1280 23 0 0 23 0 23 23 0 inf 0
-kmem-1536 1536 4 0 0 4 0 4 4 0 inf 0
-kmem-16 16 137 0 0 5 0 5 5 0 inf 0
+inpcbpl 112 50 0 35 1 0 1 1 0 inf 0
+kcredpl 192 111 0 17 5 0 5 5 0 inf 0
+kmem-100 100 129 0 9 24 0 24 24 0 inf 0
+kmem-1024 1024 116 0 40 77 0 77 77 0 inf
1
+kmem-12 12 301 0 28 7 0 7 7 0 inf 0
+kmem-128 128 56 0 6 13 0 13 13 0 inf 0
+kmem-1280 1280 31 0 7 24 0 24 24 0 inf 0
+kmem-1536 1536 16 0 0 16 0 16 16 0 inf 0
+kmem-16 16 163 0 22 5 0 5 5 0 inf 0
kmem-1792 1792 2 0 0 2 0 2 2 0 inf 0
-kmem-20 20 169107 0 0 6765 0 6765 6765 0 inf 0
+kmem-20 20 387780 0 2 15512 0 15512 15512 0 inf 0
kmem-2048 2048 5 0 0 5 0 5 5 0 inf 0
-kmem-2304 2304 6 0 0 6 0 6 6 0 inf 0
-kmem-24 24 193 0 0 10 0 10 10 0 inf 0
+kmem-2304 2304 8 0 2 6 0 6 6 0 inf 0
+kmem-24 24 217 0 17 10 0 10 10 0 inf 0
kmem-2560 2560 4 0 0 4 0 4 4 0 inf 0
-kmem-28 28 20 0 0 2 0 2 2 0 inf 0
+kmem-28 28 29 0 4 2 0 2 2 0 inf 0
kmem-2816 2816 2 0 0 2 0 2 2 0 inf 0
kmem-3072 3072 1 0 0 1 0 1 1 0 inf 0
-kmem-32 32 2112 0 0 132 0 132 132 0 inf 0
+kmem-32 32 2116 0 2072 132 0 132 132 0 inf
124
kmem-3328 3328 1 0 0 1 0 1 1 0 inf 0
kmem-3584 3584 4 0 0 4 0 4 4 0 inf 0
-kmem-36 36 2094 0 0 150 0 150 150 0 inf 0
+kmem-36 36 2104 0 2050 150 0 150 150 0 inf
139
kmem-3840 3840 1 0 0 1 0 1 1 0 inf 0
-kmem-40 40 129 0 0 11 0 11 11 0 inf 0
-kmem-44 44 6 0 0 1 0 1 1 0 inf 0
-kmem-48 48 31 0 0 4 0 4 4 0 inf 0
-kmem-512 512 107 0 0 107 0 107 107 0 inf 0
-kmem-56 56 3 0 0 1 0 1 1 0 inf 0
+kmem-40 40 164 0 41 11 0 11 11 0 inf 0
+kmem-44 44 12 0 0 2 0 2 2 0 inf 0
+kmem-48 48 35 0 1 4 0 4 4 0 inf 0
+kmem-512 512 112 0 24 110 0 110 110 0 inf
22
+kmem-56 56 4 0 1 1 0 1 1 0 inf 0
kmem-64 64 17 0 0 3 0 3 3 0 inf 0
-kmem-72 72 206 0 0 30 0 30 30 0 inf 0
-kmem-768 768 79 0 0 79 0 79 79 0 inf 0
-kmem-8 8 2 0 0 1 0 1 1 0 inf 0
-kmem-84 84 86 0 0 15 0 15 15 0 inf 0
-ksiginfo 48 29 0 0 1 0 1 1 0 inf 0
-kvakernel 4096 48151 0 0 1505 0 1505 1505 0 inf 0
-kvakmem 4096 600 0 0 19 0 19 19 0 inf 0
-lockf 64 5 0 0 1 0 1 1 0 inf 0
-lwppl 672 63 0 0 11 0 11 11 0 inf 0
-mbpl 256 258 0 0 18 0 18 18 1 inf
1
-mclpl 2048 206 0 0 107 0 107 107 4 8030
4
-mutex 64 384 0 0 7 0 7 7 0 inf 0
-ncache 128 25468 0 0 822 0 822 822 0 inf 0
+kmem-72 72 232 0 72 30 0 30 30 0 inf 0
+kmem-768 768 110 0 34 79 0 79 79 0 inf
3
+kmem-8 8 2 0 2 1 0 1 1 0 inf
1
+kmem-84 84 93 0 4 15 0 15 15 0 inf 0
+ksiginfo 48 58 0 29 1 0 1 1 0 inf 0
+kvakernel 4096 102837 0 21 3213 0 3213 3213 0 inf 0
+kvakmem 4096 707 0 6 22 0 22 22 0 inf 0
+lockf 64 5 0 4 1 0 1 1 0 inf 0
+lwppl 672 70 0 9 11 0 11 11 0 inf 0
+mbpl 256 355 0 111 18 1 17 18 1 inf 0
+mclpl 2048 310 0 125 107 4 103 107 4 8030
10
+mutex 64 400 0 31 7 0 7 7 0 inf 0
+ncache 128 26672 0 15446 822 0 822 822 0 inf
62
nfsreqcachepl 56 64 0 0 1 0 1 1 0 inf 0
-nfsrvdescpl 184 1494911 0 1494910 1 0 1 1 0 inf 0
+nfsrvdescpl 184 5997280 0 5997280 1 0 1 1 0 inf
1
pcache 576 86 0 14 11 0 11 11 0 inf 0
pcachecpu 64 81 0 0 2 0 2 2 0 inf 0
-pcglarge 512 43 0 9 5 0 5 5 0 inf 0
-pcgnormal 128 925 0 68 29 0 29 29 0 inf 0
+pcglarge 512 108 0 43 9 0 9 9 0 inf 0
+pcgnormal 128 2391 0 1741 52 14 38 52 0 inf
1
pdict16 44 19 0 4 1 0 1 1 0 inf 0
pdict32 60 9 0 1 1 0 1 1 0 inf 0
-pdppl 4096 45 0 31 37 0 37 37 0 inf
23
-phpool-0 32 44996 0 74 354 0 354 354 0 inf 0
-phpool-64 36 9072 0 1 81 0 81 81 0 inf 0
-piperd 192 7 0 0 1 0 1 1 0 inf 0
-pipewr 192 6 0 0 1 0 1 1 0 inf 0
+pdppl 4096 103 0 87 39 21 18 37 0 inf
2
+phpool-0 32 99755 0 269 784 0 784 784 0 inf 0
+phpool-64 36 19556 0 1 175 0 175 175 0 inf 0
+piperd 192 19 0 12 1 0 1 1 0 inf 0
+pipewr 192 23 0 14 1 0 1 1 0 inf 0
plimitpl 196 3 0 0 1 0 1 1 0 inf 0
-pmappl 96 37 0 0 1 0 1 1 0 inf 0
-pnbufpl 1024 169046 0 0 42262 0 42262 42262 0 inf 0
-procpl 448 21 0 0 3 0 3 3 0 inf 0
+pmappl 96 46 0 27 1 0 1 1 0 inf 0
+pnbufpl 1024 387713 0 2 96928 0 96928 96928 0 inf 0
+procpl 448 28 0 9 3 0 3 3 0 inf 0
propdict 32 85 0 17 1 0 1 1 0 inf 0
propnmbr 32 17 0 4 1 0 1 1 0 inf 0
propstng 20 80 0 8 1 0 1 1 0 inf 0
-pstatspl 268 21 0 0 2 0 2 2 0 inf 0
-ptimerpl 164 5 0 2 1 0 1 1 0 inf 0
-ptimerspl 140 5 0 2 1 0 1 1 0 inf 0
-pvpl 20 3881 0 0 20 0 20 20 0 inf 0
-ractx 24 521 0 0 4 0 4 4 0 inf 0
-rndsample 528 1953 0 1952 1 0 1 1 0 inf 0
-rtentpl 184 29 0 0 2 0 2 2 0 inf 0
-sigacts 2056 36 0 0 36 0 36 36 0 inf 0
-socket 332 57 0 0 5 0 5 5 0 inf 0
-synpl 188 2 0 2 1 0 1 1 0 inf
1
-tcpcbpl 500 20 0 7 2 0 2 2 0 inf 0
-tcpipqepl 44 3061 0 3061 1 0 1 1 0 inf
1
+pstatspl 268 28 0 9 2 0 2 2 0 inf 0
+ptimerpl 164 11 0 8 1 0 1 1 0 inf 0
+ptimerspl 140 11 0 8 1 0 1 1 0 inf 0
+pvpl 20 5202 0 20 26 0 26 26 0 inf 0
+ractx 24 521 0 434 4 0 4 4 0 inf 0
+rndsample 528 4152 0 4151 1 0 1 1 0 inf 0
+rtentpl 184 30 0 0 2 0 2 2 0 inf 0
+sigacts 2056 50 0 29 41 0 41 41 0 inf
20
+socket 332 61 0 20 6 0 6 6 0 inf
1
+synpl 188 5 0 5 1 1 0 1 0 inf 0
+tcpcbpl 500 26 0 13 2 0 2 2 0 inf 0
+tcpipqepl 44 7993 0 7993 1 0 1 1 0 inf
1
tmpfs_dirent_0xcbebc004 140 1 0 0 1 0 1 1 0 inf 0
-tstilepl 52 63 0 0 1 0 1 1 0 inf 0
-uarea 12288 63 0 0 63 0 63 63 0 inf 0
-ufsdir 264 26 0 0 2 0 2 2 0 inf 0
-vmembt 28 9401 0 0 65 0 65 65 0 inf 0
-vmmpepl 84 475 0 0 10 0 10 10 0 inf 0
-vmsppl 212 37 0 0 2 0 2 2 0 inf 0
-vnodepl 172 32713 0 0 1423 0 1423 1423 0 inf 0
-wapblinopl 20 57933 0 57933 1 0 1 1 0 inf
1
+tstilepl 52 68 0 11 1 0 1 1 0 inf 0
+uaoeltpl 84 2 0 0 1 0 1 1 0 inf 0
+uarea 12288 68 0 11 65 0 65 65 0 inf
8
+ufsdir 264 31 0 31 3 0 3 3 0 inf
3
+vmembt 28 19279 0 0 133 0 133 133 0 inf 0
+vmmpepl 84 577 0 0 13 0 13 13 0 inf 0
+vmsppl 212 41 0 0 3 0 3 3 0 inf 0
+vnodepl 172 32731 0 0 1424 0 1424 1424 0 inf 0
+wapblinopl 20 118147 0 118147 1 0 1 1 0 inf
1
-In use 228132K, total allocated 230402K; utilization 99.0%
+In use 438950K, total allocated 450688K; utilization 97.4%
>How-To-Repeat:
1. prepare NFS server and NFS client
2. mount_nfs exported directory on client with "-i,-T"
3. copy NetBSD src tree on NFS client
% rsync -a /local/src /nfs/1/ &
% rsync -a /local/src /nfs/2/ &
% rsync -a /local/src /nfs/3/ &
% rsync -a /local/src /nfs/4/ &
4. watch "vmstat -m -w30" on NFS server
>Fix:
No idea.
--
Takahiro HAYASHI
Home |
Main Index |
Thread Index |
Old Index