Subject: Re: RAIDFrame and RAID-5
To: Greg Oster <oster@cs.usask.ca>
From: Greg A. Woods <woods@weird.com>
List: current-users
Date: 10/27/2003 15:08:57
[ On Wednesday, September 10, 2003 at 07:50:43 (-0600), Greg Oster wrote: ]
> Subject: Re: RAIDFrame and RAID-5
>
> "Thomas Hertz" writes:
> >
> > I haven't been able to get a kernel core dump, since the system just
> > freezes. I have noted that just moments before the system freezes, it's
> > not possible to start new processes. The already running processes, will
> > continue to run normally for some minute more. Most of the time the
> > console prints out a few "cannot allocate Tx mbuf" for the various
> > network interfaces just before the final freeze.
>
> Hmm... I wonder which one of the "out of kernel memory" problems
> applies to this case...
I just this morning encountered a system freeze that appears to have
been caused by RAIDframe.
Oddly enough it recovered all by itself.
I was moving about 2GB of mostly files from one RAIDframe RAID-5 array
to another.
Things were going well for the first 600MB or so, then suddenly the
system "froze". It was not pingable.
After a few minutes (~10 minutes) I connected to the console and found
it did not respond to <CR>. I sent a BREAK to see if DDB would respond,
and sure enough it did (the rlogind warnings appear approx every five
minutes, they're triggered by netsaint checks):
[Mon Oct 27 10:34:09 2003]Oct 27 10:34:09 proven rlogind[21028]: Can't get peer name of remote host: Socket is not connected
[-- break #1 sent -- `\z' -- Mon Oct 27 10:46:50 2003]
[Mon Oct 27 10:46:50 2003]Stopped at cpu_Debugger+0x4: movl %ebp,%esp
[Mon Oct 27 10:46:50 2003] db> trace
[Mon Oct 27 10:46:55 2003] cpu_Debugger(c0b25c60,cfe57574,cfe57574,c0b2ae50,c0b25ba0) at cpu_Debugger+0x4
[Mon Oct 27 10:46:55 2003] comintr(c0b37600) at comintr+0xf4
[Mon Oct 27 10:46:55 2003] Xintr4() at Xintr4+0x7e
[Mon Oct 27 10:46:55 2003] --- interrupt ---
[Mon Oct 27 10:46:55 2003] idle(cfe57574,2,c01e8fdc,cfe57574) at idle+0x20
[Mon Oct 27 10:46:55 2003] bpendtsleep(c04afc0c,118,c0395140,2,0) at bpendtsleep
[Mon Oct 27 10:46:55 2003] sys_select(cfe57574,cfe92f80,cfe92f78,3) at sys_select+0x304
[Mon Oct 27 10:46:55 2003] syscall_plain(bfbf001f,bfbf001f,1f,bfbf001f,bfbfdc28) at syscall_plain+0xa7
[Mon Oct 27 10:46:55 2003] db> cont
[Mon Oct 27 10:47:03 2003]
[Mon Oct 27 10:51:43 2003]
[Mon Oct 27 10:51:43 2003]NetBSD/i386-1.6.1_STABLE (proven) (tty00)
[Mon Oct 27 10:51:43 2003]
[Mon Oct 27 10:51:43 2003]Current local time is 10:51AM on Monday, 27 October 2003
[Mon Oct 27 10:51:43 2003]
[Mon Oct 27 10:51:43 2003]login:
[Mon Oct 27 10:51:44 2003]Oct 27 10:51:31 proven rlogind[21112]: Can't get peer name of remote host: Socket is not connected
I had better things to do before dealing with it, such as make coffee,
so I went off and left it alone for a while longer.
When I came back it was, as shown above, alive and continuing to copy
files. The remainder of the copy went smoothly with no interruption.
However a couple of xterm windows had closed (ones I had clicked the
mouse in, and so the server had killed them since their client host was
not responding). Perhaps in this case one or more of the client xterm
processes did actually receive the TCP RST and died thus freeing some
memory?
Other than that I have no idea what might have made it wake up again.
Note this is on my development system running a 1.6.1_STABLE kernel from
about a month ago. The system has 320MB of RAM. I have one RAID-1 set
and two RAID-5 sets:
total memory = 319 MB
avail memory = 275 MB
using 1000 buffers containing 32720 KB of memory
$ df -kl
Filesystem 1K-blocks Used Avail %Cap Mounted on
/dev/raid2a 2064398 725146 1236032 36% /
/dev/raid2e 2064430 323784 1637424 16% /var
/dev/raid2f 3502094 2017442 1309546 60% /opt
/dev/sd0a 89410 10810 74128 12% /stand-sd0
/dev/sd1a 89410 10810 74128 12% /stand-sd1
/dev/sd5c 12186596 4755508 6821758 41% /cvs
/dev/sd4c 12186596 8256340 3320926 71% /work
/dev/raid0a 21879470 13045850 8396030 60% /home
/dev/raid1a 53389172 29480268 21239444 58% /save
mfs:134 248111 5 235700 0% /tmp
kernfs 1 1 0 100% /kern
procfs 4 4 0 100% /proc
fdesc 1 1 0 100% /fdesc
/dev/ccd0d 17503052 4515472 12112426 27% /build
I must say this is much better behaviour than I've ever seen before with
a 1.5W kernel, and the improvements are probably due to improvements in
UVM. A 10-15 minute freeze beats a panic() any day! ;-)
> You might want to add KMEMSTAT (or whatever it is) to the kernel
> config, and then do a bunch of "vmstat -m" while causing the machine
> to crash. That might indicate whether you're actually out of kernel
> memory or not....
I can only show you what it looks like now, i.e. a couple of hours after
it came back to life:
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 898 1406 20139288 1280 57
32 648 1528 6076834 640 6998
64 1232 2864 8508149 320 11595
128 681 663 7127857 160 223812
256 588 1940 11128577 80 2498438
512 628 420 2856066 40 870884
1024 490 358 1291630 20 390591
2048 29 43 5938818 10 3236424
4096 45 25 3751919 5 3729945
8192 20 43 932269 5 927994
16384 9 0 4028603 5 0
32768 4 0 6874 5 0
65536 10 0 7701 5 0
131072 5 0 138411 5 0
262144 3 0 6 5 0
524288 0 0 1 5 0
Memory usage type by bucket size
Size Type(s)
16 devbuf, pcb, routetbl, soname, namecache, UFS mount, shm, proc,
ip_moptions, in_multi, exec, miscfs mount, VM swap, UVM amap,
UVM aobj, temp, USB, RAIDframe, pipe
32 devbuf, pcb, routetbl, fragtbl, ifaddr, soname, session, vnodes,
MFS node, ether_multi, exec, VM swap, UVM amap, temp, USB, RAIDframe,
pipe
64 devbuf, pcb, routetbl, soname, NFS req, lockf, NFS daemon, in_multi,
exec, NFS diroff, UVM amap, temp, DMA map, USB, RAIDframe, NDP, pipe
128 devbuf, pcb, routetbl, ifaddr, soname, cred, iov, ttys, VM swap,
UVM amap, temp, USB, USB device, RAIDframe, inodedep, pipe
256 devbuf, pcb, routetbl, ifaddr, vnodes, VM map, file desc, Export Host,
NFS srvsock, NFS daemon, exec, NFS srvdesc, NFS diroff, UVM amap,
temp, DMA map, USB, RAIDframe, NDP, newblk, pipe
512 devbuf, pcb, ifaddr, ioctlops, mount, NFS mount, UFS mount, file desc,
NFS daemon, UVM amap, temp, USB device, RAIDframe, key mgmt, pipe
1024 devbuf, shm, file, file desc, NQNFS Lease, ttys, exec, VM page,
UVM amap, UVM aobj, temp, RAIDframe, key mgmt, pipe
2048 ifaddr, UFS mount, file, proc, exec, UVM amap, temp, RAIDframe
4096 devbuf, namecache, UFS mount, file, UVM amap, temp, RAIDframe
8192 devbuf, UFS mount, proc, UVM amap, temp, RAIDframe, indirdep
16384 devbuf, namecache, UFS mount, UVM amap, temp, RAIDframe, pagedep,
indirdep
32768 UFS mount, UVM amap, temp, RAIDframe
65536 MSDOSFS mount, UVM amap, temp, RAIDframe
131072 NFS node, namecache, UFS mount, ISOFS mount, UVM amap, RAIDframe,
inodedep
262144 UVM amap
524288 RAIDframe
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
devbuf 343 128K 128K 49092K 545 0 0 16,32,64,128,256,512,1024,4096,8192,16384
pcb 91 16K 19K 49092K 152846 0 0 16,32,64,128,256,512
routetbl 128 9K 10K 49092K 10542 0 0 16,32,64,128,256
fragtbl 0 0K 1K 49092K 60 0 0 32
ifaddr 42 8K 8K 49092K 42 0 0 32,128,256,512,2048
soname 12 1K 1K 49092K 108251 0 0 16,32,64,128
ioctlops 0 0K 1K 49092K 499 0 0 512
cred 53 7K 17K 49092K 2985249 0 0 128
session 57 2K 2K 49092K 61573 0 0 32
iov 0 0K 1K 49092K 479 0 0 128
mount 18 9K 9K 49092K 22 0 0 512
NFS req 0 0K 1K 49092K 4543 0 0 64
NFS mount 4 2K 2K 49092K 4 0 0 512
NFS node 1 128K 128K 49092K 1 0 0 131072
vnodes 91 8K 43K 49092K 31050 0 0 32,256
namecache 4 149K 149K 49092K 4 0 0 16,4096,16384,131072
UFS mount 37 284K 284K 49092K 40 0 0 16,512,2048,4096,8192,16384,32768,131072
shm 5 5K 5K 49092K 6 0 0 16,1024
VM map 4 1K 1K 49092K 4 0 0 256
file 0 0K 4K 49092K 11560 0 0 1024,2048,4096
file desc 29 11K 34K 49092K 57440 0 0 256,512,1024
lockf 21 2K 7K 49092K 289898 0 0 64
proc 10 19K 19K 49092K 1247 0 0 16,2048,8192
MFS node 1 1K 1K 49092K 1 0 0 32
NQNFS Lease 1 1K 1K 49092K 1 0 0 1024
Export Host 7 2K 2K 49092K 29 0 0 256
NFS srvsock 2 1K 1K 49092K 2 0 0 256
NFS daemon 69 7K 7K 49092K 69 0 0 64,256,512
ip_moptions 1 1K 1K 49092K 1 0 0 16
in_multi 18 1K 1K 49092K 18 0 0 16,64
ether_multi 9 1K 1K 49092K 9 0 0 32
ISOFS mount 1 128K 128K 49092K 1 0 0 131072
MSDOSFS mount 1 64K 64K 49092K 1 0 0 65536
ttys 331 215K 215K 49092K 331 0 0 128,1024
exec 1 1K 13K 49092K 1575332 0 0 16,32,64,256,1024,2048
miscfs mount 1 1K 1K 49092K 1 0 0 16
NFS srvdesc 0 0K 1K 49092K 820729 0 0 256
NFS diroff 83 9K 9K 49092K 83 0 0 64,256
VM swap 5 1K 1K 49092K 5 0 0 16,32,128
VM page 1 1K 1K 49092K 1 0 0 1024
UVM amap 2123 1664K 2319K 49092K 37154862 0 0 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,131072,262144
UVM aobj 2 2K 2K 49092K 2 0 0 16,1024
temp 68 114K 131K 49092K 4162039 0 0 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
DMA map 349 67K 67K 49092K 349 0 0 64,256
USB 15 2K 2K 49092K 15 0 0 16,32,64,128,256
USB device 4 2K 2K 49092K 4 0 0 128,512
RAIDframe 781 160K 2422K 49092K 23383637 0 0 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,131072,524288
key mgmt 462 308K 731K 49092K 1043727 0 0 512,1024
NDP 1 1K 1K 49092K 3 0 0 64,256
pagedep 1 16K 16K 49092K 1 0 0 16384
inodedep 1 128K 207K 49092K 47888 0 0 128,131072
newblk 1 1K 1K 49092K 1 0 0 256
indirdep 0 0K 344K 49092K 6533 0 0 8192,16384
pipe 0 0K 1K 49092K 21423 0 0 16,32,64,128,256,512,1024
Memory Totals: In Use Free Requests
3670K 1915K 71933003
Memory resource pool statistics
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
phpool 40 6772 0 6128 27 6 21 24 0 inf 0
pcgpool 76 3195 0 3190 2 1 1 2 0 inf 0
pmappl 68 880701 0 880597 47 45 2 10 0 inf 0
pdppl 4096 3455 0 3351 1648 1540 108 583 0 inf 4
vmsppl 188 880701 0 880597 86 80 6 28 0 inf 0
vmmpepl 64 24244269 0 24243118 481 460 21 79 0 inf 0
vmmpekpl 64 1277696 0 1276760 22 6 16 16 0 inf 0
uaoeltpl 84 164 0 129 1 0 1 1 0 inf 0
aobjpl 52 1 0 0 1 0 1 1 0 inf 0
amappl 40 11247213 0 11246520 61 52 9 31 0 inf 0
mbpl 256 19224 10295 19087 203 186 17 44 1 inf 0
mclpl 2048 7827 0 7745 1872 1827 45 93 4 16384 4
sockpl 168 438465 0 438268 83 72 11 29 0 inf 0
ttypl 284 66 0 0 5 0 5 5 0 inf 0
rndsample 528 664160 0 664139 4 0 4 4 0 inf 0
procpl 460 880721 0 880601 240 224 16 75 0 inf 0
pgrppl 24 76912 0 76831 1 0 1 1 0 inf 0
pcredpl 24 880721 0 880601 7 6 1 4 0 inf 0
plimitpl 156 100359 0 100346 1 0 1 1 0 inf 0
rusgepl 72 880601 0 880601 90 89 1 1 0 inf 1
filepl 52 43011549 0 43011099 44 38 6 12 0 inf 0
cwdipl 12 880703 0 880599 2 1 1 2 0 inf 0
fdescpl 124 880703 0 880599 44 40 4 19 0 inf 0
pnbufpl 1024 1213 0 1211 29 28 1 3 0 inf 0
vnodepl 152 19240 0 0 740 0 740 740 0 inf 0
ncachepl 72 19240 0 0 344 0 344 344 0 inf 0
sdpcpool 104 1375837 0 1375836 694 693 1 54 0 inf 0
pagedeppl 64 389226 0 389226 141 141 0 14 0 inf 0
inodedeppl 84 2022916 0 2022916 1827 1827 0 793 0 inf 0
newblkpl 32 1692008 0 1692008 41 40 1 1 0 inf 1
bmsafemappl 32 90837 0 90836 33 32 1 2 0 inf 0
allocdirectpl 64 1034459 0 1034459 668 668 0 45 0 inf 0
indirdeppl 28 6150 0 6149 46 45 1 1 0 inf 0
allocindirpl 52 657549 0 657521 1107 1104 3 71 0 inf 2
freefragpl 32 51625 0 51625 90 90 0 5 0 inf 0
freeblkspl 108 1391327 0 1391327 1162 1162 0 729 0 inf 0
freefilepl 28 1555439 0 1555439 384 384 0 191 0 inf 0
diraddpl 32 1145331 0 1145331 345 345 0 42 0 inf 0
mkdirpl 28 439242 0 439242 95 95 0 12 0 inf 0
dirrempl 32 1736505 0 1736505 483 483 0 255 0 inf 0
newdirblkpl 16 825 0 825 58 58 0 1 0 inf 0
ffsinopl 260 60396139 0 60390722 1433 1071 362 1281 0 inf 0
nfsnodepl 324 505 0 208 33 8 25 25 0 inf 0
nfsvapl 100 505 0 208 8 0 8 8 0 inf 0
cd9660nopl 148 0 0 0 0 0 0 0 0 inf 0
msdosnopl 140 0 0 0 0 0 0 0 0 inf 0
scxspl 152 42506188 0 42506188 5 1 4 4 0 inf 4
extent 20 169282 0 160721 45 2 43 43 0 inf 0
bufpl 104 6661198 0 6661198 125 102 23 50 0 inf 23
ccdpl 120 1347332 0 1347332 284 284 0 56 0 inf 0
raidpl 116 21396162 0 21396162 5 3 2 2 0 inf 2
brtpl 32 0 0 0 0 0 0 0 0 inf 0
rtentpl 132 543 0 475 3 0 3 3 0 inf 0
rttmrpl 32 0 0 0 0 0 0 0 0 inf 0
ipqepl 44 53458 0 53458 190 190 0 1 0 inf 0
ipflowpl 72 0 0 0 0 0 0 0 0 inf 0
inpcbpl 96 285627 0 285516 58 55 3 8 0 inf 0
tcpcbpl 320 150269 0 150169 187 178 9 26 0 inf 0
synpl 188 61457 0 61457 169 169 0 3 0 inf 0
sigapl 1540 880703 0 880599 1118 1066 52 292 0 inf 0
swp vnx 20 0 0 0 0 0 0 0 0 inf 0
swp vnd 108 0 0 0 0 0 0 0 0 inf 0
pipepl 128 396778 0 396704 13 10 3 4 0 inf 0
In use 7078K, total allocated 7712K; utilization 91.8%
--
Greg A. Woods
+1 416 218-0098 VE3TCP RoboHack <woods@robohack.ca>
Planix, Inc. <woods@planix.com> Secrets of the Weird <woods@weird.com>