Subject: '... RPC: Unable to send; errno = No buffer space available' in -current
To: None <current-users@netbsd.org>
From: Markus W Kilbinger <mk@kilbi.de>
List: current-users
Date: 01/09/2007 12:02:50
--cu1uLm8q3Y
Content-Type: text/plain; charset=us-ascii
Content-Description: message body text
Content-Transfer-Encoding: 7bit
Hi!
I'm having a redundant problem with one router/firewall (running
ipfilter/ipnat and squid) on a i386/-current machine, which
recurrently 'looses' its internal network interface 'wm0' while
displaying the above mentioned error message. The external network
interface 'wm1' remains active (at least ping-able).
First some excerpt of the machines dmesg:
NetBSD 4.99.7 (RADIX) #50: Mon Jan 8 12:13:04 MET 2007
[...]
wm0 at pci2 dev 10 function 0: Intel i82545EM 1000BASE-T Ethernet, rev. 1
wm0: interrupting at ioapic0 pin 22 (irq 5)
wm0: Ethernet address 00:07:e9:0e:67:c7
makphy0 at wm0 phy 1: Marvell 88E1011 Gigabit PHY, rev. 3
makphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
wm1 at pci2 dev 11 function 0: Intel i82545EM 1000BASE-T Ethernet, rev. 1
wm1: interrupting at ioapic0 pin 23 (irq 9)
wm1: Ethernet address 00:07:e9:0e:68:4b
makphy1 at wm1 phy 1: Marvell 88E1011 Gigabit PHY, rev. 3
makphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
[...]
Enabling ipfilter.
Setting sysctl variables:
kern.maxfiles: 1772 -> 4096
kern.autonicetime: 0 -> 0
kern.sbmax: 262144 -> 1048576
net.inet.tcp.sendspace: 32768 -> 131072
net.inet.tcp.recvspace: 32768 -> 131072
net.inet.udp.sendspace: 9216 -> 36864
net.inet.udp.recvspace: 41600 -> 166400
net.inet6.udp6.sendspace: 9216 -> 36864
net.inet6.udp6.recvspace: 42080 -> 168320
net.inet.tcp.ecn.enable: 0 -> 1
net.inet.ipsec.ecn: 0 -> 1
net.inet6.tcp6.ecn.enable: 1 -> 1
net.inet6.ipsec6.ecn: 0 -> 1
The machines kernel (already) has:
options NMBCLUSTERS=32768
options LARGE_NAT
options KMEMSTATS
In the state of lost 'wm0':
# netstat -m
1142 mbufs in use:
944 mbufs allocated to data
192 mbufs allocated to packet headers
6 mbufs allocated to socket names and addresses
10 calls to protocol drain routines
'vmstat -m' output is attached at the end of this mail.
Can anybody explain what is going wrong here?
(Maybe/of course I have to provide more (specific) infos..., just tell me!)
Any comment/help appreciated,
Markus.
--cu1uLm8q3Y
Content-Type: text/plain
Content-Description: vmstat -m output
Content-Disposition: inline;
filename="radix-vmstat.hangs"
Content-Transfer-Encoding: 7bit
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 201 55 783123 1280 0
32 717 51 8591 640 0
64 3673 39 20734 320 44
128 729 7 75544 160 0
256 247 25 103854 80 0
512 749 563 2409376 40 361799
1024 557 1083 501175 20 73259
2048 32 26 1746 10 219
4096 161 4 308663 5 0
8192 10 3 86 5 0
16384 5 0 5 5 0
32768 24 0 38 5 0
65536 37 0 37 5 0
131072 12 0 610402 5 0
262144 9 0 9 5 0
Memory usage type by bucket size
Size Type(s)
16 acpi, ifaddr, namecache, soname, packet tags, shm, pipe, sysctldata,
in_multi, ip_moptions, UFS mount, routetbl, temp, pcb, devbuf, exec,
VM swap, kernfs mount, RAIDframe, NETSMBDEV, NDP
32 acpi, AGP, ifmedia, ether_multi, ifaddr, soname, blist, proc,
in_multi, fragtbl, routetbl, temp, pcb, devbuf, exec, kevent, VM swap,
UVM aobj, RAIDframe, ip6_options, tcpcongctl
64 acpi, ifaddr, vnodes, pipe, in_multi, routetbl, temp, pcb, DMA map,
devbuf, NFS diroff, NFS req, NDP
128 prop dictionary, acpi, ifaddr, soname, ttys, sysctlnode, ip_moptions,
routetbl, temp, devbuf, kevent, VM swap, inodedep, RAIDframe,
key mgmt, NDP, ip6_options
256 acpi, AGP, ifaddr, vnodes, sysctldata, proc, UFS mount, routetbl,
temp, DMA map, devbuf, exec, kevent, ioctlops, file desc, newblk,
NFS daemon, NFS diroff, nfs_export
512 ifaddr, vnodes, proc, temp, pcb, devbuf, ioctlops, file desc, VM page,
VM map, NFS mount, NFS srvsock, NFS daemon, RAIDframe
1024 acpi, ttys, sysctlnode, proc, temp, devbuf, exec, kevent, file desc,
UVM aobj, RAIDframe, key mgmt
2048 ifaddr, sysctlnode, proc, in_multi, UFS mount, temp, devbuf, exec,
RAIDframe
4096 mount, sysctlnode, temp, DMA map, devbuf, RAIDframe
8192 sysctlnode, temp, devbuf, RAIDframe
16384 vmem, sysctlnode
32768 UFS mount, temp, RAIDframe
65536 UFS mount, temp, pagedep, RAIDframe
131072 devbuf, MSDOSFS mount, RAIDframe
262144 namecache, blist, UFS mount, UFS quota, inodedep, NFS node,
ISOFS mount
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
prop dictionary 6 1K 1K 78644K 14 0 0 128
acpi 1419 90K 92K 78644K 3495 0 0 16,32,64,128,256,1024
AGP 2 1K 1K 78644K 2 0 0 32,256
ifmedia 16 1K 1K 78644K 16 0 0 32
ether_multi 10 1K 1K 78644K 11 0 0 32
ifaddr 31 8K 8K 78644K 31 0 0 16,32,64,128,256,512,2048
mount 10 40K 44K 78644K 20 0 0 4096
vnodes 2067 135K 135K 78644K 6164 0 0 64,256,512
namecache 4 769K 769K 78644K 4 0 0 16,262144
soname 28 4K 4K 78644K 4537 0 0 16,32,128
packet tags 0 0K 1K 78644K 168411 0 0 16
ttys 585 381K 381K 78644K 585 0 0 128,1024
shm 2 1K 1K 78644K 2 0 0 16
pipe 0 0K 1K 78644K 22 0 0 16,64
vmem 4 56K 56K 78644K 4 0 0 16384
blist 2 141K 141K 78644K 2 0 0 32,262144
sysctldata 1 1K 1K 78644K 3 0 0 16,256
sysctlnode 79 122K 122K 78644K 108 0 0 128,1024,2048,4096,8192,16384
proc 8 3K 4K 78644K 11 0 0 32,256,512,1024,2048
in_multi 57 4K 4K 78644K 57 0 0 16,32,64,2048
ip_moptions 2 1K 1K 78644K 7 0 0 16,128
UFS mount 16 581K 581K 78644K 17 0 0 16,256,2048,32768,65536,262144
fragtbl 0 0K 1K 78644K 11 0 0 32
routetbl 218 13K 13K 78644K 6924 0 0 16,32,64,128,256
temp 1036 575K 855K 78644K 3276397 0 0 16,32,64,128,256,512,1024,2048,4096,8192,32768,65536
pcb 179 14K 14K 78644K 4818 0 0 16,32,64,512
DMA map 651 546K 546K 78644K 651 0 0 64,256,4096
devbuf 362 190K 190K 78644K 556 0 0 16,32,64,128,256,512,1024,2048,4096,8192,131072
exec 4 1K 53K 78644K 3979 0 0 16,32,256,1024,2048
kevent 18 13K 17K 78644K 28109 0 0 32,128,256,1024
ioctlops 0 0K 1K 78644K 24 0 0 256,512
file desc 37 12K 12K 78644K 243 0 0 256,512,1024
VM swap 3 1K 1K 78644K 3 0 0 16,32,128
VM page 1 1K 1K 78644K 1 0 0 512
VM map 4 2K 2K 78644K 4 0 0 512
UVM aobj 2 2K 2K 78644K 2 0 0 32,1024
UFS quota 1 256K 256K 78644K 1 0 0 262144
newblk 1 1K 1K 78644K 1 0 0 256
inodedep 1 256K 257K 78644K 6 0 0 128,262144
pagedep 1 64K 64K 78644K 1 0 0 65536
NFS mount 4 2K 3K 78644K 13 0 0 512
NFS srvsock 2 1K 1K 78644K 2 0 0 512
NFS daemon 5 3K 3K 78644K 5 0 0 256,512
NFS diroff 2 1K 1K 78644K 59 0 0 64,256
NFS req 0 0K 1K 78644K 7786 0 0 64
NFS node 1 256K 256K 78644K 1 0 0 262144
nfs_export 7 2K 2K 78644K 7 0 0 256
kernfs mount 1 1K 1K 78644K 1 0 0 16
MSDOSFS mount 1 128K 128K 78644K 1 0 0 131072
ISOFS mount 1 256K 256K 78644K 1 0 0 262144
RAIDframe 103 3884K 4012K 78644K 1220915 0 0 16,32,128,512,1024,2048,4096,8192,32768,65536,131072
NETSMBDEV 1 1K 1K 78644K 1 0 0 16
key mgmt 154 132K 1214K 78644K 88669 0 0 128,1024
NDP 9 1K 1K 78644K 12 0 0 16,64,128
ip6_options 2 1K 1K 78644K 654 0 0 32,128
tcpcongctl 2 1K 1K 78644K 2 0 0 32
Memory totals: In Use Free Requests
8932K 1469K 4823383
Memory resource pool statistics
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
rf_dqd_pl 116 1791687 0 1791687 6 3 3 3 2 8 3
rf_funclist_pl 12 508662 0 508662 1 0 1 1 1 2 1
rf_daglist_pl 236 508662 0 508662 7 4 3 3 2 8 3
rf_dagh_pl 76 1119052 0 1119052 1 0 1 1 1 3 1
rf_dagnode_pl 444 5148843 0 5148843 50 29 21 23 15 57 21
rf_mcpair_pl 12 610390 0 610390 1 0 1 1 1 2 1
rf_stripelock_pl 28 109186 0 109186 1 0 1 1 1 2 1
rf_rad_pl 328 490278 0 490278 7 3 4 4 3 11 4
rf_vple_pl 12 60 0 0 1 0 1 1 1 2 0
rf_pda_pl 40 3073959 0 3073959 1 0 1 1 1 2 1
rf_asm_pl 288 1729442 0 1729442 9 3 6 6 5 14 6
rf_asmhdr_pl 16 1711058 0 1711058 1 0 1 1 1 2 1
rf_alloclist_pl 168 1119054 0 1119052 8 4 4 4 3 11 3
extent 20 6 0 0 1 0 1 1 0 inf 0
pnbufpl 1024 65 0 61 12 4 8 8 0 inf 7
sigapl 2052 3175 0 3045 162 24 138 139 0 inf 8
mclpl 2048 1555 0 760 613 211 402 557 4 16384 4
mbpl 256 3185 0 953 142 1 141 141 1 inf 1
buf8k 8192 51980 0 42943 2486 759 1727 2456 1 1 0
buf4k 4096 6027 0 5276 259 211 48 88 1 1 1
buf2k 2048 506 0 392 15 11 4 8 1 1 0
buf1k 1024 26036 0 19534 383 275 108 176 1 1 0
kmem-4 4 2224 0 156 17 0 17 17 0 inf 0
kmem-8 8 723 0 144 12 2 10 10 0 inf 0
kmem-12 12 73 0 36 2 1 1 2 0 inf 0
kmem-16 16 59 0 30 2 1 1 2 0 inf 0
kmem-20 20 114 0 102 4 0 4 4 0 inf 3
kmem-24 24 81 0 42 3 1 2 3 0 inf 0
kmem-28 28 507 0 87 24 0 24 24 0 inf 0
kmem-32 32 517 0 70 29 1 28 28 0 inf 0
kmem-36 36 48 0 24 3 1 2 2 0 inf 0
kmem-40 40 542 0 27 44 1 43 43 0 inf 0
kmem-44 44 42 0 33 2 0 2 2 0 inf 0
kmem-48 48 27 0 21 2 1 1 2 0 inf 0
kmem-56 56 69 0 60 6 2 4 4 0 inf 2
kmem-64 64 1661 0 114 198 4 194 194 0 inf 0
kmem-72 72 27 0 24 3 2 1 2 0 inf 0
kmem-84 84 30 0 27 3 2 1 2 0 inf 0
kmem-100 100 109 0 87 19 0 19 19 0 inf 14
kmem-128 128 75 0 48 16 5 11 12 0 inf 4
propdict 72 51 0 8 1 0 1 1 0 inf 0
pdict32 64 3 0 0 1 0 1 1 0 inf 0
pdict16 48 5 0 0 1 0 1 1 0 inf 0
propnmbr 36 7 0 2 1 0 1 1 0 inf 0
propstng 20 6 0 4 1 0 1 1 0 inf 0
ataspl 56 1791982 0 1791982 1 0 1 1 0 inf 1
rttmrpl 32 305 0 305 2 1 1 1 0 inf 1
rtentpl 136 797 0 689 4 0 4 4 0 inf 0
vnodepl 168 78442 0 14592 2662 0 2662 2662 0 inf 0
lockfpl 52 21672 0 21667 1 0 1 1 0 inf 0
ncachepl 84 5442518 0 5379343 1331 0 1331 1331 0 inf 0
biopl 124 2128682 0 2128682 14 10 4 11 0 inf 4
bufpl 124 48462 0 32058 859 0 859 859 0 inf 0
sockpl 216 93443 0 93149 88 11 77 77 0 inf 55
ttypl 264 117 0 0 8 0 8 8 0 inf 0
shmmp 12 2 0 0 1 0 1 1 0 inf 0
pipepl 100 1588 0 1370 7 0 7 7 0 inf 1
vmembtpl 28 4666 0 1049 25 0 25 25 0 inf 0
ptimerspl 140 60 0 50 1 0 1 1 0 inf 0
ptimerpl 124 61 0 50 1 0 1 1 0 inf 0
ksiginfo 44 145323 0 145322 1 0 1 1 0 inf 0
siginfo 128 9131 0 9131 4 3 1 1 0 inf 1
savppl 52 1 0 0 1 0 1 1 0 inf 0
sastackpl 24 16 0 0 1 0 1 1 0 inf 0
saupcpl 1612 268741 0 268740 1 0 1 1 0 inf 0
sadatapl 40 1 0 0 1 0 1 1 0 inf 0
raspl 20 272 0 186 1 0 1 1 0 inf 0
sessionpl 40 265 0 196 1 0 1 1 0 inf 0
rusgepl 72 3046 0 3046 4 3 1 1 0 inf 1
pstatspl 224 3192 0 3046 12 3 9 9 0 inf 0
plimitpl 176 52 0 42 1 0 1 1 0 inf 0
pgrppl 16 404 0 333 1 0 1 1 0 inf 0
procpl 420 3192 0 3046 24 7 17 18 0 inf 0
lwppl 148 3196 0 3046 6 0 6 6 0 inf 0
knotepl 72 9278 0 9255 1 0 1 1 0 inf 0
kqueuepl 260 10338 0 10335 1 0 1 1 0 inf 0
fdescpl 284 3175 0 3045 12 2 10 10 0 inf 0
cwdipl 16 3175 0 3045 1 0 1 1 0 inf 0
filepl 60 353484 0 352915 12 2 10 10 0 inf 1
kauthcredpl 100 12099 0 11992 5 2 3 4 0 inf 0
rndsample 528 28001 0 27996 1 0 1 1 0 inf 0
ractx 24 13610 0 2979 64 0 64 64 0 inf 0
vmmpepl 88 411346 0 407195 205 111 94 196 0 inf 3
vmsppl 224 3174 0 3045 9 1 8 8 0 inf 0
aobjpl 56 1 0 0 1 0 1 1 0 inf 0
uaoeltpl 84 16 0 12 1 0 1 1 0 inf 0
anonpl 16 162696 0 35982 503 0 503 503 0 inf 1
amappl 48 32759 0 30419 31 2 29 29 0 inf 0
ufsdirpl 264 15646 0 15646 3 2 1 1 0 inf 1
dino1pl 128 5434452 0 5370621 2061 0 2061 2061 0 inf 0
ffsinopl 200 5434452 0 5370621 3194 0 3194 3194 0 inf 0
newdirblkpl 20 2 0 2 1 1 0 1 0 inf 0
dirrempl 44 14364 0 14364 8 5 3 3 0 inf 3
mkdirpl 32 100 0 100 3 3 0 1 0 inf 0
diraddpl 40 15724 0 15723 5 4 1 2 0 inf 0
freefilepl 40 14218 0 14218 5 3 2 3 0 inf 2
freeblkspl 176 16638 0 16638 20 18 2 11 0 inf 2
freefragpl 44 23881 0 23881 5 4 1 2 0 inf 1
allocindirpl 64 57930 0 57930 402 401 1 240 0 inf 1
indirdeppl 32 1150 0 1150 3 2 1 1 0 inf 1
allocdirectpl 80 72693 0 72692 17 16 1 8 0 inf 0
bmsafemappl 36 10391 0 10389 1 0 1 1 0 inf 0
newblkpl 36 130623 0 130623 2 1 1 1 0 inf 1
inodedeppl 92 31150 0 31148 13 7 6 7 0 inf 5
pagedeppl 72 10056 0 10055 7 4 3 4 0 inf 2
sdpcpool 124 90301 0 90300 119 68 51 63 0 inf 50
nfsvapl 112 169 0 164 1 0 1 1 0 inf 0
nfsnodepl 280 169 0 164 3 1 2 2 0 inf 1
in6pcbpl 156 13180 0 13152 2 0 2 2 0 inf 0
tcpcbpl 392 36702 0 36626 137 21 116 116 0 inf 99
sackholepl 20 2994 0 2994 3 2 1 1 0 inf 1
synpl 184 16080 0 16080 4 3 1 1 0 inf 1
tcpipqepl 44 22372 0 22372 4 3 1 1 0 inf 1
ipqepl 44 11 0 11 2 1 1 1 0 inf 1
inmltpl 32 4 0 1 1 0 1 1 0 inf 0
inpcbpl 116 75461 0 75361 42 8 34 34 0 inf 28
kvakmem 4096 581 0 107 15 0 15 15 0 inf 0
kvakernel 4096 11976 0 715 365 2 363 363 0 inf 2
pdppl 4096 217 0 79 149 11 138 139 0 inf 0
pmappl 76 3174 0 3045 3 0 3 3 0 inf 0
pcgpool 140 3462 0 2570 106 16 90 104 0 inf 8
phpool-128 164 17 0 0 1 0 1 1 0 inf 0
phpool-64 100 14 0 3 1 0 1 1 0 inf 0
phpool-32 68 413 0 4 7 0 7 7 0 inf 0
phpool-16 52 88 0 6 2 0 2 2 0 inf 0
phpool-8 44 292 0 8 4 0 4 4 0 inf 0
phpool-4 40 60 0 5 1 0 1 1 0 inf 0
phpool-0 36 4063 0 1483 30 0 30 30 0 inf 0
In use 126736K, total allocated 171446K; utilization 73.9%
--cu1uLm8q3Y--