Subject: Re: NMBCLUSTERS again
To: None <6bone@6bone.informatik.uni-leipzig.de>
From: Darren Reed <darrenr@netbsd.org>
List: tech-net
Date: 11/06/2007 11:23:15
6bone@6bone.informatik.uni-leipzig.de wrote:
> I applied the patch.
> The used mbufs are still increasing.
Ok, I'm starting to wonder what the real cause is here...
Although there isn't complete synchronisation between
running different commands (netstat -m vs ipfstat),
what I would expect is there to be some correlation
in the increase in mbufs vs an increase on some other
counter. But it doesn't happen. It may be that there
is a weird interaction between IPFilter and some other
part of the kernel that we're not seeing.
In some circumstances, I've observed the mbuf counter
going backwards - but obviously not far enough. Does
anyone have a way of being able to examine the mbufs
that it believes are allocated?
If a crashdump is made, what do we look for with gdb?
Darren
diff -u netstat-m-[45]
--- netstat-m-4 2007-11-06 17:18:07.000000000 +0100
+++ netstat-m-5 2007-11-06 17:18:37.000000000 +0100
@@ -1,6 +1,6 @@
-Tue Nov 6 17:18:07 CET 2007
-2378 mbufs in use:
- 2341 mbufs allocated to data
+Tue Nov 6 17:18:37 CET 2007
+2384 mbufs in use:
+ 2347 mbufs allocated to data
5 mbufs allocated to packet headers
32 mbufs allocated to socket names and addresses
0 calls to protocol drain routines
diff -u ipfstat-[45]
--- ipfstat-4 2007-11-06 17:18:07.000000000 +0100
+++ ipfstat-5 2007-11-06 17:18:37.000000000 +0100
@@ -1,52 +1,52 @@
bad packets: in 0 out 0
- IPv6 packets: in 92637671 out 92505651
- input packets: blocked 2562 passed 93909497 nomatch
54702546 counted 0 short 0
-output packets: blocked 6 passed 93447714 nomatch
54326544 counted 0 short 0
+ IPv6 packets: in 92750479 out 92618455
+ input packets: blocked 2563 passed 94023840 nomatch
54768249 counted 0 short 0
+output packets: blocked 6 passed 93561692 nomatch
54391883 counted 0 short 0
input packets logged: blocked 0 passed 0
output packets logged: blocked 0 passed 0
packets logged: input 0 output 0
log failures: input 0 output 0
fragment state(in): kept 0 lost 0 not fragmented 0
fragment state(out): kept 0 lost 0 not fragmented 0
-packet state(in): kept 2351 lost 2562
+packet state(in): kept 2351 lost 2563
packet state(out): kept 0 lost 0
ICMP replies: 0 TCP RSTs sent: 0
Invalid source(in): 0
-Result cache hits(in): 38959650 (out): 38870114
-IN Pullups succeeded: 70410 failed: 0
-OUT Pullups succeeded: 80574 failed: 0
+Result cache hits(in): 39008278 (out): 38918740
+IN Pullups succeeded: 70416 failed: 0
+OUT Pullups succeeded: 80589 failed: 0
IN m_pullup succeeded: 64434 failed: 0
OUT m_pullup succeeded: 65129 failed: 0
IN m_pulldown succeeded: 0 failed: 0
OUT m_pulldown succeeded: 21 failed: 0
Fastroute successes: 0 failures: 0
TCP cksum fails(in): 0 (out): 0
-IPF Ticks: 66126
+IPF Ticks: 66186
Packet log flags set: (0)
none
bad packets: in 0 out 0
- IPv6 packets: in 159559651 out 159313827
- input packets: blocked 4981 passed 161859760 nomatch
95174584 counted 0 short 0
-output packets: blocked 6 passed 160994208 nomatch
94471535 counted 0 short 0
+ IPv6 packets: in 159629247 out 159383274
+ input packets: blocked 4984 passed 161930378 nomatch
95214176 counted 0 short 0
+output packets: blocked 6 passed 161064355 nomatch
94510727 counted 0 short 0
input packets logged: blocked 0 passed 0
output packets logged: blocked 0 passed 0
packets logged: input 0 output 0
log failures: input 0 output 0
fragment state(in): kept 0 lost 0 not fragmented 0
fragment state(out): kept 0 lost 0 not fragmented 0
-packet state(in): kept 4427 lost 4981
+packet state(in): kept 4431 lost 4984
packet state(out): kept 0 lost 0
ICMP replies: 0 TCP RSTs sent: 0
Invalid source(in): 0
-Result cache hits(in): 66372351 (out): 66202597
-IN Pullups succeeded: 117064 failed: 0
-OUT Pullups succeeded: 135520 failed: 0
+Result cache hits(in): 66403357 (out): 66233527
+IN Pullups succeeded: 117070 failed: 0
+OUT Pullups succeeded: 135527 failed: 0
IN m_pullup succeeded: 107148 failed: 0
-OUT m_pullup succeeded: 108329 failed: 0
+OUT m_pullup succeeded: 108330 failed: 0
IN m_pulldown succeeded: 0 failed: 0
OUT m_pulldown succeeded: 31 failed: 0
Fastroute successes: 0 failures: 0
TCP cksum fails(in): 0 (out): 0
-IPF Ticks: 109359
+IPF Ticks: 109419
Packet log flags set: (0)
none
diff -u netstat-s-[45] | more
--- netstat-s-4 2007-11-06 17:18:07.000000000 +0100
+++ netstat-s-5 2007-11-06 17:18:37.000000000 +0100
@@ -1,6 +1,6 @@
-Tue Nov 6 17:18:07 CET 2007
+Tue Nov 6 17:18:37 CET 2007
ip:
- 2329686 total packets received
+ 2330711 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
@@ -15,13 +15,13 @@
0 malformed fragments dropped
0 fragments dropped after timeout
0 packets reassembled ok
- 1968813 packets for this host
+ 1969678 packets for this host
2 packets for unknown/unsupported protocol
0 packets forwarded (0 packets fast forwarded)
- 360871 packets not forwardable
+ 361031 packets not forwardable
0 redirects sent
0 packets no matching gif found
- 1695430 packets sent from this host
+ 1696130 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
@@ -30,17 +30,17 @@
0 datagrams that can't be fragmented
0 datagrams with bad address in header
icmp:
- 27361 calls to icmp_error
+ 27367 calls to icmp_error
0 errors not generated because old message was icmp
Output histogram:
echo reply: 171
- destination unreachable: 27359
+ destination unreachable: 27365
49 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Input histogram:
- destination unreachable: 9452
+ destination unreachable: 9458
source quench: 25
routing redirect: 1
echo: 171
@@ -48,20 +48,20 @@
171 message responses generated
0 path MTU changes
tcp:
- 53012 packets sent
- 23788 data packets (8049540 bytes)
+ 53054 packets sent
+ 23818 data packets (8058705 bytes)
10 data packets (5891 bytes) retransmitted
- 20165 ack-only packets (24419 delayed)
+ 20175 ack-only packets (24438 delayed)
0 URG only packets
0 window probe packets
0 window update packets
- 9049 control packets
+ 9051 control packets
0 send attempts resulted in self-quench
- 91275 packets received
- 39211 acks (for 8066090 bytes)
- 260 duplicate acks
+ 91331 packets received
+ 39243 acks (for 8075259 bytes)
+ 262 duplicate acks
0 acks for unsent data
- 24672 packets (674438 bytes) received in-sequence
+ 24693 packets (675700 bytes) received in-sequence
40 completely duplicate packets (4373 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
@@ -74,12 +74,12 @@
0 discarded for bad header offset fields
0 discarded because packet too short
140 connection requests
- 8760 connection accepts
- 8900 connections established (including accepts)
- 8932 connections closed (including 3 drops)
+ 8762 connection accepts
+ 8902 connections established (including accepts)
+ 8934 connections closed (including 3 drops)
0 embryonic connections dropped
0 delayed frees of tcpcb
- 30122 segments updated rtt (of 29813 attempts)
+ 30152 segments updated rtt (of 29843 attempts)
23 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts (resulting in 0 dropped connections)
@@ -87,15 +87,15 @@
4 keepalive probes sent
0 connections dropped by keepalive
1363 correct ACK header predictions
- 13681 correct data packet header predictions
- 75379 PCB hash misses
- 28915 dropped due to no socket
+ 13689 correct data packet header predictions
+ 75405 PCB hash misses
+ 28926 dropped due to no socket
0 connections drained due to memory shortage
1 PMTUD blackhole detected
4 bad connection attempts
- 8771 SYN cache entries added
+ 8773 SYN cache entries added
0 hash collisions
- 8760 completed
+ 8762 completed
0 aborted (no space to build PCB)
2 timed out
0 dropped due to overflow
@@ -112,18 +112,18 @@
0 packets with ECN CE bit
0 packets ECN ECT(0) bit
udp:
- 1246189 datagrams received
+ 1246742 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
- 27359 dropped due to no socket
- 55138 broadcast/multicast datagrams dropped due to no socket
+ 27365 dropped due to no socket
+ 55174 broadcast/multicast datagrams dropped due to no socket
0 dropped due to full socket buffers
- 1163692 delivered
- 9294528 PCB hash misses
- 1194668 datagrams output
+ 1164203 delivered
+ 9298940 PCB hash misses
+ 1195202 datagrams output
ip6:
- 160166271 total packets received
+ 160235863 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with bad options
@@ -133,11 +133,11 @@
0 fragments dropped after timeout
0 fragments that exceeded limit
0 packets reassembled ok
- 8335652 packets for this host
- 151629719 packets forwarded
+ 8339554 packets for this host
+ 151695308 packets forwarded
1 packet not forwardable
0 redirects sent
- 8288300 packets sent from this host
+ 8292152 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
104 output packets discarded due to no route
@@ -147,25 +147,25 @@
5 packets that violated scope rules
0 multicast packets which we don't join
Input packet histogram:
- hop by hop: 221068
- TCP: 151587432
- UDP: 8100301
+ hop by hop: 221074
+ TCP: 151652909
+ UDP: 8104307
fragment: 72
- ICMP6: 215822
- PIM: 36595
+ ICMP6: 215902
+ PIM: 36615
Mbuf statistics:
- 34753 one mbufs
+ 34772 one mbufs
two or more mbuf:
lo0 = 54351
- gif5 = 1865
- 160075302 one ext mbufs
+ gif5 = 1867
+ 160144873 one ext mbufs
0 two or more ext mbufs
0 packets whose headers are not continuous
0 tunneling packets that can't find gif
0 packets discarded due to too many headers
0 failures of source address selection
- 88402856 forward cache hit
- 71450166 forward cache miss
+ 88440997 forward cache hit
+ 71481549 forward cache miss
icmp6:
238 calls to icmp6_error
11 errors not generated because old message was icmp6 or so
@@ -178,23 +178,23 @@
multicast listener query: 8360
multicast listener report: 46179
router advertisement: 2104
- neighbor solicitation: 34769
- neighbor advertisement: 28471
+ neighbor solicitation: 34792
+ neighbor advertisement: 28486
0 messages with bad code fields
0 messages < minimum length
281 bad checksums
0 messages with bad length
Input packet histogram:
- unreach: 47671
+ unreach: 47700
time exceed: 42926
echo: 6853
multicast listener query: 8360
- multicast listener report: 210133
- multicast listener done: 681
+ multicast listener report: 210137
+ multicast listener done: 683
router solicitation: 284
router advertisement: 381
- neighbor solicitation: 28939
- neighbor advertisement: 30463
+ neighbor solicitation: 28954
+ neighbor advertisement: 30483
#143: 1250
Histogram of error messages to be generated:
104 no route
@@ -220,20 +220,20 @@
0 bad redirect messages
0 path MTU changes
tcp6:
- 53012 packets sent
- 23788 data packets (8049540 bytes)
+ 53054 packets sent
+ 23818 data packets (8058705 bytes)
10 data packets (5891 bytes) retransmitted
- 20165 ack-only packets (24419 delayed)
+ 20175 ack-only packets (24438 delayed)
0 URG only packets
0 window probe packets
0 window update packets
- 9049 control packets
+ 9051 control packets
0 send attempts resulted in self-quench
- 91275 packets received
- 39211 acks (for 8066090 bytes)
- 260 duplicate acks
+ 91331 packets received
+ 39243 acks (for 8075259 bytes)
+ 262 duplicate acks
0 acks for unsent data
- 24672 packets (674438 bytes) received in-sequence
+ 24693 packets (675700 bytes) received in-sequence
40 completely duplicate packets (4373 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
@@ -246,12 +246,12 @@
0 discarded for bad header offset fields
0 discarded because packet too short
140 connection requests
- 8760 connection accepts
- 8900 connections established (including accepts)
- 8932 connections closed (including 3 drops)
+ 8762 connection accepts
+ 8902 connections established (including accepts)
+ 8934 connections closed (including 3 drops)
0 embryonic connections dropped
0 delayed frees of tcpcb
- 30122 segments updated rtt (of 29813 attempts)
+ 30152 segments updated rtt (of 29843 attempts)
23 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts (resulting in 0 dropped connections)
@@ -259,15 +259,15 @@
4 keepalive probes sent
0 connections dropped by keepalive
1363 correct ACK header predictions
- 13681 correct data packet header predictions
- 75379 PCB hash misses
- 28915 dropped due to no socket
+ 13689 correct data packet header predictions
+ 75405 PCB hash misses
+ 28926 dropped due to no socket
0 connections drained due to memory shortage
1 PMTUD blackhole detected
4 bad connection attempts
- 8771 SYN cache entries added
+ 8773 SYN cache entries added
0 hash collisions
- 8760 completed
+ 8762 completed
0 aborted (no space to build PCB)
2 timed out
0 dropped due to overflow
@@ -284,7 +284,7 @@
0 packets with ECN CE bit
0 packets ECN ECT(0) bit
udp6:
- 8021478 datagrams received
+ 8025335 datagrams received
0 with incomplete header
0 with bad data length field
84 with bad checksum
@@ -292,10 +292,10 @@
5 dropped due to no socket
0 multicast datagrams dropped due to no socket
2 dropped due to full socket buffers
- 8021387 delivered
- 8077726 datagrams output
+ 8025244 delivered
+ 8081501 datagrams output
pim6:
- 36595 messages received
+ 36615 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 messages received with bad version
@@ -303,13 +303,13 @@
0 bad registers received
0 registers sent
rip6:
- 36595 messages received
+ 36615 messages received
0 checksum calculations on inbound
0 messages with bad checksum
0 messages dropped due to no socket
0 multicast messages dropped due to no socket
0 messages dropped due to full socket buffers
- 36595 delivered
- 34752 datagrams output
+ 36615 delivered
+ 34771 datagrams output
TP not configured