tech-net archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: NetBSD 5.1 TCP performance issue (lots of ACK)
On Tue, Oct 25, 2011 at 12:58:29PM -0400, Greg Troxel wrote:
>
> Looking at the trace you provided, I am mostly seeing correct
> every-other ack behavior. I continue to wonder if the bad pcap trace is
> masking something else. Try setting net.bpf.maxbufsize larger, but I am
> still not used to seeing 0-len captures even if packets are dropped.
>
> In counting packets, I concur that something seems wrong. But I am
> unable to find much fine-grained oddness.
>
> Big buffers should not be an issue.
But it looks like there are, as I get twice the speed between NetBSD
and linux than between 2 NetBSD, on stricly identical hardware.
I reran some tests. I collected pcap traces on both client and
server, with
xen1:/domains#tcpdump -n -p -i wm0 -w netbsd-client.pcap host xen2-priv
tcpdump: listening on wm0, link-type EN10MB (Ethernet), capture size 96 bytes
^C
698848 packets captured
701273 packets received by filter
2342 packets dropped by kernel
and
xen2:/domains#tcpdump -n -p -i wm0 -w netbsd-server.pcap host xen1-priv
tcpdump: listening on wm0, link-type EN10MB (Ethernet), capture size 96 bytes
^C
565269 packets captured
565345 packets received by filter
0 packets dropped by kernel
(net.bpf.maxbufsize was set to 4194304).
I ran this after a fresh reboot of both client and server, and
netstat -s shows:
on client:
tcp:
241942 packets sent
5227 data packets (857181 bytes)
0 data packets (0 bytes) retransmitted
228294 ack-only packets (229090 delayed)
0 URG only packets
0 window probe packets
8415 window update packets
6 control packets
0 send attempts resulted in self-quench
459790 packets received
2818 acks (for 857137 bytes)
0 duplicate acks
0 acks for unsent data
456840 packets (655320993 bytes) received in-sequence
6 completely duplicate packets (0 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
259 out-of-order packets (370132 bytes)
0 packets (0 bytes) of data after window
0 window probes
3 window update packets
0 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
3 connection requests
1 connection accept
4 connections established (including accepts)
15 connections closed (including 0 drops)
0 embryonic connections dropped
0 delayed frees of tcpcb
2821 segments updated rtt (of 1189 attempts)
0 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts (resulting in 0 dropped connections)
0 keepalive timeouts
0 keepalive probes sent
0 connections dropped by keepalive
98 correct ACK header predictions
455263 correct data packet header predictions
166 PCB hash misses
82 dropped due to no socket
0 connections drained due to memory shortage
0 PMTUD blackholes detected
0 bad connection attempts
1 SYN cache entries added
0 hash collisions
1 completed
0 aborted (no space to build PCB)
0 timed out
0 dropped due to overflow
0 dropped due to bucket overflow
0 dropped due to RST
0 dropped due to ICMP unreachable
1 delayed free of SYN cache entries
0 SYN,ACKs retransmitted
0 duplicate SYNs received for entries already in the cache
0 SYNs dropped (no route or no space)
0 packets with bad signature
0 packets with good signature
0 sucessful ECN handshakes
0 packets with ECN CE bit
0 packets ECN ECT(0) bit
and on server:
tcp:
323882 packets sent
321397 data packets (656445809 bytes)
5 data packets (12476 bytes) retransmitted
2364 ack-only packets (2753 delayed)
0 URG only packets
0 window probe packets
6 window update packets
110 control packets
0 send attempts resulted in self-quench
242229 packets received
229309 acks (for 656075639 bytes)
0 duplicate acks
0 acks for unsent data
5135 packets (853013 bytes) received in-sequence
489 completely duplicate packets (0 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
0 out-of-order packets (0 bytes)
0 packets (0 bytes) of data after window
0 window probes
7299 window update packets
0 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
107 connection requests
7 connection accepts
8 connections established (including accepts)
65660 connections closed (including 0 drops)
106 embryonic connections dropped
0 delayed frees of tcpcb
229310 segments updated rtt (of 22778 attempts)
1 retransmit timeout
0 connections dropped by rexmit timeout
0 persist timeouts (resulting in 0 dropped connections)
133 keepalive timeouts
133 keepalive probes sent
0 connections dropped by keepalive
6 correct ACK header predictions
4991 correct data packet header predictions
32 PCB hash misses
9 dropped due to no socket
0 connections drained due to memory shortage
0 PMTUD blackholes detected
0 bad connection attempts
7 SYN cache entries added
0 hash collisions
7 completed
0 aborted (no space to build PCB)
0 timed out
0 dropped due to overflow
0 dropped due to bucket overflow
0 dropped due to RST
0 dropped due to ICMP unreachable
7 delayed free of SYN cache entries
0 SYN,ACKs retransmitted
0 duplicate SYNs received for entries already in the cache
0 SYNs dropped (no route or no space)
0 packets with bad signature
0 packets with good signature
0 sucessful ECN handshakes
0 packets with ECN CE bit
0 packets ECN ECT(0) bit
I still have the bad-len packets on the server side, but not on the
client side. I wonder if this could be because of tso4 on the interface.
traces are available in ftp://ftp-asim.lip6.fr/outgoing/bouyer/
I also transfered the same file using ttcp instead of through
glusterfs:
xen1:/home/bouyer>ttcp -s -r -l 65536
ttcp-r: buflen=65536, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 192.168.1.2
ttcp-r: 655360000 bytes in 6.61 real seconds = 96871.52 KB/sec +++
ttcp-r: 14856 I/O calls, msec/call = 0.46, calls/sec = 2248.63
ttcp-r: 0.0user 1.4sys 0:06real 21% 0i+0d 0maxrss 0+16pf 7627+2csw
xen2:/home/bouyer>ttcp -t -l 65536 xen1-priv < /glpool/truc
ttcp-t: buflen=65536, nbuf=2048, align=16384/0, port=5001 tcp -> xen1-priv
ttcp-t: socket
ttcp-t: connect
ttcp-t: 655360000 bytes in 6.60 real seconds = 96899.55 KB/sec +++
ttcp-t: 10000 I/O calls, msec/call = 0.68, calls/sec = 1514.06
ttcp-t: -1.9user 5.3sys 0:06real 80% 0i+0d 0maxrss 0+16pf 1394+171csw
I also got pcap traces:
xen1:/domains#tcpdump -n -p -i wm0 -w netbsd-ttcpclient.pcap host xen2-priv
tcpdump: listening on wm0, link-type EN10MB (Ethernet), capture size 96 bytes
^C
690857 packets captured
694270 packets received by filter
3249 packets dropped by kernel
xen2:/domains#tcpdump -n -p -i wm0 -w netbsd-ttcpserver.pcap host xen1-priv
tcpdump: listening on wm0, link-type EN10MB (Ethernet), capture size 96 bytes
^C
546336 packets captured
546595 packets received by filter
0 packets dropped by kernel
There is again the IP bad-len 0 in the server-side trace but not in the
client side trace.
netstat -s:
client:
tcp:
241714 packets sent
249 data packets (19529 bytes)
0 data packets (0 bytes) retransmitted
227005 ack-only packets (225830 delayed)
0 URG only packets
0 window probe packets
14459 window update packets
1 control packet
0 send attempts resulted in self-quench
453104 packets received
219 acks (for 19484 bytes)
0 duplicate acks
0 acks for unsent data
451342 packets (653225521 bytes) received in-sequence
0 completely duplicate packets (0 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
368 out-of-order packets (532864 bytes)
0 packets (0 bytes) of data after window
0 window probes
0 window update packets
0 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 connection requests
2 connection accepts
2 connections established (including accepts)
17 connections closed (including 0 drops)
0 embryonic connections dropped
0 delayed frees of tcpcb
219 segments updated rtt (of 202 attempts)
0 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts (resulting in 0 dropped connections)
0 keepalive timeouts
0 keepalive probes sent
0 connections dropped by keepalive
46 correct ACK header predictions
451213 correct data packet header predictions
352 PCB hash misses
174 dropped due to no socket
0 connections drained due to memory shortage
0 PMTUD blackholes detected
0 bad connection attempts
2 SYN cache entries added
0 hash collisions
2 completed
0 aborted (no space to build PCB)
0 timed out
0 dropped due to overflow
0 dropped due to bucket overflow
0 dropped due to RST
0 dropped due to ICMP unreachable
2 delayed free of SYN cache entries
0 SYN,ACKs retransmitted
0 duplicate SYNs received for entries already in the cache
0 SYNs dropped (no route or no space)
0 packets with bad signature
0 packets with good signature
0 sucessful ECN handshakes
0 packets with ECN CE bit
0 packets ECN ECT(0) bit
and server:
tcp:
305529 packets sent
305129 data packets (655917613 bytes)
0 data packets (0 bytes) retransmitted
200 ack-only packets (216 delayed)
0 URG only packets
0 window probe packets
3 window update packets
197 control packets
0 send attempts resulted in self-quench
242418 packets received
226234 acks (for 655384706 bytes)
0 duplicate acks
0 acks for unsent data
225 packets (13905 bytes) received in-sequence
1534 completely duplicate packets (0 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
1 out-of-order packet (0 bytes)
0 packets (0 bytes) of data after window
0 window probes
14338 window update packets
0 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
197 connection requests
4 connection accepts
6 connections established (including accepts)
65747 connections closed (including 0 drops)
195 embryonic connections dropped
0 delayed frees of tcpcb
226236 segments updated rtt (of 39837 attempts)
0 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts (resulting in 0 dropped connections)
244 keepalive timeouts
244 keepalive probes sent
0 connections dropped by keepalive
0 correct ACK header predictions
88 correct data packet header predictions
24 PCB hash misses
8 dropped due to no socket
0 connections drained due to memory shortage
0 PMTUD blackholes detected
0 bad connection attempts
4 SYN cache entries added
0 hash collisions
4 completed
0 aborted (no space to build PCB)
0 timed out
0 dropped due to overflow
0 dropped due to bucket overflow
0 dropped due to RST
0 dropped due to ICMP unreachable
4 delayed free of SYN cache entries
0 SYN,ACKs retransmitted
0 duplicate SYNs received for entries already in the cache
0 SYNs dropped (no route or no space)
0 packets with bad signature
0 packets with good signature
0 sucessful ECN handshakes
0 packets with ECN CE bit
0 packets ECN ECT(0) bit
So:
- it seems ack is not the issue, we have about the same number of ack
with ttcp, and we can run full speed.
- the hardware and network can get more than 90MB/s as ttcp manages
to do it.
- glusterfs can also do it, as a linux client can get data from the
netbsd server at more than 90MB/s, and the NetBSD client can get data
from a linux server at more than 90MB/s. the linux box involved
is strictly identical (hardware-wise) to the NetBSD boxes, and
connected to the same gigabit switch
I don't know where to look next, any idea welcome.
--
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
NetBSD: 26 ans d'experience feront toujours la difference
--
Home |
Main Index |
Thread Index |
Old Index