tech-net archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
NetBSD 5.1 TCP performance issue (lots of ACK)
Hello,
I've been playing with glusterfs a bit today, and found some performance
differences between NetBSD and linux, which I tracked down to our TCP
stack. Basically, between a NetBSD/linux pair, performances are much
better than between 2 NetBSD hosts. It doesn't matter if linux is client
or server, so this point some issue outside of glusterfs.
So I have done some packet capture and found a strange TCP behavior between
2 NetBSD hosts.
The setup: 192.168.1.2 is a NetBSD 5.1 glusterfs server, 192.168.1.1 is a
NetBSD 5.1 glusterfs client and 192.168.1.3 is a linux (RHEL6) glusterfs
client. The clients read a 640Mb file from the server (with
dd if=file of=/dev/null bs=64k). All 3 hosts are strictly identical
hardware (same CPUs, ram, motherboard, gigabit network adapter and hard disk).
The linux client can read at 95MB/s from the NetBSD server, the NetBSD
client only 50MB/s. (But in other tests, a NetBSD client can read at 90MB/s
out of a linux server, no neither the NetBSD server nor the NetBSD client
is the bottleneck in a NetBSD/NetBSD setup).
Attached are the tcptrace outputs for both client. The problem is, the
NetBSD client is sending 242873 packets to the server for the file read,
while the linux client sends only 34581 packets), so I suspect there's
something wrong with our TCP ack code (it looks like we ack way too often).
The relevant part of sysctl.conf of the client is:
net.inet.tcp.sendbuf_auto=0
net.inet.tcp.recvbuf_auto=0
kern.sbmax=4194304
net.inet.tcp.sendbuf_max=1048576
net.inet.tcp.recvbuf_max=1048576
net.inet.tcp.sendspace=524288
net.inet.tcp.recvspace=524288
net.inet.tcp.abc.enable=0
net.inet.ip.ifq.maxlen=512
Any idea of a sysctl setup (or something else) that could help ?
I actually suspect it's also the reason why I see a lower data rate
over my ADSL connection when downloading from ftp.fr.netbsd.org than
when talking do some linux server ...
--
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
NetBSD: 26 ans d'experience feront toujours la difference
--
================================
TCP connection 3:
host e: 192.168.1.3:1020
host f: 192.168.1.2:24009
complete conn: yes
first packet: Mon Oct 17 16:53:29.631487 2011
last packet: Mon Oct 17 16:53:45.620919 2011
elapsed time: 0:00:15.989432
total packets: 121632
filename: re.pcap
e->f: f->e:
total packets: 34581 total packets: 87051
ack pkts sent: 34580 ack pkts sent: 87051
pure acks sent: 29555 pure acks sent: 1835
sack pkts sent: 0 sack pkts sent: 0
dsack pkts sent: 0 dsack pkts sent: 0
max sack blks/ack: 0 max sack blks/ack: 0
unique bytes sent: 845208 unique bytes sent: 121406760
actual data pkts: 5024 actual data pkts: 85214
actual data bytes: 845208 actual data bytes: 121406760
rexmt data pkts: 0 rexmt data pkts: 0
rexmt data bytes: 0 rexmt data bytes: 0
zwnd probe pkts: 0 zwnd probe pkts: 0
zwnd probe bytes: 0 zwnd probe bytes: 0
outoforder pkts: 0 outoforder pkts: 0
pushed data pkts: 5024 pushed data pkts: 3411
SYN/FIN pkts sent: 1/1 SYN/FIN pkts sent: 1/1
req 1323 ws/ts: Y/Y req 1323 ws/ts: Y/Y
adv wind scale: 7 adv wind scale: 7
req sack: Y req sack: Y
sacks sent: 0 sacks sent: 0
urgent data pkts: 0 pkts urgent data pkts: 0 pkts
urgent data bytes: 0 bytes urgent data bytes: 0 bytes
mss requested: 1460 bytes mss requested: 1460 bytes
max segm size: 424 bytes max segm size: 1448 bytes
min segm size: 132 bytes min segm size: 4 bytes
avg segm size: 168 bytes avg segm size: 1424 bytes
max win adv: 195200 bytes max win adv: 525568 bytes
min win adv: 5888 bytes min win adv: 524928 bytes
zero win adv: 0 times zero win adv: 0 times
avg win adv: 188284 bytes avg win adv: 525482 bytes
max owin: 841 bytes max owin: 195197 bytes
min non-zero owin: 1 bytes min non-zero owin: 1 bytes
avg owin: 53 bytes avg owin: 101337 bytes
wavg owin: 37 bytes wavg owin: 54793 bytes
initial window: 140 bytes initial window: 164 bytes
initial window: 1 pkts initial window: 1 pkts
ttl stream length: 845208 bytes ttl stream length: 656063384 bytes
missed data: 0 bytes missed data: 534656624 bytes
truncated data: 694488 bytes truncated data: 118850660 bytes
truncated packets: 5024 pkts truncated packets: 85180 pkts
data xmit time: 12.909 secs data xmit time: 12.909 secs
idletime max: 5634.8 ms idletime max: 5675.2 ms
throughput: 52860 Bps throughput: 7592938 Bps
RTT samples: 2952 RTT samples: 6551
RTT min: 0.0 ms RTT min: 0.1 ms
RTT max: 0.9 ms RTT max: 40.1 ms
RTT avg: 0.1 ms RTT avg: 1.1 ms
RTT stdev: 0.2 ms RTT stdev: 0.8 ms
RTT from 3WHS: 0.0 ms RTT from 3WHS: 0.8 ms
RTT full_sz smpls: 1 RTT full_sz smpls: 5239
RTT full_sz min: 0.2 ms RTT full_sz min: 0.1 ms
RTT full_sz max: 0.2 ms RTT full_sz max: 5.9 ms
RTT full_sz avg: 0.2 ms RTT full_sz avg: 1.0 ms
RTT full_sz stdev: 0.0 ms RTT full_sz stdev: 0.4 ms
post-loss acks: 0 post-loss acks: 0
segs cum acked: 2074 segs cum acked: 78665
duplicate acks: 1 duplicate acks: 1
triple dupacks: 0 triple dupacks: 0
max # retrans: 0 max # retrans: 0
min retr time: 0.0 ms min retr time: 0.0 ms
max retr time: 0.0 ms max retr time: 0.0 ms
avg retr time: 0.0 ms avg retr time: 0.0 ms
sdv retr time: 0.0 ms sdv retr time: 0.0 ms
TCP connection 5:
host i: 192.168.1.1:1021
host j: 192.168.1.2:24009
complete conn: yes
first packet: Mon Oct 17 17:42:46.460094 2011
last packet: Mon Oct 17 17:42:59.649114 2011
elapsed time: 0:00:13.189019
total packets: 342816
filename: xen1.3.pcap
i->j: j->i:
total packets: 242873 total packets: 99943
ack pkts sent: 242872 ack pkts sent: 99943
pure acks sent: 237850 pure acks sent: 2398
sack pkts sent: 1316 sack pkts sent: 0
dsack pkts sent: 0 dsack pkts sent: 0
max sack blks/ack: 1 max sack blks/ack: 0
unique bytes sent: 844892 unique bytes sent: 138607868
actual data pkts: 5021 actual data pkts: 97543
actual data bytes: 844892 actual data bytes: 138683996
rexmt data pkts: 0 rexmt data pkts: 54
rexmt data bytes: 0 rexmt data bytes: 76128
zwnd probe pkts: 0 zwnd probe pkts: 0
zwnd probe bytes: 0 zwnd probe bytes: 0
outoforder pkts: 0 outoforder pkts: 96
pushed data pkts: 5021 pushed data pkts: 3683
SYN/FIN pkts sent: 1/1 SYN/FIN pkts sent: 1/1
req 1323 ws/ts: Y/Y req 1323 ws/ts: Y/Y
adv wind scale: 7 adv wind scale: 7
req sack: Y req sack: Y
sacks sent: 1316 sacks sent: 0
urgent data pkts: 0 pkts urgent data pkts: 0 pkts
urgent data bytes: 0 bytes urgent data bytes: 0 bytes
mss requested: 1460 bytes mss requested: 1460 bytes
max segm size: 408 bytes max segm size: 1448 bytes
min segm size: 132 bytes min segm size: 20 bytes
avg segm size: 168 bytes avg segm size: 1421 bytes
max win adv: 525568 bytes max win adv: 525568 bytes
min win adv: 28160 bytes min win adv: 524160 bytes
zero win adv: 0 times zero win adv: 0 times
avg win adv: 466678 bytes avg win adv: 525530 bytes
max owin: 1345 bytes max owin: 521953 bytes
min non-zero owin: 1 bytes min non-zero owin: 1 bytes
avg owin: 15 bytes avg owin: 128037 bytes
wavg owin: 117 bytes wavg owin: 196253 bytes
initial window: 140 bytes initial window: 164 bytes
initial window: 1 pkts initial window: 1 pkts
ttl stream length: 844892 bytes ttl stream length: 656063700 bytes
missed data: 0 bytes missed data: 517455832 bytes
truncated data: 694262 bytes truncated data: 135757788 bytes
truncated packets: 5021 pkts truncated packets: 97532 pkts
data xmit time: 13.060 secs data xmit time: 13.060 secs
idletime max: 253.4 ms idletime max: 256.1 ms
throughput: 64060 Bps throughput: 10509338 Bps
RTT samples: 2572 RTT samples: 49184
RTT min: 0.0 ms RTT min: 0.5 ms
RTT max: 0.8 ms RTT max: 116.6 ms
RTT avg: 0.1 ms RTT avg: 1.8 ms
RTT stdev: 0.1 ms RTT stdev: 0.9 ms
RTT from 3WHS: 0.0 ms RTT from 3WHS: 1.5 ms
RTT full_sz smpls: 1 RTT full_sz smpls: 46752
RTT full_sz min: 0.2 ms RTT full_sz min: 0.5 ms
RTT full_sz max: 0.2 ms RTT full_sz max: 7.0 ms
RTT full_sz avg: 0.2 ms RTT full_sz avg: 1.8 ms
RTT full_sz stdev: 0.0 ms RTT full_sz stdev: 0.7 ms
post-loss acks: 0 post-loss acks: 31
For the following 5 RTT statistics, only ACKs for
multiply-transmitted segments (ambiguous ACKs) were
considered. Times are taken from the last instance
of a segment.
ambiguous acks: 0 ambiguous acks: 44
RTT min (last): 0.0 ms RTT min (last): 0.7 ms
RTT max (last): 0.0 ms RTT max (last): 2.3 ms
RTT avg (last): 0.0 ms RTT avg (last): 1.3 ms
RTT sdv (last): 0.0 ms RTT sdv (last): 0.4 ms
segs cum acked: 2451 segs cum acked: 48234
duplicate acks: 1 duplicate acks: 418
triple dupacks: 0 triple dupacks: 8
max # retrans: 0 max # retrans: 1
min retr time: 0.0 ms min retr time: 4.2 ms
max retr time: 0.0 ms max retr time: 43.4 ms
avg retr time: 0.0 ms avg retr time: 17.7 ms
sdv retr time: 0.0 ms sdv retr time: 10.0 ms
Home |
Main Index |
Thread Index |
Old Index