Re: Pathological TCP behavior running ls(1) over SSH

To: darrenr%netbsd.org@localhost
Subject: Re: Pathological TCP behavior running ls(1) over SSH
From: gson%gson.org@localhost (Andreas Gustafsson)
Date: Mon, 28 Jan 2008 19:35:13 +0200
Darren Reed wrote:
> > This looks similar to what Tomas Svensson reported in 2002 [...]
>
> This is a different problem/bug to what you are seeing.

I think the cause of all those tinygrams being sent in the first place
is probably the same, but the lack of fast retransmit when one of them
is lost is a new and separate issue.

> I'm going to bet that this is somehow tied up with SACK.
> All of the retransmitted (duplicate) ACKs include 1 or more
> ranges of bytes from SACK.
> 
> So it seems worthwhile testing to see if that is a cause here:
> sysctl -w net.inet.tcp.sack.enable=0

I think you're right.  If I disable SACK, the pauses disappear, and
tcpdump shows that fast retransmit is now happening:

  # tcpdump -n -p -i re0 port 65299
  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  listening on re0, link-type EN10MB (Ethernet), capture size 96 bytes
  19:15:15.397996 IP 10.0.1.254.65299 > 91.152.94.125.22: P 
928726461:928726493(32) ack 108106078 win 33580
  19:15:15.398954 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1:33(32) ack 32 win 
33580
  19:15:15.406027 IP 91.152.94.125.22 > 10.0.1.254.65299: P 33:65(32) ack 32 
win 33580
  19:15:15.406172 IP 91.152.94.125.22 > 10.0.1.254.65299: P 65:97(32) ack 32 
win 33580
  19:15:15.406284 IP 91.152.94.125.22 > 10.0.1.254.65299: P 97:129(32) ack 32 
win 33580
  19:15:15.406395 IP 91.152.94.125.22 > 10.0.1.254.65299: P 129:161(32) ack 32 
win 33580
  19:15:15.406501 IP 91.152.94.125.22 > 10.0.1.254.65299: P 161:193(32) ack 32 
win 33580
  19:15:15.406607 IP 91.152.94.125.22 > 10.0.1.254.65299: P 193:225(32) ack 32 
win 33580
  19:15:15.406710 IP 91.152.94.125.22 > 10.0.1.254.65299: P 225:257(32) ack 32 
win 33580
  19:15:15.406819 IP 91.152.94.125.22 > 10.0.1.254.65299: P 257:289(32) ack 32 
win 33580
  19:15:15.406923 IP 91.152.94.125.22 > 10.0.1.254.65299: P 289:321(32) ack 32 
win 33580
  19:15:15.407024 IP 91.152.94.125.22 > 10.0.1.254.65299: P 321:353(32) ack 32 
win 33580
  19:15:15.407130 IP 91.152.94.125.22 > 10.0.1.254.65299: P 353:385(32) ack 32 
win 33580
  19:15:15.407238 IP 91.152.94.125.22 > 10.0.1.254.65299: P 385:417(32) ack 32 
win 33580
  19:15:15.407344 IP 91.152.94.125.22 > 10.0.1.254.65299: P 417:449(32) ack 32 
win 33580
  19:15:15.407450 IP 91.152.94.125.22 > 10.0.1.254.65299: P 449:481(32) ack 32 
win 33580
  19:15:15.407557 IP 91.152.94.125.22 > 10.0.1.254.65299: P 481:513(32) ack 32 
win 33580
  19:15:15.407664 IP 91.152.94.125.22 > 10.0.1.254.65299: P 513:545(32) ack 32 
win 33580
  19:15:15.407770 IP 91.152.94.125.22 > 10.0.1.254.65299: P 545:577(32) ack 32 
win 33580
  19:15:15.407877 IP 91.152.94.125.22 > 10.0.1.254.65299: P 577:609(32) ack 32 
win 33580
  19:15:15.407981 IP 91.152.94.125.22 > 10.0.1.254.65299: P 609:641(32) ack 32 
win 33580
  19:15:15.408086 IP 91.152.94.125.22 > 10.0.1.254.65299: P 641:673(32) ack 32 
win 33580
  19:15:15.408196 IP 91.152.94.125.22 > 10.0.1.254.65299: P 673:705(32) ack 32 
win 33580
  19:15:15.408309 IP 91.152.94.125.22 > 10.0.1.254.65299: P 705:737(32) ack 32 
win 33580
  19:15:15.408417 IP 91.152.94.125.22 > 10.0.1.254.65299: P 737:769(32) ack 32 
win 33580
  19:15:15.408529 IP 91.152.94.125.22 > 10.0.1.254.65299: P 769:801(32) ack 32 
win 33580
  19:15:15.408636 IP 91.152.94.125.22 > 10.0.1.254.65299: P 801:833(32) ack 32 
win 33580
  19:15:15.408748 IP 91.152.94.125.22 > 10.0.1.254.65299: P 833:865(32) ack 32 
win 33580
  19:15:15.408851 IP 91.152.94.125.22 > 10.0.1.254.65299: P 865:897(32) ack 32 
win 33580
  19:15:15.408955 IP 91.152.94.125.22 > 10.0.1.254.65299: P 897:929(32) ack 32 
win 33580
  19:15:15.409076 IP 91.152.94.125.22 > 10.0.1.254.65299: P 929:961(32) ack 32 
win 33580
  19:15:15.409179 IP 91.152.94.125.22 > 10.0.1.254.65299: P 961:993(32) ack 32 
win 33580
  19:15:15.409283 IP 91.152.94.125.22 > 10.0.1.254.65299: P 993:1025(32) ack 32 
win 33580
  19:15:15.409386 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1025:1057(32) ack 
32 win 33580
  19:15:15.409496 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1057:1089(32) ack 
32 win 33580
  19:15:15.409599 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1089:1121(32) ack 
32 win 33580
  19:15:15.409704 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1121:1153(32) ack 
32 win 33580
  19:15:15.409810 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1153:1185(32) ack 
32 win 33580
  19:15:15.409912 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1185:1217(32) ack 
32 win 33580
  19:15:15.410016 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1217:1249(32) ack 
32 win 33580
  19:15:15.410129 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1249:1281(32) ack 
32 win 33580
  19:15:15.410235 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1281:1313(32) ack 
32 win 33580
  19:15:15.410340 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1313:1345(32) ack 
32 win 33580
  19:15:15.410442 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1345:1377(32) ack 
32 win 33580
  19:15:15.410547 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1377:1409(32) ack 
32 win 33580
  19:15:15.410652 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1409:1441(32) ack 
32 win 33580
  19:15:15.410760 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1441:1473(32) ack 
32 win 33580
  19:15:15.410868 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1473:1505(32) ack 
32 win 33580
  19:15:15.410915 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 65 win 33548
  19:15:15.411045 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1505:1537(32) ack 
32 win 33580
  19:15:15.411151 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1537:1569(32) ack 
32 win 33580
  19:15:15.411258 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1569:1601(32) ack 
32 win 33580
  19:15:15.411363 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1601:1633(32) ack 
32 win 33580
  19:15:15.411472 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1633:1665(32) ack 
32 win 33580
  19:15:15.411577 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1665:1697(32) ack 
32 win 33580
  19:15:15.411680 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1697:1729(32) ack 
32 win 33580
  19:15:15.411787 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1729:1761(32) ack 
32 win 33580
  19:15:15.411890 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1761:1793(32) ack 
32 win 33580
  19:15:15.411998 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1793:1825(32) ack 
32 win 33580
  19:15:15.412103 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1825:1857(32) ack 
32 win 33580
  19:15:15.412210 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1857:1889(32) ack 
32 win 33580
  19:15:15.412317 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1889:1921(32) ack 
32 win 33580
  19:15:15.412422 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1921:1953(32) ack 
32 win 33580
  19:15:15.412528 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1953:1985(32) ack 
32 win 33580
  19:15:15.412632 IP 91.152.94.125.22 > 10.0.1.254.65299: P 1985:2017(32) ack 
32 win 33580
  19:15:15.412736 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2017:2049(32) ack 
32 win 33580
  19:15:15.412841 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2049:2081(32) ack 
32 win 33580
  19:15:15.412945 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2081:2113(32) ack 
32 win 33580
  19:15:15.413049 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2113:2145(32) ack 
32 win 33580
  19:15:15.413154 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2145:2177(32) ack 
32 win 33580
  19:15:15.413261 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2177:2209(32) ack 
32 win 33580
  19:15:15.413363 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2209:2241(32) ack 
32 win 33580
  19:15:15.413467 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2241:2273(32) ack 
32 win 33580
  19:15:15.413574 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2273:2305(32) ack 
32 win 33580
  19:15:15.413676 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2305:2337(32) ack 
32 win 33580
  19:15:15.413782 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2337:2369(32) ack 
32 win 33580
  19:15:15.413889 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2369:2401(32) ack 
32 win 33580
  19:15:15.413999 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2401:2433(32) ack 
32 win 33580
  19:15:15.414106 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2433:2465(32) ack 
32 win 33580
  19:15:15.414214 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2465:2497(32) ack 
32 win 33580
  19:15:15.414321 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2497:2529(32) ack 
32 win 33580
  19:15:15.414424 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2529:2561(32) ack 
32 win 33580
  19:15:15.414728 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 129 win 33548
  19:15:15.414758 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2561:2657(96) ack 
32 win 33580
  19:15:15.414808 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2657:2689(32) ack 
32 win 33580
  19:15:15.418871 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 193 win 33548
  19:15:15.423151 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 257 win 33548
  19:15:15.423196 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2689:2913(224) ack 
32 win 33580
  19:15:15.426749 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 321 win 33548
  19:15:15.430929 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 385 win 33548
  19:15:15.435875 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 449 win 33548
  19:15:15.438661 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 513 win 33548
  19:15:15.442611 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33548
  19:15:15.444684 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.447092 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.449249 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.450701 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.450737 IP 91.152.94.125.22 > 10.0.1.254.65299: . 577:2037(1460) ack 
32 win 33580
  19:15:15.453314 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.454569 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.456774 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.458572 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.461077 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.462520 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.465030 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.466516 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.468888 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.470495 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.472925 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.474519 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.476489 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.478482 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.481476 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.482523 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.484860 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.486519 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.488804 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.490554 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.492553 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.494643 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.496628 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.498713 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.500539 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.502519 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.504926 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.506460 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.508479 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.510917 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.513033 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.515209 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.516429 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.518457 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.520450 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.522449 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.524437 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.526450 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.528440 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.530446 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.532464 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.534480 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 577 win 33580
  19:15:15.537238 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 2049 win 32108
  19:15:15.537266 IP 91.152.94.125.22 > 10.0.1.254.65299: P 2049:2913(864) ack 
32 win 33580
  19:15:15.542212 IP 10.0.1.254.65299 > 91.152.94.125.22: . ack 2913 win 32716
  ^C
  143 packets captured
  145 packets received by filter
  0 packets dropped by kernel
  #

-- 
Andreas Gustafsson, gson%gson.org@localhost
Follow-Ups:
- Re: Pathological TCP behavior running ls(1) over SSH
  - From: Zafer Aydogan
References:
- Pathological TCP behavior running ls(1) over SSH
  - From: Andreas Gustafsson
- Re: Pathological TCP behavior running ls(1) over SSH
  - From: Darren Reed
Prev by Date: Re: Pathological TCP behavior running ls(1) over SSH
Next by Date: Re: Pathological TCP behavior running ls(1) over SSH
Previous by Thread: Re: Pathological TCP behavior running ls(1) over SSH
Next by Thread: Re: Pathological TCP behavior running ls(1) over SSH
Indexes:
Home | Main Index | Thread Index | Old Index