Re: BSD doesn't track window size changes correctly.

[Dennis Ferguson <dennis.c.ferguson%gmail.com@localhost>]

On 3 Apr, 2012, at 00:16 , Darren Reed wrote:

Ah, that's a trace that is easier to look at.  I see no evidence
that NetBSD has shrunk its window significantly anywhere in that
sequence, however, other than the unavoidable window scaling jitter.
More than this, it seems like FreeBSD is actually tracking the
window shrinkage that jitter causes.

> 09:03:38.020855 IP NetBSD.ssh > FreeBSD.35421: Flags [.], ack 230249, win 
> 295, options [nop,nop,TS val 249 ecr 1076458], length 0

The right edge of the window for this ack is 230249+295*32 = 239689.
That's the furthest right it ever extends.

> 09:03:38.053904 IP NetBSD.ssh > FreeBSD.35421: Flags [.], ack 232917, win 
> 211, options [nop,nop,TS val 249 ecr 1076458], length 0

The right edge of this window is 239669

> 09:03:38.080020 IP NetBSD.ssh > FreeBSD.35421: Flags [.], ack 235568, win 
> 128, options [nop,nop,TS val 249 ecr 1076463], length 0

The right edge of this window is 239664

> 09:03:38.112052 IP NetBSD.ssh > FreeBSD.35421: Flags [.], ack 238424, win 39, 
> options [nop,nop,TS val 249 ecr 1076464], length 0

The right edge of this window is 239672

> 09:03:38.222896 IP NetBSD.ssh > FreeBSD.35421: Flags [.], seq 
> 4438371:4439811, ack 238424, win 39, options [nop,nop,TS val 249 ecr 
> 1076479], length 1440

The right edge of this window is 239672

> 09:03:38.223740 IP FreeBSD.35421 > NetBSD.ssh: Flags [.], seq 239664:239672, 
> ack 4436931, win 8280, options [nop,nop,TS val 1076502 ecr 249,nop,nop,sack 1 
> {4438371:4439811}], length 8

FreeBSD agrees the right edge of the window is 239672 at this point since it
sends 8 more bytes to fill it up.  It doesn't seem to remember that the window
had been advertised as open to 239689 a few packets earlier (it didn't send 25
bytes), however, so the evidence seems to be that FreeBSD is in fact tracking
the window shrinkage jitter.  FreeBSD has also figured out that 4436931:4438371
is lost and wants it.


> 09:03:38.225524 IP NetBSD.ssh > FreeBSD.35421: Flags [.], seq 
> 4439811:4441251, ack 238424, win 39, options [nop,nop,TS val 249 ecr 
> 1076479], length 1440

239672

> 09:03:38.226304 IP FreeBSD.35421 > NetBSD.ssh: Flags [.], ack 4436931, win 
> 8280, options [nop,nop,TS val 1076502 ecr 249,nop,nop,sack 1 
> {4438371:4441251}], length 0
> 09:03:38.416891 IP NetBSD.ssh > FreeBSD.35421: Flags [.], ack 239664, win 0, 
> options [nop,nop,TS val 249 ecr 1076479], length 0

239664

> 09:03:38.437943 IP NetBSD.ssh > FreeBSD.35421: Flags [.], seq 
> 4436931:4438371, ack 239672, win 0, options [nop,nop,TS val 249 ecr 1076502], 
> length 1440

This packet is the interesting one.  It is a retransmission of the missing
data that FreeBSD asked for 200 ms earlier.  It also ack's the 8 bytes that
FreeBSD sent in the same packet, which means that it has accepted and is
ack'ing data beyond the window edge the immediately previous packet advertised
(and the 239689 thing above suggests that FreeBSD would have noted this).  The
right hand edge of the window has moved to 239672, but is still closed.

About 1 ms later FreeBSD blasts out beyond the edge of the window.  This is
clearly incorrect.  It is no longer sack'ing for the data in the packet
above so this must be in response to receiving the packet above.

> 09:03:38.439168 IP FreeBSD.35421 > NetBSD.ssh: Flags [.], seq 239672:241112, 
> ack 4441251, win 7740, options [nop,nop,TS val 1076524 ecr 249], length 1440
> 09:03:38.439290 IP FreeBSD.35421 > NetBSD.ssh: Flags [.], seq 241112:242552, 
> ack 4441251, win 8139, options [nop,nop,TS val 1076524 ecr 249], length 1440
> 09:03:38.439412 IP FreeBSD.35421 > NetBSD.ssh: Flags [.], seq 242552:243992, 
> ack 4441251, win 8139, options [nop,nop,TS val 1076524 ecr 249], length 1440
> 09:03:38.439535 IP FreeBSD.35421 > NetBSD.ssh: Flags [.], seq 243992:245432, 
> ack 4441251, win 8139, options [nop,nop,TS val 1076524 ecr 249], length 1440

Somehow I think it is unlikely this problem has much to do with windows
shrinking.  There is evidence that FreeBSD is actually tracking window
shrinkage (it could have sent 25 bytes instead of 8, but didn't) so it
appears that FreeBSD is fine with that.  There is no evidence that
NetBSD ever advertised a window as big as FreeBSD thinks it now is.  The
unique things about the NetBSD packet which prompted the insanity are
that (1) it was a retransmission of missing data, and (2) NetBSD
acknowledges data from FreeBSD in the packet which the previous
packet NetBSD sent indicated was outside the window.

I think you really want to look at what the FreeBSD code does when
it starts off knowing that its neighbor's window is closed and then
gets a packet that does those two things.

Dennis Ferguson