kern/51753: tcp SACK causes SSH disconnects

[marcotte%panix.com@localhost]

>Number:         51753
>Category:       kern
>Synopsis:       tcp SACK causes SSH disconnects
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Dec 30 09:50:00 +0000 2016
>Originator:     Brian Marcotte
>Release:        7.0
>Organization:
Public Access Networks, Corp.
>Environment:
NetBSD trinity.nyc.access.net 7.0.2 NetBSD 7.0.2 (PANIX-XEN-STD) #1: Mon Nov 21 12:57:01 EST 2016  root%juggler.panix.com@localhost:/misc/obj/misc/devel/netbsd/7.0.2/src/sys/arch/i386/compile/PANIX-XEN-STD i386
>Description:
Ever since we started upgrading to NetBSD-7, we've been getting weird
SSH disconnects:

  client: Corrupted MAC on input. Disconnecting: Packet corrupt
  server: panix5 sshd[23482]: error: Received disconnect from x.x.x.x:
          2: Packet corrupt

It turns out, just replacing the kernel only and keeping the NetBSD-6
userland will cause the problem to show up. SSH client/server versions
don't appear to matter.

I traced this down to a change in the kernel between 2013-Nov-12 and
2013-Nov-13. I suspect the problem is in one of these files commited
on that day:

  sys/netinet/tcp_congctl.c   1.18
  sys/netinet/tcp_congctl.h   1.7
  sys/netinet/tcp_input.c     1.330
  sys/netinet/tcp_sack.c      1.29
  sys/netinet/tcp_subr.c      1.251
  src/sys/netinet/tcp_var.h   1.171

The above commits added "cubic" congestion control but also moved SACK
code around.


>How-To-Repeat:
In our case, certain types of terminal output can cause the problem.

I can now get it to happen somewhat reliably by compiling a NetBSD
kernel.

It may be that there must be some other network problem for this to
happen as I've not seen anyone else report this problem.


>Fix:
I don't know how to fix it but turning off SACK seems to be a
workaround:

    sysctl -w net.inet.tcp.sack.enable=0