I installed Windows 95 on my PC and immediately my TELNET character
echo latency went to pot (as compared to Windows 3.1 with an ancient
version of Trumpet winsock).  First I suspected Windoze, so I went
around turning things off:  header compression, modem compression,
error correction, FIFOs, and I even switched from PPP to SLIP and
then to CSLIP.  No dice.

Then I remembered the bizarre NetBSD<->NetBSD SLIP echo latency I
had a while ago (the 1.1/1.2 default of net.inet.tcp.rfc1323=1
means that NetBSD puts "high performance" options in the TCP header,
which rules out header compression; beware!), so I pointed my
Windows 95 telnet at an OSF/1 machine.  Imagine my surprise when
I got snappy character echo.  Solaris was fine too, but Linux was
slow.

After staring at a tcpdump of two short sessions, one NetBSD 1.2
and one Solaris, I have the following understanding:

NetBSD:
    Windows sends char 1
    200 microseconds later, NetBSD sends an ack
    900 microseconds after that, NetBSD echoes the character
    Windows sends char 2 (and the ack for char 1's echo)
    150 microseconds later, NetBSD sends an ack
    950 microseconds after that, NetBSD echoes the character

    Then NetBSD seems to "get smart":
    Windows sends char 3
    200 microseconds later, NetBSD sends an ack
    Windows sends char 4
    200 microseconds later, NetBSD echoes char 3 and acks char 4

    From then on, everything is "fine":  Windows gets very fast
    acks, and NetBSD isn't wasting packets....except that the
    echo time is around 100 >>milliseconds<<, which is very
    annoying (especially since significant latency is then added
    by the SLIP line).  And it gets really bad whenever I *stop*
    typing:  the last character takes a *long* time to echo back.
    I guess I'm seeing the "slow timeout" first-hand!

    This is particularly ironic since every NetBSD TELNET packet
    has TOS 2 ("low latency, please") in the IP header!

Solaris:
    Windows sends char 1
    1 millisecond later, Solaris echoes and acks the character
    Windows sends chars 2,3
    1.5 milliseconds later, Solaris echoes and acks both
    Windows sends char 4
    1.3 milliseconds later, Solaris echoes and acks the character

    ...and so on.  The bad news is that Windows needs to wait
    much longer for its acks, but the good news is that the
    human sees the echoes MUCH faster.

All of these packets (Windows 95, NetBSD, Solaris) have the "push"
bit set (except for the non-data-carrying acks); section 4.2.2.14
of RFC 1122 seems to suggest that this is right.

NetBSD's handling of "push" in tcp_input() and tcp_output() looks
a little weird to me.  In particular, it looks like:

1. There is no way for an application to assert "push".
2. "Push" is sent exactly when the most recent user byte is transmitted.
3. On receive, a waiting application is awakened for each packet,
   even if "push" is not set.
4. On receive, "push" results in "send an ACK *immediately*" (which
   I see nowhere in RFC 793 or 1122).

Not one of these seems to violate any rule, but only #2 seems to
have much to do with the spirit of the header bit.  And, as far as
I can tell, my problem is being caused by #4 (since the char 3 ack
rushed out, the char 3 echo had nothing to piggyback on) aggravated
by #1 (since telnetd couldn't push the char 3 echo, it wanted to
piggyback on something).

In fact, I hacked up a telnetd that turns on TCP_NODELAY, and, sure
enough, character echo is nice and snappy (though tcpdump reveals
that we're in the character,ack,echo regime, not the character,ack+echo
one which would be ideal).

Am I missing something, or is it the case that NetBSD telnetd is
setting IP_TOS to ask the rest of the net to expedite TELNET packets,
then not making sure they leave the local machine in a hurry, while
underneath it TCP is being blindingly fast with acks nobody is
itching to receive?

In case it would be helpful, the tcpdump I described above is
available at "http://www.cs.cmu.edu/~davide/log.txt".

Dave Eckhardt
davide+@cs.cmu.edu