For quite some time (I observed this in the summer too, but figured it
would go away), -current TCP has been performing very badly for short
file transfers. An example will show what I'm talking about.

Here is the complete transcript of a session with a http server. The
server (which I wrote) makes a single 5000-byte write to the
socket. (client on 2648, server on 8002). You'll note that the second
packet sent from port 8002 is not MTU-sized - this causes the receiver
to not send an ack immediately.


17:35:44.587118 www.2648 > www.8002: S 1662450133:1662450133(0) win 8192 <mss 1496,nop,wscale 0,nop,nop,timestamp 2420641 0>
17:35:44.587312 www.8002 > www.2648: S 1662524212:1662524212(0) ack 1662450134 win 8976 <mss 1496,nop,wscale 0,nop,nop,timestamp 2420641 2420641>
17:35:44.587403 www.2648 > www.8002: . ack 1 win 8976 <nop,nop,timestamp 2420641 2420641>
17:35:44.587972 www.2648 > www.8002: P 1:34(33) ack 1 win 8976 <nop,nop,timestamp 2420641 2420641>
17:35:44.588195 www.8002 > www.2648: . ack 34 win 16384 <nop,nop,timestamp 2420641 2420641>
17:35:44.588638 www.8002 > www.2648: . 1:1485(1484) ack 34 win 16384 <nop,nop,timestamp 2420641 2420641>
17:35:44.588694 www.8002 > www.2648: P 1485:2049(564) ack 34 win 16384 <nop,nop,timestamp 2420641 2420641>
17:35:44.693400 www.2648 > www.8002: . ack 2049 win 8976 <nop,nop,timestamp 2420642 2420641>
17:35:44.693562 www.8002 > www.2648: . 2049:3533(1484) ack 34 win 16384 <nop,nop,timestamp 2420642 2420641>
17:35:44.693633 www.8002 > www.2648: FP 3533:5001(1468) ack 34 win 16384 <nop,nop,timestamp 2420642 2420641>
17:35:44.693757 www.2648 > www.8002: . ack 5002 win 6024 <nop,nop,timestamp 2420642 2420642>
17:35:44.694270 www.2648 > www.8002: F 34:34(0) ack 5002 win 8976 <nop,nop,timestamp 2420642 2420642>
17:35:44.694372 www.8002 > www.2648: . ack 35 win 16384 <nop,nop,timestamp 2420642 2420642>

This has a pretty serious performance impact. It means that a short
TCP session transferring 2000+ bytes on local network or the loopback
interface takes a couple hundred milliseconds, rather than a couple
milliseconds. It means that NetBSD is horrible for webservers.

I figure there are two possible causes:
 1. TCP's cwnd gets set to 2048, rather than a multiple of the MTU
 2. TCP gets confused about how much data is available in the socket
    buffer. This might happen if it only looked at how much was in the
    first mbuf in the chain. This doesn't happen at mbuf boundaries
    after the first one, though.

This also happens in BSDI 2.0.1, so I assume it comes from 4.4Lite. I
can't make this problem happen on OSF1, SunOS 4 or 5, or HP-UX.

--
Trevor Blackwell         tlb@eecs.harvard.edu          (617) 495-8912