Re: Sftp: performance enhancing changes

[pgut001%cs.auckland.ac.nz@localhost (Peter Gutmann)]

Bill Sommerfeld <sommerfeld%east.sun.com@localhost> writes:
>>>o Ignoring channel window makes for a screaming
>>>  sftp transfer. (unacceptable, breaks multiplexing.)
>>
>>It's only unacceptable if you're actually doing multiplexing.  If you're just
>>using SFTP as a secure FTP replacement (which is what many people seem to be
>>using it for), it's a simple, quick fix for the SFTP performance problems.
>
>My understanding based on reading the earlier discussion is that this is a
>red herring -- the fix (which doesn't involve violating the specs) is to
>pipeline read/write requests and ensure the channel window is substantially
>larger than the TCP window).

Two comments on this..

1. It doesn't involve violating the spec.  You just advertise a maximum-size
   window (for the SSH level) and read a file as a single large chunk rather
   than lots of little bits and pieces (at the SFTP level).  Neither violate
   the spec.

2. Pipelining reads/writes and whatnot is a nice theoretical fix, but the fact
   that this problem has been around for what, five years now without anyone
   fixing it, and is enough of a problem that it comes up again and again in
   newsgroups, mailing lists, etc etc, indicates that just because a
   theoretical fix exists doesn't mean that implementations are able or
   willing to make use of it (heck, Tanenbaum's 1981 edition of Computer
   Networks covered this problem, and yet it's still present in stuff released
   this year, which indicates that it's not going to go away any time soon).
   
   Let me provide a concrete example of where this sort of thing leads... the
   motivation for me originally doing the handbrake writeup was a conversation
   with someone at a large US financial institution who were using SFTP to
   transfer some sort of large money-related files around (batch EDI
   transactions I assume).  They were doing this over fixed-bandwidth, very
   expensive leased lines (slow, high-latency, the perfect environment for the
   handbrake to take effect), and found that they were getting only a fraction
   of the expected throughput.  Their options were to either pay for more
   expensive leased lines, or (as they eventually found via trial and error)
   switch to SSL.  In the end they dumped SSH and switched to SSL instead,
   which, even though it does exactly the same thing as SSH, ran at the full
   link speed.  I offered to do them a handbrake-free STFP on top of my
   handbrake-free SSH code, but by then they had committed to SSL.
   
So although it's possible to say "We have a theoretical fix which, if
correctly applied by both sides and under ideal conditions, should work OK",
the fact that anything except carefully-optimised, tuned SSH implementations
provide poor throughput isn't really helping anyone except the SSL vendors,
whose implementations don't have this problem.  Since there are a large number
of SSH implementations out there (surprisingly so, it's turning up as an add-
on to all sorts of Windows apps that do file/data transfer, which is nice to
see), more and more stuff is being SSH-enabled by people who don't know about
the handbrake and don't know or care about spending a lot of time doing the
necessary tuning to get adequate performance.  The situation is just going to
get worse...

Peter.