Re: pkg/47424 (pkgsrc "make fetch" fails to fetch fotoxx-13.01.tar.gz for graphics/fotoxx)

[Robert Elz <kre%munnari.OZ.AU@localhost>]

The following reply was made to PR pkg/47424; it has been noted by GNATS.

From: Robert Elz <kre%munnari.OZ.AU@localhost>
To: gnats-bugs%NetBSD.org@localhost
Cc: 
Subject: Re: pkg/47424 (pkgsrc "make fetch" fails to fetch fotoxx-13.01.tar.gz 
for graphics/fotoxx)
Date: Fri, 11 Jan 2013 19:15:47 +0700

     Date:        Wed,  9 Jan 2013 18:17:28 +0000 (UTC)
     From:        wiz%NetBSD.org@localhost
     Message-ID:  <20130109181728.B809F63ED1A%www.NetBSD.org@localhost>
 
   | Distfile on nbftp is enough for submitter; he'll investigate ftp(1) 
further.
 
 I have done, and I don't think it is a bug in ftp(1), I think it is a bug
 in the HTTP server that's serving the file (though I am certainly no
 expert on the intricacies  of HTTP file transfers, nor do I particularly
 want to become one...)
 
 The difference between using ftp and wget (or so it appears to me) is that
 the http client in ftp(1) does
 
        Connection: close
 
 in the header of the GET request, and wget does
 
        Connection: Keep-Alive
 
 instead.   The server (at www.kornelix.com) (which appears to be Apache, though
 I have no idea which version) responds to the wget request with a header that
 includes
 
        Content-Length: 213182
        Connection: Keep-Alive
 
 whereas with the ftp request, the first of those is present in the reply
 header, but there is no "Connection:" field at all.
 
 Either way, in both cases, the server seems to send the data in the file,
 and then stop (implementing keep-alive type connections).  NetBSD's ftp
 client is assuming it will get a FIN to conclude the transfer, so after
 all the data has arrived, it just sits and waits for that FIN.  The server
 on the other hand appears to be waiting for the next request.
 
 Deadlock...
 
 Eventually after almost 3 minutes) I assume their server gets tired
 of waiting for a new request, and closes the connection.  That (finally)
 provides the FIN that ftp(1) has been waiting for, it is happy, and the
 connection completed properly.
 
 But that 3 minutes is too much for pkgsrc, which tells ftp to give up
 after 2 minutes.   When that happens, ftp never bothers to flush the last
 (partially filled) buffer, and just aborts, leaving those final 1902
 bytes missing from the file (if there's any bug in ftp(1) it would be
 that, it did receive the data, it could have written it to the file before
 quitting, and had it done do, pkgsrc would have verified the file checksum,
 I guess, if not on the same attempt as when ftp failed, then next time
 it went to look and found the file already existing) and all would have been
 OK.   But demanding that processes clean up fully when they are failing,
 is probably too much to exoect.
 
 With wget, the client TCP (ie: my system running wget) sends the first FIN,
 immediately after receiving the end of the file data, which is what you'd
 expect with a client doing Keep-Alive (why it bothers when it has only one
 file to fetch I have no idea, but it does, I guess just to be consistent with
 usages when it is fetching entire trees of files.)
 
 The real problem here appears to be the HTTP server that seems to be
 implementing keep-alive connection mode, when just the opposite was
 requested by the client.
 
 I have complete tcpdump (binary form) dumps of the two transactions,
 if anyone else (someone who speaks more http than I do) would like to
 take a look and confirm (or refute) my analysis.  They're each about 2.3MB
 big (and probably do not compress much, as most of that will be the file
 data, which was a .gz file, though I haven't tried to see).
 
 I can e-mail one of both of those, or make them available for ftp (but
 not for http...) if someone would like to take a look - or this seems to
 be consistently repeatable enough, that you could make your own trace
 (other than the 3 minute timeout, actually about 150 secs idle) all of
 this happens fairly quickly, it is not a very big file to transfer.
 
 kre