Subject: pkg/36774: pkg_install may prematurely abort file transfer
To: None <,,>
From: None <>
List: pkgsrc-bugs
Date: 08/12/2007 16:25:01
>Number:         36774
>Category:       pkg
>Synopsis:       pkg_install may prematurely abort file transfer
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    pkg-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Aug 12 16:25:00 +0000 2007
>Originator:     Havard Eidnes
>Release:        NetBSD 3.0_STABLE
	I Try...
System: NetBSD 3.0_STABLE NetBSD 3.0_STABLE (STEGG.MP) #1: Mon Jul 10 16:13:40 CEST 2006 i386
Architecture: i386
Machine: i386
	At home, accessing an FTP server over a wireless network
	with experienced maximum capacity in the area of 900Kbit/s,
	but sometimes significantly lower, a pkg_add of the
	openoffice2 package (implying a file transfer from the ftp
	server via PKG_PATH) would not complete, but rather be
	aborted by pkg_install because it thinks the file transfer
	took "too long".

	The reason appears to be that there is a constant time-out
	between pkg_add and the ftp sub-process, and there appears to
	be no progress detection for the ftp sub-process, so the
	constand time-out may fire even though the file transfer
	is still busy transferring the file for the package.

	This problem was experienced both with an older version of
	pkg_install as well as pkg_install-20070804.

	Here is the output from "pkg_add -v" for a failed attempt
	using the newer pkg_install:

stegg# pkg_add -v openoffice2
increasing RLIMIT_NOFILE to max. 1772 open files
trying PKG_PATH
Spawning FTP coprocess
ftp -detv

ftp> prompt off

ftp> nlist openoffice2 /var/tmp/pkg.21190a
pkg_add: nlist failed!
Reusing FDs 4/5 for communication to FTP coprocess

ftp> nlist openoffice2.tbz /var/tmp/pkg.21190b
pkg_add: nlist failed!
Reusing FDs 4/5 for communication to FTP coprocess

ftp> nlist openoffice2.tgz /var/tmp/pkg.21190c
pkg_add: nlist failed!
Reusing FDs 4/5 for communication to FTP coprocess

ftp> nlist openoffice2-*.t[bg]z /var/tmp/pkg.21190d

ftp> cd .
best match: ''
'[0-9]*.t[bg]z' expanded to ''
Trying to fetch
Reusing FDs 4/5 for communication to FTP coprocess

ftp> nlist openoffice2-2.2.1.tgz /var/tmp/pkg.21190f

ftp> cd .
best match: ''
Reusing FDs 4/5 for communication to FTP coprocess
unpackURL '' to '/var/tmp/instmp.21190e'

ftp> get openoffice2-2.2.1.tgz "| ( cd /var/tmp/instmp.21190e; tar --use-compress-program gzip -vvxp -f - | tee /dev/stderr )"
pkg_add: expect: poll() timeout

ftp> cd .
gzip: (stdin): unexpected end of file

tar: End of archive volume 1 reached
tar: ustar vol 1, 263 files, 143504894 bytes read, 0 bytes written in 601 secs (238776 bytes/sec)
OpenOffice.org2.2.1/program/libsvx680bi.sotar: Unexpected EOF on archive file
pkg_add: Cannot fetch file (-1!=226)!
Error on unpackURL('', '/var/tmp/instmp.21190e')
pkg_add: unable to fetch `' by URL
pkg_add: unable to open table of contents file `+CONTENTS' - not a package?
pkg_add: can't chdir back to '/var/tmp/instmp.21190e'
ftp: No control connection for command.

	Transferring the openoffice-2.2.1.tgz file manually got it
	transferred in 16:25 minutes:

ftp> get openoffice2-2.2.1.tgz
local: openoffice2-2.2.1.tgz remote: openoffice2-2.2.1.tgz
229 Entering Extended Passive Mode (|||59246|)
150 Opening BINARY mode data connection for 'openoffice2-2.2.1.tgz' (110846667 bytes).
100% |*************************************|   105 MB  109.86 KB/s    00:00 ETA
226 Transfer complete.
110846667 bytes received in 16:25 (109.86 KB/s)

	while the only constant poll() timeout in pkg_install appears to
	specify 1 hour as the maximum to wait for ftp to transfer a file:

    rc = poll(set, 1, 60*60*1000);    /* seconds until next message from tar */

	I seemed to have the impression that pkg_add timed out before one
	hour had passed.

	See above for conditions.
	Sorry, I don't know, but rather obviously using a constant
	time-out for the file transfer is bound to trigger issues
	such as this, so a better "progress" detector for the file
	transfer process is IMHO needed.