FETCH_OUTPUT_ARGS only used for resumed transfers?

[Brian de Alwis <brian.de.alwis%usask.ca@localhost>]

I have been trying to use FETCH_CMD=curl as curl comes with a default  
set of certificates and thus a good candidate for fetching files using  
https.  But I've encountered a few problems and in the process of  
trying to get it to work, I've found the description of  
FETCH_OUTPUT_ARGS to be unclear, and also found that the example using  
wget in the pkgsrc guide doesn't actually work.

curl, by default, sends its output to stdout.  curl can be directed to  
save the content to an explicitly specified file using '-o filename'.   
mk/fetch/fetch describes FETCH_OUTPUT_ARGS as:

#       FETCH_OUTPUT_ARGS
#                       These options specify the name of the local file
#                       that will hold the contents of the fetched file.

Does that mean that the options in FETCH_OUTPUT_ARGS should also  
encode the name of the local file?  From looking at the example in the  
pkgsrc guide and the mk/fetch/fetch code, I think this is better  
worded as:

# FETCH_OUTPUT_ARGS
#       These options are used to specify the name of the local file
#       that will hold the contents of the fetched file.

Unfortunately the example using wget in the pkgsrc guide is incorrect  
as FETCH_OUTPUT_ARGS is listed as:

> FETCH_OUTPUT_ARGS=     -O -nc


These options are reversed, I think, as they cause the fetched content  
to be saved to a file named '-nc'.  But wget seems to have a bug when  
'-nc' is specified with '-O': the '-nc' check is done *after* the  
output file has been created:

> $ wget --passive-ftp -nc -O bzr-1.2.tar.gz 
> http://bazaar-vcs.org/releases/src/bzr-1.2.tar.gz
> File `bzr-1.2.tar.gz' already there; not retrieving.

It's probably better that the example not use '-nc' and avoid any  
possible confusion.  (Patch below)


Having figured out the above, FETCH_OUTPUT_ARGS doesn't actually seem  
to be used except for resumed fetches (which is not the default).  mk/ 
fetch/fetch does the fetch using:

          if ${TEST} -n "$resume"; then
                fetch_cmd="${FETCH_CMD} ${FETCH_BEFORE_ARGS} $ 
{FETCH_RESUME_ARGS} ${FETCH_OUTPUT_ARGS} $outputfile $site$file"
          else
                fetch_cmd="${FETCH_CMD} ${FETCH_BEFORE_ARGS} $site 
$file ${FETCH_AFTER_ARGS}"
          fi

I think it makes more sense to specify `${FETCH_OUTPUT_ARGS}  
$outputfile' for both cases (patch appended below).  After all, the  
two commands should be mirrors except for the FETCH_RESUME_ARGS, right?

With this change, I can use curl to fetch files using the following:

FETCH_CMD=              curl
FETCH_BEFORE_ARGS=
FETCH_OUTPUT_ARGS=      -o
FETCH_RESUME_ARGS=

Without the patch, curl cannot be used as a fetch command.

Two other questions:

  * Should mk/fetch/fetch verify that FETCH_OUTPUT_ARGS is defined?   
Bad Things happen if it is not.

  * Can the resume support actually be used?   Whenever I interrupt a  
fetch, bmake causes the fetched file to be removed.


=== modified file 'mk/fetch/fetch'
--- mk/fetch/fetch      2008-01-09 15:38:08 +0000
+++ mk/fetch/fetch      2008-03-01 00:36:32 +0000
@@ -99,8 +99,9 @@
 #                      previous file transfer.
 #
 #      FETCH_OUTPUT_ARGS
-#                      These options specify the name of the local file
-#                      that will hold the contents of the fetched file.
+#                      These options are used to specify the name of
+#                      the local file that will hold the contents of the
+#                      fetched file.
 #
 ######################################################################

@@ -262,7 +263,7 @@
          if ${TEST} -n "$resume"; then
 		fetch_cmd="${FETCH_CMD} ${FETCH_BEFORE_ARGS} ${FETCH_RESUME_ARGS} $ 
{FETCH_OUTPUT_ARGS} $outputfile $site$file"
          else
-		fetch_cmd="${FETCH_CMD} ${FETCH_BEFORE_ARGS} $site$file $ 
{FETCH_AFTER_ARGS}"
+		fetch_cmd="${FETCH_CMD} ${FETCH_BEFORE_ARGS} ${FETCH_OUTPUT_ARGS}  
$outputfile $site$file ${FETCH_AFTER_ARGS}"
          fi
          ${TEST} -z "$verbose" || ${ECHO} "$fetch_cmd"
          $fetch_cmd )


--- doc/guide/files/faq.xml     2008-01-09 15:38:08 +0000
+++ doc/guide/files/faq.xml     2008-03-01 00:08:58 +0000
@@ -286,7 +286,7 @@
 FETCH_CMD=             wget
 FETCH_BEFORE_ARGS=     --passive-ftp
 FETCH_RESUME_ARGS=     -c
-FETCH_OUTPUT_ARGS=     -O -nc
+FETCH_OUTPUT_ARGS=     -O
 </programlisting>

 </sect1>

--
Brian de Alwis | HCI Lab | University of Saskatchewan