Re: make fetch-list

[Joerg Sonnenberger <joerg%britannica.bec.de@localhost>]

On Wed, Jul 15, 2009 at 06:25:26PM -0400, Martin S. Weber wrote:
> On Wed, Jul 15, 2009 at 07:33:23PM +0200, Joerg Sonnenberger wrote:
> > (...)
> > This brings me to the more interesting question of how "make fetch-list"
> > should behave. 
> 
> What's wrong with how it's behaving? The checksum part lacking
> after an attempted download?

Not supporting the FAILOVER_FETCH logic is one of the issues.

> > I think the current logic is too complicated for the
> > purpose of "running on a second machine without pkgsrc" 
> 
> Why's that? Too hard to collect shell-stuff-only that performs the
> equivalent task of pkgsrc with various variables either set or not
> set?

The current script tries to mimic a lot of the pkgsrc internal logic,
duplicating it.  It also hard-wires too many pathes that don't make
sense for this purpose.

> I used to use it for a place where I did have pkgsrc on a machine
> but didn't have internet all the time. I created several fetch-list.sh's
> and ran all of them when I got online. Once that was finished I
> knew I had the necessary distfiles [well, most of the time] and
> could build the packages (which was taking longer than the fetching)
> without having to have internet access.

Right, this is what I expect it to be used for. IMO the script should
essentially be:
- the definition of a fetch function that gets five arguments:
  - output name
  - url
  - expected file size
  - SHA1 hash
  - RMD160 hash
- for missing file, call the above function in a loop with the correct
  arguments

The goal of making it a function is to make it easy to customize, e.g.
to fetch via wget on a Linux box etc. The fragment can be choosen easily
to provide fetching via wget and checksum verification via openssl(1);
the file size check is trivial to implement in sh(1).

The only interaction with the pkgsrc infrastructure that would remain is
the building of the URL list and the extraction of the checksums. If
done correctly, it can even be processed by a human on a non-UNIX system.

> The other usage scenario I had was a unix machine which had access
> to the internet but my netbsd servers were locked away behind in
> some LAN where I couldn't access the internet. I used temporary
> space on that first server, uploading the scripts to there via LAN,
> running these and then lateron pushing the downloads from that
> gateway machine back on the LAN. (note that I had no control
> over whether I could or not directly access the internet).

Heh. I would have hacked the FETCH_CMD to issue the command via ssh :)

Joerg