pkgsrc-Users archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

FETCH_USING won't drive aria2 in parallel in pkgsrc



I like and use aria2 (aria2c) as a download tool.  In my opinion it has
smarter defaults and a superior feature list vis-a-vis wget, curl, or fetch.

The chief feature I like about aria2 is that it can use multiple download
sites as sources for a single file download.  It also seems to intelligently
adapt when some of the sites are faster than others.  The result is that I
can often speed up downloads by a factor of 3-4 times when using aria2.  I
max out it's 'splitting' options (-x 16 -s 16).  This usually results in
faster downloads even when I've only got one source for the file (because of
the bandwidth-delay product issue being mitigated by multiple streams).

Right now, I have a script that parses MASTER_SITES out of the Makefile in a
given pkgsrc package directory. It can then feed the information back to
aria2 in a similar fashion to "make fetch". However, this is hard. There are
a lot of weird exceptions to deal with. 

I know pkgsrc has a FETCH_USING parameter that allows you to override it's
default file-downloader.  However, I don't believe this will take advantage
of aria2's parallel capabilities.  It won't allow it to branch out to all
the master_sites at once since it'll only get passed one site at a time as
they fail or time out. 

So, my real question is this: is there any reasonable way to use aria2 with
pkgsrc and take advantage of the parallelism? 

-Swift


Home | Main Index | Thread Index | Old Index