Swift Griggs <swiftgriggs%gmail.com@localhost> writes: > On Thu, 7 Apr 2016, Greg Troxel wrote: >> Currently, I don't think so. But it could be reasonable to modify the >> infrastructure so that some FETCH_USING targets are configured to get all >> the master_sites values instead. > > That sounds reasonable. However, it might break other download clients like > curl, so I see your point. Perhaps, at some point there could be a new > variable in /etc/mk.conf like FETCH_PARALLEL=aria2c -x16 -s16 Yes, that's what I meant about it being configured per-program. But still there is the complexity/usefulness tradeoff. >> I have a script to update packages and one thing it does is run make >> fetch in the source directory of every installed package. I am hardly >> ever annoyed waiting for this. > > Wow. I'm surprised by that. I've been a NetBSD user since 1998, with a lot > of different internet providers (and I use NetBSD both at home and for my > jobs). I've often noticed situations where I've benefited from simply > grabbing the file with aria2c and dumping it in /usr/pkgsrc long before it > would be able to download the file itself. I fire it off and do something else. >> What kinds of things do you download that take lots of time, and what >> kind of speedup do you get? > > I have a few use cases, and some metrics. Right now I'm on a Comcast cable > modem at home that's 50 Mbit/s down 20 up. At work I'm on a Sprint OC3. I > usually get 1-3 Mbit/s to TNF's FTP site. The site only allows a max of 2 > connections, but even that makes a difference. Here's a test I just now > performed. One download with a single stream using wget, the other with two > streams using aria2 (3 total tests with smoothed averages across all): > > file == ftp://ftp.netbsd.org/pub/pkgsrc/stable/pkgsrc.tar.gz > aria2c == 2.8 MB/s (avg) > wget == 1.9 MB/s (avg) It's not clear to me that's enough to make this worthwhile :-) > Case A: There are zillion download sites listed, but one near the top of the > list is unreliable and is very slow or disconnect-happy. The serial download > method hits it in the same order every time and screws the whole process. > Aria2 would mitigate this by place that site lower in the download queue and > focusing on the sites that pass traffic. Yes, that's a fair point. > Case B: The pkgsrc port has a huge file and it either hangs or simply takes > forever (ala Libreoffice or Firefox). Aria2 may help here by simply > shortening the time needed to grab the file. Yes, but it doesn't address the larger parallelization of fetching everything needed. W.r.t. bittorrent or similar, I suspect it needs someone to really drive this, and that it's harder than it seems. I don't have any spare time budget for this, especially since it solves a problem I don't really have. But good luck if you want to think about it!
Attachment:
signature.asc
Description: PGP signature