[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: Automatic mirror selection.
Alistair Crooks wrote:
> On Thu, May 23, 2013 at 04:55:41PM +0200, Julian H. Stacey wrote:
> > Dmitriy Tochansky wrote:
> > > Is there any mechanism to choose mirror automaticly from list?
> > I don't know.
> > I've often wondered similar for FreeBSD sites too,
> > maybe this is something we could share among the BSDs ?
> > It should be possible to knock up a script that would generate a
> > sensible default at least some of the time, to spread the load away
> > from central servers, eg
> > With
> > http://www.berklix.net/cgi-bin/printenv
> > I see my
> > REMOTE_ADDR="188.8.131.52"
> > & nslookup
> > 184.108.40.206
> > p5DCBD01B.dip0.t-ipconnect.de
> > so in my case defaults of
> > http://www.de.netbsd.org
> > http://www.de.freebsd.org
> > would be good.
> > If a script couldnt calculate a match, before giving up & returning the root
> > http://www.netbsd.org, it could also try a
> > ping -c 1 using the 2 letter country sub domain, in case it had been
> > created since the script was last updated.
> pkgsrc has a sorting mechanism for master sites for distfiles - from
> By default a list of distribution sites will be randomly
> intermixed to prevent huge load on servers which holding
> popular packages (for example, SourceForge.net mirrors).
> Thus, every time when you need to fetch yet another distfile
> all the mirrors will be tried in new (random) order. You can
> turn this feature off by setting MASTER_SORT_RANDOM=NO (for
> PKG_DEVELOPERs it's already disabled).
> You can overwrite some of the major distribution sites to fit
> to sites that are close to your own. By setting one or two
> variables you can modify the order in which the master sites
> are accessed. MASTER_SORT contains a whitespace delimited
> list of domain suffixes. MASTER_SORT_REGEX is even more
> flexible, it contains a whitespace delimited list of regular
> expressions. It has higher priority than MASTER_SORT. Have a
> look at pkgsrc/mk/defaults/mk.conf to find some examples.
> This may save some of your bandwidth and time.
> You can change these settings either in your shell's
> environment, or, if you want to keep the settings, by editing
> the mk.conf file, and adding the definitions there.
> and from pkgsrc/mk/defaults/mk.conf
> #MASTER_SORT= .ac.at .at .de .ch .se .fi .no
> # Whitespace delimited list of preferred download locations.
> # MASTER_SITES and PATCH_SITES will be reordered according to the hosts'
> # address last components.
> # Possible: list of domain name suffixes
> # Default: none
> #MASTER_SORT_REGEX= ftp://.*/
> # Same as MASTER_SORT, but takes a regular expression for more
> # flexibility in matching. Regexps defined here have higher priority
> # than MASTER_SORT. This example would prefer ftp transfers over
> # anything else.
> # Possible: Regexps as in awk(1)
> # Default: none
> I'm sure this could be extended for binary package managers
I had guessed Dimitry meant redirection to be decided at the sending
end via automatic web server redirect; but selection at the
receiving end, of where to pull distfiles & packages from is also
interesting, presumably more important than web pages).
(BTW FreeBSD has similar MASTER_SORT_REGEX etc. I've no idea
about packaged binaries, I only build from source.)
WIBNI (Wouldn't It Be Nice If :-) there might be a summer of code (or
similar) project to develop a multi-BSD tool that could do automatic
locality sniffing, traceroute, r-dns, sub domain mapping, ping -c
1, etc & automaticaly emit mk & sh variables that could be included
by /etc/make.conf for distfiles & binary package fetching. The corollary
for the web server cgi-bin sending end could probably also use a common
database of known net names to determine probably nearest server.
(eg database (or auto increment dnslookup) knowlege that
ftp8.de.freebsd.org exists, but no ftp9.de.freebsd.org so
ftp9 would get a ping -c 1, & if that failed, default to
ftp.freebsd.org ... or to adjacent countries eg .au or .fr
etc, rather than transatlantic.
Sorry I'm not using netbsd specific host names, I tried, but from
http://www.de.netbsd.org/ + top right mirrors list, all I tried fail at
Thu May 23 18:59:58 CEST 2013 with
The requested URL /cgi-bin/redirect.cgi was not found on this server.
This also fails:
Julian Stacey, BSD Unix Linux C Sys Eng Consultant, Munich http://berklix.com
Reply below not above, like a play script. Indent old text with "> ".
Send plain text. No quoted-printable, HTML, base64, multipart/alternative.
Main Index |
Thread Index |