Re: Automatic mirror selection.

["Julian H. Stacey" <jhs%berklix.com@localhost>]

Hi,
Alistair Crooks wrote:
> On Thu, May 23, 2013 at 04:55:41PM +0200, Julian H. Stacey wrote:
> > Dmitriy Tochansky wrote:
> > > Is there any mechanism to choose mirror automaticly from list?
> > 
> > I don't know. 
> > I've often wondered similar for FreeBSD sites too,
> > maybe this is something we could share among the BSDs ?
> > 
> > It should be possible to knock up a script that would generate a
> > sensible default at least some of the time, to spread the load away
> > from central servers, eg
> > 
> > With 
> >     http://www.berklix.net/cgi-bin/printenv
> > I see my
> >     REMOTE_ADDR="93.203.208.27"
> > & nslookup 
> >     93.203.208.27
> >     p5DCBD01B.dip0.t-ipconnect.de
> > so in my case defaults of
> >     http://www.de.netbsd.org
> >     http://www.de.freebsd.org
> > would be good.
> > If a script couldnt calculate a match, before giving up & returning the root
> > http://www.netbsd.org, it could also try a 
> >     ping -c 1 using the 2 letter country sub domain, in case it had been 
> > created since the script was last updated.
> 
> pkgsrc has a sorting mechanism for master sites for distfiles - from 
> pkgsrc.txt:
> 
>       By default a list of distribution sites will be randomly
>       intermixed to prevent huge load on servers which holding
>       popular packages (for example, SourceForge.net mirrors). 
>       Thus, every time when you need to fetch yet another distfile
>       all the mirrors will be tried in new (random) order.  You can
>       turn this feature off by setting MASTER_SORT_RANDOM=NO (for
>       PKG_DEVELOPERs it's already disabled).
> 
>       You can overwrite some of the major distribution sites to fit
>       to sites that are close to your own.  By setting one or two
>       variables you can modify the order in which the master sites
>       are accessed.  MASTER_SORT contains a whitespace delimited
>       list of domain suffixes.  MASTER_SORT_REGEX is even more
>       flexible, it contains a whitespace delimited list of regular
>       expressions.  It has higher priority than MASTER_SORT.  Have a
>       look at pkgsrc/mk/defaults/mk.conf to find some examples. 
>       This may save some of your bandwidth and time.
> 
>       You can change these settings either in your shell's
>       environment, or, if you want to keep the settings, by editing
>       the mk.conf file, and adding the definitions there.
> 
> and from pkgsrc/mk/defaults/mk.conf
> 
>       #MASTER_SORT=   .ac.at .at .de .ch .se .fi .no
>       # Whitespace delimited list of preferred download locations.
>       # MASTER_SITES and PATCH_SITES will be reordered according to the hosts'
>       # address last components.
>       # Possible: list of domain name suffixes
>       # Default: none
> 
>       #MASTER_SORT_REGEX=  ftp://.*/
>       # Same as MASTER_SORT, but takes a regular expression for more
>       # flexibility in matching.  Regexps defined here have higher priority
>       # than MASTER_SORT.  This example would prefer ftp transfers over
>       # anything else.
>       # Possible: Regexps as in awk(1)
>       # Default: none
> 
> I'm sure this could be extended for binary package managers
> 
> Regards,
> Alistair

I had guessed Dimitry meant redirection to be decided at the sending
end via automatic web server redirect; but selection at the
receiving end, of where to pull distfiles & packages from is also
interesting, presumably more important than web pages).
  (BTW FreeBSD has similar MASTER_SORT_REGEX etc.  I've no idea
  about packaged binaries, I only build from source.)

WIBNI (Wouldn't It Be Nice If :-) there might be a summer of code (or
similar) project to develop a multi-BSD tool that could do automatic
locality sniffing, traceroute, r-dns, sub domain mapping, ping -c
1, etc & automaticaly emit mk & sh variables that could be included
by /etc/make.conf for distfiles & binary package fetching. The corollary
for the web server cgi-bin sending end could probably also use a common
database of known net names to determine probably nearest server.
        (eg database (or auto increment dnslookup) knowlege that
        ftp8.de.freebsd.org exists, but no ftp9.de.freebsd.org so
        ftp9 would get a ping -c 1, & if that failed, default to
        ftp.freebsd.org ... or to adjacent countries eg .au or .fr
        etc, rather than transatlantic.

Sorry I'm not using netbsd specific host names, I tried, but from
http://www.de.netbsd.org/ + top right mirrors list, all I tried fail at 
Thu May 23 18:59:58 CEST 2013 with
        The requested URL /cgi-bin/redirect.cgi was not found on this server.
        
http://www.netbsd.org/cgi-bin/redirect.cgi?location=http%3A%2F%2Fwww.se.NetBSD.org%2F
This also fails:
        http://www.se.netbsd.org/
This OK
        http://www.it.netbsd.org/

Cheers,
Julian
-- 
Julian Stacey, BSD Unix Linux C Sys Eng Consultant, Munich http://berklix.com
 Reply below not above, like a play script.  Indent old text with "> ".
 Send plain text.  No quoted-printable, HTML, base64, multipart/alternative.