Re: Automatic mirror selection.

[Alistair Crooks <agc%pkgsrc.org@localhost>]

On Thu, May 23, 2013 at 04:55:41PM +0200, Julian H. Stacey wrote:
> Dmitriy Tochansky wrote:
> > Is there any mechanism to choose mirror automaticly from list?
> 
> I don't know. 
> I've often wondered similar for FreeBSD sites too,
> maybe this is something we could share among the BSDs ?
> 
> It should be possible to knock up a script that would generate a
> sensible default at least some of the time, to spread the load away
> from central servers, eg
> 
> With 
>       http://www.berklix.net/cgi-bin/printenv
> I see my
>       REMOTE_ADDR="93.203.208.27"
> & nslookup 
>       93.203.208.27
>       p5DCBD01B.dip0.t-ipconnect.de
> so in my case defaults of
>       http://www.de.netbsd.org
>       http://www.de.freebsd.org
> would be good.
> If a script couldnt calculate a match, before giving up & returning the root
> http://www.netbsd.org, it could also try a 
>       ping -c 1 using the 2 letter country sub domain, in case it had been 
> created since the script was last updated.

pkgsrc has a sorting mechanism for master sites for distfiles - from pkgsrc.txt:

        By default a list of distribution sites will be randomly
        intermixed to prevent huge load on servers which holding
        popular packages (for example, SourceForge.net mirrors). 
        Thus, every time when you need to fetch yet another distfile
        all the mirrors will be tried in new (random) order.  You can
        turn this feature off by setting MASTER_SORT_RANDOM=NO (for
        PKG_DEVELOPERs it's already disabled).

        You can overwrite some of the major distribution sites to fit
        to sites that are close to your own.  By setting one or two
        variables you can modify the order in which the master sites
        are accessed.  MASTER_SORT contains a whitespace delimited
        list of domain suffixes.  MASTER_SORT_REGEX is even more
        flexible, it contains a whitespace delimited list of regular
        expressions.  It has higher priority than MASTER_SORT.  Have a
        look at pkgsrc/mk/defaults/mk.conf to find some examples. 
        This may save some of your bandwidth and time.

        You can change these settings either in your shell's
        environment, or, if you want to keep the settings, by editing
        the mk.conf file, and adding the definitions there.

and from pkgsrc/mk/defaults/mk.conf

        #MASTER_SORT=   .ac.at .at .de .ch .se .fi .no
        # Whitespace delimited list of preferred download locations.
        # MASTER_SITES and PATCH_SITES will be reordered according to the hosts'
        # address last components.
        # Possible: list of domain name suffixes
        # Default: none

        #MASTER_SORT_REGEX=  ftp://.*/
        # Same as MASTER_SORT, but takes a regular expression for more
        # flexibility in matching.  Regexps defined here have higher priority
        # than MASTER_SORT.  This example would prefer ftp transfers over
        # anything else.
        # Possible: Regexps as in awk(1)
        # Default: none

I'm sure this could be extended for binary package managers

Regards,
Alistair