Subject: Master sites randomization
To: None <tech-pkg@netbsd.org>
From: Mike M. Volokhov <mishka@apk.od.ua>
List: tech-pkg
Date: 06/02/2005 11:41:07
--Signature=_Thu__2_Jun_2005_11_41_07_+0300_Ai/v1chpTm1yQSYU
Content-Type: text/plain; charset=US-ASCII
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings!

As I've previously noted, all pkgsrc users have used some predefined
list of distribution sites, known as MASTER_SITE_FOO. The problem is
that list is hardcoded, and "make fetch" always tries to download sites
in order from first to last. Thus, latest sites at list are likely never
tried, and first sites are under heavy load from the pkgsrc users.

This problem can be solved by randomization of site lists, and is
extremely actual for the sites like Sourceforge or OOo mirror list.

Fortunatelly, current NetBSD make(1) already have a variable
randomization modifier - ':Ox'. It's action is shown by the following
example:

	LIST=3D			uno due tre quattro
	RANDOM_LIST=3D		${LIST:Ox}
	STATIC_RANDOM_LIST:=3D	${LIST:Ox}

	all:
		@echo "${RANDOM_LIST}"
		@echo "${RANDOM_LIST}"
		@echo "${STATIC_RANDOM_LIST}"
		@echo "${STATIC_RANDOM_LIST}"

may produce output similar to:

	quattro due tre uno
	tre due quattro uno
	due uno quattro tre
	due uno quattro tre

Thus, we can define our master sites at mk/bsd.sites.mk, using the
rule, when primary distribution site is placed first, and the rest
mirrors are randomly shuffled; some sites, like sf.net, may not have
main distribution site and thus just all mirrors will be randomized.
For example:

	MASTER_SITE_APACHE_MIRRORS+=3D	\
		http://mirrors.midco.net/pub/apache.org/ \
		.....
	MASTER_SITE_APACHE+=3D		\
		http://www.apache.org/dist/
		${MASTER_SITE_APACHE_MIRRORS:Ox}

	MASTER_SITE_SOURCEFORGE_MIRRORS+=3D	\
		http://cogent.dl.sourceforge.net/sourceforge/ \
		.....
	MASTER_SITE_SOURCEFORGE+=3D	\
		${MASTER_SITE_SOURCEFORGE_MIRRORS:Ox}

But before doing this change, the ':Ox' feature must be integrated into
bootstrap/bmake, which is based on ancient bmake-3, which is dates from
Oct 2001.

I can pullup the ':Ox' into this bmake version. Or possible someone is
working on updating it (say, after the pkgsrc freeze will be over)?

Any comments please?

--
Mishka.

--Signature=_Thu__2_Jun_2005_11_41_07_+0300_Ai/v1chpTm1yQSYU
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (NetBSD)

iD8DBQFCnsYnHwrNSUJZSkURApCPAJ4nGr5/VtcyuemaNOi71XJX7OSC0ACaAzlt
kBYH2YCZI06s00ECq0TaTq0=
=p48l
-----END PGP SIGNATURE-----

--Signature=_Thu__2_Jun_2005_11_41_07_+0300_Ai/v1chpTm1yQSYU--