Subject: Rework of the fetch target (was: CVS commit: pkgsrc/mk)
To: Frederick Bruckman <fredb@netbsd.org>
From: Bernd Ernesti <netbsd@arresum.inka.de>
List: tech-pkg
Date: 01/27/2002 17:01:38
Hi,

I am not exactly sure that it is this change, but doing a make fetch-list
is now a real mess.

Even if have the distfile, you see this kind of output, which makes it
impossible to see which distfiles you need to get:

pkgsrc/graphics/jhead> make fetch-list
test -d /home/temp/distfiles/ || /bin/mkdir -p /home/temp/distfiles/; cd /h=
ome/temp/distfiles/; unsorted_sites=3D"ftp://ftp.info-zip.org/pub/infozip/s=
rc/  ftp://ftp.icce.rug.nl/infozip/src/  ftp://uiarchive.cso.uiuc.edu/pub/p=
ackages/zip/src/  ftp://ftp.fu-berlin.de/pub/unix/tools/archivers/unzip/ ft=
p://ftp.netbsd.org/pub/NetBSD/packages/distfiles/  ftp://ftp.freebsd.org/pu=
b/FreeBSD/distfiles/";  sites=3D" $unsorted_sites";					 file=3D"unzip542.t=
ar.gz";						 bfile=3D"unzip542.tar.gz";						 if [ "X" !=3D "X" ]; then			=
	 for d in "" ; do	 if [ "X$d" =3D "X" -o "X$d" =3D "X/home/temp/distfiles"=
 ]; then continue; fi;  if [ -f $d//$bfile ]; then	 echo "Using $d//$bfile"=
;  /bin/rm -f $bfile;			 /bin/ln -s $d//$bfile $bfile;  break;					 fi;				=
		 done;							 fi;						 if [ ! -f $file -a ! -f $bfile -a ! -h $bfile ]; =
then	 echo "=3D> $bfile doesn't seem to exist on this system.";  if [ ! -w =
/home/temp/distfiles//. ]; then 			 echo "=3D> Can't download to /home/temp=
/distfiles/ (permission denied?).";  exit 1; 					 fi; 							 for site in =
$sites; do					 echo "=3D> Attempting to fetch $bfile from ${site}.";  if /=
usr/bin/ftp  ${site}${bfile} ; then  if [ -n "" -a -f /home/source/pkgsrc/a=
rchivers/unzip/distinfo -a -f /home/temp/distfiles//$bfile ]; then  alg=3D`=
/usr/bin/awk 'NF =3D=3D 4 && $2 =3D=3D "('$file')" && $3 =3D=3D "=3D" {prin=
t $1;}' /home/source/pkgsrc/archivers/unzip/distinfo`;  if [ -z "$alg" ]; t=
hen		 alg=3DSHA1; fi;				 CKSUM=3D`/usr/pkg/bin/digest $alg < /home/temp/di=
stfiles//$bfile`;  CKSUM2=3D`/usr/bin/awk '$1 =3D=3D "'$alg'" && $2 =3D=3D =
"('$file')" {print $4;}' </home/source/pkgsrc/archivers/unzip/distinfo`;  i=
f [ "$CKSUM" =3D "$CKSUM2" -o "$CKSUM2" =3D "IGNORE" ]; then  break;			 els=
e				 echo "=3D> Checksum failure - trying next site.";  fi;				 elif [ ! -=
f /home/temp/distfiles//$bfile ]; then  echo "=3D> FTP didn't fetch expecte=
d file, trying next site." ;  else					 break;				 fi;					 fi						 done;	=
						 if [ ! -f /home/temp/distfiles//$bfile ]; then  echo "=3D> Couldn't =
fetch $bfile - please try to retrieve this"; echo "=3D> file manually into =
/home/temp/distfiles/ and try again.";  exit 1;						 fi;							 fi;
test -d /home/temp/distfiles/jhead-1.5 || /bin/mkdir -p /home/temp/distfile=
s/jhead-1.5; cd /home/temp/distfiles/jhead-1.5; unsorted_sites=3D"http://ww=
w.sentex.net/~mwandel/jhead/ ftp://ftp.netbsd.org/pub/NetBSD/packages/distf=
iles/jhead-1.5/ ftp://ftp.freebsd.org/pub/FreeBSD/distfiles/jhead-1.5/";  s=
ites=3D" $unsorted_sites";					 file=3D"jhead-1.5/jhead.zip";						 bfile=
=3D"jhead.zip";						 if [ "X" !=3D "X" ]; then				 for d in "" ; do	 if [ =
"X$d" =3D "X" -o "X$d" =3D "X/home/temp/distfiles" ]; then continue; fi;  i=
f [ -f $d/jhead-1.5/$bfile ]; then	 echo "Using $d/jhead-1.5/$bfile";  /bin=
/rm -f $bfile;			 /bin/ln -s $d/jhead-1.5/$bfile $bfile;  break;					 fi;		=
				 done;							 fi;						 if [ ! -f $file -a ! -f $bfile -a ! -h $bfile ]=
; then	 echo "=3D> $bfile doesn't seem to exist on this system.";  if [ ! -=
w /home/temp/distfiles/jhead-1.5/. ]; then 			 echo "=3D> Can't download to=
 /home/temp/distfiles/jhead-1.5 (permission denied?).";  exit 1; 					 fi; =
							 for site in $sites; do					 echo "=3D> Attempting to fetch $bfile f=
rom ${site}.";  if /usr/bin/ftp  ${site}${bfile} ; then  if [ -n "" -a -f /=
home/source/pkgsrc/graphics/jhead/distinfo -a -f /home/temp/distfiles/jhead=
-1.5/$bfile ]; then  alg=3D`/usr/bin/awk 'NF =3D=3D 4 && $2 =3D=3D "('$file=
')" && $3 =3D=3D "=3D" {print $1;}' /home/source/pkgsrc/graphics/jhead/dist=
info`;  if [ -z "$alg" ]; then		 alg=3DSHA1; fi;				 CKSUM=3D`/usr/pkg/bin/=
digest $alg < /home/temp/distfiles/jhead-1.5/$bfile`;  CKSUM2=3D`/usr/bin/a=
wk '$1 =3D=3D "'$alg'" && $2 =3D=3D "('$file')" {print $4;}' </home/source/=
pkgsrc/graphics/jhead/distinfo`;  if [ "$CKSUM" =3D "$CKSUM2" -o "$CKSUM2" =
=3D "IGNORE" ]; then  break;			 else				 echo "=3D> Checksum failure - tryi=
ng next site.";  fi;				 elif [ ! -f /home/temp/distfiles/jhead-1.5/$bfile =
]; then  echo "=3D> FTP didn't fetch expected file, trying next site." ;  e=
lse					 break;				 fi;					 fi						 done;							 if [ ! -f /home/temp/dis=
tfiles/jhead-1.5/$bfile ]; then  echo "=3D> Couldn't fetch $bfile - please =
try to retrieve this"; echo "=3D> file manually into /home/temp/distfiles/j=
head-1.5 and try again.";  exit 1;						 fi;							 fi;
Bernd

On Sun, Jan 06, 2002 at 10:03:24PM +0200, Frederick Bruckman wrote:
>=20
> Module Name:	pkgsrc
> Committed By:	fredb
> Date:		Sun Jan  6 20:03:24 UTC 2002
>=20
> Modified Files:
> 	pkgsrc/mk: bsd.pkg.mk
>=20
> Log Message:
> Re-work the fetch target(s), as discussed on "tech-pkg". The primary
> motivation is to make the order that sites are hit more sane, and
> especially to prevent hitting sites that aren't ever expected to have
> the sought after file.
>=20
> Now, ${MASTER_SITE_OVERRIDE} is always hit first. If that fails, then
> the sites designated for that file, then ${MASTER_SITE_BACKUP}, are hit
> (by default), but with the order subject to ${MASTER_SORT_REGEX} and
> ${MASTER_SORT_AWK}. The "designated sites" are usually ${MASTER_SITES}
> for files in ${DISTFILES}, and ${PATCH_SITES} for files in ${PATCH_FILES}.
> However, defining a variable `${SITES_foo}' in the package "Makefile"
> overrides that for file "foo". [The use of ${MASTER_SITES_foo} and
> ${PATCH_SITES_foo}, which is currently only used by a couple of packages
> for the same purpose, is deprecated, and will be shortly unsupported.]
>=20
> Also eliminate redundancy in the do-fetch and fetch-list-one-pkg targets,
> by making them use a single, common macro (escaped with `:Q' in the
> fetch-list-one-pkg case), so "make fetch-list-one-pkg | sh" now does
> exactly the same thing as "make do-fetch".
>=20
>=20
> To generate a diff of this commit:
> cvs rdiff -r1.894 -r1.895 pkgsrc/mk/bsd.pkg.mk
>=20
> Please note that diffs are not public domain; they are subject to the
> copyright notices on the relevant files.
>=20
>=20