On Thu, Apr 28, 2005 at 12:14:47PM -0400, Jan Schaumann wrote:
> Jan Schaumann <jschauma@netmeister.org> wrote:
> > John Klos <john@ziaspace.com> wrote:
> > 
> > > After 25 hours, a 60 MHz m68060 Amiga is still doing up and down
> > > dependency files. It'd be nice if all of this work could be done on a
> > > fast machine and the slow machines could just do the actual
> > > compiling...

Is it taking this long just extracting the ${DEPENDSTREEFILE} or
producing the ${SUPPORTSFILE} and ${DEPENDSFILE} from ${DEPENDSTREEFILE}?

I wonder if something has put back in some of the recursive make calls which
were specifically removed to keep this step at a somewhat reasonable
level.  Then again, maybe thats just how long 5,000 make calls take on
such a system.


> > That should be possible -- you should be able to run 'make bulk-cache'
> > in /usr/pkgsrc on a different machine and copy over the files.  We'd
> > probably need some glue and a flag to run the bulk-build without
> > rebuilding this, but that should not be all too complex.
> 
> How about the attached patch?
> 
> This way, on a fast machine you can
> 
> cd /usr/pkgsrc
> make BATCH=yes bulk-cache
> 
> then copy the files ${BULK_DBFILE} ${DEPENDSTREEFILE} ${INDEXFILE}
> ${ORDERFILE} ${SUPPORTSFILE} and ${DEPENDSFILE} onto the slower machine,
> set PRESERVE_BULK_CACHE=yes in your build.conf and save a few hours/days
> of work.

well.... you have to be careful here.  The dependency tree is operating
system, operating system version, and machine architecture dependent.

For example, the acroread package pulls in the compat_linux stuff on
NetBSD-i386, but not on NetBSD-sparc or Linux or Solaris.

On NetBSD, you get a different set between 1.6 and 2.0 due to native threads.

Others include libgetopt.  math/libffm is used on NetBSD-alpha to
get some faster replacements for some libm functions.

Some of the extensions (Xv maybe) in recent X.org or XFree86 will
also cause issues.

You get the picture.  I'm sure there are others.

-Dan

> -Jan
> 
> -- 
> ``Life is too short to stay entirely sober.'' -- Chuck Swiger

> Index: bsd.bulk-pkg.mk
> ===================================================================
> RCS file: /cvsroot/pkgsrc/mk/bulk/bsd.bulk-pkg.mk,v
> retrieving revision 1.79
> diff -b -u -r1.79 bsd.bulk-pkg.mk
> --- bsd.bulk-pkg.mk	11 Apr 2005 11:18:01 -0000	1.79
> +++ bsd.bulk-pkg.mk	28 Apr 2005 16:07:32 -0000
> @@ -160,7 +160,9 @@
>  # Note:  we have to install the BULK_PREREQ packages _before_
>  # extracting the depends tree because some packages like
>  # xpkgwedge only become DEPENDS if it is installed
> -bulk-cache:
> +bulk-cache:	${BULK_DBFILE} ${DEPENDSTREEFILE} ${INDEXFILE} ${ORDERFILE} ${SUPPORTSFILE} ${DEPENDSFILE}
> +
> +${BULK_DBFILE}:
>  	@${ECHO_MSG} "BULK> Installing BULK_PREREQ packages"
>  .for __prereq in ${BULK_PREREQ}
>  	cd ${PKGSRCDIR}/${__prereq} && ${MAKE} bulk-install
> @@ -168,38 +170,75 @@
>  	${RM} -f ${BULK_DBFILE}
>  	${TOUCH} ${BULK_DBFILE}
>  .if !defined(SPECIFIC_PKGS)
> -	@${ECHO} "This file is unused for a full pkgsrc bulk build" >> ${BULK_DBFILE}
> -	@${ECHO} "It is only used for a SPECIFIC_PKGS bulk build" >> ${BULK_DBFILE}
> -	@${ECHO_MSG} "BULK> Building complete pkgsrc dependency tree (this may take a while)."
> -	cd ${PKGSRCDIR} && ${SETENV} BMAKE=${MAKE} ${SH} mk/bulk/printdepends ${BROKENFILE} > ${DEPENDSTREEFILE}
> -	@${ECHO_MSG} "BULK> Generating package name <=> package directory cross reference file"
> -	@${ECHO_MSG} "      (this may take a while)."
> -	cd ${PKGSRCDIR} && ${SETENV} BMAKE=${MAKE} ${SH} mk/bulk/printindex ${BROKENFILE} > ${INDEXFILE}
> +	@${ECHO} "This file is unused for a full pkgsrc bulk build" 	\
> +							>> ${BULK_DBFILE}
> +	@${ECHO} "It is only used for a SPECIFIC_PKGS bulk build" 	\
> +							>> ${BULK_DBFILE}
>  .else
>  	@${ECHO_MSG} "BULK> Extracting database for SPECIFIC_PKGS subset of pkgsrc"
>  	@${ECHO_MSG} "      along with their dependencies"
> -.for __tmp__ in ${SUBDIR} ${BULK_PREREQ} ${PERL5_PKGSRCDIR:C|.*/([^/]*/[^/]*)$|\1|} pkgtools/pkglint
> -	cd ${PKGSRCDIR}/${__tmp__} && ${SETENV} BMAKE=${MAKE} AWK=${AWK} EXPR=${EXPR} ${SH} ../../mk/scripts/mkdatabase -a -f ${BULK_DBFILE}
> -.endfor
> +.  for __tmp__ in ${SUBDIR} ${BULK_PREREQ} 				\
> +		${PERL5_PKGSRCDIR:C|.*/([^/]*/[^/]*)$|\1|} pkgtools/pkglint
> +	cd ${PKGSRCDIR}/${__tmp__} && 					\
> +		${SETENV} BMAKE=${MAKE} AWK=${AWK} EXPR=${EXPR} 	\
> +			${SH} ../../mk/scripts/mkdatabase -a -f ${BULK_DBFILE}
> +.  endfor
> +.endif
> +
> +${DEPENDSTREEFILE}:	${BULK_DBFILE}
> +.if !defined(SPECIFIC_PKGS)
> +	@${ECHO_MSG} "BULK> Building complete pkgsrc dependency tree (this may take a while)."
> +	cd ${PKGSRCDIR} && ${SETENV} BMAKE=${MAKE} 			\
> +		${SH} mk/bulk/printdepends ${BROKENFILE} > ${DEPENDSTREEFILE}
> +.else
>  	@${ECHO_MSG} "BULK> Extracting dependency tree file"
> -	${AWK} '/^(build_)?depends/ {pkgs[$$2] = 1; cat=$$2; sub(/\/.*/, "", cat); \
> -		for(i=3; i<=NF; i=i+1){ \
> +	${AWK} '							\
> +		/^(build_)?depends/ {					\
> +			pkgs[$$2] = 1;					\
> +			cat=$$2; 					\
> +			sub(/\/.*/, "", cat); 				\
> +			for(i=3; i<=NF; i=i+1) { 			\
>  			listed[$$2] = 1; \
>  			sub(/[^:]*:\.\.\/\.\.\//, "", $$i); \
>  			sub(/[^:]*:\.\./, cat , $$i); \
>  			print $$i " " $$2; \
> -		}} END{ \
> -		for(pkg in pkgs) {if( pkg in listed ) {} else{ print pkg " " pkg;}} \
> +			}						\
> +		}							\
> +		END{ 							\
> +			for(pkg in pkgs) {				\
> +				if( pkg in listed ) {} 			\
> +				else{ print pkg " " pkg;}		\
> +			} 						\
>  		}' \
>  		${BULK_DBFILE} | ${_SORT} -u > ${DEPENDSTREEFILE}
> +.endif
> +
> +${INDEXFILE}:	${DEPENDSTREEFILE}
> +	@${ECHO_MSG} "BULK> Generating package name <=> package directory cross reference file"
> +	@${ECHO_MSG} "      (this may take a while)."
> +.if !defined(SPECIFIC_PKGS)
> +	cd ${PKGSRCDIR} && ${SETENV} BMAKE=${MAKE} ${SH} mk/bulk/printindex ${BROKENFILE} > ${INDEXFILE}
> +.else
>  	@${ECHO_MSG} "BULK> Extracting package name <=> package directory cross reference file"
>  	${AWK} '/^index/ {print $$2 " " $$3 " "}' ${BULK_DBFILE} > ${INDEXFILE}
>  .endif
> +
> +${ORDERFILE}:	${DEPENDSTREEFILE}
>  	@${ECHO_MSG} "BULK> Sorting build order."
>  	${TSORT} ${DEPENDSTREEFILE} > ${ORDERFILE}
> -	@${ECHO_MSG} "BULK> Generating up and down dependency files."
> -	${SETENV} SORT=${_SORT:Q} ${AWK} -f ${PKGSRCDIR}/mk/bulk/tflat up ${DEPENDSTREEFILE} > ${SUPPORTSFILE}
> -	${SETENV} SORT=${_SORT:Q} ${AWK} -f ${PKGSRCDIR}/mk/bulk/tflat down ${DEPENDSTREEFILE} > ${DEPENDSFILE}
> +
> +${SUPPORTSFILE}:	${ORDERFILE}
> +	@${ECHO_MSG} "BULK> Generating up dependency file."
> +	${SETENV} SORT=${_SORT:Q} 					\
> +	${AWK} -f ${PKGSRCDIR}/mk/bulk/tflat up ${DEPENDSTREEFILE} 	\
> +			> ${SUPPORTSFILE}
> +
> +
> +${DEPENDSFILE}:	${ORDERFILE}
> +	@${ECHO_MSG} "BULK> Generating down dependency file."
> +	${SETENV} SORT=${_SORT:Q} 					\
> +	${AWK} -f ${PKGSRCDIR}/mk/bulk/tflat down ${DEPENDSTREEFILE} 	\
> +			> ${DEPENDSFILE}
>  
>  # remove the bulk cache files
>  clean-bulk-cache:
> Index: build
> ===================================================================
> RCS file: /cvsroot/pkgsrc/mk/bulk/build,v
> retrieving revision 1.48
> diff -b -u -r1.48 build
> --- build	14 Jan 2005 15:32:34 -0000	1.48
> +++ build	28 Apr 2005 16:07:32 -0000
> @@ -276,6 +276,9 @@
>  # Create the bulk cache files
>  cd ${USR_PKGSRC}
>  if [ "x$restart" != "xyes" ]; then
> +	if [ "x${PRESERVE_BULK_CACHE}" != "xyes" ]; then
> +		rm -f $BULK_DBFILE $DEPENDSTREEFILE $DEPENDSFILE $SUPPORTSFILE $INDEXFILE $ORDERFILE
> +	fi
>  	cd ${USR_PKGSRC} && ${BMAKE} bulk-cache $makeargs
>  	if [ $? != 0 ]; then
>  		echo "$0: Cache creation failed.  Aborting build."
> Index: build.conf-example
> ===================================================================
> RCS file: /cvsroot/pkgsrc/mk/bulk/build.conf-example,v
> retrieving revision 1.28
> diff -b -u -r1.28 build.conf-example
> --- build.conf-example	22 Mar 2005 16:42:52 -0000	1.28
> +++ build.conf-example	28 Apr 2005 16:07:32 -0000
> @@ -36,6 +36,13 @@
>  #CVS_FLAGS="-rnetbsd-2005Q1"
>  #CVS_FLAGS="-A"				# pkgsrc-current
>  
> +# preserve bulk-cache files
> +# can be set to 'yes' or 'no'
> +# useful on slow architectures, where you have generated the
> +# bulk-cache files on a faster machine by running 'make bulk-cache' from
> +# USR_PKGSRC
> +PRESERVE_BULK_CACHE=no
> +
>  # prune distfiles to remove those which are out of date
>  # can be set to 'yes' or 'no'.
>  PRUNEDISTFILES=no




--