Subject: minor improvements in bulk build
To: None <tech-pkg@netbsd.org>
From: Jan Schaumann <jschauma@netmeister.org>
List: tech-pkg
Date: 11/18/2006 21:09:19
--TRYliJ5NKNqkz5bu
Content-Type: multipart/mixed; boundary="+QahgC5+KEYLbs62"
Content-Disposition: inline


--+QahgC5+KEYLbs62
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hi,

The attached patch tries to improve performance of the bulk builds ever
so slightly by reducing the number of forks we perform:

(1) if we use bulk timestamps, we currently call find(1), pipe the
output to grep to exclude some files, the run find again to count the
files.

My patch eliminates grep by excluding files via find(1) itself, then
uses the files we found to count them rather than run find(1) again.
(I rely on the fact that filenames do not contain spaces etc, which I
think is reasonable in our pkgsrc tree.)

(2) current package count is calculated by running 'ls | grep | wc',
which can be simplified into 'find | wc'

Does that look ok?  I haven't completed a bulk-build with this patch
yet, but I do have one running, and it hasn't bailed out yet. ;-)

-Jan

--=20
   It's psychosomatic. You need a lobotomy. I'll get a saw.
		  -- Calvin

--+QahgC5+KEYLbs62
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=diff

Index: bsd.bulk-pkg.mk
===================================================================
RCS file: /cvsroot/pkgsrc/mk/bulk/bsd.bulk-pkg.mk,v
retrieving revision 1.125
diff -b -u -r1.125 bsd.bulk-pkg.mk
--- bsd.bulk-pkg.mk	13 Oct 2006 23:14:45 -0000	1.125
+++ bsd.bulk-pkg.mk	19 Nov 2006 05:35:43 -0000
@@ -301,8 +301,8 @@
 			uptodate=0; \
 		elif [ "${USE_BULK_TIMESTAMPS}" = "yes" ]; then \
 			${SHCOMMENT} "Check files of this package"; \
-			newfiles="`${FIND} . -type f -newer "${REF}" -print | ${EGREP} -v -e ./work -e COMMENT -e DESCR -e README.html -e CVS -e '^\./\.' || ${TRUE}`"; \
-			nnewfiles="`${FIND} . -type f -newer "${REF}" -print | ${EGREP} -v -e ./work -e COMMENT -e DESCR -e README.html -e CVS -e '^\./\.' | ${WC} -l`"; \
+			newfiles="`${FIND} . -type f -newer "${REF}" ! \( -name ./work -or -name README.html -or -name DESCR -or -name COMMENT -or -name Repository -or -name Entries -or -name Root -or -name Tag \) -print || ${TRUE}`"; \
+			nnewfiles="`echo ${newfiles} | ${WC} -w`"; \
 			if [ "$$nnewfiles" -gt 0 ]; then \
 				${BULK_MSG} 1>&2 "Package ${PKGNAME} ($$newfiles) modified since last 'make package' re-packaging..."; \
 				uptodate=0; \
@@ -351,7 +351,7 @@
 		${ECHO} '###' ; \
 		${ECHO} '###' `date`: ; \
 		${ECHO} '### ${MAKE} ${.TARGET} for ${PKGNAME}' ; \
-		${ECHO} '### Current pkg count: ' `${LS} -l ${PKG_DBDIR} | ${GREP} '^d' | ${WC} -l` installed packages: `${LS} ${PKG_DBDIR} | ${GREP} -v pkgdb.byfile.db`; \
+		${ECHO} '### Current pkg count: ' `${FIND} ${PKG_DBDIR} -maxdepth 1 -type d ! -name '.' -print | ${WC} -l` installed packages: `${LS} ${PKG_DBDIR} | ${GREP} -v pkgdb.byfile.db`; \
 		${ECHO} '###' ; \
 	fi \
 	) 2>&1 | ${TEE} -a ${_BUILDLOG:Q}

--+QahgC5+KEYLbs62--

--TRYliJ5NKNqkz5bu
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (NetBSD)

iD8DBQFFX+b/fFtkr68iakwRAlOeAKDXNg+qjgPMtop9CqqZ5WFVYTv3aACbBTpx
qW6hwiPh6NoQLwN2pWZQzEM=
=I5S2
-----END PGP SIGNATURE-----

--TRYliJ5NKNqkz5bu--