Subject: xsrc/22456: UNPRIVED build hacks for /usr/xsrc and netbsd-1-6
To: None <gnats-bugs@gnats.netbsd.org>
From: Greg A. Woods <woods@weird.com>
List: netbsd-bugs
Date: 08/12/2003 14:50:44
>Number:         22456
>Category:       xsrc
>Synopsis:       UNPRIVED build hacks for /usr/xsrc and netbsd-1-6
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    xsrc-manager
>State:          open
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Tue Aug 12 18:51:00 UTC 2003
>Closed-Date:
>Last-Modified:
>Originator:     Greg A. Woods
>Release:        2003/08/10
>Organization:
Planix, Inc.; Toronto, Ontario; Canada
>Environment:
System: NetBSD 1.6.1_STABLE
Architecture: alpha
Machine: alph
>Description:

	I've just finished doing a relatively painless and apparently
	successful UNPRIVED "make release" of /usr/xsrc using a few very
	quick and dirty hacks.

	Note that I've used a patched version of the "NetBSD.x11.dist"
	mtree to facilitate "maketars" support of a METALOG file.  In
	theory this should be moved to the xsrc tree and installed in
	$DESTDIR/usr/X11R6/etc/mtree or similar so that pkgsrc can use
	it from there, but for now it was a lot easier for me to
	maintain one shared copy of it in my pkgsrc tree and to not
	install it.

	I still have to tweak checkflist to work (or write a custom one
	that will only work within $DESTDIR/usr/X11R6 since all the xsrc
	stuff does still go exclusively into there).  Ultimately I think
	all this sets/lists stuff should go away and "tags=" keywords
	should be included in the metalog so that pax (with equivalent
	features matching mtree's '-I' and '-E') (or similar for
	whatever pkg tool for syspkgs) can directly select installed
	files based on their tag name(s) (which would be supplied
	directly by the makefile that installs them, with a suitable
	default to the [x]base set of course).  I'm sick and fed up with
	maintaining the same information in two places, especially when
	failure to do it exactly right every time and all of the time
	adversely affects the usability of my resulting binary sets.

	It would also be useful (to me, at least) to have support for
	makesrctars in xsrc as well, but I've not even hacked on that
	yet.

	Note I'm using a separate $DESTDIR/METALOG.X11 file.  In theory
	this makes it possible for xsrc to be built and installed into
	an otherwise empty $DESTDIR but I haven't tried that (yet).

	Note I'm doing this on the netbsd-1-6 branch (on alpha) with the
	head of the xsrc tree.  There are some changes specific to
	netbsd-1-6 "maketars".

	I'd also like to get "lndir" support or similar working for
	always building xsrc in a shadow tree in $OBJDIR, but I haven't
	yet hacked on that aspect of it.  I try to build my xsrc on at
	least three different platforms now and scheduling each is
	turning into a bit of a pain since switching platforms requires
	either full source tree copies (that have to by rsync'ed), or a
	full "make clean" and re-build.

>How-To-Repeat:

	apply the following patches to your xsrc, src, and pkgsrc trees

	do a "make all" in /usr/xsrc

	make sure your $TOOLDIR is up-to-date from a system build
	(presumably you'll do a whole "./build.sh -U" for your src tree
	such that TOOLDIR is fully populated)

	set PKGSRCDIR, TOOLDIR, RELEASEDIR, and DESTDIR in the
	environment as appropriate, and set USETOOLS=yes, e.g.:

		PKGSRCDIR=/usr/pkgsrc
		USETOOLS=yes
		TOOLDIR=/home/proven/woods/NetBSD-1.6.x-alpha-tools
		RELEASEDIR=/home/proven/woods/NetBSD-1.6.x-alpha-release
		DESTDIR=/home/proven/woods/NetBSD-1.6.x-alpha-destdir
		export PKGSRCDIR USETOOLS TOOLDIR RELEASEDIR DESTDIR

	make sure $TOOLDIR is at the head of your PATH:

		PATH=$TOOLDIR:$PATH

	Run the final "install" and "release" steps:

		make release
	
>Fix:

	WARNING: these changes, as-is, force you to use $TOOLDIR and to
	at least make use of the full facilities of an UNPRIVED build.
	I think this is a good thing, but perhaps not everyone will.  :-)

	Note the following diffs come from src, xsrc, and pkgsrc:

Index: distrib/sets/maketars
===================================================================
RCS file: /cvs/master/m-NetBSD/main/src/distrib/sets/maketars,v
retrieving revision 1.24.2.1
diff -c -r1.24.2.1 maketars
*** distrib/sets/maketars	23 Jul 2002 09:16:32 -0000	1.24.2.1
--- distrib/sets/maketars	12 Aug 2003 00:04:35 -0000
***************
*** 21,26 ****
--- 21,27 ----
  tars=$RELEASEDIR
  dest=$DESTDIR
  metalog=
+ justx11=false
  
  # handle args
  while : ; do
***************
*** 29,34 ****
--- 30,36 ----
  		lists="$xlists $nlists"
  		;;
  	-x*)
+ 		justx11=true
  		lists=$xlists
  		;;
  	-a*)
***************
*** 109,120 ****
  trap "/bin/rm -rf $SDIR ; exit 0" 0 2 3 13		# EXIT INT QUIT PIPE
  
  if [ -n "$metalog" ]; then
! 	echo "parsing $metalog"
! 	(
! 		cat ${DESTDIR}/etc/mtree/NetBSD.dist
! 		echo "/unset all"
! 		sed -e "s,^\.$dest,.," < $metalog
! 	) | ${MTREE} -D -k all -N ${DESTDIR}/etc | \
  	    sed -e 's,\(.*\) \(\..*\),\2 \1,' > $SDIR/metalog
  fi
  
--- 111,125 ----
  trap "/bin/rm -rf $SDIR ; exit 0" 0 2 3 13		# EXIT INT QUIT PIPE
  
  if [ -n "$metalog" ]; then
! 	cp ${DESTDIR}/etc/mtree/NetBSD.dist $SDIR/metalog.input
! 	chmod u+w $SDIR/metalog.input
! 	if ${justx11}; then
! 		echo ./usr/X11R6
! 		cat ${PKGSRCDIR}/mk/NetBSD.x11.dist
! 	fi >> $SDIR/metalog.input 
! 	echo "/unset all" >> $SDIR/metalog.input
! 	sed -e "s,^\.$dest,.," < $metalog >> $SDIR/metalog.input
! 	${MTREE} -D -k all -N ${DESTDIR}/etc -f $SDIR/metalog.input | \
  	    sed -e 's,\(.*\) \(\..*\),\2 \1,' > $SDIR/metalog
  fi
  


Index: xc/config/cf/NetBSD.cf
===================================================================
RCS file: /cvs/master/m-NetBSD/main/xsrc/xc/config/cf/NetBSD.cf,v
retrieving revision 1.106
diff -c -r1.106 NetBSD.cf
*** xc/config/cf/NetBSD.cf	24 Jun 2003 18:50:51 -0000	1.106
--- xc/config/cf/NetBSD.cf	8 Aug 2003 16:52:43 -0000
***************
*** 577,586 ****
  #define InstDatFlags		-m 0444 -o root -g wheel
  #define InstKmemFlags		-m 2555 -o root -g kmem
  
! #ifdef  InstallCmd
! # undef  InstallCmd
! # define InstallCmd 		/usr/bin/install
! #endif
  
  #ifndef ExtraFilesToClean
  # define ExtraFilesToClean 	*.core
--- 577,584 ----
  #define InstDatFlags		-m 0444 -o root -g wheel
  #define InstKmemFlags		-m 2555 -o root -g kmem
  
! #undef  InstallCmd
! #define InstallCmd 		nbinstall -U -M $(DESTDIR)/METALOG.X11
  
  #ifndef ExtraFilesToClean
  # define ExtraFilesToClean 	*.core
Index: Makefile
===================================================================
RCS file: /cvs/master/m-NetBSD/main/xsrc/Makefile,v
retrieving revision 1.40
diff -c -r1.40 Makefile
*** Makefile	21 Jul 2003 11:09:45 -0000	1.40
--- Makefile	11 Aug 2003 23:24:47 -0000
***************
*** 116,121 ****
--- 116,125 ----
  
  # release goo
  #
+ SETSENV=	DESTDIR=${DESTDIR:Q} \
+ 		MACHINE=${MACHINE:Q} MACHINE_ARCH=${MACHINE_ARCH:Q} \
+ 		MAKE=${MAKE:Q} MTREE=${MTREE:Q} PAX=${PAX:Q} 
+ 
  .if !defined(DESTDIR)
  release snapshot:
  	@echo setenv DESTDIR before doing that!
***************
*** 136,157 ****
  #
  .endif # INSTALL_DONE or BUILD_DONE
  #
! 	${INSTALL} -d -m 755 -o root -g wheel ${RELEASEDIR}/${MACHINE}/binary/sets
  .if (${MACHINE} == "amiga") && ${USE_XF86_4:Uno} == "no"
  	cd ${XCDIR}/programs/Xserver/hw/netbsd/amiga && \
  		${INSTALL} -c -m 644 INSTALL.X11 ${RELEASEDIR}/${MACHINE}
  .endif
  .if defined(METALOG.add) && !exists(${DESTDIR}/etc/master.passwd)
  	cd ${NETBSDSRCDIR}/distrib/sets && \
! 	    sh ./maketars -x -d ${DESTDIR:S,^$,/,} -N ${NETBSDSRCDIR}/etc -t ${RELEASEDIR}/${MACHINE}/binary/sets
  .else
  	cd ${NETBSDSRCDIR}/distrib/sets && \
! 	    sh ./maketars -x -d ${DESTDIR:S,^$,/,} -t ${RELEASEDIR}/${MACHINE}/binary/sets
  .endif
! 	cd ${RELEASEDIR}/${MACHINE}/binary/sets && \
! 		cksum -o 1 *.tgz >BSDSUM && \
! 		cksum *.tgz >CKSUM && \
! 		cksum -m *.tgz >MD5 && \
! 		cksum -o 2 *.tgz >SYSVSUM
  #
  .endif # DESTDIR and RELEASEDIR check
--- 140,163 ----
  #
  .endif # INSTALL_DONE or BUILD_DONE
  #
! .if !exists(${RELEASEDIR}/binary/sets)
! 	${INSTALL} -d -m 755 ${RELEASEDIR}/binary/sets
! .endif
  .if (${MACHINE} == "amiga") && ${USE_XF86_4:Uno} == "no"
  	cd ${XCDIR}/programs/Xserver/hw/netbsd/amiga && \
  		${INSTALL} -c -m 644 INSTALL.X11 ${RELEASEDIR}/${MACHINE}
  .endif
  .if defined(METALOG.add) && !exists(${DESTDIR}/etc/master.passwd)
  	cd ${NETBSDSRCDIR}/distrib/sets && \
! 	    ${SETSENV} sh -x ./maketars -x -M ${DESTDIR}/METALOG.X11 -d ${DESTDIR:S,^$,/,} -t ${RELEASEDIR}/binary/sets
  .else
  	cd ${NETBSDSRCDIR}/distrib/sets && \
! 	    ${SETSENV} sh -x ./maketars -x -M ${DESTDIR}/METALOG.X11 -d ${DESTDIR:S,^$,/,} -t ${RELEASEDIR}/binary/sets
  .endif
! 	cd ${RELEASEDIR}/binary/sets && \
! 		cksum -o 1 x*.tgz >>BSDSUM && \
! 		cksum x*.tgz >>CKSUM && \
! 		cksum -m x*.tgz >>MD5 && \
! 		cksum -o 2 x*.tgz >>SYSVSUM
  #
  .endif # DESTDIR and RELEASEDIR check


Index: NetBSD.x11.dist
===================================================================
RCS file: /cvs/master/m-NetBSD/main/pkgsrc/mk/NetBSD.x11.dist,v
retrieving revision 1.30
diff -c -c -r1.30 NetBSD.x11.dist
*** NetBSD.x11.dist	22 Jul 2003 23:21:19 -0000	1.30
--- NetBSD.x11.dist	12 Aug 2003 00:06:08 -0000
***************
*** 99,104 ****
--- 99,108 ----
                  ..
                  iso8859-9
                  ..
+                 iso8859-10
+                 ..
+                 iso8859-15
+                 ..
                  ja
                  ..
                  ja.JIS
***************
*** 117,122 ****
--- 121,128 ----
                  ..
                  zh_TW
                  ..
+                 zh_TW.Big5
+                 ..
              ..
              proxymngr
              ..
***************
*** 170,182 ****
--- 176,196 ----
                      ..
                      fujitsu
                      ..
+                     macintosh
+                     ..
                      nec
                      ..
+                     sgi
+                     ..
+                     sharp
+                     ..
                      sony
                      ..
                      sun
                      ..
                  ..
+ 		types
+ 		..
              ..
              xserver
              ..
***************
*** 282,291 ****
      ..
      share
          doc
-             html
-             ..
          ..
          examples
          ..
  	fonts
  	..
--- 296,307 ----
      ..
      share
          doc
          ..
          examples
+             rc.d
+             ..
+ 	    ssh-askpass
+ 	    ..
          ..
  	fonts
  	..
>Release-Note:
>Audit-Trail:
>Unformatted: