Subject: Re: Dynamic PLISTs
To: None <tech-pkg@NetBSD.org>
From: Julio M. Merino Vidal <jmmv@menta.net>
List: tech-pkg
Date: 04/08/2004 00:50:48
So here is a preliminar patch that does this; it was not so difficult ;)
(mainly because the dynamic PLIST stuff was already in place, thanks to
pkgviews). Though yes, there may still be some issues to be "fixed".
Basically, the patch changes nothing if a package is not modified. The
dynamic PLISTs (together with staged installs) are only activated when
there are no PLIST files available (i.e., 'mv PLIST PLIST.old').
Aside this, fixes may be needed in packages, like modifying post-install
targets to install files under ${STAGEDIR}${PREFIX}/whatever, instead of
simply ${PREFIX}/whatever.
The variable that points to the stage dir is named STAGEDIR, as shown above.
I would prefer DESTDIR, as commonly used by third party packages, but it
seems it's already in use in the defs* files for something else.
I've already converted several packages locally to see how this works,
including perl58, glib2... and all worked fine so far (will try more
tomorrow; it's late now ;). Though, for example, I had to add a patch for
m4, because its staged install is broken.
Note that the patch does not change the way pkgsrc works. I mean, it doesn't
try to 'make package' and install the resulting .tgz with pkg_add, as
OpenBSD ports do. This could be added in a future if needed, but I wanted
something simple (few modifications) to start with.
As you can see, using staged installs and dynamic PLISTs can be very easy
if the package itself supports the destdir stuff (if I'm not missing
something important). And it can surely save us from a lot of maintenance
headaches (as said in the previous mail: no more shared dirs, no mtree,
no *-dirs packages, less files in pkgsrc, staged installs)...
And, if a package can't be converted (mainly because it could mean lots of
local patches, one can still go with a static PLIST). I see no problem in
doing this.
Comments? Thanks.
Index: bsd.pkg.mk
===================================================================
RCS file: /cvsroot/pkgsrc/mk/bsd.pkg.mk,v
retrieving revision 1.1437
diff -u -r1.1437 bsd.pkg.mk
--- bsd.pkg.mk 5 Apr 2004 08:06:07 -0000 1.1437
+++ bsd.pkg.mk 7 Apr 2004 22:31:42 -0000
@@ -137,10 +137,6 @@
PKG_FAIL_REASON+= "PLIST_TYPE must be \`\`dynamic'' or \`\`static''."
.endif
-.if (${PKG_INSTALLATION_TYPE} == "overwrite") && (${PLIST_TYPE} != "static")
-PKG_FAIL_REASON+= "PLIST_TYPE must be \`\`static'' for \`\`overwrite'' packages."
-.endif
-
.if empty(USE_BUILDLINK2:M[nN][oO]) && empty(USE_BUILDLINK3:M[nN][oO])
PKG_FAIL_REASON+= "Please undefine USE_BUILDLINK2 or USE_BUILDLINK3."
.endif
@@ -629,18 +625,38 @@
. endif
. elif exists(${PKGDIR}/PLIST.${OPSYS})
PLIST_SRC= ${PKGDIR}/PLIST.${OPSYS}
-. else
+. elif exists(${PKGDIR}/PLIST)
PLIST_SRC= ${PKGDIR}/PLIST
+. else
+# No PLISTs found, so we will generate it on the fly after installation.
+# Enable the staged installation to achieve this.
+PLIST_SRC= # empty
+PLIST_TYPE= dynamic
+STAGEDIR= ${WRKDIR}/.stage
. endif
. endif
_PLIST_SRC= ${PLIST_SRC}
.elif ${PLIST_TYPE} == "dynamic"
_PLIST_SRC= # empty, since we're using a dynamic PLIST
+STAGEDIR= # empty, as the depot dir is used instead
.endif
DLIST= ${WRKDIR}/.DLIST
DDIR= ${WRKDIR}/.DDIR
+.if !empty(STAGEDIR) && ${PKG_PHASE} == "install"
+INSTALLATION_DIRS+= ${STAGEDIR}${PREFIX}
+NO_MTREE= yes
+# Set some variables that are commonly used during installation for staged
+# installs.
+MAKE_ENV+= DESTDIR=${STAGEDIR}
+.endif
+
+.if (${PKG_INSTALLATION_TYPE} == "overwrite") && (${PLIST_TYPE} != "static") && \
+ empty(STAGEDIR)
+PKG_FAIL_REASON+= "PLIST_TYPE must be \`\`static'' for \`\`overwrite'' packages."
+.endif
+
# Set PLIST_SUBST to substitute "${variable}" to "value" in PLIST
PLIST_SUBST+= OPSYS=${OPSYS} \
@@ -2299,6 +2315,15 @@
# post-configure. These targets typically edit the files generated by
# the do-configure target that are used during the build phase.
+.if !empty(STAGEDIR)
+_CONFIGURE_PREREQ+= subst-destdir
+SUBST_CLASSES+= destdir
+SUBST_MESSAGE.destdir= "Fixing destdir references."
+SUBST_FILES.destdir= `${FIND} . -name Makefile.in -print`
+SUBST_SED.destdir= \
+ -e 's|^DESTDIR[ ]*=.*|DESTDIR = ${STAGEDIR}|'
+.endif
+
.if defined(USE_PKGLOCALEDIR)
_PKGLOCALEDIR= ${PREFIX}/${PKGLOCALEDIR}/locale
REPLACE_LOCALEDIR_PATTERNS?= # empty
@@ -2676,11 +2701,11 @@
case $$dir in \
/*) ;; \
*bin|*bin/*|*libexec|*libexec/*) \
- ${INSTALL_PROGRAM_DIR} ${PREFIX}/$$dir ;; \
+ ${INSTALL_PROGRAM_DIR} ${STAGEDIR}${PREFIX}/$$dir ;; \
*man/*) \
- ${INSTALL_MAN_DIR} ${PREFIX}/$$dir ;; \
+ ${INSTALL_MAN_DIR} ${STAGEDIR}${PREFIX}/$$dir ;; \
*) \
- ${INSTALL_DATA_DIR} ${PREFIX}/$$dir ;; \
+ ${INSTALL_DATA_DIR} ${STAGEDIR}${PREFIX}/$$dir ;; \
esac; \
done
.endif # INSTALLATION_DIRS
@@ -2717,7 +2742,23 @@
# that the real config files and rc.d scripts aren't \
# listed in the PLIST. \
#
+.if !empty(STAGEDIR)
+ @${RM} -f ${PLIST}
+.endif
${_PKG_SILENT}${_PKG_DEBUG}cd ${.CURDIR} && ${MAKE} ${MAKEFLAGS} ${PLIST}
+.if !empty(STAGEDIR)
+. if defined(PKG_DEVELOPER) && !empty(PKG_DEVELOPER:M[Yy][Ee][Ss])
+ @${ECHO_MSG} "${_PKGSRC_IN}> Checking for files installed outside staging area"
+ ${_PKG_SILENT}${_PKG_DEBUG}cd ${.CURDIR} && ${MAKE} ${MAKEFLAGS} print-PLIST >${WRKDIR}/.unstaged_files
+ @if [ "$$(${WC} -l ${WRKDIR}/.unstaged_files | ${SED} -e 's| *||' | ${CUT} -d ' ' -f 1)" -gt 1 ]; then \
+ ${ECHO_MSG} "*** SOME FILES WERE INSTALLED OUTSIDE THE STAGING AREA!!!"; \
+ ${ECHO_MSG} "*** PLEASE REVIEW ${WRKDIR}/.unstaged_files MANUALLY"; \
+ fi
+. endif
+ @${ECHO_MSG} "${_PKGSRC_IN}> Dumping staged files into ${PREFIX}"
+ @cd ${STAGEDIR}${PREFIX} && ${PAX} -rwpe . ${PREFIX}
+ @${RM} -rf ${STAGEDIR}
+.endif
${_PKG_SILENT}${_PKG_DEBUG}cd ${.CURDIR} && ${MAKE} ${MAKEFLAGS} post-install-script
${_PKG_SILENT}${_PKG_DEBUG}newmanpages=`${EGREP} -h \
'^([^@/]*/)*man/([^/]*/)?(man[1-9ln]/.*\.[1-9ln]|cat[1-9ln]/.*\.0)(\.gz)?$$' \
@@ -5066,13 +5107,13 @@
[ "$$ignore" = "yes" ] || ${ECHO} "$$i"; \
done )
_GENERATE_PLIST= \
- ${FIND} ${PREFIX} \! -type d -print | ${SORT} | \
- ${SED} -e "s|^${PREFIX}/||" | \
+ ${FIND} ${STAGEDIR}${PREFIX} \! -type d -print | ${SORT} | \
+ ${SED} -e "s|^${STAGEDIR}${PREFIX}/||" | \
${_PLIST_IGNORE_CMD}; \
- ${FIND} ${PREFIX} -type d -print | ${SORT} -r | \
- ${GREP} -v "^${PREFIX}$$" | \
+ ${FIND} ${STAGEDIR}${PREFIX} -type d -print | ${SORT} -r | \
+ ${GREP} -v "^${STAGEDIR}${PREFIX}$$" | \
${_PLIST_IGNORE_CMD} | \
- ${SED} -e "s|^${PREFIX}/|@unexec ${RMDIR} -p %D/|" \
+ ${SED} -e "s|^${STAGEDIR}${PREFIX}/|@unexec ${RMDIR} -p %D/|" \
-e "s,$$, 2>/dev/null || ${TRUE},";
.else
_GENERATE_PLIST= ${CAT} ${_PLIST_SRC}; ${GENERATE_PLIST}
Index: bsd.pkg.install.mk
===================================================================
RCS file: /cvsroot/pkgsrc/mk/bsd.pkg.install.mk,v
retrieving revision 1.55
diff -u -r1.55 bsd.pkg.install.mk
--- bsd.pkg.install.mk 12 Jan 2004 12:58:23 -0000 1.55
+++ bsd.pkg.install.mk 7 Apr 2004 22:31:42 -0000
@@ -277,6 +277,7 @@
FILES_SUBST_SED= ${FILES_SUBST:S/=/@!/:S/$/!g/:S/^/ -e s!@/}
INSTALL_SCRIPTS_ENV= PKG_PREFIX=${PREFIX}
+INSTALL_SCRIPTS_ENV+= PKG_STAGEDIR=${STAGEDIR}
.PHONY: pre-install-script post-install-script
Index: install/install
===================================================================
RCS file: /cvsroot/pkgsrc/mk/install/install,v
retrieving revision 1.30
diff -u -r1.30 install
--- install/install 8 Jan 2004 14:19:29 -0000 1.30
+++ install/install 7 Apr 2004 22:31:43 -0000
@@ -126,18 +126,20 @@
-n "${MAKE_DIRS_PERMS}" -o -n "${OWN_DIRS_PERMS}" ]; then
eval set -- ${MAKE_DIRS} ${OWN_DIRS}
for dir; do
- if [ -d "${dir}" ]; then
+ if [ -d "${PKG_STAGEDIR}${dir}" ]; then
continue
fi
if [ "${_PKG_CONFIG}" = "YES" ]; then
- ${MKDIR} "${dir}"
- if [ "${PKG_INSTALLATION_TYPE}" = "pkgviews" ]; then
- case "${dir}" in
+ ${MKDIR} "${PKG_STAGEDIR}${dir}"
+ if [ -n "${PKG_STAGEDIR}" ]; then
+ :
+ elif [ "${PKG_INSTALLATION_TYPE}" = "pkgviews" ]; then
+ case "${PKG_STAGEDIR}${dir}" in
${PKG_PREFIX}|${PKG_PREFIX}/*) ;;
- *) ${ECHO} "${PKGBASE}" > "${dir}/.pkgsrc" ;;
+ *) ${ECHO} "${PKGBASE}" > "${PKG_STAGEDIR}${dir}/.pkgsrc" ;;
esac
else
- ${ECHO} "${PKGBASE}" > "${dir}/.pkgsrc"
+ ${ECHO} "${PKGBASE}" > "${PKG_STAGEDIR}${dir}/.pkgsrc"
fi
fi
done
@@ -146,19 +148,21 @@
dir="$1"; owner="$2"; group="$3"; mode="$4"
shift; shift; shift; shift
if [ "${_PKG_CONFIG}" = "YES" ]; then
- if [ ! -d "${dir}" ]; then
- ${MKDIR} "${dir}"
+ if [ -n "${PKG_STAGEDIR}" ]; then
+ :
+ elif [ ! -d "${PKG_STAGEDIR}${dir}" ]; then
+ ${MKDIR} "${PKG_STAGEDIR}${dir}"
if [ "${PKG_INSTALLATION_TYPE}" = "pkgviews" ]; then
- case "${dir}" in
+ case "${PKG_STAGEDIR}${dir}" in
${PKG_PREFIX}|${PKG_PREFIX}/*) ;;
- *) ${ECHO} "${PKGBASE}" > "${dir}/.pkgsrc" ;;
+ *) ${ECHO} "${PKGBASE}" > "${PKG_STAGEDIR}${dir}/.pkgsrc" ;;
esac
else
- ${ECHO} "${PKGBASE}" > "${dir}/.pkgsrc"
+ ${ECHO} "${PKGBASE}" > "${PKG_STAGEDIR}${dir}/.pkgsrc"
fi
- ${CHOWN} -R "${owner}" "${dir}"
- ${CHGRP} -R "${group}" "${dir}"
- ${CHMOD} -R "${mode}" "${dir}"
+ ${CHOWN} -R "${owner}" "${PKG_STAGEDIR}${dir}"
+ ${CHGRP} -R "${group}" "${PKG_STAGEDIR}${dir}"
+ ${CHMOD} -R "${mode}" "${PKG_STAGEDIR}${dir}"
fi
fi
done
On Wed, 7 Apr 2004 15:48:47 +0200
"Julio M. Merino Vidal" <jmmv@menta.net> wrote:
> Hi all,
>
> while testing pkgsrc under Linux, I hit some PLIST problems because some
> packages install different files depending on the build system. This is
> a common situation that has been found by many other people in the past,
> and for which we have ugly PLIST hacks to workaround it (see libgtop2,
> gnome2-applets, icu, for example). And I'm sure there are lots of
> problems like this remaining in our tree, which can only be fixed by
> trying packages and/or looking at bulk build results; eww.
>
> I wonder why we insist on keeping static PLISTs, where almost all other
> packaging systems out there use dynamic ones (e.g., Gentoo's portage,
> Debian packages, RPMs... and yes, even pkgviews).
>
> Having them could make maintenance a lot easier, specially when it comes
> to portability. No more PLIST hacks, no more need to handle shared
> directories with ugly ${RMDIR} lines, no more need for *-dirs packages...
> And also, lots of free inodes! ;)
>
> OTOH, with static PLISTs, one can easily grep for files, but this can be
> worked around by generating file lists when doing bulk builds (a thing we
> do very often). Think for example about DEBs; you can search individual
> files from the Debian's website...
>
> Yes, doing dynamic PLISTs mean staged installs as a prerequisite, and
> this is what, IIRC, everybody says is difficult. Almost all packages out
> there support it with no problems, specially if they use automake, as the
> DESTDIR feature is provided by it. Also, considering that other packaging
> systems use this feature, one can assume that they ask authors to fix
> their packages to support it (I, for one, got some messages about this for
> my own programs, when RPM packages were created for them by a third party).
>
> And, for those packages that don't support staged installs... well, we
> could either fix them (which could require many patches), or continue to
> use static PLISTs (which could mean that the staged install is disabled
> for that specific package). I don't see any problem in keeping them for
> packages that can't be easily converted.
>
> Am I missing the real reason for why we don't do this? If not, and people
> agrees, could we do it? (We could simply add support for staged installs
> and dynamic lists, but keep the default behavior as is now. Then, start
> converting packages one by one, when needed/wanted.)
>
> Thanks.
>
> --
> Julio M. Merino Vidal <jmmv@menta.net>
> The NetBSD Project - http://www.NetBSD.org/
>
--
Julio M. Merino Vidal <jmmv@menta.net>
The NetBSD Project - http://www.NetBSD.org/