Proposing EARLY_PRINT_PLIST_AWK (was: please fix print-PLIST for python-3.x)

[Leonardo Taccari <leot%NetBSD.org@localhost>]

Thomas Klausner writes:
> Hi!
>

Hello Thomas!

> Can someone please fix the print-PLIST target so it produces a
> correctly sorted output even for python-3.x?
>
> It works fine for python-2.x but the non-munged and munged parts need
> different sorting for python-3.x.
> [...]

Elaborating a bit further the problem (only for the
record/completeness, feel free to skip it and go near the end of the
email if you are in hurry!)... print-PLIST basically generates the
file/directory lists via `find' and then pipe the corresponding output
to `sort'.

So, for example, given a package that installs a `foo.py', `foo.pyc'
and `foo.pyo' and the egg file in ${PYSITELIB}, the initial file list
generated by the `find' and then sorted will be something like (assuming
/usr/pkg as PREFIX), for PYTHON_VERSION_DEFAULT=27:

 some/path/to/.destdir/usr/pkg/./lib/python2.7/site-packages/foo-1.2.3-py2.7.egg-info
 some/path/to/.destdir/usr/pkg/./lib/python2.7/site-packages/foo.py
 some/path/to/.destdir/usr/pkg/./lib/python2.7/site-packages/foo.pyc
 some/path/to/.destdir/usr/pkg/./lib/python2.7/site-packages/foo.pyo

...with the final PLIST properly sorted:

 @comment $NetBSD$
 ${PYSITELIB}/${EGG_FILE}
 ${PYSITELIB}/foo.py
 ${PYSITELIB}/foo.pyc
 ${PYSITELIB}/foo.pyo

However, for Python 3.x, e.g. PYTHON_VERSION_DEFAULT=38 instead we will
have:

 some/path/to/.destdir/usr/pkg/./lib/python2.7/site-packages/__pycache__/foo.cpython-38.opt-1.pyc
 some/path/to/.destdir/usr/pkg/./lib/python2.7/site-packages/__pycache__/foo.cpython-38.pyc
 some/path/to/.destdir/usr/pkg/./lib/python2.7/site-packages/foo-1.2.3-py3.8.egg-info
 some/path/to/.destdir/usr/pkg/./lib/python2.7/site-packages/foo.py

...and - always for Python 3.x - the corresponding PLIST will not
be properly sorted like the one generated with
PYTHON_VERSION_DEFAULT=27:

 @comment $NetBSD$
 ${PYSITELIB}/foo.pyo
 ${PYSITELIB}/foo.pyc
 ${PYSITELIB}/${EGG_FILE}
 ${PYSITELIB}/foo.py


To address this problem I would like to introduce
EARLY_PRINT_PLIST_AWK that is similar to PRINT_PLIST_AWK but actually
do the substitutions before sorting the file/directory list.

The attached patch in this email implements it and adjust
lang/python/extension.mk to use it so that PLIST generated via
print-PLIST are sorted also for Python 3.x packages.

I would like to continue testing it for other 2 weeks and if there are
no negative feedbacks/comments nor problems I would like to commit it
in three separate commits (one to mk/, another to mention it in The
pkgsrc Guide and the latter to adjust lang/python/extension.mk).


Any feedbacks/comments are welcomed!

Index: lang/python/extension.mk
===================================================================
RCS file: /cvsroot/pkgsrc/lang/python/extension.mk,v
retrieving revision 1.56
diff -u -p -r1.56 extension.mk
--- lang/python/extension.mk	2 May 2019 22:06:15 -0000	1.56
+++ lang/python/extension.mk	27 May 2020 10:39:23 -0000
@@ -74,10 +74,10 @@ FILES_SUBST+=	PYVERSSUFFIX=${PYVERSSUFFI
 .if empty(_PYTHON_VERSION:M2?)
 PLIST_AWK+=		-f ${PKGSRCDIR}/lang/python/plist-python.awk
 PLIST_AWK_ENV+=		PYVERS="${PYVERSSUFFIX:S/.//}"
-PRINT_PLIST_AWK+=	/^[^@]/ && /[^\/]+\.py[co]$$/ {
-PRINT_PLIST_AWK+=	gsub(/__pycache__\//, "")
-PRINT_PLIST_AWK+=	gsub(/opt-1\.pyc$$/, "pyo")
-PRINT_PLIST_AWK+=	gsub(/\.cpython-${_PYTHON_VERSION}/, "")}
+EARLY_PRINT_PLIST_AWK+=	/^[^@]/ && /[^\/]+\.py[co]$$/ {
+EARLY_PRINT_PLIST_AWK+=	gsub(/__pycache__\//, "")
+EARLY_PRINT_PLIST_AWK+=	gsub(/opt-1\.pyc$$/, "pyo")
+EARLY_PRINT_PLIST_AWK+=	gsub(/\.cpython-${_PYTHON_VERSION}/, "")}
 .endif
 
 DISTUTILS_BUILDDIR_IN_TEST_ENV?=	no
Index: mk/plist/print-plist.mk
===================================================================
RCS file: /cvsroot/pkgsrc/mk/plist/print-plist.mk,v
retrieving revision 1.35
diff -u -p -r1.35 print-plist.mk
--- mk/plist/print-plist.mk	18 Apr 2020 10:54:21 -0000	1.35
+++ mk/plist/print-plist.mk	27 May 2020 10:39:23 -0000
@@ -94,6 +94,9 @@ print-PLIST:
 	*)		genlinks=0 ;;					\
 	esac;								\
 	${_PRINT_PLIST_FILES_CMD}					\
+	 | ${AWK} '							\
+		${EARLY_PRINT_PLIST_AWK}				\
+		{ print $$0; }'						\
 	 | ${_PRINT_PLIST_LIBTOOLIZE_FILTER}				\
 	 | ${SORT}							\
 	 | ${AWK} '							\
@@ -124,6 +127,9 @@ print-PLIST:
 		{ print $$0; }'
 	${RUN}\
 	for i in `${_PRINT_PLIST_DIRS_CMD}				\
+			| ${AWK} '					\
+				${EARLY_PRINT_PLIST_AWK}		\
+				{ print $$0; }'				\
 			| ${SORT} -r					\
 			| ${AWK} '					\
 				/emul\/linux\/proc/ { next; }		\