Subject: and recursive broken marks
To: None <,>
From: Todd Vierling <>
List: pkgsrc-bulk
Date: 10/15/2005 11:20:09
You two have the most infrastructure commits here, so I figured at least one
of you would be able to provide some thoughts on these points, in case you
aren't subscribed to pkgsrc-bulk.

One of the annoyances about Interix's fork() problems is that invoking bmake
recursively is *expensive*.  I'd like to avoid doing that where feasible --
particularly in deep loops.  Today I've been observing some really bad
behavior in the "marking package that requires XXX as broken" phase; it can
take many hours to complete if the list is very long.  This is mostly thanks
to the following block:

pkgerr="-1"; \
pkgignore=`(cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} show-var VARNAME=PKG_FAIL_REASON)`; \
pkgskip=`(cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} show-var VARNAME=PKG_SKIP_REASON)`; \
if [ ! -z "$${pkgignore}$${pkgskip}" -a ! -f ${PKGSRCDIR}/$$pkgdir/${BROKENFILE} ]; then \
         ${ECHO_MSG} "BULK> $$pkgname ($$pkgdir) may not be packaged because:" >> ${PKGSRCDIR}/$$pkgdir/${BROKENFILE};\
         ${ECHO_MSG} "BULK> $$pkgignore" >> ${PKGSRCDIR}/$$pkgdir/${BROKENFILE};\
         ${ECHO_MSG} "BULK> $$pkgskip" >> ${PKGSRCDIR}/$$pkgdir/${BROKENFILE};\
        if [ -z "`(cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} show-var VARNAME=BROKEN)`" ]; then \
                pkgerr="0"; \
        else \
                pkgerr="1"; \
        fi; \
fi; \

Is the extra check to see if the package is already PKG_FAIL, PKG_SKIP, or
BROKEN necessary?  Note that there are three bmake invocations here, and
that's killing me when something with a crapload of dependencies, such as
groff, breaks.  For the currently running build, it started marking groff
dependencies at 1:48AM this morning, and was only about half done when I
^C'd it and restarted the bulk build.

The only thing I can think that this does is change the "breaks" column of
the final report, by lowering the numbers for packages that are themselves
broken -- but is that really needed?  After all, it's dependency breakage,
and the only real reason for the "breaks" column is to indicate a severity
of the dependency breakage.  If that number includes packages that are
themselves broken, I don't really see that as a vital problem.  Eliminating
this looped check should speed up bulk builds on all platforms, not just
Interix, but perhaps not as dramatically on others.

Now, I don't really understand the magic going on with the output
$PKGSRCDIR/$BROKENFILE data from just this block of code, but maybe you
understand its format more readily.  Are the values there used for anything
other than the final bulk report?


One of the additional things I plan to do to reduce recursive bmake
multiplicity is add a "show-vars-eval" target, which is similar to
"show-var"/"show-vars" but outputs backtick evaluatable shell assignment
code for multiple variables in only one invocation, e.g.:

$ make show-vars-eval VARS='BROKEN:pkgbroken PKG_FAIL_REASON:pkgfail'
broken=This\ package\ is\ broken.
pkgfail=This\ package\ fails\ miserably.

...using :Q on the output, of course, to escape spaces and metas.


And one final note:  Does USE_BULK_CACHE=no still work, or should I expect
that to break spectacularly now?  It's been forced =yes in mk/bulk/build for
a long time, so I don't know if setting it =no in a full bulk build will
work at all.  8-)

-- Todd Vierling <> <> <>