bsd.bulk-pkg.mk and recursive broken marks

[Todd Vierling <tv%duh.org@localhost>]

You two have the most infrastructure commits here, so I figured at least one
of you would be able to provide some thoughts on these points, in case you
aren't subscribed to pkgsrc-bulk.

One of the annoyances about Interix's fork() problems is that invoking bmake
recursively is *expensive*.  I'd like to avoid doing that where feasible --
particularly in deep loops.  Today I've been observing some really bad
behavior in the "marking package that requires XXX as broken" phase; it can
take many hours to complete if the list is very long.  This is mostly thanks
to the following block:

=====
pkgerr="-1"; \
pkgignore=`(cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} show-var 
VARNAME=PKG_FAIL_REASON)`; \
pkgskip=`(cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} show-var 
VARNAME=PKG_SKIP_REASON)`; \
if [ ! -z "$${pkgignore}$${pkgskip}" -a ! -f 
${PKGSRCDIR}/$$pkgdir/${BROKENFILE} ]; then \
         ${ECHO_MSG} "BULK> $$pkgname ($$pkgdir) may not be packaged because:" 
>> ${PKGSRCDIR}/$$pkgdir/${BROKENFILE};\
         ${ECHO_MSG} "BULK> $$pkgignore" >> 
${PKGSRCDIR}/$$pkgdir/${BROKENFILE};\
         ${ECHO_MSG} "BULK> $$pkgskip" >> ${PKGSRCDIR}/$$pkgdir/${BROKENFILE};\
        if [ -z "`(cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} show-var 
VARNAME=BROKEN)`" ]; then \
                pkgerr="0"; \
        else \
                pkgerr="1"; \
        fi; \
fi; \
=====

Is the extra check to see if the package is already PKG_FAIL, PKG_SKIP, or
BROKEN necessary?  Note that there are three bmake invocations here, and
that's killing me when something with a crapload of dependencies, such as
groff, breaks.  For the currently running build, it started marking groff
dependencies at 1:48AM this morning, and was only about half done when I
^C'd it and restarted the bulk build.

The only thing I can think that this does is change the "breaks" column of
the final report, by lowering the numbers for packages that are themselves
broken -- but is that really needed?  After all, it's dependency breakage,
and the only real reason for the "breaks" column is to indicate a severity
of the dependency breakage.  If that number includes packages that are
themselves broken, I don't really see that as a vital problem.  Eliminating
this looped check should speed up bulk builds on all platforms, not just
Interix, but perhaps not as dramatically on others.

Now, I don't really understand the magic going on with the output
$PKGSRCDIR/$BROKENFILE data from just this block of code, but maybe you
understand its format more readily.  Are the values there used for anything
other than the final bulk report?

=====

One of the additional things I plan to do to reduce recursive bmake
multiplicity is add a "show-vars-eval" target, which is similar to
"show-var"/"show-vars" but outputs backtick evaluatable shell assignment
code for multiple variables in only one invocation, e.g.:

$ make show-vars-eval VARS='BROKEN:pkgbroken PKG_FAIL_REASON:pkgfail'
broken=This\ package\ is\ broken.
pkgfail=This\ package\ fails\ miserably.

...using :Q on the output, of course, to escape spaces and metas.

=====

And one final note:  Does USE_BULK_CACHE=no still work, or should I expect
that to break spectacularly now?  It's been forced =yes in mk/bulk/build for
a long time, so I don't know if setting it =no in a full bulk build will
work at all.  8-)

-- 
-- Todd Vierling <tv%duh.org@localhost> <tv%pobox.com@localhost> 
<todd%vierling.name@localhost>