pkgsrc-Bulk archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: bsd.bulk-pkg.mk and recursive broken marks

Todd Vierling wrote:

You two have the most infrastructure commits here, so I figured at least one
of you would be able to provide some thoughts on these points, in case you
aren't subscribed to pkgsrc-bulk.

One of the annoyances about Interix's fork() problems is that invoking bmake
recursively is *expensive*.  I'd like to avoid doing that where feasible --
particularly in deep loops.  Today I've been observing some really bad
behavior in the "marking package that requires XXX as broken" phase; it can
take many hours to complete if the list is very long.  This is mostly thanks
to the following block:

=====
pkgerr="-1"; \
pkgignore=`(cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} show-var 
VARNAME=PKG_FAIL_REASON)`; \
pkgskip=`(cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} show-var 
VARNAME=PKG_SKIP_REASON)`; \
if [ ! -z "$${pkgignore}$${pkgskip}" -a ! -f 
${PKGSRCDIR}/$$pkgdir/${BROKENFILE} ]; then \
         ${ECHO_MSG} "BULK> $$pkgname ($$pkgdir) may not be packaged because:" 
>> ${PKGSRCDIR}/$$pkgdir/${BROKENFILE};\
         ${ECHO_MSG} "BULK> $$pkgignore" >> 
${PKGSRCDIR}/$$pkgdir/${BROKENFILE};\
         ${ECHO_MSG} "BULK> $$pkgskip" >> ${PKGSRCDIR}/$$pkgdir/${BROKENFILE};\
        if [ -z "`(cd ${PKGSRCDIR}/$$pkgdir && ${MAKE} show-var 
VARNAME=BROKEN)`" ]; then \
                pkgerr="0"; \
        else \
                pkgerr="1"; \
        fi; \
fi; \
=====

Is the extra check to see if the package is already PKG_FAIL, PKG_SKIP, or
BROKEN necessary?  Note that there are three bmake invocations here, and
that's killing me when something with a crapload of dependencies, such as
groff, breaks.  For the currently running build, it started marking groff
dependencies at 1:48AM this morning, and was only about half done when I
^C'd it and restarted the bulk build.

The only thing I can think that this does is change the "breaks" column of
the final report, by lowering the numbers for packages that are themselves
broken -- but is that really needed?  After all, it's dependency breakage,
and the only real reason for the "breaks" column is to indicate a severity
of the dependency breakage.  If that number includes packages that are
themselves broken, I don't really see that as a vital problem.  Eliminating
this looped check should speed up bulk builds on all platforms, not just
Interix, but perhaps not as dramatically on others.

Now, I don't really understand the magic going on with the output
$PKGSRCDIR/$BROKENFILE data from just this block of code, but maybe you
understand its format more readily.  Are the values there used for anything
other than the final bulk report?
It has been several years now, but I think it is only for the final report. I vaguely recall that had something to do with not wanting all the compat_linux packages to make the build report look bad on pmax or some such thing. 


=====

And one final note:  Does USE_BULK_CACHE=no still work, or should I expect
that to break spectacularly now?  It's been forced =yes in mk/bulk/build for
a long time, so I don't know if setting it =no in a full bulk build will
work at all.  8-)
It will probably still "work" although my feeling was it never worked correctly to begin with. The whole bulk cache thing started becuase
a) the dependency ordering was wrong. The bulk code would decide on a rebuild if dependencies had been repackaged, but unless you built in the right order, you could run the same build over and over on the same source tree and keep getting things rebuilt.
b) without the bulk cache, there were zillions (ok, 100's of thousands) of calls to make while recursively processing the dependencies. I wanted to call make once per package and then have the entire up/down flattened tree avaialble for quick lookup
c) the recursive broken marks are actually faster than the old way. In the old way, if something like png were broken, you might go to build a package, install 27 dependencies, find out that #28 (png) was broken and then deinstall all 27. That took forever on the pmax.
d) a restart of a bulk build was _painful_ and involved potentially thousands of make calls.
e) it became easier to deinstall packages not listed as dependencies at the start of the build of a particular package instead of deinstalling all packages at the end of a the build of a particular package. While somewhat hard to quantify, this led to less instances of deinstalling perl and promptly reinstalling it. When you get into the kde or gnome packages on a slower system with a slower drive, this can be a big help.
so, not using the bulk cache probably still works, but you'll be left with these issues. 

-Dan
Home | Main Index | Thread Index | Old Index