Re: pkg_comp - asking for setup tips

[Robert Elz <kre%munnari.OZ.AU@localhost>]

    Date:        Mon, 13 Jul 2009 18:40:35 -0400
    From:        "Martin S. Weber" <Ephaeton%gmx.net@localhost>
    Message-ID:  <20090713224035.GE506%agamemnon.entropie.local@localhost>

  | Well, the perceived cost is more spam as well as doing a find on whole
  | LOCALBASE before and after installation of each package. This might be
  | fine for you quadcore on a RAID

I assure you I have nothing like that (and perhaps for this, more importantly,
not huge quantities of RAM either, so if the whole of the /usr/pkg inode
set was kept in cache it would be probably be affecting other things).
What's more, my build system is also moderately busy doing real work (which
is why I can't just upgrade it and not care if things break, which is why
it's stuck at a 4.0 based system - see one of my recent PR's ..)

What might help is that I frequently removeroot/makeroot so the actual
/usr/pkg that it is dealing with tends to be just the package and its 
dependencies, rather than everything useful that I might want to be using.

A much bigger PITA (that pkg_comp also invokes) is doing make clean
(with CLEANDEPENDS set) all the time - that tends to take far more time
and resources (from casual observation, not measurement) than anything
related to PKG_DEVELOPER, especially when playing in one of the packages
that gets involved with the gnome libraries.   The clean stage (even though
there's essentially nothing to clean - all the dependecnies were satisfied
from binary packages) I have seen take about 5 times longer than actually
building the package...

I have tried making CLEANDEPENDS go away, but so far, not successfully
(with pkg_comp, a "removeroot/makeroot" cycle can be much faster, and
certainly at least as effective, as one of those deep CLEANDEPENDS
recursive make sequences.

  | I don't see why the sloppyness of pkgsrc *developers* should come as another
  | level of punishment to pkgsrc *users* (There ARE more of the latter than of
  | the former, aren't there?).

On the rhetorical question, I'm actually not sure - I'm certainly just
a user, but in general, I'm not sure that there aren't more developers
than others ... and in any case I don't really see it as punishment, it
is helping make sure that the packages that are built are bug free
(since I'm using pkg_comp, as apparently are you, I really want the binary
packages to be correct, so if a PLIST error causes a file to get skipped
from being added to the binary package, I really want to know about it).

Sure, it would be better if all those problems were fixed before the
package was committed, but ...   (and I did say that just having PKG_DEVELOPER
default to on, with the ability to explicitly disable it, so if it
really does cost you too much you could do without it, ought to be
good enough.)

  | I've suggested following improvement to pkg_chk back in 2005 [1], using 
tsort
  | to determine the ordering. That'll work fine so long you don't change the
  | requirements of a pkg between its installed and pkgsrc version. Something
  | that'll likely happen if you only update between the -Q branches. So that
  | should be changed to use pkgsrc's Makefile's infos and basically you should
  | be fine. (Building packages in parallel isn't supported anyways).

Yes, I use tsort, but I generally have no installed version to check
against - remember, this is all using pkg_comp, the pkg_chk has to also
run inside the pkg_comp arena or it certainly gets the dependencies
wrong (the installed system is quite likely a quite different system
version than that being used inside the pkg_comp arena, and the dependencies
are quite possibly different out there).   The pkg_comp arena starts off
empty lots of times, so there's nothing installed to check against.

Running pkg_chk just raw on my main system was an early bug which certainly
caused the dependency list to be miscalculated, and then for stuff to
miss being correctly detected as a dependency, so the (t)sort order list
is missing data, and the effect you described certainly occurs.

All that interests me is whether my binary packages are up to date wrt
pkgsrc, all those that are not get (re-)moved (just moved for me, but
to somewhere they'll never be used by accident, I actually remove them
manually much later).   The dependency data comes from "make show-depends"
(or some such target) which I then run recursively on all the dependencies
until nothing new gets added to the list - then tsort it all.   This usually
takes a few minutes (depending of course on how long it has been since
it was last run, and so how many packages have actually been updated),
then I filter out all the stuff that's still up to date, and so installing
from the binary package I have is correct, then start rebuilding from
the packages with no dependencies out to the ones that have lots, all of
which should be rebuilt and available as binary packages (and often just
installed as well, though I do do a pkg_delete on every package, inside
the pkg_comp of course, immediately after building it, which allows some
sets of mutually exclusive packages, like all the vim-* packages, to
bet automatically rebuilt in one pass without problems).

I run all this a lot, which is why I tend to generate so many PR's on
(often) relatively trivial problems and often fairly quickly after the
problem is introduced - it's all automated enough that I can just start
it running and go home and forget it

kre

ps: I can send copies of my scripts to anyone who wants them, but don't ask
unless you're prepared to go through them line by line and fix stuff before
using - they're not intended for use by anyone but me, and are likely to
have my local (weird) pathnames and stuff embedded at the strangest places...