Subject: pkg/34974: pkgsrc sometimes compiles packages it doesn't need to compile
To: None <email@example.com, firstname.lastname@example.org,>
From: None <kre@munnari.OZ.AU>
Date: 11/02/2006 13:00:01
Note: There was a bad value `' for the field `Class'.
It was set to the default value of `sw-bug'.
>Synopsis: pkgsrc sometimes compiles packages it doesn't need to compile
>Arrival-Date: Thu Nov 02 13:00:00 +0000 2006
>Originator: Robert Elz
>Release: NetBSD 3.99.15 (pkgsrc current today)
Prince of Songkla University
System: NetBSD jade.coe.psu.ac.th 3.99.15 NetBSD 3.99.15 (GENERIC-1.696-20060125) #8: Wed Jan 25 04:59:39 ICT 2006 email@example.com:/usr/obj/current/kernels/JADE_ASUS i386
When using the "bin-install" pkgsrc target, and a reasonably complete
set of binary packages (so only packages that have been updated should
ever need to be rebuilt), pkgsrc sometimes recompiles and rebuilds a
package that does not need to be recompiled 9that is, for which a
binary package already exists, and which is unchanged).
I have seen this happen many times over the pasy year(s) - ie: this
is nothing new, I'm just getting around to remorting it as I noticed
it again today in a reasonably simple case.
To make this happen, remove (move) binary packages (if any) for
security/opencdk and security/wireshark - but have an existing
package for security/gnutls.
if you don't have the binary packages already, this is a hard state
to arrive in, as gnutls depends upon opencdk.
It happened for me today, as opencdk was updated a day or two ago,
so my old binary package was removed, and wireshard was updates
today (so its binary package was rmeoved), but gnutls hasn't been
updated for a month and a half now.
Then build (using the bin-install target) security/wireshark.
pkgsrc notices the dependency upon gnutls, and attempts to install the
bnary package (which is as it should). The pkg_install fails, as
gnutls requires opencdk, and there is no binary package available for
that one (yet). At this stage pkgsrc falls back to a source build,
and attempts the source build of gnutls - that again needs opencdk
as a dependency, but a source build of that succeeds without problems.
At this point, a binary install of gnutls would work just fine, but
pkgsrc has already abandoned that attempt (tried it, it failed...)
So goes ahead and rebuilds gnutls from source, making a new binary
package (which overwrites the one that was there, which is really not
a very nice thing to happen - and in some cases could be a real problem)
I suspect that the only way to fix this involves a rework of the way
that pkgsrc handles dependencies - instead of doing recursive builds,
it really needs to be iterative - that is, pkgsrc needs to find what
dependencies need to exist for the package being built (the one on the
original user command line), and then for all dependencies of that
package, hen combine that list into a sensible form (if A requires
C>=2 and B reqiores C>=3 and we need both A and B, then we need
C>=3 (the C>=2 just gets absorbed). Once we have that list, it needs
to be sorrted into depenndcncy order, after which the packages can be
built (whether that's a bin-install build, or just install, or
whaveer other has been requested) in the order that means no
(further) recursion happens - everything already exists when it is
needed. With a scheme like this, the build of wireshark would have
discovered that opencdk 9as well as gnutls) was needed, and that as
opencdk is needed by gnutls, opencdk would be built first. There
the pkg_install fails, as there is (was) no binary packate, and a
source build is done. Then gnutls comes next, now the binary install
succeeds, as all dependencies are already installed, and n new
compilation is required. After that, the compilation of wireshark
can be successfully accomplished.
NOTE: the answer here is **NOT** "gnutls really needsd to be
recompiled anyway so it could get correct dependency info in the
binary package" - or if that is the answer, then that just exposes
a similar (but much harder to fix, and much more stupid) but in
pkgsrc - as if instead of simply building wireshark, I had first
built opencdk (which I might have, and would have, had I gotten around
to updating my binary package collection a day earlier), then the
bvnary-install of the (old) gnutls package would have succeeded (it
might have issued a strange error-looking type of message about the
correct vesion of the binary package not being found - but it would
have successfully installed from the binary package - and would not
have been rebuilt. That's inconsistent with the (assumed here) view
that the binary package needed to be rebuilt.
Also note tha the 3 packages opencdk, gnutls, and wireshark (and
whetaver other dependencies they have which aren't relevant here)
are not the point f this PR - they just happened to form today's
example, so nothing specific to any of thse packages can possibly be
relevant to this PR (ie: bumping PKGREVISION of gnutls would be
a truly stupid response to this PR, whether or not it might happen
to be called for because of some other reasons).
pkgsrc probably needs a major reweite, and that should probably be
done in sh, rather than in make - it uses almost none of the
true facilities of make, and continuing to use it just because once,
many years ago, when the project started (as FreeBSD ports) make
seemed like the appropriate tool, does not cause that to continue
to be true... As a set of shell scripts I suspect pkgsrc could be
many times more flexible, faster, and MUCH simpler to maintain.