Subject: pkg/34974: pkgsrc sometimes compiles packages it doesn't need to compile
To: None <,,>
From: None <kre@munnari.OZ.AU>
List: pkgsrc-bugs
Date: 11/02/2006 13:00:01
	Note: There was a bad value `' for the field `Class'.
	It was set to the default value of `sw-bug'.

>Number:         34974
>Category:       pkg
>Synopsis:       pkgsrc sometimes compiles packages it doesn't need to compile
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    pkg-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Nov 02 13:00:00 +0000 2006
>Originator:     Robert Elz
>Release:        NetBSD 3.99.15 (pkgsrc current today)
	Prince of Songkla University
System: NetBSD 3.99.15 NetBSD 3.99.15 (GENERIC-1.696-20060125) #8: Wed Jan 25 04:59:39 ICT 2006 i386
Architecture: i386
Machine: i386
	When using the "bin-install" pkgsrc target, and a reasonably complete
	set of binary packages (so only packages that have been updated should
	ever need to be rebuilt), pkgsrc sometimes recompiles and rebuilds a
	package that does not need to be recompiled 9that is, for which a
	binary package already exists, and which is unchanged).

	I have seen this happen many times over the pasy year(s) - ie: this
	is nothing new, I'm just getting around to remorting it as I noticed
	it again today in a reasonably simple case.

	To make this happen, remove (move) binary packages (if any) for
	security/opencdk and security/wireshark - but have an existing
	package for security/gnutls.

	if you don't have the binary packages already, this is a hard state
	to arrive in, as gnutls depends upon opencdk.

	It happened for me today, as opencdk was updated a day or two ago,
	so my old binary package was removed, and wireshard was updates
	today (so its binary package was rmeoved), but gnutls hasn't been
	updated for a month and a half now.

	Then build (using the bin-install target) security/wireshark.

	pkgsrc notices the dependency upon gnutls, and attempts to install the
	bnary package (which is as it should).  The pkg_install fails, as
	gnutls requires opencdk, and there is no binary package available for
	that one (yet).   At this stage pkgsrc falls back to a source build,
	and attempts the source build of gnutls - that again needs opencdk
	as a dependency, but a source build of that succeeds without problems.

	At this point, a binary install of gnutls would work just fine, but
	pkgsrc has already abandoned that attempt (tried it, it failed...)
	So goes ahead and rebuilds gnutls from source, making a new binary
	package (which overwrites the one that was there, which is really not
	a very nice thing to happen - and in some cases could be a real problem)

	I suspect that the only way to fix this involves a rework of the way
	that pkgsrc handles dependencies - instead of doing recursive builds,
	it really needs to be iterative - that is, pkgsrc needs to find what
	dependencies need to exist for the package being built (the one on the
	original user command line), and then for all dependencies of that
	package, hen combine that list into a sensible form (if A requires
	C>=2 and B reqiores C>=3 and we need both A and B, then we need
	C>=3 (the C>=2 just gets absorbed).  Once we have that list, it needs
	to be sorrted into depenndcncy order, after which the packages can be
	built (whether that's a bin-install build, or just install, or
	whaveer other has been requested) in the order that means no
	(further) recursion happens - everything already exists when it is
	needed.  With a scheme like this, the build of wireshark would have
	discovered that opencdk 9as well as gnutls) was needed, and that as
	opencdk is needed by gnutls, opencdk would be built first.   There
	the pkg_install fails, as there is (was) no binary packate, and a
	source build is done.  Then gnutls comes next, now the binary install
	succeeds, as all dependencies are already installed, and n new
	compilation is required.  After that, the compilation of wireshark
	can be successfully accomplished.

	NOTE: the answer here is **NOT** "gnutls really needsd to be
	recompiled anyway so it could get correct dependency info in the
	binary package" - or if that is the answer, then that just exposes
	a similar (but much harder to fix, and much more stupid) but in
	pkgsrc - as if instead of simply building wireshark, I had first
	built opencdk (which I might have, and would have, had I gotten around
	to updating my binary package collection a day earlier), then the
	bvnary-install of the (old) gnutls package would have succeeded (it
	might have issued a strange error-looking type of message about the
	correct vesion of the binary package not being found - but it would
	have successfully installed from the binary package - and would not
	have been rebuilt.   That's inconsistent with the (assumed here) view
	that the binary package needed to be rebuilt.

	Also note tha the 3 packages opencdk, gnutls, and wireshark (and
	whetaver other dependencies they have which aren't relevant here)
	are not the point f this PR - they just happened to form today's
	example, so nothing specific to any of thse packages can possibly be
	relevant to this PR (ie: bumping PKGREVISION of gnutls would be
	a truly stupid response to this PR, whether or not it might happen
	to be called for because of some other reasons).

	pkgsrc probably needs a major reweite, and that should probably be
	done in sh, rather than in make - it uses almost none of the
	true facilities of make, and continuing to use it just because once,
	many years ago, when the project started (as FreeBSD ports) make
	seemed like the appropriate tool, does not cause that to continue
	to be true...   As a set of shell scripts I suspect pkgsrc could be
	many times more flexible, faster, and MUCH simpler to maintain.