Re: Version requirements for dependancies

[Robert Elz <kre%munnari.OZ.AU@localhost>]

    Date:        Mon, 10 Nov 2008 13:38:33 -0500
    From:        Chris Ross <cross+netbsd%distal.com@localhost>
    Message-ID:  <1226342313.3523.11.camel%zalamar.mm-corp.net@localhost>

  | I'm not sure where the fault lies, here, but I think something
  | somewhere should tell mail/spamprobe that it doesn't always *require*
  | the most recent version available.  Is there anything that I can do to
  | request this, or investigate it more?

You're absolutely right, the dependency system in pkgsrc is about as
broken as it is possible to be, while still kind of working (though at
a cost to all of the users).

In reply, Jeremy C. Reed <reed%reedmedia.net@localhost> said:

  | The policy is a new version is required if the ABI changes, making  existing
  | dependencies to no longer work with new packages (or new  dependencies to no
  | longer work with old packages).

  | Also if an API changes (meaning it needs newer dependency), then that will
  | be required too. 

You misinterpreted the problem.   You're assuming Chris is asking about
needing to rebuild spamprobe when db4 is updated.   That isn't it at all.

Rather, spamprobe is being updated, he wants the latest version of that
(quite logically, that's the kind of package you're likely to always want
the latest version of, whereas for many others, if the version you have works
for what you need, and has no bugs (or security issues) of concern, then
updating just because there's something newer is not always desirable.)

The problem is that when he goes to update spamprobe, pkgsrc decides that
he must also upgrade db4.  It isn't spamprobe (or its authors, or its
pkg, or its pkgsrc maintainer) who have decided that, rather it is the
db4 pkgsrc maintainer, almost certainly without even considering spamprobe,
and probably without even knowing that spamprobe exists, or uses db4, who
has decreed that spamprobe must use the newer version.  So, the old db4
needs to get deleted, along with all kind of other stuff that didn't need
upgrading at all (and for which it is possible the pkgsrc packages may even
be temporarily broken) and all recompiled, with the new db4.    Absurd.

Regardless of whether the ABI has changed (that is no reason for requiring
the new version, if spamprobe could be built against the old one yesterday,
and link against it, and use it, the same will be true today - as we're
compiling from source here, the ABI is largely irrelevant), or the API
has changed (in that case, since we have no information that spamprobe
source has actually been upgraded to meet the new API requirements,
attempting to compile it with the new db4 is more likely to fail than
using the old one where we know spamprobe has been used successfully).

Either way, the old db4 would do, and is perhaps even better than, the
new one (of course, the new one may have bug or performance fixes that
might make it preferable, but certainly not essential.)

The problem here is that pkgsrc is optimised towards making it easier for
the pkgsrc developer, and not making things correct, or easier for the
end user.   Rather than requiring every package that uses db4 to
explicitly state which version of db4 is the minimum that works, and
perhaps which is the latest known to work, pkgsrc puts that information
in one place (by default) - in db4's buildlink4.mk file, where everyone
else can obtain it - which makes packages like spamprobe much easier to
create, "db4 is needed, .include db4/buildlink3.mk, we're done" - which
is great for the developer, really easy to get things "right".

Except it isn't right, as there is no way that db4's buildlink file can
possibly know which version of db4 is appropriate for any other package in
pkgsrc, let alone for all or them - even less that all of them require
the same one.

Now, before everyone jumps up and down, yes, I know, packages in pkgsrc
can override the buildlink3 version requirement, and explicitly say which
version they require, so Chris, you can file a PR against spamprobe, and
ask that it do exactly that - have it specify which version of db4 is
required, then the next time you upgrade spamprobe, you won't get this
problem (or perhaps the time after that, as the version pkgsrc starts
out requiring might still be more than the version you have installed now.)

The problem is that this should be mandatory for all packages.   If a
package uses some other as a dependency, the author of that package should
be stating, explicitly, just which version of the dependency is required.
The dependency info in buildlink3.mk files MUST go!   All that should
remain there is incompatibility info - what db4 does know, is that its
ABI changed between version x and y, or that its API altered between versions
p and q.  That can go in the buildlink3.mk files - then if a binary is
being installed which was linked against version x, and version y of db4
is there instead, the install can fail, and require recompilation.   Or
if a package is being compiled from source, and says it requires >= db4 
version p, and version q (q>p) is installed, the compile can at least warn
that the API has changed, and the dependent package might need source
changes (I wouldn't auto fail that case, as most API changes are visible to
only a small subset of client applications.)

In reply to Jeremy's message, Steven M. Bellovin 
<smb%cs.columbia.edu@localhost> said:

  | When I create a package, I sometimes set the dependencies to what I have on
  | my system at that point -- that's what I've built against, that's what I've
  | tested against, that's what I know works.  It might work with some earlier
  | versions, but I can't easily test that.

That's fine, and that's exactly what you should do - provided that you state
it explicitly in your pkgsrc Makefile, and don't rely upon getting the
required version information out of some other dependency's buildlink3.mk
file.  No more than that can reasonably be asked.   What matter is that that
is the version you tested against - not only don't you know that it works
with earlier versions, you also don't know that it works with later ones.

As we have it now, the earlier version problem is mostly OK, but (unless
you have it explicit in your Makefile) you're at the mercy of the dependencies,
which will eventually not only claim that a never version of their package
works for yours, but even mandate that a newer version be used - all with
testing by absolutely no-one.   Again, absurd.

For earlier versions you're not testing against, if someone else has such
a version installed, and suspect that your package will work OK with that
version, they can test it by overriding the version setting in your package's
Makefile when they build your package.   If it fails, well, they knew that
was a possibility when they started.   If it works, then they can file a PR,
and request that the required version be changed lower, indicating that
they have tested a lower version and it all works fine.   Unless you have
some knowledge to the contrary (some hidden bug that is rarely seen that
you know of in that earlier version perhaps) then you're likely to simply
update your package, relying on the other user's testing being no less than
you would have done had you had the older version of the dependency when
you created the package.

Much the same when a newer version appears - if it claims to be API
compatible, there's nothing to do, your package should compile and link
against either the old or new versions.  If the API has changed then your
package may need updating - or might not - depending upon whether you use
the particular API that changed or not.   Only investigation will figure
that out.


Before I finish, let me also say that I know this is all not just trivial.

It isn't merely a matter of getting rid of the (bogus) dependency info
in buildlink3.mk files, and adding it to every (yes *every*) package that
uses the buildlink3.mk files (so lots of work for every pkgsrc maintainer,
though a lot of it could be automated for the change by just starting with
the assumption that the current buildlink3.mk requirement is the correct thing
for every package that includes the buildlink3.mk and doesn't override it,
which is what we're doing now anyway) - the bigger problems come with dealing
with binary packages, especially those with multiple levels of libraries,
with intermixed dependencies.   It is vital that all dependencies have been
built with the exact same versions of everything they depend upon.
Currently I don't believe this is checked (and I'm pretty sure I have a
whole set of binary packages that have been linked against a mixed set of
Xfree and Xorg libraries in obscene ways, because I keep altering my
X11_VERSION setting backwards and forwards...) - rather pkgsrc is
relying upon its current behaviour of "force everyone to update everything
to the latest version, all at once" which more or less (mostly) ensures that
everything is getting compiled with a consistent set of underlying libraries.
But that's really all just fluke - but if we alter things the way I'm 
suggesting above (ie: make pkgsrc correct) that "fluke" will go way out the
window, and require a real solution to the mixed library dependency problem.

That is, at the very least, check all dependent packages and make sure that
they were all built with the same (exact same) set of dependency versions.
Even better, do that for shared library packages only (dependencies which
are shared libraries) or their equivalent (interpreters are another similar
problem I think) and don't worry about the others.

kre