Date:        Fri, 15 Oct 2004 23:27:04 +0100
    From:        Alistair Crooks <agc@pkgsrc.org>
    Message-ID:  <20041015222703.GF26364@nef.pbox.org>

  | What do you see as being the underlying problem?

The problem is that there are 2 different problems, pkgviews handles
one of them quite nicely, but either doesn't help the other at all, or
actually makes it slightly worse.

The first is the one you're thinking of I believe - that is, where a
new version of some library comes out (libfoo changes from version 1 to 2)
which the user doesn't much care about.   Sometime later some new package
appears, which uses libfoo.   Because the current version of libfoo is 2
when the new package is created, that's the one it depends upon (nothing
else has been verified to work).   This new package I want to install.
Without package views I get "make update hell" in that all 93 packages
I had installed that used libfoo now need to be removed and recompiled, so
I can do the upgrade that is needed to install the new package.

pkgviews handles this just fine - the new library, and new package, just
go in a different view than the existing stuff, and all is fine.

The second problem starts out the same way - a new version of libfoo
comes out (changes from version 1 to 2 - and note here, I'm just
counting, these aren't library "major" version numbers).   I see the
change list for what happened to libfoo, and it looks to me as if
there's a big been fixed that has been bothering me in many of the 93
packages I have installed that are using libfoo.   So, I want to actively
upgrade the library that is used in all 93 of them, but many of those
things are large, and I really don't want to (and there's absolutely
no need to) recompile any of them (that would be "package update hell".)
What I want to do is have libfoo recompiled, and reinstalled, and then
for everything to simply use the new one, which is the one real benefit
of using shared libraries - it is why we use the things in the first
place (without this, they're probably not worth the cost).

This one pkgviews doesn't help with at all.    This isn't a condemnation
of pkgviews, it is just not something it is designed to do.   Pkgviews
can make this a little worse, as there might actually be several old versions
of the old library installed (different views) all of which probably need
to be updated (and reconciled into now using a single view), rather than
juse the one that could exist without pkgviews (but this is relatively minor).

In theory, "make replace" should help here, but that's broken (or just
way too simplistic for its own good), and isn't safe to go using without
understanding much more about what will happen in each particular case
than your average pkgsrc user should ever need to understand.

pkg_hack on the other hand handled this just fine - it should be incorporated
into pkgsrc (after being renamed to something more appropriate, like
pkg_saviour) - not to replace pkgviews in any way, but to handle this
one specific problem, allowing libraries to safely be upgraded in place.

"Safely" that is, assuming that the library author is doing the right
thing - this doesn't prevent bugs - but nor does anything else, producing
bad library upgrades because the API is in some way altered, is just a bug,
just like producing upgrades that simply dump core in some cases.

Lastly on this topic Jeremy Reed branched out into a different area...

  | I think the problem is that BUILDLINK_DEPENDS and BUILDLINK_RECOMMENDED
  | aren't being used correctly. I think the complaint is that dependencies
  | required are bumped without any real reason. 

This is a different problem - it leads in some cases to the first
problem above, where the new package could have actually safely used
the old library version, but knowing when that's true is way too hard.
In particular ...

  | It would be nice to have a test machine available for pkgsrc developers that
  | always had the "old" dependencies installed (or packaged for quick
  | reinstalls).

is way too much to ask of anyone - quite aside from that there isn't "the"
old version, but lots of old versions (some of these libraries seem to get
updated every week).

The real problem here is the attempt to centralise the dependency information.

That's absurd - the DEPENDS info should be removed from all of the
buildlink files and moved into the individual package makefiles the
way it used to be.

There is *no* circumstance where a new version of some required package
can possibly require any other (existing and unchanging) package to now
require the new version.

If an existing package worked well enough to distribute with the old
version of the required package when that version was the only version,
then that cannot possibly be rendered untrue merely because a new version
has appeared - whatever the reason for the new version, including security
fixes.

Claiming that we must suddenly start forcing upgrades is simply nonsense.
By all means issue an advisory, put the broken package in the vulnerabilities
file, and do whatever else is reasonable to attempt to convince people that
they should upgrade (and having pkg_hack included with pkgsrc to make that
relatively easy for library upgrades won't hurt) - but that's as far as
it is reasonable to go.

Of course, I keep saying this over and over - no-one actually ever gives
any reasons why it isn't correct, but nothing changes...   And yes, I know
that going back and putting the dependency info in all of the packages
that have now had it removed is not going to be any small job, but at
least a start should be made (while retaining the info in the buildlink
files until everything else is fixed - but never (*never*) changing the
dependency info that is in the buildlink files, whatever happens to the
package that the buildlink file includes).

Lastly, all the above relates to source use of pkgsrc - binary packages
have a whole different set of problems, which should be solved differently.
I don't use binary packages a lot (that is, I make them but don't often
install them) - but as I recall, pkg_hack worked fine for them as well, for
its purpose.

The depends info in binary packages needs to be (MUST be) the version that
the package was actually compiled against, any time where the version installed
affects the compilation in any way (as in, libraries, or anything that
provides header files) - certainly it cannot allow installing using an
older version of the library (or whatever) - that's totally unsafe, yet
is, I believe, what we currently have.   [On the other hand, run time
dependencies where the thing compiled doesn't alter in the slightest,
it just wants to use the thing at run time - eg: as how exmh depends upon
nmh should allow the binrary package to use any version at least as new
as the version stated in the package to be required - and yes, this means
we need two different kinds of run time dependencies, not just one].

kre