Date:        Sun, 25 Jan 2004 23:28:34 +0100
    From:        Quentin Garnier <netbsd@quatriemek.com>
    Message-ID:  <20040125232834.4d4c5377.netbsd@quatriemek.com>

  | >   | It doesn't require two separate versions.  If a package's DEPENDS
  | >   | contains:
  | >   | 
  | >   | 	zlib>=1.1.4nb1:../../devel/zlib
  | >   | 	zlib>=1.2.1:../../devel/zlib
  | >   | 
  | >   | then it simply states that any zlib that is used as a dependency for
  | >   | the package must satisfy _both_ of the restrictions listed in
  | >   DEPENDS.
  | > 
  | > But why would a package's depends say that?   Surely that would be a
  | > bug, it needs only the 2nd of those two, doesn't it?
  | 
  | Well, the 2nd is the one to consider.

Where do the two of them come from?   If they're both in the package's
Makefile, then yes, I agree, the second is the one that matters - but the
right way to achieve that is simplt to delete the irrelevant first one
from the makefile isn't it.

If there's just one in the Makefile, then where is the other one coming
from (which ever of the two is the "other")?   If package X says that
"I need zlib at least 1.1.4nb1" then that's what it means, and that is
all it requires.

  | And the change you're commenting
  | was precisely made in order to be sure such a dependency is preserved
  | through the buildlink inclusion process.

But this should be irrelevant - buildlink files should not (ever) be
specifying dependency version information - they cannot possibly know.

Now if it happened that you were talking about what should end up in
a constructed binary package - in the case where a package depends on
1.1.4nb1 (or more) and a dependency depends upon 1.2.1 (or more) of the
same thing (zlib in this case) - then the answer isn't 1.2.1, it is
"whatever version of zlib is actually installed when the package was
built".

That is, if the system has 1.3.0 installed (I have no idea if zlib has
such a version, this is fiction for the purposes of example) and a package
is built, then the binary package needs to say "zlib=1.3.0" (or just
perhaps "zlib>=1.3.0") - the constructed binary won't necessarily work
with 1.2.1 (let alone 1.1.4nb1) as downgrading libraries that way isn't
generally possible - regardless of what versions would have been OK to
compile against.

  | Sure, we could require people to rebuild all installed packages that don't
  | match the version in pkgsrc, but trust me on that, we would get a lot of
  | angry users.

Who said anything like that?   Certainly wasn't me - I want less
recompiling, not more.

  | We need a pkgsrc package to expose a minimum requirement in its buildlink2
  | file to say one of two things (and that's exactly what Johnny was saying):
  | 
  |   o Packages linked against earlier versions are unsafe or cannot be
  |     trusted

Rubbish.   You cannot possibly say that as any kind of general statement.
No matter how broken/unsafe the dependency package is, my package
may be using the thing in an entirely safe way.   Only each individual
package can possibly know if it needs to be relinked against a new
version of some dependency because of what was fixed there.

For sure, having this claim made in the buildlink file is the cheap
way out - it means one line gets changed, and every package that
uses that one now demands recompiling with the newer version - but it
is the wrong way.

  |   o Packages using that version of the buildlink file might not link with
  |     the installed version

I'm not sure what that one means, so I can't comment on that.  (It
seems like it has some relationship with major version bumps in libraries,
in which case it simply makes no sense - my package linked against the
old library yesterday, it will still link against it today, if the old
one is still what is installed - if a newer one is installed, then my
package might, or might not, still work - one hopes it usually will, but
there are no guarantees - but in any case, the compile time dependency is
still what my package claims it is, not what you believe that I really
should be using for whatever reason).

  | Of course, in an ideal world, each package should describe in the Makefile
  | all of its requirements.

Oh good - we agree, that's the point I am trying to make.  What's more,
per-buildlink, that's exactly what packages did.

  | Frankly, that would not be maintainable.

But I don't agree here.   It worked fine before.

  | The current "security fix or major bump" motto is supposed to be enough in
  | most cases (as long as software authors use library versions in a sane
  | way), but a package can still specify a more restrictive requirement, e.g.
  | when a minor bump of a library introduces a new functionality that the
  | package uses.

Yes, but that's not what is of interest to me - I want a package to
be able to specify a lesser version than someone or other believes
I should have foisted upon me.   I am pretty sure that jlam understands
this point...

  | Well, that's again another debate, and would require quite an overhelm of
  | pkgsrc.

Yes, but that is what is happening - in fact, happening all the time it
seems (and that's a good thing).   I'm just trying to make a point about
where pkgsrc should be aiming - I'm not expecting any kind of sudden
conversion over the coming weekend!

  | Right now we need buildlink to handle dependencies that way, and
  | we can't afford to require the rebuild of everything and its mother all
  | the time.

Good.   I don't want rebuilds of everything.   I would prefer rebuilds of
(almost) nothing.   pkgsrc should have the pkg_hack stuff (brilliant stuff,
with one glaring defect - its name suggests it is a kludge, if it had been
called pkg_saviour instead, it would probably be part of pkgsrc proper by now).

But that's not what we're talking about.   What matters is that the
ultimate objective here is to make pkgsrc be as good as is possible (it is
already very good, with just a couple of problems).   Before buildlink,
packages did dependencies properly, each said
	DEPENDS+=zlib>=1.1.4nb1
(or whatever), in the makefile.   That was correct - the package knows
which version is required (or more likely, which version the package
was first built to use - older ones perhaps also work, but are untested,
hence cannot be specified sanely).

With buildlink, those dependencies have started to become centralised,
so that (most often) now pkgsrc is telling the packages which versions
they require.   That's simply broken, and we need to work on going back
to the earlier state of affairs.   The rest of buildlink is great,
this "centralise the dependency info" isn't a property of buildlink
really, it just happens to be a convenient mis-use of the common
include file.

  | Say you're buildling X (let's choose another letters so we don't get
  | confused), which depends on Y and Z.  Both Y and Z depends on W, with
  | different requirements (it might happen, I gave an example earlier).  The
  | most restrictive requirement must be recorded in order to have X working
  | (and sometimes even *linking*) properly.

Recorded where?   What are we talking about here?   If we're talking
about pkgsrc (which is what I thought we were talking about) all that
matters is that an adequate version of W is installed that satisfies
both Y and Z - X doesn't depend upon W at all (I believe from your
example) so should have no dependency for W in it at all.

Even if we're talking about binary packages, you're wrong.   Here there
are several choices as to what should be done - one is for the binary
package for X to not mention W at all, just Y and Z.   If Y and Z are
installed (which they need to be for X to be installed) then W will
be installed as well - and that W will be (of necessity) a suitable
version for both Y and Z (otherwise Y and Z couldn't both be installed,
and then X couldn't be installed either).

Or, we could note that for X to be installed, W needs to be installed
as well, but the version of W isn't the dependency info from either the
Y or Z pkgsrc - say Y says W>=2, and Z says W>=3 - and let's assume
that W.4 is installed when X was built - the dependency we would have
to include in X (if anything is to be included at all) is W>=4 (or W==4)
not (W>=3) as there's no guarantee that this compiled version of X
will work if there's only W.3 on the system (even if that is fine for
both Y and Z).

  | But still, you agree we need them even if only for security
  | considerations.

No, I agree nothing of the kind.   The security considerations are
nonsense.

If package Q is using package R, and package R is revised because of
a security bug being fixed, then if that fix makes a difference to Q,
Q should update its dependency version on R (and at the same time,
bump its PKGREVISION).   If the fix to R makes no difference to Q,
one way or the other, then Q does nothing (and its PKGREVISION does
not change).

Not only is this more correct, it also avoids those mass "bump the
PKGREVISION of everything that does a .include of
 ../../category/pkg/buildlinkN.mk when "pkg" gets updated - where
in some cases the otherpkg.nb(n+1) is identical bit for bit (timestamps
excluded) to otherpkg.nb(n).

  | > Next, with this new world order, how do I explicitly override the
  | > BUILDLINK_DEPENDS line in a buildlink3.mk file?   I do that quite often
  | > in practice (or did, until now).
  | 
  | BUILDLINK_DEPENDS.foo= foo>=1.2.3
  | 
  | in the Makefile, just like with bl2.  Then, later buildlink3 inclusions
  | will add (+=) other requirements, usually less restrictive, which will be
  | ignored in the end.  But in the case somewhere in the path a package
  | requires foo>=1.2.4, thanks to += it will get recorded.

No, that's exactly what I don't want.   The "usually less restrictive"
is the problem - in practice, where it matters to me, they are
"usually more restrictive" and I want to ignore them.

If some other package requires foo>=1.2.4, and that package is installed,
then of necessity, I will have foo.1.2.4 (or better) installed, and my
requirement for foo>=1.2.3 will be trivially satisfied, right?   We need
no buildlink magic to cause that to happen.   (And this is true regardless
of whether that other package is a depencency of mine or not).

On the other hand, if that other package is not installed, and I don't
want it installed, and all I have is foo.1.2.3 then I should not need
to replace foo, just because pkgsrc claims that foo.1.2.7 is now available,
and what's more, fixes a security bug in some obscure function that
I never use.

  | So this is rant about tiff?

No, much older than that, I think it was png that first drove me
to question buildink version setting (there were several png updates,
and they all updated the BUILDLINK_DEPENDS line, and I kept on compiling
packages just fine against the older version of png, by explicitly
setting a BUILDLINK_DEPENDS on the older version - even more than tiff,
recompiling png means reinstalling close to everything).
[Note, it was buildlink v1 (or just "buildlink") at the time].

It may have actually been something even earlier than that when
I first encountered this (this isn't the first time I have objected
to setting the dependency version numbers in the buildlink files I
don't believe).

  | At
  | that price, it's as bad as doing 'make replace' for the new A, since
  | you're the one making the decision that C is ok with A.

No, that's me the luser doing the install, who most likely knows nothing
about what versions work with what other versions.   That one I agree is
not rational.   What I want is for the maintainer of package C to decide
which version of A is OK for C.   If that person isn't sure, and says
"well, new version of A, that might be important, perhaps I'd better to
depend upon that one, to be safe" then that's OK - and when that happens
someone who actually knows that it wasn't required can object to the
change and have it backed out (if appropriate).

What's important is that it is a package by package decision, not
one arbitrary decision for everyone made by someone who has no clue
about just how the various packages use the dependency info.


Johnny Lam <jlam@buildlink.org> said:

  | I think explicitly overriding the BUILDLINK_DEPENDS.<pkg> line is a
  | workaround for the problem you subsequently describe. 

Ah, OK, BUILD_LINK_DEPENDS.<pkg>=xxx *after* the .include of the buildlink
file for pkg, rather than before it, as before - yes, that would probably work.

Thanks.

 |  Yes, updating using pkgsrc can suck, and I'm trying to slowly modify pkgsrc
  | to the point where updating doesn't suck.

Great, and it is all much appreciated - really - even if it doesn't
always seem like that.

Believe it or not, that's what I am trying to assist with, the only way
I can right now - which is by making some observations about the
overall direction and goals.   One of those is that it really is improper
for the buildlink files to be specifying required versions.

On the other hand, believe me, I understand the "how to we get to there
from here" issues, and don't claim that now that those thins are endemic
(in buildlink2 in any case) that removing them is going to be trivial.

That's why I questioned this particular change - it seemed to me to be
a step in the wrong direction, more deliberately expecting to be
getting multiple versions as requirements for one package (which makes
no sense at all) - rather than looking at why that is happening in
any particular case where it current happens (and causes the wrong thing
to happen) and fixing that particular case, until the root cause of the
effect (the DEPENDS lines that shouldn't exist in the first place) can
eventually be removed.

kre