Date:        Mon, 21 May 2001 12:25:26 +0300
    From:        Jukka Marin <jmarin@pyy.jmp.fi>
    Message-ID:  <20010521122526.A29172@pyy.jmp.fi>

On the netbsd-users list, Jukka Marin said...

  | I updated some pkgsrc package, which removed an older SUSE linux emulation
  | package and installed a new one (6.4).  Now I can no longer run Applix
  | because it can't find the library versions it wants (ld-linux.so.1,
  | libdl.so.1, etc.).

[...]

  | I also wanted to install avifile, but it depends on a new pth version -
  | and a thousand other packages depend on pth, so installing one new
  | package would mean rebuilding almost everything (and leaving the system
  | unusable, like several times before (because some of the packages fails
  | to build)).

[...]

The problem here is that the way libraries are handled with pkgsrc is
simply wrong.   Note that it isn't all that much of a pkgsrc problem
in the way that the software is written (though I suspect a smallish
enhancement will be needed to pkgtools) - the problem is with the way
the packages are put together and used.

Libraries are a very special case as an installed object, as they're used
by other applications, rather than just by the users of the system, which
is what applies to almost all other things that are installed by pkgsrc.

That is, nothing breaks if I remove the ncurses man pages, except that I
can no longer read the things - nothing breaks if I remove the ncurses
include files, except that I can no longer compile programs that need them.
But if I remove the ncurses .so files, all kinds of other applications
simply stop working.

The solution to this is to recognise that difference - the way to do
that is to make the .so files be a pkg of their own (one or more .so
libraries installed as a group - and absolutely nothing else in that pkg).
Then the rest of the library - man pages, .a files ... all the stuff
needed for compiling new programs should be a separate package that
DEPENDS upon the .so package (or perhaps even packages in some cases).
(And then any application that uses these libraries should be a third
package, that BUILD-DEPENDS upon the .a library package, and RUN-DEPENDS
upon the .so library package).

Next, only in one circumstance should anything ever CONFLICTS with a
.so type of library package - that's when a minor upgrade of that library
is being done which preserves the ABI (and keeps the same library version
number .. so the kind of problem Jukka referred to wouldn't happen, the
new one would be a drop in replacement for the old).

That is - only human intervention (a manual pkg_delete) would *ever* remove
a .so type library package, or change it in any incompatible way.
Typically those packages would only every be installed as the result of
one of the DEPENDS clauses of some other package.   One enhancement that
would be useful to pkgtools (unless this exists already) would be to mark
packages that are installed because of a DEPENDS rather than a human
"make install" or "pkg_add", and then later to be able to go through the
pkg db and locate any such packages that are no longer needed by anything,
so they can be deleted if desired.  Installed packages should be marked as
depending upon the particular version of a .so library type package that
exists when they're installed - no ">=" dependencies allowed in the db
(which isn't the same as what is allowed in the pkg Makefile).  To make
this easier, perhaps dependencies on .so libraries could be entered in
pkgsrc as LIB_DEPENDS or something, with >= allowed in the Makefile, and
pkgtools converting that to an explicit reference on the current installed
library version.

In pkgsrc, all the old versions of the .so library packages would be retained
while it is still possible to keep them (while the distfiles that are needed
for them can still be obtained somewhere - and for all the ones where the
distfile can be copied to the netbsd site, that is really forever).
Whenever there's an upgraded version of a .so library package, it goes in
a new pkgsrc directory, as a separate package of its own, distinct from
the previous one (and most certainly *does not* CONFLICT with the previous 
one).

On the other hand, the .a library, with its include files, and man pages,
can be upgraded whenever a new version appears - all that affects is
compiling new stuff - if some old package no longer compiles with the new
versions of libraries that it used to use, then it can be marked as
broken until it gets upgraded (or removed).

The effect of all this is that when something is being upgraded, the .so
libraries will never be deleted, as nothing ever conflicts with them,
unless there's a ABN conserving minor upgrade going on.   But eventually
their usefulness will be terminated, when everything that used to use them
has gone, so perhaps the (currently empty) /etc/monthly could go through
the pkg db (once a month obviously) and report on any packages that were
installed as dependencies, and which are no longer depended upon by
anything (or perhaps this would be done only for things installed as
LIB_DEPEND if that gets adopted).

With this, the vast majority of the "I wanted to upgrade X, but X needs
a new Y, and A B C D E F all depend on the old Y (almost exclusively
because they use its .so library) so A B C D E F were all deleted, and
now they don't get reinstalled, because something about the new Y is
incompatible" problems will go away.   All those packages (A..F) will
depend upon the Y1.so package, and when X needs a new Y, that will depend
upon Y2.so which won't conflict with Y1.so - it will just cause Y to be
deleted (nothing will have a RUN_DEPENDS on Y, it provides nothing used at
run time, so that will cause nothing to be deleted).  When the new Y
is installed as X BUILD_DEPENDS upon that, it will cause Y2.so to be
installed as a DEPENDS, and then X will have a LIB_DEPENDS on Y2.so).
There's probably a way to automate that so all packages needing lbraries
don't have to have two explicit dependencies on the two parts of the
library - but without requiring X to be deleted any time Y is deleted,
only if Y2.so is deleted.

The hard work in implementing this is going through all the packages
containing libraries (all that insatll a shared object) and splitting
them into two.   That could be done for each individual package, or some
pkgtools work could be done to automate it (one pkgsrc dir installing
two separate binary packages - and producing two distinct packages/*.tgz
files as well).

But the current situation is broken - and we're seeing more and more
reports (complaints) from people who are experiencing the problems.
The root cause of the brokeness is the lumping together of the .so part
of a library install, and all the rest of the stuff (includes, .a, man
pages) as one indivisible package - split them into two parts, and I
think that most of the problems go away.  Conceptually it isn't that hard.

Of course, doing all the work to upgrade all the packages will take
some time - and should be allowed to take time - upgrading them as they
would have been upgraded anyway - don't go on a binge of upgrading all
the packages in one swoop - that will cause problems for people, all of
whose current pkg installations would need to be redone from scratch.
But if each library is converted at the time it would have been ugraded
anyway, that will cause no pain for people that they wouldn't have had
to experience because of the upgrade anyway (and then in the future, much
less of it).

I also don't underestimate the problem of determining whether a library
upgrade preserves the ABI or not.   Especially of some third party maintained
library, and even more where the library maintainer simply doesn't understand
the issue, or doesn't care - perhaps one of those oddball shared library
haters, who believes that their library should only be installed as a .a,
and hence isn't interested in preserving the ABI (or even documenting whether
it has changed).   But there's always the "if in doubt, bump the major number,
make it a new package, leave the old one installed" fallback for whenever
it isn't clear that the ABI is preserved.

Opinions?

kre