Subject: Re: When DEPENDS can be upgraded in place
To: NetBSD Packages Technical Discussion List <tech-pkg@netbsd.org>
From: Greg A. Woods <woods@weird.com>
List: tech-pkg
Date: 09/09/2000 13:01:46
[ On Friday, September 8, 2000 at 23:25:55 (-0500), Frederick Bruckman wrote: ]
> Subject: Re: When DEPENDS can be upgraded in place
>
> No, it's not irrelevant what the author uses. Suppose an author, one
> day, decides to support NetBSD, maybe even distribute binaries for it.
> He's supposed to maintain a different version number for NetBSD (maybe
> for every vendor)? I don't think so.

Well, iff the author is also the maintainer of the pkgsrc module then
`OK' yes the pkgsrc shared library binary can have the same version
identifier as what the author chooses.  Otherwise `No way'.  Well
*maybe* -- iff the pkgsrc module does not include any patches or options
or whatever that will change the binary library module(s) in any
way.....

In order for this whole picture to fit together properly it is parmount
that the pkgrsc library modules have unique run-time version numbers,
and that includes uniqueness in the major number field too (in fact the
major number is of course the only thing that really matters).  In the
special case of shared libraries it may even be necessary to go one step
further and require the shared library major number be unique for every
*variant* of a given release version of a package -- at least if there's
ever a chance that two different parent packages will require variants
of the library -- e.g. the library compiled with conflicting
configuration options that cause different library binaries to be
produced.  This is, in fact, quite possible right now with graphics/gd.

Pkgsrc (and its other siblings) is a special system that's in a
different universe from that where the author supplies binaries or
source intended to be installed directly by the end user without the
benefit *and* knowledge of pkgsrc.  The job of pkgsrc is in fact to foil
the smart self-configuration systems commonly used in freeware today and
to create an environment where all of the pieces have unique hard-coded
shapes that only fit together into a running system in one very
predicatable way.  For example pkgsrc must not allow a package configure
script to find any add-on run-time component (eg. shared library) that
it itself did not explicitly tell, or expect, that configure script to
find (otherwise binary packages cannot work).  To this end, given the
third-party nature of other shared libraries it manages, pkgsrc must
also be explicit about the versions of those run-time components that
the application is permitted to find at build-time and use at run-time.

> I _don't_ want obsolete shared library turds laying around on my
> system, but I may be odd man out on this. The problem I have with the
> present system, is that an innocuous change to "glib" (perhaps a minor
> change to the PLIST), requires me to deinstall and reinstall a couple
> dozen packages to test.

Exactly.  We both have exactly the same goals.  I've worked out my
proposal to the point of doing live real-world (manual) experiments with
it and I'm almost 100% sure that it will work and I'm about 90% sure
that it is the only practical way to make it work too.

> So what's the answer? Never upgrade anything? Or only use commercial
> software, so you have someone to sue if something goes wrong?

Yes the only practical answer right at the moment is in fact to plan to
*never* upgrade anything.  You take a release CD with binary packages
and that's got to be your production environment until the next release
of the CD is available and then when you have time to take the
production machine out of service long enough to either swap a new one
in, or to rebuild it in place, you've got to stick to exactly what
you've got.  Of course this isn't really very much different than
running commercial software since even then you probably have already
agreed not to sue the vendor for damages caused by their bugs.....

Naturally anyone in the know will not find this a satisfactory way to
run a production environment.  When someone reports to BUGTRAQ a remote
root exploit in one of those packages you damn well have to upgrade it
or suffer the likely consequences.  Right now this often means taking
the system offline for unplanned maintenance and de-installing many
packages so that the new release of the package in question can be
installed with the new pakcages it depends upon.  My experience of doing
this even with a separate build and test system has left me with many
frustrations, and though some of them have had to do with accidentally
sup'ing pkgsrc in an inconsistent state (eg. where a dependent package
was upgraded but all of the applications using it have not yet been
upgraded), the majority of the problems have been with the issues you
describe above.

I think the real answer is to split all packages that supply both
development (i.e. which fulfill BUILD_DEPENDS needs) and also run-time
(i.e. which fulfill RUN_DEPENDS needs) components into to separate
packages.  The former will conflict with other versions of itself unless
the installed components contain the pkgsrc version identifier in their
pathnames, while the latter will allow multiple versions of themselves
to be installed simultaneously (which implies that they *must* include
the pkgsrc version identifier somewhere in their pathnames, eg. in the
shared library major-number component for shared libraries).

Packages which use shared libraries from other packages will depend at
run-time on the components of the run-time portion of the library
package, but will require the appropriate developer portion to be
installed on the package build machine at build time.

In order to clean up the remnants of old unnecessary run-time packages
you need only run a daily/weekly/monthly script that'll parse through
the pkg dependency tree and figure out if any of them are still required
by remaining applications.

Note that this concept should be extended to other things that have
version dependencies too -- for example the various interpreters which
support different language features in their various incarnations.

In order to successfully support fine-grained upgrades of applications
the pkgsrc system must become so tightly integrated that applications
know not only what system and add-on components they require to function
correctly, but also what versions of those components that they require,
and of course they must be careful at runtime to actually use the
versions of the components that they were explicitly told to use.  It is
not sufficient for a package to simply choose the first matching generic
name of the component it is using.  The alternative is to go the road of
DLLs and simply ignore the version identifier completely and always
assume a black-box world of components, and simply wish good luck to the
user who makes the fatal mistake of installing a new component that
breaks applications requiring the old component of the same name (or
vice versa).

A slightly "harder", but even more robust additional process to the
above would be for all development packages to be used by a given
application package be built in total isolation from any packages
installed on the build system, and to always build them every time, and
always immediately create binary packages from them and then delete them
as soon as the final application's binary package has been built (all
the while still doing as above for packaging build-time and run-time
components in separate binary packages).  This could be done by always
setting up a pkg build system with a unique private pkg install
hierarchy (eg. /usr/my-pkg) and then always manually doing as I say
above to build binary packages.  This is my plan for the next system I
build for creating and testing binary packages.  This way at least then
I'll have a stable multi-user general purpose system that can at the
same time be used for building and QA'ing binary packages targeted for
other systems with the same base system release.

> > Note also you're talking about system libraries -- I'm talking about
> > pkgsrc libraries.
> 
> I'll say just one more thing. Consider, it's been proposed, from time
> to time, to "pkgize" the base sets.

Yes, of course (and hopefully it happens sooner rather than later.  The
difference I'm talking about is who owns the code in question -- it's
the "third-party" nature of pkgsrc modules which is critical here.

-- 
							Greg A. Woods

+1 416 218-0098      VE3TCP      <gwoods@acm.org>      <robohack!woods>
Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>