The problem:
-----------

Upgrading a package often requires updating a web of dependencies.
That's fine, except when those dependencies have dependents which are
unrelated to the package of interest -- then you've got to update all
of them, too. Moreover, the current methods demands that those
dependents be deleted before anything is built. If anything goes wrong
with any link in the chain, the builder could be left with a severely
degraded system.


Some ideas:
----------

First, some background. PR pkg/10835, with the fetching title "Package
upgrade procedure sucks", called for leaving old files from any old,
out-of-date package in place, much as if you'd simply done a "make &&
make install" without a package system. That's a good idea, to let the
package system get out of the way while still keeping all it's other
benefits, but the devil's in the details...

I have discussed all this, somewhat, with some of the other "pkgsrc"
developers at various times. Notably, it was pointed out by Hubert
Feyrer, that a naive implementation would lead to a "flag day" in
"pkgsrc". I do seem to remember that all were agreeable to no keeping
old headers at all; the only files we really need to keep are the
shared libraries in those packages which contain shared libraries. In
ELF, now, that means we need to keep the file, and the major link, but
not the "so" link. (The idea being to keep the minimum to run the
dependents against -- never to build against.)


A solution:
----------

So, this implementation follows the brainstorm I had nearly a year
ago (below)...

The first part of the idea, in brief, is to store these needed
files in a way that doesn't require any extension to the format
of ${PKG_DBDIR}, via the mechanism of creating packages "on the fly"
with reserved names. So when "libfoo-1.0" is deleted, "libfoo-SO-1.0"
appears, containing only the shared libraries formerly held by
"libfoo-1.0", while carrying over all the dependencies formerly
held by "libfoo-1.0". Despite it's dubious origin, this "libfoo-SO-1.0",
looks just like a regular package to the package tools: the dependents
reported by "pkg_info -R" are accurate, "pkg_info" over such
dependents accurately reports their dependence on "libfoo-SO-1.0",
and "pkg_info -L" lists the (few) files it contains.

As you can imagine, there are lots of ways to go wrong. With that in
mind, the second part of the idea described in the following,

    http://mail-index.netbsd.org/tech-pkg/2001/05/31/0003.html
    http://mail-index.netbsd.org/tech-pkg/2001/05/31/0004.html

is to implement all of this as wrappers around the ordinary package
tools (with an eye to updating the package tools, eventually). The
solution to the problem described in the second message, that of
binaries depending on multiple versions of the same shared library,
I came to realize, is to put extra checks in "pkgsrc/mk" to force
incompletely upgrade packages to be declared out of date, and rebuilt.
That part is contained in the same make fragment, for inclusion in
${MAKECONF} or "/etc/mk.conf", that invokes the wrappers.

So, please have a look at

    ftp://ftp.netbsd.org/pub/NetBSD/misc/fredb/pkg_hack.tar.gz

"make dependall && make install" will install a few files into
"/usr/local": the "pkg_hack"  script, "pkgdb", a helper utility
written in C, and some supporting files. You'll need to set
${BSDSRCDIR} somewhere, so that the helper utility can be staticly
linked against the package tools' library (which isn't built or
installed shared by the base system). Finally, you may wish to
include "pkg_hack.mk"  in "/etc/mk.conf", as shown in the installed
example "/etc/mk.conf".


Benefits:
--------

Chiefly, "make install" (or "make package") installs (or packages)
the package indicated, and just the dependencies it needs, regardless
of any out-of-date package(s) that may or may not be installed. The
"delete" step is implicit, and is only performed after the package
is built, to minimize the impact of the update-in-progress.

Unlike "make update" (phew), there's no building packages multiple
times, and there's no attempt to keep state in the work directories.

Most importantly, packages not rebuilt will largely not be affected.
There are still a couple of worms in this apple: "a.out" programs
may be broken by having their shared libraries replaced by "ELF" ones,
and certain actions can lead to dependencies being "fixed", where
they should be wildcards (which is only an issue if you intend to
use the binary packages on a non-pkg_hack'd system.

"pkg_hack -a" likewise installs a binary package, even though an older
(or newer!) version my already be installed.


Caveats:
-------

You must be root to install or package with this. All the targets in
"pkgsrc/mk" aren't overrideable, so where I managed to wedge it all in
happens to break the just-in-time "su" magic.

You need a fairly new "pkgsrc", no more than a couple of months old
(1.5.3 tag OK).


More:
----

In the tarball, there's also a man page for a little utility I'm
working on, "pkgorder", to spit out dependents and dependencies in
order. I originally thought it would be necessary to make "pkg_hack"
work at all, but now I think it's separable, and not absolutely
necessary.

I think I've said enough. I hope people try this, and find it as
useful as I have.


Frederick