Cover me, I'm going in.  :)

Reading this thread has struck a chord with me as I, too, have had many
incidents with the NetBSD pkg system.  I have this rant regarding it
which is more than a little bit of a troll, isn't entirely fair, and
concludes with "Save the diffs, burn the rest to the ground."

While not the most political way of expressing my feelings, it does leave
little question as to where I stand.

Let me see if I can't explain myself more eloquently.  In this analysis,
I assume that the user will take the recommended defaults in all cases.
While I use sendmail as the primary example, it is not unique.  The same
applies to bind, openssl, and other base packages.  Here I'm examining
only the binary package system.  Pkgsrc has a whole other set of issues,
which should be examined separately.


Anyone reading this thread would not be surprised to hear the NetBSD pkg
system has issues.  These issues stem from the fact that it tries to
serve many different interests at once, none fully nor even well.

The pkg tools attempt to keep the packages in a separate location to aid
in managing the system files vs the package files.  This is done so that
it is easy to upgrade the system and the packages without them stomping
all over each other.  That seems to be the only virtue, itself of
questionable value.

So, let us explore the potential virtues of the package system, and how
the NetBSD pkg tools handle them.

The pkg tools add pre-built software to the system.  They install
everything under /usr/pkg.

This includes the rc.d scripts.  Even if a user has the wherewithal to
add "gdm=YES" to their /etc/rc.conf, the system by default doesn't find
the rc.d script.  Nor could it, due to a chicken-egg issue.  Even if 
rcorder were told to look in /usr/{X11R6|pkg|local}/etc/rc.d the scripts
would remain undetected as those partitions aren't mounted when rcorder
runs -- it's an rc.d script which mounts them in the first place.  By
default, packages are inaccessible to the system at boot time.


This also includes the etc configuration files.  The pkg tools install
all files under $PREFIX.  $PREFIX, in the pre-built binary packages, is
set to either /usr/pkg or /usr/X11R6.  When the user or sysadmin uses
the package system, he ends up with three etc directories to manage.
For simple workstation or server installs, this adds complexity to the
user or sysadmins job.  The argument for this behavior is hybrid systems
where /etc contains local system stuff, but /usr/pkg is shared among 
multiple machines.  I think this is a poor choice.  I'll explain why
later in the post.


By keeping the packages separate, the pkg tools could make it easy to
uninstall a package by deleting the pkg directory.  E.g, pkg_add
sendmail could create a /usr/pkg/sendmail directory with bin, lib, etc,
foo under it and symlinks to the files therein in /usr/bin, /usr/lib,
/etc/sendmail, and /my/foo.  De-installation of sendmail can be
trivially enforced by deleting the /usr/pkg/sendmail directory.  

Our pkg_add doesn't do this.  It creates a new set of bin,lib,etc,foo 
directories under /usr/pkg and mixes all packages therein.  The pkg
tools are still needed to manage the files and ensure that all the
files are present or removed as directed.

If pkg_add kept the packages separate, the pkg tools could maintain
multiple versions of a package (say, sendmail 8.8 and 8.9), the active
one being chosen by a symlink.  Imagine having /usr/pkg/sendmail8.8 and
/usr/pkg/sendmail8.9 as installs, then /usr/pkg/sendmail as a symlink to
the active version.  Now let /usr/bin/sendmail point to
/usr/pkg/sendmail/bin/sendmail which in turn points to
/usr/pkg/sendmail8.9/bin/sendmail.


Since the pkg_tools don't provide for version management, perhaps they
could provide some security management.  Sendmail is part of the base
install.  When a security alert is issued for sendmail, pkg_add could
upgrade sendmail to a more patched version, or patch the installed
version.  

Our pkg_add can't do this.  Pkg_add (again, by default) doesn't
overwrite the base system version.  In the case of a security update,
the old vulnerable version is still present.  And since it lives in the
base system, it, intuitively, is all too easy to execute by mistake.

Even in the case of upgrading for feature sets, the old base package is
still present.  By having two copies "active" (one in /usr/bin, the other
in /usr/pkg/bin), the system exhibits inconsistent behavior.  Scripts
that search the path (assuming /usr/pkg/bin is before /usr/bin) use the
new version.  Scripts that have /usr/bin/sendmail hard-coded (as is good
and common practice) use the old version.  Similarly, scripts that are
hard-coded for path (e.g. #!/usr/bin/perl) don't find their executable
because it's not in the sacred base system.


In each of these situations, the pkg tools only go half-way to servicing
the needs of the user or system administrator.  After pkg_add is
completed, there is additional work to be done by the user/admin.

pkg_add does tell the user that some additional work is needed to get
the package just installed to work, such as modifying a config file.
Pkg_add also automatically installs other dependent packages.  These two
features work at odds with one another.  Installing multiple packages
shows potentially dozens of these important messages flying by the user
before action can be taken.  The user is left with what ever made it in
the scroll back buffer [assuming they're not using a teletype with reams
of paper :) ], or going through all the messages of all installed
packages to figure out which need additional work.

Some of this can be done by the package when installed.  ap-php4
requires entries in the apache httpd.conf file in order to work.
Why can't the package add those entries?  It knows it's being installed
for apache.  The httpd.conf file is in a known location in the installed
pkg database.  A simple set of scripts (perl, or awk/sed/grep) can see
if it's already present and if not add it, and a diff of the change can
be saved to facilitate uninstalling.

pkg_add also allows the user to pull packages from the Internet by
setting PKG_PATH to ftp://example.com/my/package/repository.  This is
nice, as it allows the user to pull from a (hopefully) well-maintained
set of packages.  But, pkg_add fails to tell them a download is in
progress or how long it will take, or which dependent packages are also
being pulled.  Turning on verbose mode goes overboard in the opposite
direction.


Earlier in this post I mentioned that pkg tools attempt to service
different interests.  I'd like to explore them at this point.

There are at least 4 type of systems out there in the real-world:

1. Single-user workstations
2. Multi-user "mainframe-style" work-servers.
3. Service servers. (dns, smtp, ntp, nfs)
4. 1-2 Hybrid systems (Single-user workstations with shared /usr, /foo)


Single-user workstations (SUWs) are what many of us try to use on a
day-to-day basis.  We have a machine at home, we install an OS and
software on it, and we have little need for integration beyond network
services (smtp, dns, pop, imap, whatever.)  

This is a kind of consumer level system, and NetBSD fails miserably to
service them.  The bootable install CD may install X11 binaries, but it
takes some effort and system administration skills to even get an X on a
gray screen, much less a graphical login or a desktop environment.

As explored above, the binary packages don't help here.

This brand of end users, when told they have to copy the scripts into
/etc/rc.d are likely to balk.  "What?  Isn't that the job of the package
tools?"  What should be a simple task is no longer simple.

This also applies to professionals who just want a system up and running
in a short amount of time.  When I get a new system, I don't want to
spend two days installing packages, moving around rc.d scripts and
mucking with configuration files.  I want to get to work!


The multi-user mainframe systems are similar to end-user workstations
except that a skilled admin is managing the software.  These admins are
often handling other user issues ("I forgot my password.  Again.")
They don't have time to manually complete a package install, or diagnose
why the software "I-know-I-upgraded/installed-that!" doesn't work.

This issue is compounded for service servers.  While admins of those
systems don't have shell users, they do have far more users expecting
the services to work.  Upgrades of services need to be timely, and
reliable, and Just Work.

In both of these cases, the admins want tools they can trust to make
their jobs easier.  They do not want tools they have to double-check or
follow up with manual file moves or adding a /usr/bin/foo symlink for
the benefit of common scripts.


For they hybrid 1-2 systems -- systems with both local and remote
parts -- pkg tools put the files in places that make it possible for a
system admin to manage both local machine configuration and global
configurations.  This is an admirable goal, but it has come at the
expense of other more common installation types.  Hybrid systems are
difficult to manage.  Those who do manage them are more skilled than the
average desktop user, and have a deeper understanding of how all the
bits interact.

Earlier I said I'd explain why I thought $PREFIX for everything was a
poor choice and here it is.  The $PREFIX method as used by our pkg tools
covers the corner case of hybrid systems by default.  But, admins of
hybrid systems expect things to be more difficult.  They expect to have
to do things in the non-trivial manner.  They expect they'll need to use
additional tools or command line options for their special situation.
They're smart.  (Or at least smarter.)  We should make it possible for
them to manage their systems, but not at the expense of the majority of
systems out there.  Make the default /usr, and let the hybrid admin relocate
them with -p /usr/pkg.  This is complicated by packages which install 
in /usr/X11R6. 

When it comes to software management tools, keep these goals in mind:

	Easy things should be no-brainers.
	Difficult things should be easy.
	Impossible things should be possible.
	Explain what's going on to the user. 
	But don't scare him with too much info!
	Package, configure thyself.
`	The tools should make the admin look good.

jf
-- 
John Franklin
franklin@elfie.org
ICBM: 35°43'56"N 78°53'27"W