We are about to introduce a new form of package installation to the
packages collection's infrastructure.  This is the "staged
installation" - where a package is installed into a separate staging
area, which is under the ${WRKDIR} - and then a binary package is
produced, and that binary package is added into the final ${PREFIX}. 
In tandem with this, we shall be introducing a new form of packing
list - the dynamic inventory - which can only be used with staged
installations, and is basically a list of files, directories and other
entries which may be installed by the package, as determined from the
entries installed in the staging area.  The current installation
behaviour, known as "overwrite", has been kept, as has the current
type of PLIST, known as "static".

Staged installations use the same idea as the OpenBSD "FAKE"
installations - the packages are configured with a PREFIX of
'$${STAGE_PREFIX}${PREFIX}' - and ${STAGE_PREFIX} is left unexpanded
at configuration and build time, and expanded at installation time. 
Nothing else has been taken from OpenBSD.  I don't have any idea
whether they use dynamic packing lists or not.

Staged installations and dynamic inventories use more resources and
are slower than standard "overwrite" installations, and "static"
PLISTs, but are much more flexible to both users and maintainers of
packages. The benefits are:

1.  we can now check whether a package is about to overwrite a file,
and choose to abort the installation, or to carry on and overwrite
the file.

2.  the barriers to creating a new package are now reduced greatly,
since it's not necessary to have endless rounds of checking PLISTs -
this is all done dynamically at install-time.

3.  PLISTs are guaranteed to be correct for the machine on which the
package was installed.

4. a lot of the burden of maintaining a package when an update occurs
is in the PLIST maintenance. With staged installations and dynamic
inventories, this burden is removed.

5. GNU info files are handled automatically in staged installations
and dynamic inventories

6.  By using binary packages throughout, and moving installation-time
actions into INSTALL/DEINSTALL scripts, binary packages become
first-class citizens, and installation-time actions will be performed
exactly the same in installations from source as well as from a binary
package.


And now a FAQ about staged installs.  I've obviously been spending
way too much time in the bozosphere of management and marketing.

Q.  What is a staged install?
A.  Basically, a package is built in the normal way, but when it comes
to installing the package, a staging area is made underneath
${WRKDIR}, and the package is installed there.  Then a binary package
is made from the files in the staging area, and the staging area is
deleted.  The binary package is then installed on the host system.

Q.  It sounds like you're using a lot of resources to do this.  What's
the point?
A.  A staged install will take longer, but more is happening.  There
are also additional benefits to a staged install.  For example, one of
the things that happens is that a dynamic PLIST is generated of
everything which is installed into the staging area.  If PLIST_TYPE is
set to "dynamic", then this list is used in preference to pkg/PLIST.
We also check whether anything will be overwritten, by comparing the
staging area to the target installation area, and can either warn, or
error out if anything would be overwritten.  In addition, this would
enable us to get rid of all the shlib-handling in bsd.pkg.mk, and the
manual page manipulation code, too.

Q.  Is this the same as OpenBSD's FAKE?
A.  We've used their (very neat) idea whereby the packages are
configured with a PREFIX of '$${STAGE_PREFIX}${PREFIX}' - and
${STAGE_PREFIX} is left unexpanded at configuration and build time,
and expanded at installation time.  Apart from that, our *.mk
infrastructure has diverged far too much, and no other OpenBSD work
has been used.  We use the name "staged installation" because we
believe it better represents the new approach.

Q.  How do I activate staged installs?
A.  If INSTALLATION_TYPE is set to "staged", then a staged install
will be done.  If INSTALLATION_TYPE is set to "overwrite", the
existing behaviour is used. The default is "overwrite".

Q.  How do I use dynamic PLISTs?
A.  Dynamic PLISTs are great, since they make package creation much
simpler. If PLIST_TYPE is set to "dynamic" (and INSTALLATION_TYPE is
set to "staged"), then dynamic PLISTs are used. The default is "static".

Q.  Do any packages have to be changed?
A.  Probably. libtool packages work fine. However, there are numerous
packages like nmh which install into the GNU sysconfdir, which has to
be set properly in the package Makefile. There is a definition called
IGNORE_WHEN_INSTALLING, which defaults to "info/dir" at present, to stop
any file present in the staging area to be ignored when installing into
the ${PREFIX}.

Q.  Why have you implemented the changes in this way?
A.  So that we can migrate all packages within pkgsrc over to using
staged install with the minimum of fuss.

Q.  How do I move a package over to using staged installs?
A.  Remove any existing instance of the package.  Set
INSTALLATION_TYPE=staged, and build and install the package.  Then
test it. If it works fine, then the pkg can use staged installs.

Q.  How do I move a package over to using staged installs and dynamic PLISTs?
A.  Firstly, make sure the package can use a staged install (see
above).  Then check that there are no @exec or @unexec lines in the
pkg/PLIST file, (or that the @exec and @unexec lines simply manipulate
info files), and that the package itself doesn't do smart checks for
files already in existence (such as configuration files).  Then set
PLIST_TYPE=dynamic, and install the package, and test it.  The leading
comment in a package's installed +CONTENTS file looks like:

        @comment Dynamic NetBSD inventory for <pkgname>

if the package uses a dynamic inventory.

Q.  What about packages which have @exec and @unexec lines in their PLISTs?
A.  We can manipulate these packages, causing them to carry out actions
at install time and deinstall time by the use of the INSTALL and DEINSTALL
scripts. Because we install from a generated binary package, we can be sure
that the actions carried out at installation time from source, and those
carried out when installing from a binary package, are the same.

Q.  Why are you introducing the changes like this?
A.  Because these changes (potentially) touch every package in pkgsrc,
and backwards compatibility, and proper functional operation is
necessary at all times.  This is the best way I could think of to
introduce this.

Q.  How do I make sure I don't overwrite any files in ${LOCALBASE} or
${X11BASE}?
A.  If you set OVERWRITE_FATAL=yes, then any attempts to overwrite an
existing file will be considered fatal.  A list of all the offending
files will be displayed. The default is OVERWRITE_FATAL=no (i.e.
existing behaviour).

Q.  Do I have to modify the package tools?
A.  Yes - they need to be updated to 20010518 levels, to take
advantage of the -U switch to pkg_create.  This is done automatically
in bsd.pkg.mk.

Q.  Is there anything that can be improved?
A.  Most definitely.  We welcome feedback from users and package
maintainers, and anyone who uses the new functionality.

And one from Dan McMahill:

Q: What are some of the other benefits of this?
A: Our current system of generating PLIST's manually has several major
   flaws:

   1)  When we forget to add a file, we leave leftovers upon deinstall.
       Not a major bug until somehow the existance of such a file makes
       autoconf go wacko.

   2)  When we forget to add a file, the binary package will not have it.
       This means if we leave out bin/mozilla, the mozilla package won't
       have mozilla!  This is a biggie because if the person who made the
       bad PLIST tries out the binary pkg it will appear to work because
       they probably had a leftover copy of the binary.

   3)  Sometimes with automadeautoconfifiedlibtooled pkgs its just not
       obvious what will get installed with a pkg.  This gives problems
       when the installed files are different depending on what else
       you have on your system (see the rio500 breaks KDE2 problem from
       last week).  Also, you're more likely to overwrite some file you
       wanted during testing of the pkg.


This notice is to warn you that there may be some upheaval over the next
week as the new system becomes embedded, and to ask for your patience
as we move things forward. Right now would also be a good time to take
a snapshot of pkgsrc.

Thanks,
Alistair