Subject: Package Views Integration (finally!)
To: None <tech-pkg@netbsd.org>
From: Johnny Lam <jlam@jgrind.org>
List: tech-pkg
Date: 08/20/2003 23:00:16
Preamble
========

Those of you who follow pkgsrc-changes and source-changes may have
seen a lot of activity on the "pkgviews" and "pkgviews-mk" branches
over the past six weeks.  I've been actively trying to take the work
that Alistair Crooks did over a year ago on implementing package
views and bringing it into the main pkgsrc branch.  I'll broadly
describe what's been done, and what the next step will be.  I intend
to start merging the work from the two branches back into the main
pkgsrc branch starting on Saturday (time is short), so please take the
time to read, review and respond.


A short history of package views
================================

It was proposed as a solution to the problem of not being able to
install multiple versions of the same software simultaneously via
pkgsrc.  Alistair Crooks presented a paper on package views at
EuroBSDCon 2002 last November that described the work he did on the
"pkgviews" branch of pkgsrc where he implemented his ideas.  I highly
recommend reading it, available at:

	http://www.NetBSD.org/Documentation/software/pkgviews.pdf

to get a more full idea of package views.  Unfortunately, Al ran out
of time to devote to integrating his work into the main pkgsrc branch.
Between then and now, a lot of code was added to pkgsrc/mk that
increased the complexity and the capabilities of pkgsrc, including
buildlink2, which conflicted with Al's implementation of package
views.  Back in May of this year, Al sounded the charge for package
views integration, and a few of us pledged some time toward the end of
the summer to try to get it done.  I've had more free time than the
others these last few weeks, and I've devoted it almost exclusively to
package views.  As part of the integration, I created a new buildlink3
framework (hey, another year, another version of buildlink!) so that
we wouldn't lose the ability to isolate builds from differences in the
environment.


Package views terminology
=========================

The terminology for packages in the package views world is as
follows:  a "pkgviews" package is a package that has been converted to
build and install using package views.  An "overwrite" package is one
that hasn't.  A "depoted" package describes a pkgviews package
installed into /usr/pkg/packages in its "depot" directory.  A "viewed"
package describes a pkgviews package symlinked into a view.


What's been done?
=================

The pkgsrc/mk infrastructure and pkg_install tools changes that
implement package views are ready to be imported into pkgsrc-current.
My changes to the pkg_install tools were made on the "pkgviews"
branch, and my changes to the pkgsrc/mk infrastructure were made on
the "pkgviews-mk" branch.  The pkgsrc internals have been modified to
allow building and installing depoted packages and to automatically
add a depoted package in the the default view.  "Overwrite" packages
shouldn't be affected by the changes, and can be freely mixed with
"pkgviews" packages during the time that we migrate pkgsrc to fully
use package views.  During this migration period, we only support a
single view: the default (empty) view.

A new buildlink3 framework that takes advantage of depoted packages
will also be added to pkgsrc as part of the package views changes.
Buildlink3 uses wrapper scripts much like buildlink2, but it will no
longer symlink files into ${BUILDLINK_DIR} since we can safely refer
to only a specific package's files by passing the appropriate -I<dir>
and -L<dir> flags to the compiler, where <dir> is a subdirectory of
the package's depot directory.  A "pkgviews" package must be converted
to use buildlink3.  Some highlights of pkgviews packages:

      * fully dynamic PLISTs
      * multiple versions of the same package may be depoted and
	co-exist without problems


What's next?
============

After the pkgsrc/mk module from the "pkgviews-mk" branch and the
src/usr.sbin/pkg_install module from the "pkgviews" branch has been
merged back into the main pkgsrc branch, the first step of the
package views integration plan will be finished.  The next step of the
package views migration plan is to convert all non-USE_X11BASE and
non-USE_IMAKE packages into "pkgviews" packages.  The process
basically consists of making the following changes to the package:

    (1) Adding the line "PKG_INSTALLATION_TYPE=pkgviews" to the
	package Makefile.

    (2) Removing the USE_BUILDLINK2 definition from the package
	Makefile.

    (3) Changing references to buildlink2.mk into the corresponding
	buildlink3.mk file.

    (4) Removing the PLIST from the package (hooray for dynamic
	PLISTs!).

There are complexities with packages that have "module" packages, e.g.
perl5*/p5-* and php4/php4-*, that still need to be sorted out on a
package-by-package basis, but the work for all other packages should
be rather straightforward.

	Cheers,

	-- Johnny Lam <jlam@jgrind.org>