(This message, and by fiat the entire thread it spawns, is only about
NetBSD. If you would like to bring up something that affects
non-NetBSD, please start a new thread. I say this forcefully as a a
courtesy to pkgsrc users on other OSes to allow them to ignore the
entire thread safely.)
I've seen various comments on the NetBSD mailinglists and otherwise
about the pkgdb migration. My overall read is that people are ok with
the database moving, but some are having problems and many are concerned
about migration smoothness for others. This email is an attempt to take
a step back and question the migration assumptions while being clear
about the core difficulties.
We are getting close to branch, so while substantive technical
discussion is most welcome, I'm going to be heavyhanded about bikeshed
color choices.
Sorry this is long; I've tried to keep in compact.
The first question is if we should revert all this, vs figure out how to
move through it. My opinion, I think reasonably widely shared, is that
the basic change is something that should happen, that going back would
cause a lot of disruption, and that with easier migration for regular
users things will be ok. So I am only looking at how to ease/automate
migration.
The reason this is difficult is that on NetBSD we have multiple things
that are aware of the pkg database directory
base pkg_install
/usr/pkg/sbin pkg_install (often/usually present, and henceforth
called "pkg pkg_install"))
mk/*
The current strategy can be summarized as a synchronous change
involving:
cvs update (or equiv) to new mk files
mk files require migration and new pkg_install
build the new pkg_install (awkward with cwrappers, ccache, distcc) and
replace the package
overwrite the base pkg_install with pkg pkg_install
along with
new pkg_install exists in current, 8, and 9 so that when one updates
the OS along branches or to any (forthcoming) releases, the tools will
be configured to use the new location
there are instructions, but there is no script
My own take on requirements is:
There must be only one database.
pkg pkg_install and base pkg_install must operate on the sole database.
mk/* should refer to the sole database.
There is not an actual requirement to have newer pkg_install code at
this time, just that all things operate on the sole database.
We have historically had situations where newer pkg_install was
required, and systems with /usr/pkg/sbin in PATH before /usr/sbin
dealt with new pkg pkg_install and avoided base pkg_install just fine.
For a good user experience, we need script support to 1) report on
system configuration vs desired and 2) perform migration, at least on
systems which are configured normally.
Cleanliness arguments, like not leaving symlinks in /var/db, are very
much secondary to user experience.
The plan needs to support 8 and 9 with or without the new pkg_install,
current with the new pkg_install, on systems that are building
packages from source and those that are using pkgin or equivalent.
(Current without new pkg_install ought to work for any scheme that
supports the above, but I am making the point that old current is not
something I think we should worry about)
It would be nice to ease things for pkgsrc-curnret users, and it's
really important to make things easier for people who "cvs up -r
pkgsrc-2020Q4" once it is cut.
We don't really have a requirement to make things automated for people
who update from 2019Q3 to 2021Q2 or some other vast leap, and making
them have an intermediate update to 2020Q4 is sufficeintly
reasonable that it need not be avoided.
Observations:
After manual migration with new pkg pkg_install, With /var/db/pkg a
symlink to /usr/pkg/pkgdb, the base pkg_info works just fine.
(Additionally, I have been running a few systems with /var/db/pkg
symlinked to /someplace/with/space/pkgdb for years, on systems with
small /vars, and had zero problems from this.)
With "PKG_DBDIR=/usr/pkg/pkgdb" in /etc/pkg_install.conf, the base
pkg_info works just fine with the new db location, with no symlinks.
This leads me to:
To effect migration, there is no need to force replacement of base
pkg_install. Setting the variable in pkg_install.conf is 99% clearly
sufficient, and the symlink is very clearly sufficient.
There is no need to force the use of new pkg_install so hard that
rebuilding it is awkward. The real need is to know that the db
migration has happened.
We should have a script to check and migrate, and somehow cause mk/*
to insist on that, rather than a specific pkg_install version.
It's perfectly fine to print a warning if pkg_install is old.
And a proposed plan:
Write a script that can check if migration is done, and to make
migration happen. Migration steps consist of:
moving the db/refcount from /var/db to /usr/pkg
adding a variable in pkg_install.conf
probably, touching some migration done cookie file
on source machines, updating pkg pkg_install
The check invocation should touch the cookie file if all is ok (to
ease things for people that do things manually, or that have done it
manually already).
Have mk/* insist on migration being complete rather than a particular
version of pkg_install being installed. The real requirement is
"tools use the new db", not a particular version with a hardcoded default.
A moratorium on requiring new pkg_install for a while, at least well
into next quarter.
Later, just require the new pkg_install.
A bit later, another script to clean up the migration (remove
PKG_DBDIR from pkg_install.conf, remove cookie file).
For people using chroots (pbulk, whatever), they either need new tools
in the chroot or the pkg_install.conf line. The requirement is
simply that the tools in the chroot use the right db, plus maybe
having the cookie file.
I have the script mostly written in my head. It's 90% checking and
printing errors.
If I have this wrong, please tell me how. Also please speak up if you
think this is worse than the current "read the manual instructions and
follow them" approach, or if you think something else should be done".
I'll start writing the checking part of the script; that's clearly
useful even with the "follow instructions" approach.
Greg
Attachment:
signature.asc
Description: PGP signature