tech-pkg archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

NetBSD pkgdb migration issues and the way forward

(This message, and by fiat the entire thread it spawns, is only about
NetBSD.  If you would like to bring up something that affects
non-NetBSD, please start a new thread.  I say this forcefully as a a
courtesy to pkgsrc users on other OSes to allow them to ignore the
entire thread safely.)

I've seen various comments on the NetBSD mailinglists and otherwise
about the pkgdb migration.  My overall read is that people are ok with
the database moving, but some are having problems and many are concerned
about migration smoothness for others.  This email is an attempt to take
a step back and question the migration assumptions while being clear
about the core difficulties.

We are getting close to branch, so while substantive technical
discussion is most welcome, I'm going to be heavyhanded about bikeshed
color choices.

Sorry this is long; I've tried to keep in compact.

The first question is if we should revert all this, vs figure out how to
move through it.  My opinion, I think reasonably widely shared, is that
the basic change is something that should happen, that going back would
cause a lot of disruption, and that with easier migration for regular
users things will be ok.  So I am only looking at how to ease/automate

The reason this is difficult is that on NetBSD we have multiple things
that are aware of the pkg database directory

  base pkg_install

  /usr/pkg/sbin pkg_install (often/usually present, and henceforth
  called "pkg pkg_install"))


The current strategy can be summarized as a synchronous change

  cvs update (or equiv) to new mk files
    mk files require migration and new pkg_install

  build the new pkg_install (awkward with cwrappers, ccache, distcc) and
  replace the package

  overwrite the base pkg_install with pkg pkg_install

along with

  new pkg_install exists in current, 8, and 9 so that when one updates
  the OS along branches or to any (forthcoming) releases, the tools will
  be configured to use the new location

  there are instructions, but there is no script

My own take on requirements is:

  There must be only one database.

  pkg pkg_install and base pkg_install must operate on the sole database.

  mk/* should refer to the sole database.

  There is not an actual requirement to have newer pkg_install code at
  this time, just that all things operate on the sole database.

  We have historically had situations where newer pkg_install was
  required, and systems with /usr/pkg/sbin in PATH before /usr/sbin
  dealt with new pkg pkg_install and avoided base pkg_install just fine.

  For a good user experience, we need script support to 1) report on
  system configuration vs desired and 2) perform migration, at least on
  systems which are configured normally.

  Cleanliness arguments, like not leaving symlinks in /var/db, are very
  much secondary to user experience.

  The plan needs to support 8 and 9 with or without the new pkg_install,
  current with the new pkg_install, on systems that are building
  packages from source and those that are using pkgin or equivalent.
  (Current without new pkg_install ought to work for any scheme that
  supports the above, but I am making the point that old current is not
  something I think we should worry about)

  It would be nice to ease things for pkgsrc-curnret users, and it's
  really important to make things easier for people who "cvs up -r
  pkgsrc-2020Q4" once it is cut.

  We don't really have a requirement to make things automated for people
  who update from 2019Q3 to 2021Q2 or some other vast leap, and making
  them have an intermediate update to 2020Q4 is sufficeintly
  reasonable that it need not be avoided.


  After manual migration with new pkg pkg_install, With /var/db/pkg a
  symlink to /usr/pkg/pkgdb, the base pkg_info works just fine.
  (Additionally, I have been running a few systems with /var/db/pkg
  symlinked to /someplace/with/space/pkgdb for years, on systems with
  small /vars, and had zero problems from this.)

  With "PKG_DBDIR=/usr/pkg/pkgdb" in /etc/pkg_install.conf, the base
  pkg_info works just fine with the new db location, with no symlinks.

This leads me to:

  To effect migration, there is no need to force replacement of base
  pkg_install.  Setting the variable in pkg_install.conf is 99% clearly
  sufficient, and the symlink is very clearly sufficient.

  There is no need to force the use of new pkg_install so hard that
  rebuilding it is awkward.  The real need is to know that the db
  migration has happened.

  We should have a script to check and migrate, and somehow cause mk/*
  to insist on that, rather than a specific pkg_install version.

  It's perfectly fine to print a warning if pkg_install is old.

And a proposed plan:

  Write a script that can check if migration is done, and to make
  migration happen.  Migration steps consist of:
    moving the db/refcount from /var/db to /usr/pkg
    adding a variable in pkg_install.conf
    probably, touching some migration done cookie file
    on source machines, updating pkg pkg_install
  The check invocation should touch the cookie file if all is ok (to
  ease things for people that do things manually, or that have done it
  manually already).

  Have mk/* insist on migration being complete rather than a particular
  version of pkg_install being installed.  The real requirement is
  "tools use the new db", not a particular version with a hardcoded default.

  A moratorium on requiring new pkg_install for a while, at least well
  into next quarter.

  Later, just require the new pkg_install.

  A bit later, another script to clean up the migration (remove
  PKG_DBDIR from pkg_install.conf, remove cookie file).

  For people using chroots (pbulk, whatever), they either need new tools
  in the chroot or the pkg_install.conf line.   The requirement is
  simply that the tools in the chroot use the right db, plus maybe
  having the cookie file.

I have the script mostly written in my head.  It's 90% checking and
printing errors.

If I have this wrong, please tell me how.  Also please speak up if you
think this is worse than the current "read the manual instructions and
follow them" approach, or if you think something else should be done".

I'll start writing the checking part of the script; that's clearly
useful even with the "follow instructions" approach.


Attachment: signature.asc
Description: PGP signature

Home | Main Index | Thread Index | Old Index