Subject: make replace and keeping track of 'unsafe' operations
To: None <tech-pkg@netbsd.org>
From: Greg Troxel <gdt@ir.bbn.com>
List: tech-pkg
Date: 04/28/2003 10:43:00
First, thanks to whomever has worked on make replace.  make update,
while very nice, can occasionally be painful.

I got used to using make update, which is very nice on well-connected
fast machines.  I had 3 concerns

  builds might fail, leaving the machine in a non-working state (pkg wise).
  pkgsrc quality is amazingly high, and this is rarely a problem.

  updating something at the bottom of the dependency tree (tiff,
  gettext, etc.) took a long time.  This is a problem both due to lots
  of cpu/disk/net usage (heat, power, wear&tear) and because a lot of
  programs are missing for a long time.
  'make replace' mostly addresses this.

  if one wants to update most packages, and a package depends on two
  out-of-date packages, doing make update on one and then the other
  rebuilds the upper one twice

make replace has the potential to address all of these concerns, but
has a safety problem, in that if a package is replaced, the packages
that depend on it might or might not be in a working state.

I have a proposal for how to augment make replace and have a wrapper
around it to update multiple packages:

  pkgsrc should maintain a database (/var/db/pkg/pkgdb.unsafe or some
  such), that lists all packages which have had a dependency replaced
  (by make replace or some binary equivalent).
    make replace adds all depending packages when replacing
    packages are removed on deinstall, reinstall, etc.

  a script could make a list of all unsafe packages and create a file
  with lines for each
    pkg depending-pkg
  pair, tsort this, and get an order for make replace, and then do
  make replace on all in sequence.

  a new wrapper (or extension of pkg_chk), could make a list of all
  out of date packages and apply similar logic

Probably the last two behaviors are really a general 'update a number
of things' behavior.  Key points are to analyze dependencies and
tsort, thus doing a leaf node first.  Since 'make replace' will in
general add packages to the unsafe list, the dependency scan/tsort
probably should be repeated after each replacement.

In keeping with the minimal spirit of make replace, it makes sense to
have 3 options:

  replace packages that are out-of-date w.r.t. pkgsrc

  rebuild packages marked unsafe, rechecking the list of unsafe
  packages after each build.  (Possibly this can be computed ahead of
  time, knowing that replace will add to the unsafe list.  But see
  below.)

  combination behavior, replacing packages that are out-of-date and
  also scrubbing the unsafe list.

In the glorious future, there could perhaps be logic to refrain from
marking dependencies unsafe on some updates.  Perhaps a variable in
/usr/pkgsrc/foo/bar/Makefile could specify a range of previous
revisions that are equivalent (i.e., no interfaces changed that would
be detectable at configure or compile-time by another program, such
that one would not expect a rebuild of a dependency to result in
different behavior).

With all this, one could just kick off a 'replace-all' command, and
have a bunch of 'make replace's run sequentially, failing in a fairly
safe state if a pkg build fails, having very small windows of packages
being missing, and ending up with all pacakges safely up to date at
the end if all builds work.

        Greg Troxel <gdt@ir.bbn.com>