Re: pkg/24255 (pkg_delete removes meta data even if deinstall fails)

[Matthew Mondor <mm_lists%pulsar-zone.net@localhost>]

The following reply was made to PR pkg/24255; it has been noted by GNATS.

From: Matthew Mondor <mm_lists%pulsar-zone.net@localhost>
To: gnats-bugs%NetBSD.org@localhost
Cc: 
Subject: Re: pkg/24255 (pkg_delete removes meta data even if deinstall
 fails)
Date: Thu, 21 Apr 2011 05:49:01 -0400

 On Thu, 21 Apr 2011 03:49:39 +0000 (UTC)
 dholland%NetBSD.org@localhost wrote:
 
 > Synopsis: pkg_delete removes meta data even if deinstall fails
 > 
 > State-Changed-From-To: feedback->open
 > State-Changed-By: dholland%NetBSD.org@localhost
 > State-Changed-When: Thu, 21 Apr 2011 03:49:38 +0000
 > State-Changed-Why:
 > Problem still exists. Also, submitter wrote to the gnats junkmail address
 > to report that he can't test.
 
 Just so I have a clear view of what is ideally wanted:
 
 Currently, some files of a package might still subsist while the db
 metadata is lost/deleted.
 
 It would be possible to instead leave the meta information pending on
 error, at the risk of having some files lost/missing with inconsistent
 files/PLIST.
 
 I doubt it, but could access(2) reliably be used as a first pass before
 the unlink(2)/rmdir(2) phase to fail before performing any
 non-reversible operation?
 
 Alternatively, should packages also support a "partial/broken" state,
 where metadata would be updated to reflect actual remaining files after
 an incomplete removal?  Possibly keeping the original PLIST for
 reference but adding a PARTIAL-PLIST for such packages?
 
 
 Or should a transactional/atomic system be expected instead?
 
 In which case I could see various imperfect solutions (FFS not
 helping much here):
 
 - Obtain a global exclusive pkg advisory lock
 
 - Backup the package files and attempt to delete the original files; if
 everything is successful, delete the backup and metadata, otherwise
 restore the deleted files from the backup and preserve the metadata.
 This should take into account file permissions, flags, PaX flags, etc.
 As drawbacks, some space would be required to store the temporary
 backup, and some effort made to sync all file permissions and flags,
 code which might be duplicated among the various backup utilities
 supporting some or all of these flags (not to mention the OS-specifics).
 Perhaps acceptable with a dependency on pax(1)?
 
 - Instead of deleting files, move them to a temporary "trashcan"
 directory tree on the same filesystem (on write failure, fail
 immediately); if all files were successfully moved, ensure that their
 permissions allow deletion and recursively delete the temporary tree,
 then the metadata.
 On error move successfully-moved files back where they belong,
 restoring their permissions, and leave the metadata as-is, since we
 restored the tree state.  Seems flaky; I'm not sure this would
 succeed in all cases where package removal now fails.  We also could
 end up with a growing trash directory if files could be moved yet could
 not be deleted.
 
 - fsync(2) and release the pkg lock
 
 
 Unfortunately, even the above solutions might yield inconsistent state
 in the advent of a crash or user interrupt.  NetBSD doesn't have
 transactional FS and support syscalls, and even if it did this might
 not exist on most other pkgsrc-supported OSs...
 
 
 Other more far-fetched "atomic" support built on non-transactional FS:
 
 I guess that a variant could be using a NetBSD-supported transactional
 "trashcan" per file system along with its atomic-simulated
 auto-restore/async-delete system.  Such a trashcan might include two
 sections, one for user-deleted files and another for system-deleted
 files.  Files would be stored with their original path/flags.
 User-deleted files would require explicit "empty/restore" commands, yet
 files destined for system-removal would be enclosed in transactions and
 could be restored at crash/reboot in case of an incomplete transaction,
 with a thread/daemon responsible for asynchroneously deleting files
 deleted in complete transactions.
 Transactions metadata could be implemented on top of bdb, sqlite or a
 custom system supporting a transactional log (using fdatasync(2) at
 reasonable intervals and automatic log file rotation/removal, etc).
 pkgsrc when on NetBSD could then make use of that "atomic multi-delete
 operation", despite it being non-portable.  Perhaps that some files
 might still be dangling over time and that a nightly script should warn
 about it.
 
 A variant of the above might be directly provided by pkgsrc, with
 reliance on the required portable tools such as sqlite used for package
 metadata and transactional-log, with $PKGDIR/tmp/trash used as a
 temporary trash directory, an asynchroneous files cleanup cron event or
 daemon, with a boot rollback/recovery script.
 Unfortunately complicates pkgsrc, adds new tool dependencies and makes
 it more system-invasive...  Perhaps that instead of having a
 boot-time recovery script and a cleanup cron event/daemon, every pkg_*
 command could as a prologue perform those verification/pending
 operations as needed, to minimise system invasion, with failed
 procedures logged to stderr as a warning before attempting the
 requested operation, with some loss of performance...
 
 Other suggestions?
 
 Thanks,
 -- 
 Matt