vnode lifecycle change for ZFS

[Chuck Silvers <chuq%chuq.com@localhost>]

hi folks,

while working on updating ZFS to the current freebsd code I've discovered
that the vnode lifecycle changes in the last couple of years prevent ZFS
from working, specifically that calling vcache_get() from within VOP_FSYNC()
or VOP_PUTPAGES() while the vnode is being cleaned for reuse (vi_state ==
VS_RECLAIMING) results in a deadlock.  ZFS does this, and I don't see any
easy way to change ZFS to not do that, so I'd like to make the following
changes to the vnode lifecycle code to support ZFS:

 - split the VS_RECLAIMING state into VS_CLEANING and VS_RECLAIMING,
   using VS_CLEANING while the vnode is being cleaned and VS_RECLAIMING
   for the rest of the reuse process.  this would only be for file systems
   that indicate they need the extra state by setting a new IMNT_CLEANING flag;
   file systems that don't need this would continue to use VS_RECLAIMING while
   cleaning and avoid the extra locking needed by the extra state transition.
 - add new interfaces vcache_get_cleaner() and vrele_async_cleaner() for
   taking and releasing a reference when the vnode might be being cleaned
   as part of reuse.  my implementation has the restriction that only one
   such "cleaner" reference can be taken on a given vnode at a time
   since that is all that ZFS needs.  allowing multiple would require
   adding a counter to the vnode, which use extra memory for no benefit.

a diff implementing this is at http://ftp.netbsd.org/pub/NetBSD/misc/chs/diff.vs_cleaning.2

comments?

-Chuck