Let me toss in an old idea I had, that I never did anything with.

The basic problem here is that vnode locks are interdependent (form
graphs) and unconstrained graphs lead to loops (deadlock).  (As long
as the graph is constrained to linear or tree-structured, you are
okay.)

So, instead of the usual approaches -- "constrain the graph" or
"put in deadlock detectors" -- I once proposed that BSD try to
do away with the interdependencies entirely.

A standard "problem example" is file creation or deletion, where
UFS wants to lock the parent directory, so that no other operation
can sneak in and modify the directory "in between".  An alternative
for this particular case is to protect the directory with a capability
number (rather like the namei cache numbers): when the create or
remove op starts, it gets the number for the parent directory, and
when it goes to commit, it checks the number.  If the number has
changed, the directory has been modified, and you simply start
over.

I am not sure how far this can be pushed (and was never all that
hot on pushing it very far in the first place), but it might go
far enough.

The other real alternative, if you are sure you want to give up the
graph constraints, is to build a full-blown deadlock detector.

Chris