This is a followup to the previous note about having struct vnode contain
a struct lock.

I've found a problem in the current code which will be much worse with
this change: If a vnode on the underlying layer is cleaned, the upper
vnode needs to be reclaimed too. This is especially important with the
struct lock pointers as the upper node will want to use the lower node's
struct lock!

The only time this problem will occur, AFAICT, is if someone forcibly
unmounts the underlying fs of a nullfs. All other calls to vclean are
cleaning vnodes on the free list, and the underlying node won't get there
until (all) the upper nodes get vcleaned (due to the upper node holding a
reference on the lower). But we should still do the right thing in the
case of an unmount.

I have a two-part solution. The first part, which is fairly firm :-) is to
add a queue (probably TAILQ) head and entry in struct vnode, along with a
simple_lock to protect access.

Whenever a vnode gets stacked on top of another node, it locks the
simple_lock in the underlying node, and then inserts itself in the list.
When it gets reclaimed, it (locks the underlying simple_lock and) removes
itself.

The second half is what the underlying node does with this. I think the
best thing is that vclean would look at the list on the vnode it is
cleaning, and repeatedly call vclean on the head vnode. Then it would
vclean the given node. Oh, it would have to release the interlock first.

Thoughts?

Take care,

Bill