Revoke() and VOP_OPEN() vs. vref()

[Juergen Hannken-Illjes <hannken%eis.cs.tu-bs.de@localhost>]

Trying to fix PR 43456 "KASSERT from ptyfs null mount" I'm a bit lost.

In short we run ksh with its controlling tty on a null mounted ptyfs.
When the shell exits we get the warning "vrelel: missing VOP_CLOSE()"
and then the assertion "sn->sn_opencnt == 0" fails.

This seems to originate from exit1() revoking the controlling tty.  This
tty gets opened from cttyopen() but doesn't get an additional reference
for this open.

Is this usage correct?

I have no idea where this tty gets closed.

Some days ago I introduced layer_revoke() to adjust the reference count
before passing the op down the stack.  While this works well with the
test case from PR 43456 and some other test cases, it fails for the original
problem for this PR.

Using a big hammer and changing layer_revoke() to

        vref(lvp);
        error = LAYERFS_DO_BYPASS(vp, ap);
        vrele(lvp);

works but this way revoke() through a layered file system will always
inactivate and close the lower vnode so I'm not sure this is a real solution.

-- 
Juergen Hannken-Illjes - hannken%eis.cs.tu-bs.de@localhost - TU Braunschweig 
(Germany)