[src/trunk]: src/sys/kern Improve Christos's vn_open fix.

[dholland <dholland%NetBSD.org@localhost>]

details:   https://anonhg.NetBSD.org/src/rev/d0fa5570297b
branches:  trunk
changeset: 380014:d0fa5570297b
user:      dholland <dholland%NetBSD.org@localhost>
date:      Wed Jun 30 17:51:49 2021 +0000

description:
Improve Christos's vn_open fix.

- assert about api misuse up front (suggested by riastradh)
- restore the behavior of returning EOPNOTSUPP if ret_fd is NULL and we
  get a fd back (otherwise things like ktruss -o /dev/stderr panic)
- clear error to 0 for the EDUPFD and EMOVEFD cases so opening a
  cloner succeeds

diffstat:

 sys/kern/vfs_vnops.c |  15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)

diffs (43 lines):

diff -r c613cba0deac -r d0fa5570297b sys/kern/vfs_vnops.c
--- a/sys/kern/vfs_vnops.c      Wed Jun 30 17:34:03 2021 +0000
+++ b/sys/kern/vfs_vnops.c      Wed Jun 30 17:51:49 2021 +0000
@@ -1,4 +1,4 @@
-/*     $NetBSD: vfs_vnops.c,v 1.217 2021/06/30 11:20:32 christos Exp $ */
+/*     $NetBSD: vfs_vnops.c,v 1.218 2021/06/30 17:51:49 dholland Exp $ */
 
 /*-
  * Copyright (c) 2009 The NetBSD Foundation, Inc.
@@ -66,7 +66,7 @@
  */
 
 #include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: vfs_vnops.c,v 1.217 2021/06/30 11:20:32 christos Exp $");
+__KERNEL_RCSID(0, "$NetBSD: vfs_vnops.c,v 1.218 2021/06/30 17:51:49 dholland Exp $");
 
 #include "veriexec.h"
 
@@ -173,6 +173,9 @@ vn_open(struct vnode *at_dvp, struct pat
 
        KASSERT((nmode & (TRYEMULROOT | NOCHROOT)) == nmode);
 
+       KASSERT(ret_vp != NULL);
+       KASSERT((ret_domove == NULL) == (ret_fd == NULL));
+
        if ((fmode & (O_CREAT | O_DIRECTORY)) == (O_CREAT | O_DIRECTORY))
                return EINVAL;
 
@@ -332,9 +335,11 @@ out:
        case EDUPFD:
        case EMOVEFD:
                /* if the caller isn't prepared to handle fds, fail for them */
-               KASSERTMSG(ret_domove && ret_fd,
-                   "caller did not supply ret_domove and ret_fd for %d",
-                   error);
+               if (ret_fd == NULL) {
+                       error = EOPNOTSUPP;
+                       break;
+               }
+               error = 0;
                *ret_vp = NULL;
                *ret_domove = error == EMOVEFD;
                *ret_fd = l->l_dupfd;