kern/47439: open() does not check for invalid flags

[dholland%netbsd.org@localhost]

>Number:         47439
>Category:       kern
>Synopsis:       open() does not check for invalid flags
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Jan 12 22:25:00 +0000 2013
>Originator:     David A. Holland
>Release:        NetBSD 6.99.10 (20120731) also 6.99.16 (20130112)
>Organization:
>Environment:
System: NetBSD valkyrie 6.99.10 NetBSD 6.99.10 (VALKYRIE) #11: Sun Dec  9 
20:42:51 EST 2012  root@valkyrie:/usr/src/sys/arch/amd64/compile/VALKYRIE amd64
Architecture: x86_64
Machine: amd64
>Description:

open() does not reject invalid flags. It ignores them, which is
reasonably harmless (especially since there aren't that many unused
bits in that flags word) but is silly/stupid.

It does not appear to be possible to set the internal kernel-only flag
bits as open_setfp() ANDs with FMASK.

>How-To-Repeat:

Code reading, plus:

#include <unistd.h>
#include <fcntl.h>
#include <err.h>

#define O_STUPID 0x30000000 /* always make sure this is really invalid */

int main() {
   int fd;

   fd = open("testfile", O_CREAT|O_TRUNC|O_WRONLY|O_STUPID, 0664);
   if (fd < 0) {
      warn("O_STUPID");
   }
   else {
      warnx("O_STUPID was accepted");
   }
   return 0;
}

>Fix:

Straightforward.