Re: Open master pty (/dev/ptmx) non blocking

["David H. Gutteridge" <david%gutteridge.ca@localhost>]

On Fri, 23 Sep 2022 at 20:14:23 +0000, David Holland wrote:
> On Fri, Sep 23, 2022 at 01:39:16PM +0200, Martin Husemann wrote:
> > > Then, shouldn't the open(2) (and posix_openpt(3)) at least fail with
> > > EINVAL or something if other flags are specified?
> >
> > The man page says:
> >
> >      Note that unlike implementations on some other operating systems,
> >      posix_openpt() does not return EINVAL if the value of oflag would be
> >      deemed invalid, instead it is simply ignored.  This means it is not
> >      possible to dynamically test which open(2) flags are possible to set, and
> >      apply a fallback if EINVAL is received.
>
> That is, however, kind of a feeble excuse :-)

Agreed. But I inferred this was perhaps done this way on purpose, and
could be controversial to change. (Would also then mean downstream code
would have to special-case NetBSD releases and/or add tests which could
be cumbersome.) I hadn't got around to asking, I was dealing with
documenting the reality and working with the vte project so that NetBSD
support for gnome-terminal/mate-terminal/xfce4-terminal/et al. was
accepted.

> While my inclination would be to make it work, until someone wants to
> figure out how to do that it seems straightforward to make O_NONBLOCK
> fail:
>
> Index: tty_ptm.c
> ===================================================================
> RCS file: /cvsroot/src/sys/kern/tty_ptm.c,v
> retrieving revision 1.43
> diff -u -p -r1.43 tty_ptm.c
> --- tty_ptm.c	29 Jun 2021 22:40:53 -0000	1.43
> +++ tty_ptm.c	23 Sep 2022 20:12:07 -0000
> @@ -338,6 +338,10 @@ ptmopen(dev_t dev, int flag, int mode, s
> 	dev_t ttydev;
> 	struct mount *mp;
>
> +	if (flag & O_NONBLOCK) {
> +		return EINVAL;
> +	}
> +
> 	switch(minor(dev)) {
> 	case 0:		/* /dev/ptmx */
> 	case 2:		/* /emul/linux/dev/ptmx */

It's not just O_NONBLOCK that can be expected/desired, vte wants to set
O_CLOEXEC as well. The Linux kernel accepts and applies both of those
flags in a posix_openpt(3) call.

OpenBSD does this validation in posix_openpt(3) directly:

	/* User must specify O_RDWR in oflag. */
	if ((oflag & O_ACCMODE) != O_RDWR ||
	    (oflag & ~(O_ACCMODE | O_NOCTTY)) != 0) {
		errno = EINVAL;
		return -1;
	}

Now, that doesn't cover the case that someone simply calls open(2) on
the ptmx cloning device directly, so safer to be placed where you're
suggesting. (And I make no claims to being an authority on any of
this, I've simply looked at it a bit, and may well have missed
something.)

Regards,

Dave