lib/58266: dup3 behavior differs from FreeBSD, OpenBSD, and glibc.

[collin.funk1%gmail.com@localhost]

>Number:         58266
>Category:       lib
>Synopsis:       dup3 behavior differs from FreeBSD, OpenBSD, and glibc.
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat May 18 23:05:00 +0000 2024
>Originator:     Collin Funk
>Release:        NetBSD 10.0 amd64
>Organization:
GNU
>Environment:
NetBSD netbsd 10.0 NetBSD 10.0 (GENERIC) #0: Thu Mar 28 08:33:33 UTC 2024  mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/amd64/compile/GENERIC amd64

>Description:
Hello,

A CI job on NetBSD 9.0 and NetBSD 10.0 caught some test failures in Gnulib, one of which being dup3 [1]. I've reproduced it on a NetBSD 10.0 virtual machine and found a fix for Gnulib.

However, when looking into the test failure I noticed that dup3 on NetBSD behaves differently than FreeBSD, OpenBSD, and glibc.

>From the Linux man pages describing the return value [2]:

    If oldfd equals newfd, then dup3() fails with the error EINVAL.

The man pages for FreeBSD and OpenBSD describe this same behavior [3] [4].

On NetBSD the man pages describe a different behavior [5]:

     These calls return the new file descriptor value.  In the case of dup2()
     and dup3() this is always the same as newfd.  If an error occurs, the
     value -1 is returned and errno is set to indicate what happened.

I'm not sure if there was a reasoning behind this difference, so I figured it was worth reporting. I've taken the test case that fails and put it in a sample program attached to help reproduce the behavior. All the other test cases in Gnulib seem to work as expected besides that one.

[1] https://lists.gnu.org/archive/html/bug-gnulib/2024-05/msg00271.html
[2] https://man7.org/linux/man-pages/man2/dup3.2.html
[3] https://man.freebsd.org/cgi/man.cgi?query=dup3&sektion=3
[4] https://man.openbsd.org/dup.2
[5] https://man.netbsd.org/dup3.2
>How-To-Repeat:

I've extracted the relevant test case and placed it in this program:

=====================================================
#define _GNU_SOURCE 1
#include <unistd.h>
#include <fcntl.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <errno.h>

int
main (void)
{
  const char *file = "test-dup3.tmp";
  int fd = open (file, O_CREAT | O_TRUNC | O_RDWR, 0600);
  int result = dup3 (fd, fd, 0);
  int saved_errno = errno;

  /* Check that the file could be created.  */
  if (fd == -1)
    abort ();
  /* Cleanup the file and restore errno.  */
  (void) unlink (file);
  errno = saved_errno;
  /* Expect dup3 to fail with errno == EINVAL.  */
  if (result != -1)
    {
      (void) close (result);
      fprintf (stderr, "dup3 (fd, fd, 0) should fail with EINVAL\n");
      abort ();
    }
  printf ("%s\n", strerror (errno));
  return 0;
}
=====================================================

Here are the results on different platforms. Unfortunately, I don't have access to an OpenBSD machine but I expect it to behave like GNU/Linux and FreeBSD.

On GNU/Linux and FreeBSD:
$ gcc main.c
$ ./a.out 
Invalid argument
$ echo $?
0

On NetBSD:
$ gcc main.c 
$ ./a.out 
dup3 (fd, fd, 0) should fail with EINVAL
Abort trap (core dumped)

>Fix:
I believe that a small change in the 'dodup' function in src/sys/kern/sys_descrip.c should change the behavior to align with the other systems. I am not too familiar with NetBSD though so I may be missing something.