lib/59137: pthread_atfork deadlocks if called in atfork handler

[campbell+netbsd%mumble.net@localhost]

>Number:         59137
>Category:       lib
>Synopsis:       pthread_atfork deadlocks if called in atfork handler
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Mar 05 21:00:01 +0000 2025
>Originator:     Taylor R Campbell
>Release:        current, 10, 9, ...
>Organization:
The NetBSD Atforkatforkation
>Environment:
>Description:
What happens if a pthread_atfork prefork handler calls pthread_atfork?

POSIX is fuzzy on the semantics (and hints that pthread_atfork is destined for the dustbin of history): https://pubs.opengroup.org/onlinepubs/9799919799.2024edition/functions/pthread_atfork.html 

NetBSD's libpthread currently deadlocks in this case, because it takes a lock against itself.  It could fail with EDEADLK, though, since pthread_atfork is supposed to have the opportunity for failure, although the only failure POSIX enumerates for pthread_atfork is ENOMEM.

glibc quietly allows it, but then violates the contract by calling a postfork handler in the parent and child that doesn't correspond to any prefork handler that had been called in the parent.  It is not possible to call the prefork handler too because that would violate the ordering requirement: prefork handlers must be called in reverse order of registration.  Thus, if prefork handler A installs prefork handler B, it's too late to call B because the order has to be B-then-A; A-then-B is forbidden.

Another option would be to quietly allow it but only have it affect subsequent forks.
>How-To-Repeat:
#include <sys/wait.h>

#include <err.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

static void
prefork(void)
{
	fprintf(stderr, "prefork handler\n");
	if (pthread_atfork(NULL, NULL, NULL) == -1)
		err(1, "pthread_atfork 2");
	fprintf(stderr, "second atfork handler installed\n");
}

int
main(void)
{
	pid_t pid;
	int status;

	if (pthread_atfork(prefork, NULL, NULL) == -1)
		err(1, "pthread_atfork 1");
	alarm(1);
	fprintf(stderr, "call fork\n");
	if ((pid = fork()) == -1)
		err(1, "fork");
	if (pid == 0)
		_exit(0);
	fprintf(stderr, "fork returned\n");
	alarm(0);
	if (waitpid(pid, &status, 0) == -1)
		err(1, "waitpid");
	if (!WIFEXITED(status) || WEXITSTATUS(status) != 0)
		errx(1, "child exited 0x%x\n", status);
	fflush(stdout);
	return ferror(stdout);
}
>Fix:
Whether or not we decide to `fix' this (whether or not we consider it a bug), we should at least write down the rationale for our choice in a place that is easily findable, like this gnats bug database.