lib/59135: PTHREAD_CANCEL_ASYNCHRONOUS doesn't do much

[campbell+netbsd%mumble.net@localhost]

>Number:         59135
>Category:       lib
>Synopsis:       PTHREAD_CANCEL_ASYNCHRONOUS doesn't do much
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Mar 05 16:40:01 +0000 2025
>Originator:     Taylor R Campbell
>Release:        current, 10, 9, ...
>Organization:
The PthreadBSD Foun
>Environment:
>Description:
Under POSIX, when a thread has asynchronous cancellation configured with pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, ...), cancellation should be able to happen at any time, not just at cancellation points:

> Each thread maintains its own cancelability state, which may be encoded in two bits:
>  1. Cancelability-Enable: ...
>  2. Cancelability Type: When cancelability is enabled and the cancelability type is PTHREAD_CANCEL_ASYNCHRONOUS (as defined in <pthread.h>), new or pending cancellation requests may be acted upon at any time. When cancelability is enabled and the cancelability type is PTHREAD_CANCEL_DEFERRED (as defined in <pthread.h>), cancellation requests are held pending until a cancellation point (see below) is reached....

However, NetBSD's libpthread doesn't really do much with PTHREAD_CANCEL_ASYNCHRONOUS.
>How-To-Repeat:
#include <err.h>
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>

static void
cleanup(void *cookie)
{
	fprintf(stderr, "cleanup\n");
}

static void *
thread(void *cookie)
{
	pthread_barrier_t *bar = cookie;
	int error;

	error = pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);
	if (error)
		errc(1, error, "pthread_setcanceltype");
	pthread_cleanup_push(cleanup, NULL);
	fprintf(stderr, "loop forever\n");
	(void)pthread_barrier_wait(bar);
	for (;;) {
		__asm("");
	}
	pthread_cleanup_pop(0);
	return NULL;
}

int
main(void)
{
	pthread_barrier_t bar;
	pthread_t t;
	int error;

	error = pthread_barrier_init(&bar, NULL, 2);
	if (error)
		errc(1, error, "pthread_barrier_init");

	fprintf(stderr, "create thread\n");
	error = pthread_create(&t, NULL, &thread, &bar);
	if (error)
		errc(1, error, "pthread_create");
	(void)pthread_barrier_wait(&bar);
	fprintf(stderr, "cancel\n");
	error = pthread_cancel(t);
	if (error)
		errc(1, error, "pthread_cancel");
	(void)alarm(1);
	error = pthread_join(t, NULL);
	if (error)
		errc(1, error, "pthread_join");
	(void)alarm(0);
	fprintf(stderr, "ok\n");
	return 0;
}
>Fix:
1. Invent a new signal, say SIGCANCEL.
2. Teach libc's signal/sigaction/sigprocmask stubs to pretend it doesn't exist.
3. Teach libpthread to set up a SIGCANCEL handler that does pthread_cancel as appropriate.
4. Shake out all the bugs.
5. Shake out some more bugs.
6. ???
7. Profit!
8. ...just one or two more bugs