swapcontext() around pthreads

[Emmanuel Dreyfus <manu%netbsd.org@localhost>]

Hi

I encountered a funny portability problem when working on glusterfs.
In its 3.3. branch, it makes heavy use of swapcontext() and pthreads
to get better performance. Unfortunately the code assumes a Linux
specific behavior : a thread calling swapcontext() should not affect 
other threads. Only the calling thread context should be changed.

NetBSD has a much more rich/messy behavior. Typical usage us to call
getcontext() and makecontext() prior calling swapcontext(). 
If they all happen in the same thread, everything works like on 
Linux.

However, if getcontext() was called in thread A and swapcontext()
is called in thread B, then swapcontext causes the context of thread
A to be changed, preempting the code being executed. thread B seems to
terminate, though I suspect that could be changed with makecontext() 
and uc_link.

Attached is a test case that exhibit the behavior, and here is the output:
netbsd# ./tss
before swapcontext self = 0xbb600000
after swapcontext self = 0xbfa00000
linux# ./tss
before swapcontext self = 0x4002
after swapcontext self = 0x4002

This seems to fall into a grey area of unspecified behavior. Standards
cannot help much since that functions were removed from POSIX.1. 

Questions:
1) do you think the NetBSD behhavior is a bug?
2) it would be nice to support the Linux behavior, at least optionnally
how hard would it be to change that? Anyone has a clear vision of what 
should be done?

-- 
Emmanuel Dreyfus
manu%netbsd.org@localhost

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>
#include <err.h>
#include <sysexits.h>
#include <signal.h>
#ifdef __linux__
#include <bits/sigstack.h>
#endif
#include <ucontext.h>

char stack[66536];
ucontext_t ctx;
ucontext_t octx;

void wrap(int);
void *thread1(void *);

void
wrap(cookie)
        int cookie;
{
        printf("after swapcontext self = %p\n", (void *)pthread_self());

        return;
}

void *
thread1(arg)
        void *arg;
{

        usleep(50);

        ctx.uc_stack.ss_sp = stack;
        ctx.uc_stack.ss_size = sizeof(stack);
        
        makecontext(&ctx, (void *)*wrap, 2, (int)0xdeadbeef);

        printf("before swapcontext self = %p\n", (void *)pthread_self());
        if (swapcontext(&octx, &ctx) != 0)
                err(EX_OSERR, "swapcontext failed");
        

        errx(EX_SOFTWARE, "swapcontext returned");
        /* NOTREACHED */
        return NULL;
}


int
main(argc, argv)
        int argc;
        char **argv;
{
        int error;
        pthread_t th;

        if ((error = pthread_create(&th, NULL, *thread1, NULL)) != 0)
                err(EX_OSERR, "pthread_create returns error = %d", error);

        if (getcontext(&ctx) != 0)
                err(EX_OSERR, "getcontext failed");

        sleep(2);
        printf("main exits\n");

        return EX_OK;
}