lib/38087: pthread_mutex_trylock on an unlocked recursive mutex fails with EBUSY

[rafal%netbsd.org@localhost]

>Number:         38087
>Category:       lib
>Synopsis:       pthread_mutex_trylock fails w/unlocked recursive mutex
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Feb 22 23:10:00 +0000 2008
>Originator:     rafal%netbsd.org@localhost
>Release:        NetBSD 4.99.54
>Organization:
        Rafal's House of Hackable Hardware
>Environment:
System: NetBSD fearless-vampire-killer 4.99.54 NetBSD 4.99.54 (FVK) #6: Wed Feb 
20 23:42:11 EST 2008 
rafal@fearless-vampire-killer:/extra/sparc64/obj/sys/arch/sparc64/compile/FVK 
sparc64
Architecture: sparc64
Machine: sparc64
>Description:
        Calling pthread_mutex_trylock() on a newly created, recursive mutex
        should succeed and lock the mutex.  However, the latest changes to
        libpthread broke this; now pthread_mutex_trylock() returns EBUSY
        if any of the ancillary bits in the ptm_owner field are set (which
        is how mutexes are marked as recursive).

>How-To-Repeat:

#include <stdio.h>
#include <pthread.h>
#include <errno.h>

main()
{
        int rc;
        pthread_mutex_t my_mtx;
        pthread_mutexattr_t  attr;

        printf("pthread_self = %08x\n", pthread_self());
        pthread_mutexattr_init(&attr);
        pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE);

        rc = pthread_mutex_init(&my_mtx, &attr);
        pthread_mutexattr_destroy(&attr);
        printf("mutex init returned %d (errno %d)\n", rc, errno);

        rc = pthread_mutex_trylock(&my_mtx);
        printf("trylock returned %d (errno %d)\n", rc, errno);

        rc = pthread_mutex_lock(&my_mtx);
        printf("lock returned %d (errno %d)\n", rc, errno);
}

Compile, run, and note that you get the following output:

pthread_self = ffe00000
mutex init returned 0 (errno 0)
trylock returned 16 (errno 0)           <<< Note this
lock returned 0 (errno 0)               <<< and this (lock does succeed)

On another (sparc64) machine running a -current from 1/11/2008, I get:

pthread_self = ffe00000
mutex init returned 0 (errno 0)
trylock returned 0 (errno 0)            <<< Note this
lock returned 0 (errno 0)

>Fix:

Here's a quick hack I came up with that fixes the mis-feature, but it's
probably not the optimal fix.

Index: pthread_mutex.c
===================================================================
RCS file: /cvsroot/src/lib/libpthread/pthread_mutex.c,v
retrieving revision 1.45
diff -u -p -u -p -r1.45 pthread_mutex.c
--- pthread_mutex.c     14 Feb 2008 21:40:51 -0000      1.45
+++ pthread_mutex.c     22 Feb 2008 22:37:40 -0000
@@ -322,7 +322,7 @@ int
 pthread_mutex_trylock(pthread_mutex_t *ptm)
 {
        pthread_t self;
-       void *val;
+       void *val, *new, *next;
 
        self = pthread__self();
        val = atomic_cas_ptr(&ptm->ptm_owner, NULL, self);
@@ -333,6 +333,18 @@ pthread_mutex_trylock(pthread_mutex_t *p
                return 0;
        }
 
+       if (MUTEX_OWNER(val) == 0) {
+               new = (void *) ((uintptr_t)self | (uintptr_t)val);
+               next = atomic_cas_ptr(&ptm->ptm_owner, val, new);
+
+               if (__predict_true(next == val)) {
+#ifndef PTHREAD__ATOMIC_IS_MEMBAR
+                       membar_enter();
+#endif
+                       return 0;
+               }
+       }
+
        if (MUTEX_OWNER(val) == (uintptr_t)self && MUTEX_RECURSIVE(val)) {
                if (ptm->ptm_recursed == INT_MAX)
                        return EAGAIN;

>Unformatted:
                Sources from ~ 2/19 1600 UTC.