Re: pthread_rwlock starvation

[Sad Clouds <cryintothebluesky%googlemail.com@localhost>]

On Fri, 16 Apr 2010 19:46:32 -0700
Steve Woodford <steve%mctavish.co.uk@localhost> wrote:

> 
> On 16 Apr 2010, at 16:01, Sad Clouds wrote:
> > 
> > Any ideas?
> 
> What's the observed behaviour with a kernel compiled with "options
> HZ=1000" in the kernel config file?
> 
> Steve
> 

Same again, one CPU sitting idle for up to 30 seconds. This all happens
very randomly. I think HZ=1000 reduces the frequency of when this
happens, but doesn't fix it.

I've attached a program that reproduces the issue. Run it a couple of
times and watch top, or vmstat. Sooner or later one CPU will sit idle
100% for 10-30 seconds, until something nudges it back into work.

#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>

#define NDATA 1000
#define NLOCK 500
#define NTHREADS 2

pthread_rwlock_t lock[NLOCK];
int data[NDATA];

void *thread(void *arg)
{
        unsigned int seed = time(NULL);
        int lock_index, data_index;

        for(;;)
        {
                /* Calculate random index value */
                data_index = rand_r(&seed) % NDATA;
                lock_index = data_index % NLOCK;

                /* Get a read lock on this index */
                if(pthread_rwlock_rdlock(&(lock[lock_index])) != 0)
                {
                        printf("pthread_rwlock_wrlock() error\n");
                        exit(1);
                }
                if(data[data_index] != data_index)
                {
                        printf("Error on line %d\n", __LINE__);
                }
                if(pthread_rwlock_unlock(&(lock[lock_index])) != 0)
                {
                        printf("pthread_rwlock_wrlock() error\n");
                        exit(1);
                }

                /* Get a write lock on this index */
                data_index = rand_r(&seed) % NDATA;
                lock_index = data_index % NLOCK;

                if(pthread_rwlock_wrlock(&(lock[lock_index])) != 0)
                {
                        printf("pthread_rwlock_wrlock() error\n");
                        exit(1);
                }
                if(data[data_index] != data_index)
                {
                        printf("Error on line %d\n", __LINE__);
                }
                if(pthread_rwlock_unlock(&(lock[lock_index])) != 0)
                {
                        printf("pthread_rwlock_wrlock() error\n");
                        exit(1);
                }
        }
}

int main(void)
{
        int i;
        pthread_t tid[NTHREADS];

        /*
        Initialise locks and data
        */

        for(i = 0; i < NLOCK; i++)
        {
                if(pthread_rwlock_init(&(lock[i]), NULL) != 0)
                {
                        printf("pthread_rwlock_init() error\n");
                        exit(1);
                }

        }
        for(i = 0; i < NDATA; i++)
                data[i] = i;


        /*
        Create threads, 1 second apart
        */

        for(i = 0; i < NTHREADS; i++)
        {
                printf("Creating thread: %d\n", i);
                if(pthread_create(&tid[i], NULL, thread, NULL) != 0)
                {
                        printf("pthread_create() error\n");
                        exit(1);
                }
                sleep(1);
        }

        printf("Waiting for all threads\n", i);
        for(i = 0; i < NTHREADS; i++)
                pthread_join(tid[i], NULL);

        exit(0);
}