kern/50094: not all threads woken up when multiple threads waiting with keven on same kqueue

[cmeerw%cmeerw.org@localhost]

>Number:         50094
>Category:       kern
>Synopsis:       not all threads woken up when multiple threads waiting with keven on same kqueue
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Jul 26 19:10:00 +0000 2015
>Originator:     Christof Meerwald
>Release:        netbsd-7
>Organization:
>Environment:
NetBSD droid 7.0_RC1 NetBSD 7.0_RC1 (ODROID-C1.201507211940Z) evbarm
>Description:
as reported in http://mail-index.netbsd.org/tech-userlevel/2015/07/26/msg009262.html

I have got multiple threads waiting on a single kqueue and want to
wake up all these threads at some point (before exiting). I believe
the way to do it is to have a dummy socket/pipe added to the kqueue
and close the other end (I have also found
http://thread.gmane.org/gmane.os.netbsd.devel.kernel/28051 which seems
to agree on that approach).

But when I am actually testing this approach, some threads don't seem
to be woken up and keep waiting in the kevent syscall. Source code for
a test case is available from
http://svn.cmeerw.net/src/nginetd/trunk/test/kqtest-wakeup.cc

I have tested on my ODROID-C1 (quad core ARMv7) with netbsd-7 (build
from a few days ago) and seen the problem occur with 3 or more
threads.

This has also been confirmed in netbsd-6 on x86 in http://mail-index.netbsd.org/tech-userlevel/2015/07/26/msg009257.html
>How-To-Repeat:
compile with g++ -pthread and run specifying the number of threads (e.g. 16) as the argument. Test case should consistently terminate after a 1 second sleep.


#include <stdio.h>
#include <stdlib.h>

#include <errno.h>
#include <fcntl.h>
#include <unistd.h>

#include <arpa/inet.h>
#include <sys/event.h>
#include <sys/fcntl.h>
#include <sys/socket.h>
#include <sys/time.h>

#include <pthread.h>


namespace
{
  extern "C" void *worker(void *arg)
  {
    const int kqid = *static_cast<int *>(arg);

    struct kevent events[16];
    bool done = false;

    while (!done)
    {
      int rc_kevent = kevent(kqid, NULL, 0,
			     events, sizeof(events)/sizeof(*events),
			     NULL);
      done = rc_kevent != 0;
    }
  }
}


int main(int argc, const char * const argv[])
{
  const int nr_threads = (argc > 1) ? atoi(argv[1]) : 1;

  int kqid(kqueue());

  int syncsockets[2];
  socketpair(AF_UNIX, SOCK_STREAM, 0, syncsockets);
  {
    struct kevent const events[] = {
      { syncsockets[1], EVFILT_READ, EV_ADD, 0, 0, 0 },
    };
    kevent(kqid, events, 1, NULL, 0, NULL);
  }

  pthread_t *threads = new pthread_t[nr_threads];
  for (int i = 0; i < nr_threads; ++i)
  {
    pthread_create(threads + i, NULL, worker, &kqid);
  }

  sleep(1);

  close(syncsockets[0]);

  for (int i = 0; i < nr_threads; ++i)
  {
    pthread_join(threads[i], NULL);
  }

  close(syncsockets[1]);
  close(kqid);

  delete [] threads;

  return 0;
}
>Fix: