Re: kern/55663 (Support for EVFILT_USER in kqueue(2))

[Ruslan Nikolaev <nruslan_devel%yahoo.com@localhost>]

I greatly apologize for the delay. Recently, I had so many things to do...
Please keep in mind that my understanding may be incomplete since I have not really introduced this feature, I just ported it from FreeBSD/OpenBSD. In fact, I have far more experience with epoll() rather than kqueue(). So, let me start the discussion with the motivation first, and then we can go to specific technical details (in a separate email which I will try to prepare next as needed).

1. Why is it needed?

Quite a few applications use it. To be specific, please consider Nginx. Although, it *does* compile under NetBSD, several critical features are missing without EVFILT_USER.

Take a look at https://nginx.googlesource.com/nginx/+/refs/heads/master/src/core/ngx_thread_pool.c:

static ngx_int_t
ngx_thread_pool_init_worker(ngx_cycle_t *cycle)
{
...
    if (ngx_notify_init == NULL || ngx_notify == NULL) {
        ngx_log_error(NGX_LOG_ALERT, cycle->log, 0,
               "the configured event method cannot be used with thread pools");
        return NGX_ERROR;
    }

...


Note that if ngx_notify is *NOT* initialized to some function, Nginx will not support the thread pool mode, which is arguably a very important Nginx feature that boosts performance greatly!

So, where is this function initialized? For kqueue, Nginx provides https://nginx.googlesource.com/nginx/+/refs/heads/master/src/event/modules/ngx_kqueue_module.c

Note these lines:
static ngx_event_module_t  ngx_kqueue_module_ctx = {
...
        NULL,                              /* delete an connection */
#ifdef EVFILT_USER
        ngx_kqueue_notify_init,            /* init a notify */
        ngx_kqueue_notify,                 /* trigger a notify */
        NULL,                              /* close a notify */
#else
        NULL,                              /* init a notify */
        NULL,                              /* trigger a notify */
        NULL,                              /* close a notify */
#endif
...

So, in other words, Nginx compiles and runs but lacks the thread pool support, which is an important feature. I am not saying that they could not have implemented it differently, but quite a few applications just assume you have EVFILT_USER and do not provide any good fallback.
If we can just implement EVFILT_USER as in FreeBSD (regardless of its merits), that would take care of those cases completely.

As another example, see the bug report for mySQL: http://gnats.netbsd.org/54627
Without EVFILT_USER, it will even have uninitialized data

2. Regarding the EVFILT_USER itself, the FreeBSD man page says the following:

EVFILT_USER      Establishes a user event identified by ident which is
                 not associated with any kernel mechanism but is trig-
                 gered by user level code.  The lower 24 bits of the
                 fflags may be used for user defined flags and manipu-
                 lated using the following:

                 NOTE_FFNOP          Ignore the input fflags.

                 NOTE_FFAND          Bitwise AND fflags.

                 NOTE_FFOR           Bitwise OR fflags.

                 NOTE_FFCOPY         Copy fflags.

                 NOTE_FFCTRLMASK     Control mask for fflags.

                 NOTE_FFLAGSMASK     User defined flag mask for
                                     fflags.

                 A user event is triggered for output with the follow-
                 ing:

                 NOTE_TRIGGER        Cause the event to be triggered.

                 On return, fflags contains the users defined flags in
                 the lower 24 bits.
                 
Please let me know if EVFILT_USER may sound interesting for NetBSD. If so, I can try to recollect technical details (it has been really a while). But for the most part it was based on OpenBSD's patch (with some adaptations for NetBSD): https://www.mail-archive.com/tech%openbsd.org@localhost/msg42433.html


Thanks,
Ruslan