Re: kern/55663 (Support for EVFILT_USER in kqueue(2))

[Ruslan Nikolaev <nruslan_devel%yahoo.com@localhost>]

[I am re-sending the email, the previous message was corrupted by the 
email client ]

I greatly apologize for the delay. Recently, I had so many things to do...
Please keep in mind that my understanding may be incomplete since I have 
not really introduced this feature, I just ported it from 
FreeBSD/OpenBSD. In fact, I have far more experience with epoll() rather 
than kqueue(). So, let me start the discussion with the motivation 
first, and then we can go to specific technical details (in a separate 
email which I will try to prepare next as needed).

1. Why is it needed?

Quite a few applications use it. To be specific, please consider Nginx. 
Although, it *does* compile under NetBSD, several critical features are 
missing without EVFILT_USER.

Take a look at 
https://nginx.googlesource.com/nginx/+/refs/heads/master/src/core/ngx_thread_pool.c:

static ngx_int_t
ngx_thread_pool_init_worker(ngx_cycle_t *cycle)
{
...
    if (ngx_notify_init == NULL || ngx_notify == NULL) {
        ngx_log_error(NGX_LOG_ALERT, cycle->log, 0,
               "the configured event method cannot be used with thread 
pools");
        return NGX_ERROR;
    }

...


Note that if ngx_notify is *NOT* initialized to some function, Nginx 
will not support the thread pool mode, which is arguably a very 
important Nginx feature that boosts performance greatly!

So, where is this function initialized? For kqueue, Nginx provides 
https://nginx.googlesource.com/nginx/+/refs/heads/master/src/event/modules/ngx_kqueue_module.c

Note these lines:
static ngx_event_module_t  ngx_kqueue_module_ctx = {
...
        NULL,                              /* delete an connection */
#ifdef EVFILT_USER
        ngx_kqueue_notify_init,            /* init a notify */
        ngx_kqueue_notify,                 /* trigger a notify */
        NULL,                              /* close a notify */
#else
        NULL,                              /* init a notify */
        NULL,                              /* trigger a notify */
        NULL,                              /* close a notify */
#endif
...

So, in other words, Nginx compiles and runs but lacks the thread pool 
support, which is an important feature. I am not saying that they could 
not have implemented it differently, but quite a few applications just 
assume you have EVFILT_USER and do not provide any good fallback.
If we can just implement EVFILT_USER as in FreeBSD (regardless of its 
merits), that would take care of those cases completely.

As another example, see the bug report for mySQL: 
http://gnats.netbsd.org/54627
Without EVFILT_USER, it will even have uninitialized data

2. Regarding the EVFILT_USER itself, the FreeBSD man page says the 
following:

EVFILT_USER      Establishes a user event identified by ident which is
                 not associated with any kernel mechanism but is trig-
                 gered by user level code.  The lower 24 bits of the
                 fflags may be used for user defined flags and manipu-
                 lated using the following:

                 NOTE_FFNOP          Ignore the input fflags.

                 NOTE_FFAND          Bitwise AND fflags.

                 NOTE_FFOR           Bitwise OR fflags.

                 NOTE_FFCOPY         Copy fflags.

                 NOTE_FFCTRLMASK     Control mask for fflags.

                 NOTE_FFLAGSMASK     User defined flag mask for
                                     fflags.

                 A user event is triggered for output with the follow-
                 ing:

                 NOTE_TRIGGER        Cause the event to be triggered.

                 On return, fflags contains the users defined flags in
                 the lower 24 bits.

Please let me know if EVFILT_USER may sound interesting for NetBSD. If 
so, I can try to recollect technical details (it has been really a 
while). But for the most part it was based on OpenBSD's patch (with some 
adaptations for NetBSD): 
https://www.mail-archive.com/tech%openbsd.org@localhost/msg42433.html


Thanks,
Ruslan