Re: Funded project(s) to improve Linux emulation

[Jason Thorpe <thorpej%me.com@localhost>]

> On Nov 10, 2020, at 9:36 AM, Martin Husemann <martin%NetBSD.org@localhost> wrote:
> 
> The current state of Linux emulation is greatly affected by 
> incompatibilities in the futex emulation, so completition/merging
> of the thorpej-futex branch may be a prerequisite for some of the
> projects explained below.

Let me provide some context here.

The thorpej-futex branch contains changes for two separate things, only one of which is required for Linux emulation:

1- Changing from using a condvar internally to becoming its own synchronization object primitive and using sleepqs directly.  This is required in order to fix a class of bugs related to thread selection on wakeup.

2- Implement a new set of operations that are NetBSD extensions to the futex API, for use natively by the pthread library and possible other things as well.

(1) does indeed fix the bugs it was intended to fix (and there are test cases for them), and it has been reported to improve the situation somewhat with regard to Linux compatibility.  However, there are some issues with the Linux openjdk that lead to applications getting stuck.  The bug isn't 100% reproducible.

I have not yet written support for the PI ("priority-inheritance") futex operations.  Those should be fairly easy to implement, but will require changes outside of the futex code ... in particular, we want to share the actual guts of the priority-lending mechanism with the turnstile code used to back kernel's mutex and rwlock implementations.  Anyway, I think it's best to do as a two-step approach, doing the PI bits later (and maybe as a lower priority project; as far as I'm aware, priority-inheriting mutexes are not the default in glibc, and applications have to opt-in).

(2) is currently disabled in the branch because of a weird crasher, but the only extensions implemented are RW_WAIT and RW_HANDOFF designed for implementing direct-handoff reader/writer locks in libpthread.  I actually have a bunch of changes to libpthread that sit on top of native futexes (for pthread_mutex and pthread_cond), including support for process-shared synchronization objects.

Anyway, if people are interested in helping carry this stuff across the finish line, let me know, and I will be happy to provide guidance, etc.

-- thorpej