Current best practice for testing a new kernel API?

[Jason Thorpe <thorpej%me.com@localhost>]

Howdy folks --

I'm currently working on whipping riastradh@'s threadpool / taskqueue draft implementations into shape.  I wrote a simple test jig that lets me exercise the threadpool API from user space, in isolation (i.e. without depending on any users-of-threadpool), but I'm looking for guidance on the current best practice for integrating it into the ATF suite.

Let me describe the test jig a little first.

The threadpool API lets you create per-cpu or unbound pools of threads that run at a specified priority.  So, my test jig simply exposes the API basics as sysctl nodes:

- kern.threadpool_tester.get_unbound
- kern.threadpool_tester.put_unbound
- kern.threadpool_tester.run_unbound
- kern.threadpool_tester.get_percpu
- kern.threadpool_tester.put_percpu
- kern.threadpool_tester.run_percpu

There is a single pre-definedm, extremely simple job that simple increments a variable, and there is another sysctl node that exposes this variable so it can be examined or reset by the user space test harness.

So, for example:

- Setting kern.threadpool_tester.get_unbound=5 calls threadpool_get() with a priority of 5.  The resulting threadpool is stashed away in a table the jig maintains for later use.
- Setting kern.threadpool_tester.run_unbound=5 calls threadpool_schedule_job() with the predefined job, scheduling it on the unbound priority 5 pool previously created.
- Setting kern.threadpool_tester.put_unbound=5 calls threadpool_put() on the previously created unbound priority 5 pool.

This allows a simple shell script to exercise all the basics of the threadpool API, including pool / job reference counting and lifecycle.  There are some things the jig doesn't expose, like scheduling different jobs on the same pool while one job is already running (although this would be pretty easy to add, just define a second simple job and add another sysctl node), or scheduling a job on a remote CPU.  But it covers the basics, and I used it to find and fix a bunch of bugs in Taylor's latest draft of the code.

Now, of course I don't want the test jig present in the kernel all of the time.  Right now, I have it conditional on #ifdef THREADPOOL_TESTER.  What I was thinking of doing is having that CPP macro, if defined, build kern_threadpool.c as a module that includes only the test code, and have it installed in some place like /stand/<arch>/<version>/modules/atf/threadpool/threadpool_tester.kmod ... then the ATF script could first modload the tester module, run the tests, and then modunload it when the test is complete.

If there are objections to this approach, I'm happy to do it a different way... I thought maybe about using rump for this, but I am totally unfamiliar with how that all works, and how even to get threadpool built into it.  Guidance in this area would be greatly appreciated.

-- thorpej