Subject: Re: soc zfs: taskqueue / workqueue
To: None <thorpej@shagadelic.org, dillo@danbala.tuwien.ac.at,>
From: Oliver Gould <ogould@olix0r.net>
List: tech-kern
Date: 07/09/2007 16:34:06
Hello-

I've been looking in more detail at FreeBSD's taskqueue(9) [1], which
provides several predefined task queues [1]:

	The system provides four global taskqueues, taskqueue_fast,
	taskqueue_swi, taskqueue_swi_giant, and taskqueue_thread.
	...

Particularly, ZFS uses the thread queue:

	The thread queue can be used, for instance, by interrupt level
	routines that need to call kernel functions that do things that
	can only be done from a thread context.  (e.g., call malloc with
	the M_WAITOK flag.)

Taskqueue is used to queue up 'tasks', which consist of a function pointer
and some other data:

    struct task {
	STAILQ_ENTRY(task)       ta_link;	/* link for queue */
	u_short			 ta_pending;	/* count times queued */ 
	u_short		     	 ta_priority;	/* priority of task in queue */
	task_fn_t		 ta_func;	/* task handler */
	void			*ta_context;	/* argument for handler */
    }

Whereas, our workqueue(9) seems to be a framework for queueing up 'work'
(data) to be passed to a specified handler function within a newly created
thread for each workqueue.

In the kernel module loader, zfs_start() (which in turn creates several
kernel threads) is enqueued [2:1802] in the dedicated taskqueue thread.
This is the only place where I have found FreeBSD's ZFS to use taskqueue;
so, I think that it's best to avoid using workqueue(9) and simply use
kthread_create1() with zfs_start passed to it.

I think the general idea (as I have inferred) is that we don't want the lkm
load operation to take as long zfs_start() may take, so it is thusly
offloaded to a worker thread.  Is it worth 'wasting' a kthread on this
one task?  Are there any other options?

Thanks,
  - Oliver

[1] http://www.freebsd.org/cgi/man.cgi?query=taskqueue&manpath=FreeBSD+7-current
[2] http://netbsd-soc.sf.net/projects/zfs/src/sys/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c