Source-Changes-HG archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
[src/trunk]: src Add threadpool(9), an abstraction that provides shared pools...
details: https://anonhg.NetBSD.org/src/rev/7cf1d26eb664
branches: trunk
changeset: 446894:7cf1d26eb664
user: thorpej <thorpej%NetBSD.org@localhost>
date: Mon Dec 24 16:58:53 2018 +0000
description:
Add threadpool(9), an abstraction that provides shared pools of kernel
threads running at specific priorities, with support for unbound pools
and per-cpu pools.
Written by riastradh@, and based on the May 2014 draft, with a few changes
by me:
- Working on the assumption that a relative few priorities will actually
be used, reduce the memory footprint by using linked lists, rather than
2 large (and mostly empty) tables. The performance impact is essentially
nil, since these lists are consulted only when pools are created (and
destroyed, for DIAGNOSTIC checks), and the lists will have at most 225
entries.
- Make threadpool job object, which the caller must allocate storage for,
really opaque.
- Use typedefs for the threadpool types, to reduce the verbosity of the
API somewhat.
- Fix a bunch of pool / worker thread / job object lifecycle bugs.
Also include an ATF unit test, written by me, that exercises the basics
of the API by loading a kernel module that exposes several sysctls that
allow the ATF test script to create and destroy threadpools, schedule a
basic job, and verify that it ran.
And thus NetBSD 8.99.29 has arrived.
diffstat:
distrib/sets/lists/comp/mi | 41 +-
distrib/sets/lists/tests/mi | 3 +-
distrib/sets/lists/tests/module.mi | 4 +-
share/man/man9/Makefile | 17 +-
share/man/man9/threadpool.9 | 336 ++++++
sys/kern/files.kern | 3 +-
sys/kern/kern_threadpool.c | 1085 ++++++++++++++++++++
sys/rump/librump/rumpkern/Makefile.rumpkern | 3 +-
sys/sys/param.h | 4 +-
sys/sys/threadpool.h | 77 +
tests/kernel/Makefile | 5 +-
tests/kernel/t_threadpool.sh | 169 +++
tests/kernel/threadpool_tester/Makefile | 14 +
tests/kernel/threadpool_tester/threadpool_tester.c | 502 +++++++++
14 files changed, 2253 insertions(+), 10 deletions(-)
diffs (truncated from 2416 to 300 lines):
diff -r 87c6a470cf2b -r 7cf1d26eb664 distrib/sets/lists/comp/mi
--- a/distrib/sets/lists/comp/mi Mon Dec 24 16:04:14 2018 +0000
+++ b/distrib/sets/lists/comp/mi Mon Dec 24 16:58:53 2018 +0000
@@ -1,4 +1,4 @@
-# $NetBSD: mi,v 1.2244 2018/12/22 14:39:47 maxv Exp $
+# $NetBSD: mi,v 1.2245 2018/12/24 16:58:53 thorpej Exp $
#
# Note: don't delete entries from here - mark them as "obsolete" instead.
./etc/mtree/set.comp comp-sys-root
@@ -11825,6 +11825,19 @@
./usr/share/man/cat9/tc_syncbus.0 comp-sys-catman .cat
./usr/share/man/cat9/tc_wmb.0 comp-sys-catman .cat
./usr/share/man/cat9/tcp_congctl.0 comp-sys-catman .cat
+./usr/share/man/cat9/threadpool.0 comp-sys-catman .cat
+./usr/share/man/cat9/threadpool_get.0 comp-sys-catman .cat
+./usr/share/man/cat9/threadpool_put.0 comp-sys-catman .cat
+./usr/share/man/cat9/threadpool_percpu_get.0 comp-sys-catman .cat
+./usr/share/man/cat9/threadpool_percpu_put.0 comp-sys-catman .cat
+./usr/share/man/cat9/threadpool_percpu_ref.0 comp-sys-catman .cat
+./usr/share/man/cat9/threadpool_percpu_ref_remote.0 comp-sys-catman .cat
+./usr/share/man/cat9/threadpool_job_init.0 comp-sys-catman .cat
+./usr/share/man/cat9/threadpool_job_destroy.0 comp-sys-catman .cat
+./usr/share/man/cat9/threadpool_job_done.0 comp-sys-catman .cat
+./usr/share/man/cat9/threadpool_schedule_job.0 comp-sys-catman .cat
+./usr/share/man/cat9/threadpool_cancel_job.0 comp-sys-catman .cat
+./usr/share/man/cat9/threadpool_cancel_job_async.0 comp-sys-catman .cat
./usr/share/man/cat9/tick.0 comp-sys-catman .cat
./usr/share/man/cat9/tickadj.0 comp-sys-catman .cat
./usr/share/man/cat9/time.0 comp-obsolete obsolete
@@ -19645,6 +19658,19 @@
./usr/share/man/html9/tc_syncbus.html comp-sys-htmlman html
./usr/share/man/html9/tc_wmb.html comp-sys-htmlman html
./usr/share/man/html9/tcp_congctl.html comp-sys-htmlman html
+./usr/share/man/html9/threadpool.html comp-sys-htmlman html
+./usr/share/man/html9/threadpool_get.html comp-sys-htmlman html
+./usr/share/man/html9/threadpool_put.html comp-sys-htmlman html
+./usr/share/man/html9/threadpool_percpu_get.html comp-sys-htmlman html
+./usr/share/man/html9/threadpool_percpu_put.html comp-sys-htmlman html
+./usr/share/man/html9/threadpool_percpu_ref.html comp-sys-htmlman html
+./usr/share/man/html9/threadpool_percpu_ref_remote.html comp-sys-htmlman html
+./usr/share/man/html9/threadpool_job_init.html comp-sys-htmlman html
+./usr/share/man/html9/threadpool_job_destroy.html comp-sys-htmlman html
+./usr/share/man/html9/threadpool_job_done.html comp-sys-htmlman html
+./usr/share/man/html9/threadpool_schedule_job.html comp-sys-htmlman html
+./usr/share/man/html9/threadpool_cancel_job.html comp-sys-htmlman html
+./usr/share/man/html9/threadpool_cancel_job_async.html comp-sys-htmlman html
./usr/share/man/html9/tick.html comp-sys-htmlman html
./usr/share/man/html9/tickadj.html comp-sys-htmlman html
./usr/share/man/html9/time_second.html comp-sys-htmlman html
@@ -27665,6 +27691,19 @@
./usr/share/man/man9/tc_syncbus.9 comp-sys-man .man
./usr/share/man/man9/tc_wmb.9 comp-sys-man .man
./usr/share/man/man9/tcp_congctl.9 comp-sys-man .man
+./usr/share/man/man9/threadpool.9 comp-sys-man .man
+./usr/share/man/man9/threadpool_get.9 comp-sys-man .man
+./usr/share/man/man9/threadpool_put.9 comp-sys-man .man
+./usr/share/man/man9/threadpool_percpu_get.9 comp-sys-man .man
+./usr/share/man/man9/threadpool_percpu_put.9 comp-sys-man .man
+./usr/share/man/man9/threadpool_percpu_ref.9 comp-sys-man .man
+./usr/share/man/man9/threadpool_percpu_ref_remote.9 comp-sys-man .man
+./usr/share/man/man9/threadpool_job_init.9 comp-sys-man .man
+./usr/share/man/man9/threadpool_job_destroy.9 comp-sys-man .man
+./usr/share/man/man9/threadpool_job_done.9 comp-sys-man .man
+./usr/share/man/man9/threadpool_schedule_job.9 comp-sys-man .man
+./usr/share/man/man9/threadpool_cancel_job.9 comp-sys-man .man
+./usr/share/man/man9/threadpool_cancel_job_async.9 comp-sys-man .man
./usr/share/man/man9/tick.9 comp-sys-man .man
./usr/share/man/man9/tickadj.9 comp-sys-man .man
./usr/share/man/man9/time.9 comp-obsolete obsolete
diff -r 87c6a470cf2b -r 7cf1d26eb664 distrib/sets/lists/tests/mi
--- a/distrib/sets/lists/tests/mi Mon Dec 24 16:04:14 2018 +0000
+++ b/distrib/sets/lists/tests/mi Mon Dec 24 16:58:53 2018 +0000
@@ -1,4 +1,4 @@
-# $NetBSD: mi,v 1.798 2018/12/23 21:27:45 jakllsch Exp $
+# $NetBSD: mi,v 1.799 2018/12/24 16:58:54 thorpej Exp $
#
# Note: don't delete entries from here - mark them as "obsolete" instead.
#
@@ -2205,6 +2205,7 @@
./usr/tests/kernel/t_subr_prf tests-kernel-tests compattestfile,atf
./usr/tests/kernel/t_sysctl tests-kernel-tests compattestfile,atf
./usr/tests/kernel/t_sysv tests-kernel-tests compattestfile,atf
+./usr/tests/kernel/t_threadpool tests-kernel-tests compattestfile,atf
./usr/tests/kernel/t_time tests-obsolete obsolete
./usr/tests/kernel/t_timeleft tests-kernel-tests compattestfile,atf
./usr/tests/kernel/t_trapsignal tests-kernel-tests compattestfile,atf
diff -r 87c6a470cf2b -r 7cf1d26eb664 distrib/sets/lists/tests/module.mi
--- a/distrib/sets/lists/tests/module.mi Mon Dec 24 16:04:14 2018 +0000
+++ b/distrib/sets/lists/tests/module.mi Mon Dec 24 16:58:53 2018 +0000
@@ -1,7 +1,9 @@
-# $NetBSD: module.mi,v 1.13 2018/01/08 14:17:15 martin Exp $
+# $NetBSD: module.mi,v 1.14 2018/12/24 16:58:54 thorpej Exp $
#
# These are only made for ports doing modules.
#
+./usr/tests/kernel/threadpool_tester tests-kernel-tests compattestfile,atf
+./usr/tests/kernel/threadpool_tester/threadpool_tester.kmod tests-kernel-tests compattestfile,atf
./usr/tests/modules/Atffile tests-sys-tests atf,rump
./usr/tests/modules/Kyuafile tests-sys-tests atf,rump,kyua
./usr/tests/modules/k_helper tests-sys-tests atf,rump
diff -r 87c6a470cf2b -r 7cf1d26eb664 share/man/man9/Makefile
--- a/share/man/man9/Makefile Mon Dec 24 16:04:14 2018 +0000
+++ b/share/man/man9/Makefile Mon Dec 24 16:58:53 2018 +0000
@@ -1,4 +1,4 @@
-# $NetBSD: Makefile,v 1.433 2018/12/22 14:39:46 maxv Exp $
+# $NetBSD: Makefile,v 1.434 2018/12/24 16:58:54 thorpej Exp $
# Makefile for section 9 (kernel function and variable) manual pages.
@@ -55,7 +55,8 @@
splraiseipl.9 \
store.9 suspendsched.9 \
sysctl.9 sysmon_envsys.9 sysmon_pswitch.9 sysmon_taskq.9 tc.9 \
- tcp_congctl.9 timecounter.9 time_second.9 todr.9 ts2timo.9 tvtohz.9 \
+ tcp_congctl.9 threadpool.9 timecounter.9 time_second.9 todr.9 \
+ ts2timo.9 tvtohz.9 \
ucas.9 uiomove.9 ucom.9 userret.9 \
vattr.9 veriexec.9 vcons.9 vfs.9 vfs_hooks.9 vfsops.9 vfssubr.9 \
video.9 vme.9 \
@@ -915,6 +916,18 @@
tc.9 tc_badaddr.9 \
tc.9 TC_DENSE_TO_SPARSE.9 \
tc.9 TC_PHYS_TO_UNCACHED.9
+MLINKS+=threadpool.9 threadpool_get.9 \
+ threadpool.9 threadpool_put.9 \
+ threadpool.9 threadpool_percpu_get.9 \
+ threadpool.9 threadpool_percpu_put.9 \
+ threadpool.9 threadpool_percpu_ref.9 \
+ threadpool.9 threadpool_percpu_ref_remote.9 \
+ threadpool.9 threadpool_job_init.9 \
+ threadpool.9 threadpool_job_destroy.9 \
+ threadpool.9 threadpool_job_done.9 \
+ threadpool.9 threadpool_schedule_job.9 \
+ threadpool.9 threadpool_cancel_job.9 \
+ threadpool.9 threadpool_cancel_job_async.9
MLINKS+=todr.9 todr_gettime.9 todr.9 todr_settime.9 \
todr.9 clock_ymdhms_to_secs.9 todr.9 clock_secs_to_ymdhms.9
diff -r 87c6a470cf2b -r 7cf1d26eb664 share/man/man9/threadpool.9
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/share/man/man9/threadpool.9 Mon Dec 24 16:58:53 2018 +0000
@@ -0,0 +1,336 @@
+.\" $NetBSD: threadpool.9,v 1.1 2018/12/24 16:58:54 thorpej Exp $
+.\"
+.\" Copyright (c) 2014 The NetBSD Foundation, Inc.
+.\" All rights reserved.
+.\"
+.\" This code is derived from software contributed to The NetBSD Foundation
+.\" by Taylor R. Campbell.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\" notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\" notice, this list of conditions and the following disclaimer in the
+.\" documentation and/or other materials provided with the distribution.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
+.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+.\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+.\" PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
+.\" BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+.\" POSSIBILITY OF SUCH DAMAGE.
+.\"
+.Dd December 24, 2018
+.Dt THREADPOOL 9
+.Os
+.\"
+.Sh NAME
+.Nm threadpool
+.Nd shared pools of kthreads
+.\"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+.Sh SYNOPSIS
+.In sys/threadpool.h
+.\""""""""""""""""""""""""""""""""""""
+.Ft int
+.Fn threadpool_get "threadpool_t **poolp" "pri_t pri"
+.\"
+.Ft void
+.Fn threadpool_put "threadpool_t *pool" "pri_t pri"
+.\""""""""""""""""""""""""""""""""""""
+.Ft int
+.Fn threadpool_percpu_get "threadpool_percpu_t **pool_percpup" "pri_t pri"
+.\"
+.Ft void
+.Fn threadpool_percpu_put "threadpool_percpu_t *pool_percpu" "pri_t pri"
+.\"
+.Ft threadpool_t *
+.Fn threadpool_percpu_ref "threadpool_percpu_t *pool"
+.\"
+.Ft threadpool_t *
+.Fn threadpool_percpu_ref_remote "threadpool_percpu_t *pool" "struct cpu_info *ci"
+.\""""""""""""""""""""""""""""""""""""
+.Ft void
+.Fn threadpool_job_init "threadpool_job_t *job" "void (*fn)(threadpool_job_t *)" "kmutex_t *interlock"
+.\"
+.Ft void
+.Fn threadpool_job_destroy "threadpool_job_t *job"
+.\"
+.Ft void
+.Fn threadpool_job_done "threadpool_job_t *job"
+.\""""""""""""""""""""""""""""""""""""
+.Ft void
+.Fn threadpool_schedule_job "threadpool_t *pool" "threadpool_job_t *job"
+.\"
+.Ft void
+.Fn threadpool_cancel_job "threadpool_t *pool" "threadpool_job_t *job"
+.\"
+.Ft bool
+.Fn threadpool_cancel_job_async "threadpool_t *pool" "threadpool_job_t *job"
+.\"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+.Sh DESCRIPTION
+The
+.Nm
+abstraction is provided to share a pool of
+.Xr kthread 9
+kernel threads for medium- to long-term actions, called jobs, which can
+be scheduled from contexts that do not allow sleeping.
+.Pp
+For each priority level, there is one unbound thread pool, and one
+collection of per-CPU thread pools.
+Access to the unbound thread pools is provided by
+.Fn threadpool_get
+and
+.Fn threadpool_put .
+Access to the per-CPU thread pools is provided by
+.Fn threadpool_percpu_get
+and
+.Fn threadpool_percpu_put .
+.Pp
+Job state is stored in the
+.Vt threadpool_job_t
+object.
+Callers of the
+.Nm
+abstraction
+must allocate memory for
+.Vt threadpool_job_t
+objects, but should consider them opaque, and should not inspect or
+copy them.
+Each job represented by a
+.Vt threadpool_job_t
+object will be run only once at a time, until the action associated
+with it calls
+.Fn threadpool_job_done .
+.Pp
+Jobs are run in thread context and may take arbitrarily long to run or
+sleep arbitrarily long.
+.\" The
+.\" .Nm
+.\" abstraction is intended as a building block for cheaper abstractions,
+.\" namely
+.\" .Xr task 9
+.\" and
+.\" .Xr workqueue 9 .
+.\" It should generally not be used directly.
+.\"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+.Sh FUNCTIONS
+.\"
+.Bl -tag -width abcd
+.\""""""""""""""""""""""""""""""""""""
+.It Fn threadpool_get "poolp" "pri"
+Obtain a reference to the unbound thread pool at priority
+.Fa pri
+and store it in
+.Fa poolp .
+.Pp
+May sleep.
+.\"
+.It Fn threadpool_put "pool" "pri"
+Release the reference to the unbound thread pool
+.Fa pool
+at priority
+.Fa pri ,
+which must be the same as the priority that was passed to
+.Fn threadpool_get
+to obtain
+.Fa pool .
+.Pp
+May sleep.
+.Pp
+Do not use
+.Fn threadpool_put
+with thread pools obtained from
+.Fn threadpool_percpu_ref
+or
+.Fn threadpool_percpu_ref_remote .
+.\""""""""""""""""""""""""""""""""""""
+.It Fn threadpool_percpu_get "pool_percpup" "pri"
+Obtain a reference to the per-CPU thread pool at priority
+.Fa pri
+and store it in
+.Fa pool_percpup .
Home |
Main Index |
Thread Index |
Old Index