Source-Changes-HG archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
[src/trunk]: src/sys Make the routing table and rtcaches MP-safe
details: https://anonhg.NetBSD.org/src/rev/cea9dae3ca97
branches: trunk
changeset: 349487:cea9dae3ca97
user: ozaki-r <ozaki-r%NetBSD.org@localhost>
date: Mon Dec 12 03:55:57 2016 +0000
description:
Make the routing table and rtcaches MP-safe
See the following descriptions for details.
Proposed on tech-kern and tech-net
Overview
--------
We protect the routing table with a rwock and protect
rtcaches with another rwlock. Each rtentry is protected
from being freed or updated via reference counting and psref.
Global rwlocks
--------------
There are two rwlocks; one for the routing table (rt_lock) and
the other for rtcaches (rtcache_lock). rtcache_lock covers
all existing rtcaches; there may have room for optimizations
(future work).
The locking order is rtcache_lock first and rt_lock is next.
rtentry references
------------------
References to an rtentry is managed with reference counting
and psref. Either of the two mechanisms is used depending on
where a rtentry is obtained. Reference counting is used when
we obtain a rtentry from the routing table directly via
rtalloc1 and rtrequest{,1} while psref is used when we obtain
a rtentry from a rtcache via rtcache_* APIs. In both cases,
a caller can sleep/block with holding an obtained rtentry.
The reasons why we use two different mechanisms are (i) only
using reference counting hurts the performance due to atomic
instructions (rtcache case) (ii) ease of implementation;
applying psref to APIs such rtaloc1 and rtrequest{,1} requires
additional works (adding a local variable and an argument).
We will finally migrate to use only psref but we can do it
when we have a lockless routing table alternative.
Reference counting for rtentry
------------------------------
rt_refcnt now doesn't count permanent references such as for
rt_timers and rtcaches, instead it is used only for temporal
references when obtaining a rtentry via rtalloc1 and rtrequest{,1}.
We can do so because destroying a rtentry always involves
removing references of rt_timers and rtcaches to the rtentry
and we don't need to track such references. This also makes
it easy to wait for readers to release references on deleting
or updating a rtentry, i.e., we can simply wait until the
reference counter is 0 or 1. (If there are permanent references
the counter can be arbitrary.)
rt_ref increments a reference counter of a rtentry and rt_unref
decrements it. rt_ref is called inside APIs (rtalloc1 and
rtrequest{,1} so users don't need to care about it while
users must call rt_unref to an obtained rtentry after using it.
rtfree is removed and we use rt_unref and rt_free instead.
rt_unref now just decrements the counter of a given rtentry
and rt_free just tries to destroy a given rtentry.
See the next section for destructions of rtentries by rt_free.
Destructions of rtentries
-------------------------
We destroy a rtentry only when we call rtrequst{,1}(RTM_DELETE);
the original implementation can destroy in any rtfree where it's
the last reference. If we use reference counting or psref, it's
easy to understand if the place that a rtentry is destroyed is
fixed.
rt_free waits for references to a given rtentry to be released
before actually destroying the rtentry. rt_free uses a condition
variable (cv_wait) (and psref_target_destroy for psref) to wait.
Unfortunately rtrequst{,1}(RTM_DELETE) can be called in softint
that we cannot use cv_wait. In that case, we have to defer the
destruction to a workqueue.
rtentry#rt_cv, rtentry#rt_psref and global variables
(see rt_free_global) are added to conduct the procedure.
Updates of rtentries
--------------------
One difficulty to use refcnt/psref instead of rwlock for rtentry
is updates of rtentries. We need an additional mechanism to
prevent readers from seeing inconsistency of a rtentry being
updated.
We introduce RTF_UPDATING flag to rtentries that are updating.
While the flag is set to a rtentry, users cannot acquire the
rtentry. By doing so, we avoid users to see inconsistent
rtentries.
There are two options when a user tries to acquire a rtentry
with the RTF_UPDATING flag; if a user runs in softint context
the user fails to acquire a rtentry (NULL is returned).
Otherwise a user waits until the update completes by waiting
on cv.
The procedure of a updater is simpler to destruction of
a rtentry. Wait on cv (and psref) and after all readers left,
proceed with the update.
Global variables (see rt_update_global) are added to conduct
the procedure.
Currently we apply the mechanism to only RTM_CHANGE in
rtsock.c. We would have to apply other codes. See
"Known issues" section.
psref for rtentry
-----------------
When we obtain a rtentry from a rtcache via rtcache_* APIs,
psref is used to reference to the rtentry.
rtcache_ref acquires a reference to a rtentry with psref
and rtcache_unref releases the reference after using it.
rtcache_ref is called inside rtcache_* APIs and users don't
need to take care of it while users must call rtcache_unref
to release the reference.
struct psref and int bound that is needed for psref is
embedded into struct route. By doing so we don't need to
add local variables and additional argument to APIs.
However this adds another constraint to psref other than
reference counting one's; holding a reference of an rtentry
via a rtcache is allowed by just one caller at the same time.
So we must not acquire a rtentry via a rtcache twice and
avoid a recursive use of a rtcache. And also a rtcache must
be arranged to be used by a LWP/softint at the same time
somehow. For IP forwarding case, we have per-CPU rtcaches
used in softint so the constraint is guaranteed. For a h
rtcache of a PCB case, the constraint is guaranteed by the
solock of each PCB. Any other cases (pf, ipf, stf and ipsec)
are currently guaranteed by only the existence of the global
locks (softnet_lock and/or KERNEL_LOCK). If we've found the
cases that we cannot guarantee the constraint, we would need
to introduce other rtcache APIs that use simple reference
counting.
psref of rtcache is created with IPL_SOFTNET and so rtcache
shouldn't used at an IPL higher than IPL_SOFTNET.
Note that rtcache_free is used to invalidate a given rtcache.
We don't need another care by my change; just keep them as
they are.
Performance impact
------------------
When NET_MPSAFE is disabled the performance drop is 3% while
when it's enabled the drop is increased to 11%. The difference
comes from that currently we don't take any global locks and
don't use psref if NET_MPSAFE is disabled.
We can optimize the performance of the case of NET_MPSAFE
on by reducing lookups of rtcache that uses psref;
currently we do two lookups but we should be able to trim
one of two. This is a future work.
Known issues
------------
There are two known issues to be solved; one is that
a caller of rtrequest(RTM_ADD) may change rtentry (see rtinit).
We need to prevent new references during the update. Or
we may be able to remove the code (perhaps, need more
investigations).
The other is rtredirect that updates a rtentry. We need
to apply our update mechanism, however it's not easy because
rtredirect is called in softint and we cannot apply our
mechanism simply. One solution is to defer rtredirect to
a workqueue but it requires some code restructuring.
diffstat:
sys/net/if.c | 12 +-
sys/net/if.h | 4 +-
sys/net/if_faith.c | 6 +-
sys/net/if_mpls.c | 8 +-
sys/net/if_stf.c | 8 +-
sys/net/radix.c | 7 +-
sys/net/route.c | 776 ++++++++++++++++++++++++++++++++++++++++++------
sys/net/route.h | 59 +-
sys/net/rtbl.c | 6 +-
sys/net/rtsock.c | 30 +-
sys/netinet/if_arp.c | 6 +-
sys/netinet/if_atm.c | 10 +-
sys/netinet/in.c | 7 +-
sys/netinet/in_gif.c | 8 +-
sys/netinet/in_pcb.c | 6 +-
sys/netinet/ip_carp.c | 8 +-
sys/netinet/ip_icmp.c | 19 +-
sys/netinet/ip_input.c | 5 +-
sys/netinet/ip_output.c | 14 +-
sys/netinet6/icmp6.c | 20 +-
sys/netinet6/in6.c | 22 +-
sys/netinet6/in6_gif.c | 8 +-
sys/netinet6/in6_pcb.c | 6 +-
sys/netinet6/nd6.c | 12 +-
sys/netinet6/nd6_nbr.c | 6 +-
25 files changed, 843 insertions(+), 230 deletions(-)
diffs (truncated from 2402 to 300 lines):
diff -r a6c8c0b71c91 -r cea9dae3ca97 sys/net/if.c
--- a/sys/net/if.c Mon Dec 12 03:14:01 2016 +0000
+++ b/sys/net/if.c Mon Dec 12 03:55:57 2016 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: if.c,v 1.365 2016/12/09 02:38:14 christos Exp $ */
+/* $NetBSD: if.c,v 1.366 2016/12/12 03:55:57 ozaki-r Exp $ */
/*-
* Copyright (c) 1999, 2000, 2001, 2008 The NetBSD Foundation, Inc.
@@ -90,7 +90,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: if.c,v 1.365 2016/12/09 02:38:14 christos Exp $");
+__KERNEL_RCSID(0, "$NetBSD: if.c,v 1.366 2016/12/12 03:55:57 ozaki-r Exp $");
#if defined(_KERNEL_OPT)
#include "opt_inet.h"
@@ -1686,6 +1686,7 @@
void
ifaref(struct ifaddr *ifa)
{
+ KASSERT(!ISSET(ifa->ifa_flags, IFA_DESTROYING));
ifa->ifa_refcnt++;
}
@@ -1700,6 +1701,13 @@
}
}
+bool
+ifa_is_destroying(struct ifaddr *ifa)
+{
+
+ return ISSET(ifa->ifa_flags, IFA_DESTROYING);
+}
+
void
ifa_insert(struct ifnet *ifp, struct ifaddr *ifa)
{
diff -r a6c8c0b71c91 -r cea9dae3ca97 sys/net/if.h
--- a/sys/net/if.h Mon Dec 12 03:14:01 2016 +0000
+++ b/sys/net/if.h Mon Dec 12 03:55:57 2016 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: if.h,v 1.230 2016/12/08 01:06:35 ozaki-r Exp $ */
+/* $NetBSD: if.h,v 1.231 2016/12/12 03:55:57 ozaki-r Exp $ */
/*-
* Copyright (c) 1999, 2000, 2001 The NetBSD Foundation, Inc.
@@ -604,6 +604,7 @@
#endif
};
#define IFA_ROUTE RTF_UP /* (0x01) route installed */
+#define IFA_DESTROYING 0x2
/*
* Message format for use in obtaining information about interfaces from
@@ -1001,6 +1002,7 @@
void ifa_acquire(struct ifaddr *, struct psref *);
void ifa_release(struct ifaddr *, struct psref *);
bool ifa_held(struct ifaddr *);
+bool ifa_is_destroying(struct ifaddr *);
void ifaref(struct ifaddr *);
void ifafree(struct ifaddr *);
diff -r a6c8c0b71c91 -r cea9dae3ca97 sys/net/if_faith.c
--- a/sys/net/if_faith.c Mon Dec 12 03:14:01 2016 +0000
+++ b/sys/net/if_faith.c Mon Dec 12 03:55:57 2016 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: if_faith.c,v 1.54 2016/08/07 17:38:33 christos Exp $ */
+/* $NetBSD: if_faith.c,v 1.55 2016/12/12 03:55:57 ozaki-r Exp $ */
/* $KAME: if_faith.c,v 1.21 2001/02/20 07:59:26 itojun Exp $ */
/*
@@ -40,7 +40,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: if_faith.c,v 1.54 2016/08/07 17:38:33 christos Exp $");
+__KERNEL_RCSID(0, "$NetBSD: if_faith.c,v 1.55 2016/12/12 03:55:57 ozaki-r Exp $");
#ifdef _KERNEL_OPT
#include "opt_inet.h"
@@ -322,7 +322,7 @@
else
ret = 0;
if (rt)
- rtfree(rt);
+ rt_unref(rt);
return ret;
}
#endif
diff -r a6c8c0b71c91 -r cea9dae3ca97 sys/net/if_mpls.c
--- a/sys/net/if_mpls.c Mon Dec 12 03:14:01 2016 +0000
+++ b/sys/net/if_mpls.c Mon Dec 12 03:55:57 2016 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: if_mpls.c,v 1.28 2016/10/03 11:06:06 ozaki-r Exp $ */
+/* $NetBSD: if_mpls.c,v 1.29 2016/12/12 03:55:57 ozaki-r Exp $ */
/*
* Copyright (c) 2010 The NetBSD Foundation, Inc.
@@ -30,7 +30,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: if_mpls.c,v 1.28 2016/10/03 11:06:06 ozaki-r Exp $");
+__KERNEL_RCSID(0, "$NetBSD: if_mpls.c,v 1.29 2016/12/12 03:55:57 ozaki-r Exp $");
#ifdef _KERNEL_OPT
#include "opt_inet.h"
@@ -304,7 +304,7 @@
}
err = mpls_send_frame(m, rt1->rt_ifp, rt);
- rtfree(rt1);
+ rt_unref(rt1);
return err;
}
@@ -481,7 +481,7 @@
if (error != 0 && m != NULL)
m_freem(m);
if (rt != NULL)
- rtfree(rt);
+ rt_unref(rt);
return error;
}
diff -r a6c8c0b71c91 -r cea9dae3ca97 sys/net/if_stf.c
--- a/sys/net/if_stf.c Mon Dec 12 03:14:01 2016 +0000
+++ b/sys/net/if_stf.c Mon Dec 12 03:55:57 2016 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: if_stf.c,v 1.100 2016/12/08 05:16:33 ozaki-r Exp $ */
+/* $NetBSD: if_stf.c,v 1.101 2016/12/12 03:55:57 ozaki-r Exp $ */
/* $KAME: if_stf.c,v 1.62 2001/06/07 22:32:16 itojun Exp $ */
/*
@@ -75,7 +75,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: if_stf.c,v 1.100 2016/12/08 05:16:33 ozaki-r Exp $");
+__KERNEL_RCSID(0, "$NetBSD: if_stf.c,v 1.101 2016/12/12 03:55:57 ozaki-r Exp $");
#ifdef _KERNEL_OPT
#include "opt_inet.h"
@@ -535,10 +535,10 @@
(uint32_t)ntohl(sin.sin_addr.s_addr));
#endif
if (rt)
- rtfree(rt);
+ rt_unref(rt);
return -1;
}
- rtfree(rt);
+ rt_unref(rt);
}
return 0;
diff -r a6c8c0b71c91 -r cea9dae3ca97 sys/net/radix.c
--- a/sys/net/radix.c Mon Dec 12 03:14:01 2016 +0000
+++ b/sys/net/radix.c Mon Dec 12 03:55:57 2016 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: radix.c,v 1.46 2016/11/15 01:50:06 ozaki-r Exp $ */
+/* $NetBSD: radix.c,v 1.47 2016/12/12 03:55:57 ozaki-r Exp $ */
/*
* Copyright (c) 1988, 1989, 1993
@@ -36,7 +36,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: radix.c,v 1.46 2016/11/15 01:50:06 ozaki-r Exp $");
+__KERNEL_RCSID(0, "$NetBSD: radix.c,v 1.47 2016/12/12 03:55:57 ozaki-r Exp $");
#ifndef _NET_RADIX_H_
#include <sys/param.h>
@@ -1053,7 +1053,8 @@
{
struct delayinit *di;
- KASSERT(radix_initialized == 0);
+ if (radix_initialized)
+ return;
di = kmem_alloc(sizeof(*di), KM_SLEEP);
di->head = head;
diff -r a6c8c0b71c91 -r cea9dae3ca97 sys/net/route.c
--- a/sys/net/route.c Mon Dec 12 03:14:01 2016 +0000
+++ b/sys/net/route.c Mon Dec 12 03:55:57 2016 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: route.c,v 1.182 2016/11/15 01:50:06 ozaki-r Exp $ */
+/* $NetBSD: route.c,v 1.183 2016/12/12 03:55:57 ozaki-r Exp $ */
/*-
* Copyright (c) 1998, 2008 The NetBSD Foundation, Inc.
@@ -97,7 +97,7 @@
#endif
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: route.c,v 1.182 2016/11/15 01:50:06 ozaki-r Exp $");
+__KERNEL_RCSID(0, "$NetBSD: route.c,v 1.183 2016/12/12 03:55:57 ozaki-r Exp $");
#include <sys/param.h>
#ifdef RTFLUSH_DEBUG
@@ -117,6 +117,9 @@
#include <sys/kauth.h>
#include <sys/workqueue.h>
#include <sys/syslog.h>
+#include <sys/rwlock.h>
+#include <sys/mutex.h>
+#include <sys/cpu.h>
#include <net/if.h>
#include <net/if_dl.h>
@@ -131,6 +134,19 @@
#define rtcache_debug() 0
#endif /* RTFLUSH_DEBUG */
+#ifdef RT_DEBUG
+#define RT_REFCNT_TRACE(rt) printf("%s:%d: rt=%p refcnt=%d\n", \
+ __func__, __LINE__, (rt), (rt)->rt_refcnt)
+#else
+#define RT_REFCNT_TRACE(rt) do {} while (0)
+#endif
+
+#ifdef DEBUG
+#define dlog(level, fmt, args...) log(level, fmt, ##args)
+#else
+#define dlog(level, fmt, args...) do {} while (0)
+#endif
+
struct rtstat rtstat;
static int rttrash; /* routes not in table but not freed */
@@ -147,6 +163,105 @@
static void rt_timer_remove_all(struct rtentry *);
static void rt_timer_timer(void *);
+/*
+ * Locking notes:
+ * - The routing table is protected by a global rwlock
+ * - API: RT_RLOCK and friends
+ * - rtcaches are protected by a global rwlock
+ * - API: RTCACHE_RLOCK and friends
+ * - References to a rtentry is managed by reference counting and psref
+ * - Reference couting is used for temporal reference when a rtentry
+ * is fetched from the routing table
+ * - psref is used for temporal reference when a rtentry is fetched
+ * from a rtcache
+ * - struct route (rtcache) has struct psref, so we cannot obtain
+ * a reference twice on the same struct route
+ * - Befere destroying or updating a rtentry, we have to wait for
+ * all references left (see below for details)
+ * - APIs
+ * - An obtained rtentry via rtalloc1 or rtrequest* must be
+ * unreferenced by rt_unref
+ * - An obtained rtentry via rtcache_* must be unreferenced by
+ * rtcache_unref
+ * - TODO: once we get a lockless routing table, we should use only
+ * psref for rtentries
+ * - rtentry destruction
+ * - A rtentry is destroyed (freed) only when we call rtrequest(RTM_DELETE)
+ * - If a caller of rtrequest grabs a reference of a rtentry, the caller
+ * has a responsibility to destroy the rtentry by itself by calling
+ * rt_free
+ * - If not, rtrequest itself does that
+ * - If rt_free is called in softint, the actual destruction routine is
+ * deferred to a workqueue
+ * - rtentry update
+ * - When updating a rtentry, RTF_UPDATING flag is set
+ * - If a rtentry is set RTF_UPDATING, fetching the rtentry from
+ * the routing table or a rtcache results in either of the following
+ * cases:
+ * - if the caller runs in softint, the caller fails to fetch
+ * - otherwise, the caller waits for the update completed and retries
+ * to fetch (probably succeed to fetch for the second time)
+ */
+
+/*
+ * Global locks for the routing table and rtcaches.
+ * Locking order: rtcache_lock => rt_lock
+ */
+static krwlock_t rt_lock __cacheline_aligned;
+#ifdef NET_MPSAFE
+#define RT_RLOCK() rw_enter(&rt_lock, RW_READER)
+#define RT_WLOCK() rw_enter(&rt_lock, RW_WRITER)
+#define RT_UNLOCK() rw_exit(&rt_lock)
+#define RT_LOCKED() rw_lock_held(&rt_lock)
+#define RT_ASSERT_WLOCK() KASSERT(rw_write_held(&rt_lock))
+#else
+#define RT_RLOCK() do {} while (0)
+#define RT_WLOCK() do {} while (0)
+#define RT_UNLOCK() do {} while (0)
+#define RT_LOCKED() false
+#define RT_ASSERT_WLOCK() do {} while (0)
+#endif
+
+static krwlock_t rtcache_lock __cacheline_aligned;
+#ifdef NET_MPSAFE
Home |
Main Index |
Thread Index |
Old Index