Subject: SMP re-entrancy in kernel drivers/"bottom half?"
To: None <tech-kern@netbsd.org>
From: Jonathan Stone <jonathan@DSG.Stanford.EDU>
List: tech-kern
Date: 12/17/2003 14:06:13
Has anyone looked into allowing device drivers, and other
interrupt-priority code, run in the kernel in multiple CPUs
at the same time?

Over the past week or so I've been thinking through the implications.
We already support very limited entry simultaneous multi-CPU entry
into the kernel (e.g,. for TLB shoot-down interrupts).

It shouldn't take too much (apart from the SMOP and hairy debugging) to
allow, say, NIC driver to enter the kernel in the same way.

For discussion purposes, imagine we add:
    * an interrupt-establish time flag whereby  a driver can
      mark itself SMP-safe;
    * a spinlock to each device instance softc;
    * acquire a spinlock when raising IPL to IPL_NET or above
      (one per hardware IPL). Raising IPl requires acquiring all spinlocks
      between current IPL and target, in order to prevent deadlocks.
      release  all the spinlocks in reverse order at the corresponding splx().
 
    * Spinlocks protecting malloc()/free(), kernel printf(), and
      each protocol input queue (or just piggyback off the "at mostone
      CPU active at or above a given SPL" invariant of previous item)

   * Spinlocks(?) protecting each software-interrupt callout queue
       (or just piggyback off invariatns of modfied spl*(), yet again)

Then we'd be most of the way to a safe implementation.  This may not
buy much for many NetBSD users. but, for the mixes I run with 90%+ of
CPU time is in the kernel, 30%-40% hardware interrupts and 40%-60% at
software-interrupt priority (i.e., no benefit from userspace-only
SMP), it should be a hefty improvement.

ISTR Jason had a branch for finer-grained SMP locking; is that
still alive? Is there any overlap?