[cc'ed to Stephan and Yamamoto-san, due to discussion of SMP
interrupt/spl ordering issues in another thread]

In message <20050601012148.GA28233@pauli.thundrix.ch>,
Tonnerre writes:

>I'm currently trying to improve SMP support, whereas my next task is to
>get softinterrupts off the BKL. Ideally, one would be able to run one
>softint handler per CPU, [...]

Moving soft-interrupts out from under the big-lock may not be as
simple as one might think at first glance.  Full version follows below
(longwinded, in an effort to avoid ambiguity).

I think one must either explicitly make all code called from soft-int
context explicitly SMP-safe (e.g., by adding explicit locking); or
ensure that the SMP soft-int mechanism itself make suitable guarantees
between different soft-int levels. (and, implicitly, different hard
interrupt levels, aka spl()s.)

The former is prohibitive: too much work to get from here to there in
incremental steps.  For the latter, I see only two practiable choices,
either:

 (a) either to ensure all soft-int levels are disjoint:
     for all levels, any specific softint level X never needs to do
     splsoftY(), for X != Y).
or

 (b) Establish a strict ordering between different soft-intr levels,
     enforced by ordered locking, or by other means, to guarantee that
     at most one CPU is running at a given [soft] interrupt level at a time. 

Given that our networking code runs largely at splsoftnet and makes
heavy use of timeouts and callouts, I don't see any easy way to
achieve (a).  So that leaves me looking at option (b), at least in the
near-term.  (I think a very similar argument applies to hardware
interrupt levels vis-a-vis SPLs.)

Unless we want to make spl()s a no-op and move everything to explicit
locking, possibly also with explicit threads to handle each interrupt
or each soft-int level.  

Do we want the evolutionary approach, via several of modest steps, or
the revolutionary approach without intermediate steps?
Is there a middle ground?