Subject: Re: splx() optimization [was Re: SMP re-eetrancy in "bottom half" drivers]
To: YAMAMOTO Takashi <yamt@mwd.biglobe.ne.jp>
From: Jonathan Stone <jonathan@dsg.stanford.edu>
List: tech-kern
Date: 06/07/2005 16:38:04
In message <1118184979.987812.2086.nullmailer@yamt.dyndns.org>,
YAMAMOTO Takashi writes:

>> 
>> Because, I personally, have basically zero interest in SMP audio
>> drivers??  (btw, I feel reasonably sure I said so, the last time you
>> suggested audio.)
>
>then why don't you just lower IPL_AUDIO in your local tree?

Because then I can't commit any of these changes, until someone
actually delivers on the idea to elimiate IPLs entirely, one by one,
in IPL order.  And I see no profit in waiting for that.


>> On the other hand, I have considerable interest in delivering 10Gbit
>> of TCP traffic -- or as close as I can get. And I'm very, _very_ sure
>> that I can get closer by using more CPUs.
>
>what you're saying here is that you want performance but
>are not intersted in necessary work (== audio).
>i don't think it's reasonable.

Audio drivers have kHz sample rates and generally handle DMA buffers
with some nontrival number of samples. 10GbE ethernet NICs deliver
800,000 1500-byte packet per second. You seriously think you can put
an onus on a developer, that working on the low-rate audio problem is
a pre-requisite to tackling the 10GbE problem?

That strikes me as indefensible.


>besides, i don't think you can achieve your desired performance with
>lock-per-IPL.  the frequent locking due to spl operations will easily kill
>any performance benefits from concurrency.

Huh??  Which frequent spl() operations are these?  I wonder if you and
Jason have lost sight of what I'm actually proposing to do.

Of _course_ I expect to put explicit per-data-structure locks in the
datastructures that are touched frequently: device softc, per-protocol
input queues. (Does that really need to be said??)

More, when we get around to attacking splsoftnet(), we clearly need
locks at the boundaries (or "perimeter") between lower-half
splsoftnet() and upper-half code.  Sockets and socket-queues look like
reasonable candidates for what Jason calls "perimeters".


But I *don't* want to bite off having to make (again, purely as an
example) all the lPPP code and its mess of spltty()/splnet()/
splsoftnet() interactions all explicitly locked.

It's those non-local (in the sense of "in another module or
subsystem", or tied to another module or subsystem) which strike me
as a significant barrier to entry/progress.

It's *just* such cases where, in the short-to-medium term, I propose
to temporarily punt by introducing per-IPL locks.  Or maybe for the
IPLs where aforesaid modules get their fingers in each other.