On Tue, 2005-05-17 at 15:08, Jonathan Stone wrote:
> In message <1116318610.383363.1485.nullmailer@yamt.dyndns.org>,
> YAMAMOTO Takashi writes:
> 
> Hi,
> 
> I think we are in agreement; I had lumped the issue you raise as a
> potential misunderstanding under "finding an acceptable lock/spl
> hierarchy".
> 
> >what i meant is, even if you made all network drivers "smp-safe",
> >you still need to grab biglock for them because they can be
> >interrupted by higher level interrupts which need biglock.
> >ie. in the case of i386, it's better to tackle audio drivers first.
> 
> A 1 Gbit NIC will generate on the order of 3,000 interrupts/sec when
> filling a pipe. (I just tried a 3.0GHz P4 Xeon with an onboard wm; i
> see about 7k interrupts/sec).  So a 10Gbit NIC will generate, say,
> 30,000 to 70,000 interrupts/sec to fill the wire.
> 
> I find the NIC case for SMP compelling. I don't find audio compelling,
> but I'll beleive you if you say audio is easier.
> 
> 
> 
> >alternatives i can think of are:
> >
> >	- introduce some magic in md intr code to determine
> >	  the highest level IPL which needs biglock dynamically.
> >	  (device hotplug nightmare?)
> 
> "Nightmare" any way you take it.
> 
> >	- obsolete IPLs.
> >	  (like freebsd.  i think my old patch contains an equivalent hack.)

While there are no IPLs in FreeBSD any more - part of the concept kind
of survived the transformation.
The interrupt handlers (with exceptions) normally do not call any device
driver functions. Instead they schedules an interrupt thread that is
responsible for calling the device driver's interrupt function.
The priorities of the interrupt threads are loosely based on previous
spl levels allowing interrupt threads with better priority to interrupt
(preempt) threads with lower priority.

> 
> Yes; your old patch for i386 SMP effectively collapsed all SPLs to a
> single level. That way, no driver should interrupt another.

One level may be a little extreme since you may run into trouble with
pmap operations in drivers that need interprocessor interrupts for TLB
shootdowns.
Serial devices (and others) may also take a dim few on the latency
issues this may cause.

> OTOH, I to recall considerable unhappiness with that approach.
> several people (Jason? Christos?) observed that we can already handle
> hardclock interrupts concurrently on multiple CPUs.
> 
> Again, just for myself, tens of thousands of interrupts for a 10GbE
> NIC (or on the close order of 1e5, for multiple 10GbE NICs) are a much
> more compelling case than ~1e3 interrupts/CPU.
>