Subject: Re: SMP re-entrancy in kernel drivers/"bottom half?"
To: Jason Thorpe <firstname.lastname@example.org>
From: Jonathan Stone <email@example.com>
Date: 12/17/2003 19:18:50
In message <ECF1DBA0-3105-11D8-91BB-000A957650EC@wasabisystems.com>,
Jason Thorpe writes:
>The hierarchy of simplelocks is simply not an acceptable solution:
s/strict hierarcy/whatever we have now/, then. That was all I really
meant. I hadnt noticed the splclock() changes.
(Forgive me if I dont pay attention to Events that at 100Hz when I'm
focusing on event sthat hapen at several tens of KHz per device, with
mulpitle such devices per system.)
> 3. I'm not seeing how it would actually solve your problem. If you
> want to have 2 CPUs handling NIC interrupts, they would both have
> to acquire the IPL_NET_slock, which would stall one of them.
Err, no, I must have phrased it carelessly: thats precisely
what I *dont* want. See below.
>What have I missed?
Interrupt handler woudn't aquire the IP_NET_slock. They acquire just
the lock for a given different softc, when they enter the device
interrupt handler _for that device_. They dont need to get the
IPL_NET lock until they need to touch datastructures outside the
device softc itself (say for a NIC driver, dropping a packet on the ip
(or other) input queue.)
> 4. Having to acquire several simplelocks in a hierarchy would increase
> the interrupt latency.
Nope, the `hierarchy' (modified above to `emulate whatever we have
now') was for IPLs *above* device interrupts.
I suggested the interrupt glue will know whether an interrupt routine
was SMP-safe before entering it (from a flag provided by the device,
when it registered the interrupt).
If we changed the argument to an (SMP-safe) interrupt routine from a
void * to a struct device * (or an extension of struct device), then
the per-device spinlock could be part of struct device, and the
interrupt glue could test or acquire the per-softc lock before
entering the driver. I'm agnostic about doing that versus spinning in
the very beginning of the interrupt hanlder.
If and when the driver needs IP_NET_slock, to touch something outside
the softc, it can attempt to obtain it.
Yamamoto-san already suggested doing away wiht per-IP locks and just
using kernel_lock. smp-intr-save drivers would then attempt to acquire
kernel_lock before they reach into data structures outside the softc.
I said I though that should work, too. but it seems a (small) step in
the wrong direction, too: we want finer-grained locks, not yet more
use of kernel_lock.