On Sat, Sep 02, 2006 at 07:18:21PM +0100, David Laight wrote:
> > But you have got to stop putting mutexes in other data structures
> > like this.  It causes terrible cache behavior.  (Do I actually need
> > to explain this?)
> 
> If we loop on a normal memory read then normally the cache line will stay
> valid in both (all) of the cpus' cache.

And this is where you go wrong.

Any time the current lock holder writes to that cache line, it has to
transition to exclusive (which forces an invalidation on the other
CPU(s)) and then immediately back to shared (which causes a writeback).
This is very expensive, and it happens all the time.

> A cache snoop will only happen when the cpu holding the mutex writes to
> the cache line.  Since this ought to be a 'read' snoop, it should only
> (really) slow down the cpu that is trying to acquire the lock.

Wrong.  It slows down both, *and* it creates extra FSB traffic, which
can affect other CPUs (or cores) as well.

> Of course, if the 'lock' does point elsewhere, it makes the DEBUG and
> DIAGNOSTIC code not affect the driver structures.

Well, that's another advantage.  You can also make "simplelock"s not
affect data structure layouts directly, which in turn makes the UP & SMP
kernels more similar.