Subject: Re: inter-cpu coherency and kernel data structures
To: Jason R Thorpe <>
From: Eduardo Horvath <>
List: tech-smp
Date: 08/14/2000 08:58:16
On Sun, 13 Aug 2000, Jason R Thorpe wrote:

> Folks...
> I noticed this in the context of the Alpha port, but I think it's
> a potential issue for all MP-capable arch's.
> On the Alpha, you're supposed to issue a memory barrier to ensure
> that other "processors" see the changes you've made to a memory
> location -- basically, you drain the write buffer.  The "processor"
> may be an I/O device (such as a disk controller, DMA mapper, etc.)
> What I'm not entirely clear on is if this is strictly necessary for
> other CPUs on the system.  I was under the impression that this is
> necessary when I wrote the Alpha simple lock primitives, issuing
> an MB after the stl_c completed successfully.
> While doing some hacking in the pmap module, it occurred to me that
> this would be a good idea for PTEs, too.  But then I thought -- geez,
> where does it end?  Do you issue an MB after frobbing with any random
> linked list structure?  Etc.
> I would appreciate some input on this -- I haven't yet read the
> architecutre manual after thinking about this... and, indeed, it
> may not be much of an issue, if a global data structure is protected
> with locks, and those locks, when released, perform the necessary
> memory barriers...
> Anyway, I wanted to bring it up.

In general:

1) Shared data structures should not be accessed w/o locking (exceptions

2) The locking primitives should issue appropriate memory barriers.

3) Since the other memory modifications come before the lock operations
they should be flushed as a sideffect of the lock operation's memory

4) (Exception from #1 above.)  Any other data structures should be using
some sort of atomic operation or software synchronization
algorithm.  These should require explicit memory barrier operations.

I/O devices usually fall into #4.

Does this help any?

Eduardo Horvath