Subject: Re: inter-cpu coherency and kernel data structures
To: Jason R Thorpe <email@example.com>
From: Eduardo Horvath <firstname.lastname@example.org>
Date: 08/14/2000 08:58:16
On Sun, 13 Aug 2000, Jason R Thorpe wrote:
> I noticed this in the context of the Alpha port, but I think it's
> a potential issue for all MP-capable arch's.
> On the Alpha, you're supposed to issue a memory barrier to ensure
> that other "processors" see the changes you've made to a memory
> location -- basically, you drain the write buffer. The "processor"
> may be an I/O device (such as a disk controller, DMA mapper, etc.)
> What I'm not entirely clear on is if this is strictly necessary for
> other CPUs on the system. I was under the impression that this is
> necessary when I wrote the Alpha simple lock primitives, issuing
> an MB after the stl_c completed successfully.
> While doing some hacking in the pmap module, it occurred to me that
> this would be a good idea for PTEs, too. But then I thought -- geez,
> where does it end? Do you issue an MB after frobbing with any random
> linked list structure? Etc.
> I would appreciate some input on this -- I haven't yet read the
> architecutre manual after thinking about this... and, indeed, it
> may not be much of an issue, if a global data structure is protected
> with locks, and those locks, when released, perform the necessary
> memory barriers...
> Anyway, I wanted to bring it up.
1) Shared data structures should not be accessed w/o locking (exceptions
2) The locking primitives should issue appropriate memory barriers.
3) Since the other memory modifications come before the lock operations
they should be flushed as a sideffect of the lock operation's memory
4) (Exception from #1 above.) Any other data structures should be using
some sort of atomic operation or software synchronization
algorithm. These should require explicit memory barrier operations.
I/O devices usually fall into #4.
Does this help any?