Subject: Re: UP/SMP and atomic ops
To: Andrew Doran <firstname.lastname@example.org>
From: Manuel Bouyer <email@example.com>
Date: 11/28/2007 20:23:08
On Wed, Nov 28, 2007 at 07:06:43PM +0000, Andrew Doran wrote:
> On Wed, Nov 28, 2007 at 07:43:10PM +0100, Manuel Bouyer wrote:
> > I see that the new atomic ops that have been added to kernel, are
> > patched to behave like non-atomic ops on UP kernels.
> > Even for UP systems there are cases where we want atomic memory operations:
> > - read/write to in-memory structres shared with devices. Some device
> > controllers really behaves like a CPU.
> The atomic ops defined in sys/atomic.h are specified to work only with RAM,
> not device memory. Extending them to work reliably on device memory in a
> portable way would be .. an interesting challenge.
You mean, some systems can't do it.
> > I know that reordered
> > load/store can cause issues with esiop, and I suspect the occasional timeouts
> > I've noticed can come from this. The lack of appropriate MI interface
> > to fix this has always bothered be, and I think it's a real issue.
> Is this on x86?
> bus_space_barrier (a no-op) should be enough on x86 if the
> page is marked uncacheable, but I need to check that (i.e does PG_N imply
> strongly ordered access). bus_dmamap_sync needs an lfence, which I can see
> is missing. I will see about fixing that.
It looks strange to me to use bus_space() functions to order access to
DMA memory. If bus_dmamap_sync() does it even if there's no cache
flush or bounce buffers needed, at first glance it could be enough.
Does someone know if a device can do atomic memory operation though the
PCI bus ?
> > - read-write to data shared with the hypervisor or another domain in Xen.
> > A guest may be UP, although the hardware it's running on is SMP.
> > Domain0 can eventually know the harware's number of CPUs but I'm
> > not sure this information is available to guests (and with migration, this
> > may change).
> I feel this one is an x86/xen specific problem and should be handled there.
> I have just realised that the atomic ops will be patched on xen/amd64, will
> that cause problems for the pmap?
Right now it should not be patched AFAIK, because x86/x86/cpu is not
used. But I feel that one day we should call the patch() and errata()
functions, so we'll have to handle it.
> On a tangent, there is some provision for dynamic reconfiguration with Xen,
> isn't there? That is: adding and removing virtual CPUs while the system is
Yes, it's theorically possible, but I don't know if it's functionnal
on linux right now.
Manuel Bouyer <firstname.lastname@example.org>
NetBSD: 26 ans d'experience feront toujours la difference