Re: Am I using bus_dma right?

[Eduardo Horvath <eeh%NetBSD.org@localhost>]

Let me try to simplify these concepts.

On Thu, 23 Apr 2020, Mouse wrote:

> I'm not doing read/write DMA.  DMA never transfers from memory to the
> device.  (Well, I suppose it does to a small extent, in that the device
> reads buffer descriptors.  But the buffer descriptors are set up once
> and never touched afterwards; the code snippet I posted is not writing
> to them.)

If you are not doing DMA you don't need to do any memory synchronization 
(modulo SMP issues with other CPUs, but that's a completely different 
topic.)

> The hardware is DMAing into the memory, and nothing else.  The driver
> reads the memory and immediately writes it again, to be read by the
> driver some later time, possibly being overwritten by DMA in between.
> So an example that says "do write DMA" is not directly applicable.

If a (non CPU) device is directly reading or writing DRAM without the 
CPU having to read a register and then write its contents to memory, then 
it is doing DMA.

The problem is many modern CPUs have write-back caches which are not 
shared by I/O devices.  So when you do a read operation (from device to 
CPU) you should:

1) Do a BUS_DMASYNC_PREREAD to make sure there is no data in the cache 
that may be written to DRAM during the I/O operation.

2) Tell the hardware to do the read operation.

3) When the transaction completes issue a BUS_DMASYNC_POSTREAD to make 
sure the CPU sees the data in DRAM not stale data in the cache.


When you do a write operation you should:

1) Make sure the buffer contains all the data you want to transmit.

2) Do a BUS_DMASYNC_PREWRITE to make sure any data that may remain in the 
CPU writeback cache is flushed to memory.

3) Tell the hardware to do the write operation.

4) When the write operation completes... well it shouldn't matter.


If you have a ring buffer you should try to map it CONSISTENT which will 
disable all caching of that memory.  However, some CPUs will not allow you 
to disable caching, so you should put in the appropriate bus_dmamap_sync() 
operations so the code will not break on those machines.

When you set up the mapping for the ring buffer you should do either a 
BUS_DMASYNC_PREREAD, or if you need to initialize some structures in that 
buffer use BUS_DMASYNC_PREWRITE.  One will do a cache invalidate, the 
other one will force a writeback operation.

When you get a device interrupt, you should do a BUS_DMAMEM_POSTREAD to 
make sure anything that might have magically migrated into the cache has 
been invalidated.  Then copy the data out of the ring buffer and do 
another BUS_DMASYNC_PREREAD or BUS_DMASYNC_PREWRITE as appropriate.

> The example makes it look as though read DMA (device->memory) needs to
> be bracketed by PREREAD and POSTREAD and write DMA by PREWRITE and
> POSTWRITE.  If that were what I'm doing, it would be straightforward.
> Instead, I have DMA and the driver both writing memory, but only the
> driver ever reading it.
> 
> Your placement for PREREAD and POSTREAD confuses me because it doesn't
> match the example.  The example says
> 
> 	/* invalidate soon-to-be-stale cache blocks */
> 	bus_dmamap_sync(..., BUS_DMASYNC_PREREAD);
> 	[ do read DMA ]
> 	/* copy from bounce */
> 	bus_dmamap_sync(..., BUS_DMASYNC_POSTREAD);
> 	/* read data now in driver-provided buffer */
> 	[ computation ]
> 	/* data to be written now in driver-provided buffer */
> 	/* flush write buffers and writeback, copy to bounce */
> 	bus_dmamap_sync(..., BUS_DMASYNC_PREWRITE);
> 	[ do write DMA ]
> 	/* probably a no-op, but provided for consistency */
> 	bus_dmamap_sync(..., BUS_DMASYNC_POSTWRITE);
> 
> but what your changes would have my driver doing is
> 
> [read-direction DMA might happen here]
> PREREAD
> driver reads data from driver-provided buffer
> POSTREAD
> [read-direction DMA might happen here]
> PREWRITE
> driver writes data to driver-provided buffer
> POSTWRITE
> [read-direction DMA might happen here]

That bit is not right.

> The conceptual paradigm is
> 
> - at attach time: allocate, set up, and load the mapping
> 

Presumably you should do a BUS_DMASYNC_PREWRITE somewhere in here

> - at open time: tell hardware to start DMAing

and a BUS_DMASYNC_POSTWRUTE arond here.

> 

Here you need a BUS_DMAMEM_POSTREAD.

> - at read time (ie, repeatedly): driver reads buffer to see how much
>    has been overwritten by DMA, copying the overwritten portion out and
>    immediately resetting it to the pre-overwrite data, to be
>    overwritten again later

If you wrote anything to the ring buffer during this operation you need to 
insert a BUS_DMASYNC_PREWRITE.

> 
> - at close tiem: tell hardware to stop DMAing
> 
> The map is never unloaded; the driver is not detachable.  The system
> has no use case for that, so I saw no point in putting time into it.
> 
> The code I quoted is the "at read time" part.  My guess based on the
> manpage's example and what you've written is that I need
> 
> 	while (fewer than n samples copied)
> 		POSTWRITE
> 		POSTREAD
> 		read sample from buffer
> 		if sample isn't "impossible"
> 			write "impossible" value to buffer
> 			PREWRITE
> 		PREREAD
> 		if sample is "impossible", break


That sounds about right.

> 
> because some aspects of "write", and relatively normal "read", are
> happening outside that code segment.  But this is different enough from
> what you said (and possibly not well-paired - should the PREWRITE be
> outside the if?) that now I'm possibly even less sure of myself.  I
> could just try different permutations in the hope of finding something
> that works, but that strikes me as one of the worst possible ways to do
> it; I would prefer to understand the paradigm enough to get it right.
> 
> I am not concerned about the race between pushing the driver-written
> value to the buffer and DMA overwriting it; provided the driver's write
> gets pushed reasonably promptly, this will happen only in error
> conditions like userland ignoring the device for too long - it takes
> the hardware multiple seconds to wrap around the ring buffer.
> 
> > I always have to look up the direction, but READ is when CPU reads
> > data provided by the device.
> 
> Yes: READ corresponds to read() and WRITE to write().  One of the
> things that confuses me is that I have no write-direction DMA going on
> at all; all the DMA is in the read direction.  But there is a driver
> write to the buffer that is, to put it loosely, half of a write DMA
> operation (the "host writes the buffer" half).

When the CPU updates the contents of the ring buffer it *is* a DMA write, 
even if the device never tries to read the contents, since the update must 
be flushed from the cache to DRAM or you may end up reading stale data 
later.

Eduardo