Subject: Re: FreeBSD Bus DMA (was Re: AdvanSys board support)
To: Jonathan Stone <jonathan@DSG.Stanford.EDU>
From: Justin T. Gibbs <gibbs@plutotech.com>
List: tech-kern
Date: 06/11/1998 17:16:08
>I'm sorry to keep harping on this point, but you keep using the term
>"S/G list". I really dont think that's a good idea: it carries over
>too much baggage from SCSI controllers.
>
>On some machines, where DMA is non-cache-coherent, the hardware
>_forces_ you to keep around a list of the physical buffer pages.
>Your goal is to avoid actually building a dmamap for "simple"
>implementations.
>
>The way you do this is to postpone building the map until the driver
>calls a callback.

No. I don't know how to say this any clearer.  Perhaps a code snippet will
help:

        /*
         * Create our DMA tags.  These tags define the kinds of device
         * accessable memory allocations and memory mappings we will
         * need to perform during normal operation.
         *                     
         * Unless we need to further restrict the allocation, we rely
         * on the restrictions of the parent dmat, hence the common
         * use of MAXADDR and MAXSIZE.
         */

        /* DMA tag for mapping buffers into device visible space. */
        if (bus_dma_tag_create(bt->parent_dmat, /*alignment*/0, /*boundary*/0,
                               /*lowaddr*/BUS_SPACE_MAXADDR,
                               /*highaddr*/BUS_SPACE_MAXADDR,
                               /*filter*/NULL, /*filterarg*/NULL,
                               /*maxsize*/MAXBSIZE, /*nsegments*/BT_NSEG,
                               /*maxsegsz*/BUS_SPACE_MAXSIZE_32BIT,
                               /*flags*/BUS_DMA_ALLOCNOW,
                               &bt->buffer_dmat) != 0) {
                goto error_exit;
        }

This is the first of several tags that are created to handle allocations
by the bt driver.  This one covers buffers that need to be mapped into
bus space for SCSI I/O.  Other tags cover things like mailboxes and S/G
space.

When a BT CCB (the object that handles individual transactions) is 
allocated, we encounter code like this:

	error = bus_dmamap_create(bt->buffer_dmat, /*flags*/0,
				  &ccb->dmamap);

This occurs before any transaction is queued.  As you can see, the 
allocation of the dma map is not deferred.  Now what is contained in
the dma map is completely opaque to the client of bus_dmamap_create.
It is simply a cookie.

When a transaction comes in and a buffer needs to be loaded, we do this:

	s = splvm();
	error = bus_dmamap_load(bt->buffer_dmat, bccb->dmamap,
				csio->data_ptr,	csio->dxfer_len,
				btexecuteccb, bccb, /*flags*/0);
	if (error == EINPROGRESS) { 
		 /*
		  * So as to maintain ordering, freeze the
		  * controller queue until our mapping is
		  * returned.
		  */
	...
	splx(s);

And on completion, btexecuteccb is called:

static void                                         
btexecuteccb(void *arg, bus_dma_segment_t *dm_segs, int nseg, int error)
{
	/*
	 * Convert bus_dma_segment list to device dependent format and
	 * store it in dma'able S/G space.
	 */
}	

As "dm_segs" is provided to me as an argument to my callback, the 
requirement that this information be stored in the dma map object
is removed.

>But how is the bus-dma layer supposed to get at the map itself if the
>only way to construct the map is ot call the _driver's_ callback?

The map is constructed up front, so this question makes no sense.

>This makes no sense to me at all. It's the same issue as with the 
>outboard bus TLBs (mapping registers): if you need a virtual-to-physical
>address map for every page in the DMA buffer, you need the dmamap;
>and in your design, the bus layer itself cant' get at it.

The bus layer itself was the one that created the dma map object, so it
should have put whatever information it needed into that object so that it
can do it's job.  You cannot assume that the MI bus dma segment list will
be exactly what the implementation wants in the dma map to perform it's
job.  In implementing the FreeBSD version of bus dma bounce buffers, I use
a totally different, more space efficient, data structure than a dma
segment list in order to store my mapping info.

>I think it's entirely fair to claim that you roriginal claim is wrong:
>there is a severe portability problem with the FreeBSD design in this
>area.

I think it's entirely fair to claim that you are still very much confused.

>I think we need to reach agreement on this point before any further
>progress can be made; do you agree to the need to reach agreement?.

I think you need to become less confused first. 8-)

>I'm thinking about adding a "BUS_DMAMAP_TRIVIAL" flag which lets a
>driver and the bus_dma layer conspire to avoid constructing a map, if
>and only if *both* the bus_dma layer and the driver dont need the map
>info.  I think that's likely to be more successful than your approach.
>Perhaps we could talk about that design approach?

Constructing a map isn't the issue.  The issue is the freedom of the 
implementation (the code that implements bus_dmamap_create for a particular
arch, bus, etc.) to put whatever is convenient for it into the map object
so that map objects are completely opaque.

--
Justin