Subject: Re: FreeBSD Bus DMA (was Re: AdvanSys board support)
To: Jonathan Stone <jonathan@DSG.Stanford.EDU>
From: Justin T. Gibbs <gibbs@plutotech.com>
List: tech-kern
Date: 06/11/1998 16:32:47
>>>Perhaps the special case of a linear, no-op mapping from bus addresses
>>>to system memory addresses comes up often enough that it's worth
>>>optimizing for.
>
>Justin replied (with context snipped, because it's my text talking
>about non-x86 systems both above and below):
>
>>It happens in all cases but to an ISA bus mastering device in a system 
>>with more than 16MB of memory for x86 platforms.
>
>I think that's a question of easily verifiable fact, and from where
>I'm sitting, it's simply untrue. It may be true _FOR FREEBSD_, but
>this is, after all, a discussion about portability.

The FreeBSD x86 implementation of the bus dma interface doesn't need
any per map instance information in order to map data into bus space
so long as bounce buffers are not required.  Different implementations
may have different requirements.

>Justin, I really, sincerely, dont think you are trying to say
>something like:
>
>  }I had hoped to share an identical bus dma interface with
>  }NetBSD, but I am unwilling to do this so long as the NetBSD interface
>  }acrifices speed and memory resources [on x86] for absolutely no gain in
>  }portability [on x86].

Perhaps it would be better to say this:

I don't believe that the interface should force simple implementations to
be inefficient unless there is a portability concern. I still haven't seen
any portability issues raised here with regard to my changes.

>>You somehow believed that the FreeBSD implementation wasn't allocating
>>dma maps.
>
>
>I thought you said they didn't allocate dma maps; that the "common
>case" for drivers that communciate the device-dependent S/G data to a
>SCSI device didn't need a dmamap and that that they shared a "default"
>map.

That's not what I said.  I said that an implementation should be free
to stuff whatever it wants in it's dma map object.  In the common case
of the FreeBSD implementation, we wind up returning a "no-op map" to
the client.  Other implementations servicing other types of devices or
busses may have different requirements.

>Again, in my book, what you seem to mean by an "S/G list" is not a
>dmamap:

In NetBSD, the dma map contains the S/G list of the mapping as well as
any implementation private data required to perform the mapping.  In
FreeBSD, the dma map contains only the implementation private data.  The
MI S/G list is provided only as an argument to the callback function during
a load operation.  The FreeBSD implementation completely separates the
S/G list from the dma map, so I don't understand your concern about a
mixture of the two terms or what they mean.

>  (i)  If it's in a device-dependent format, then it DOESN'T WORK.
>       (or, to be more precise, you incur the hair of a translation
>       function from DD to an MI format, which as you pointed out is
>       nasty; and on many (most?) NetBSD platforms, you save nothing,
>       since the bus layer has to build a NetBSD-style dmamap and stash 
>       it away for the duration of the transfer, regardless.

What is 'it' here?  The dma map?  The dma map in FreeBSD is completely 
opaque, so the client needed concern itself with its contents.  If 'it'
is the S/G list, then it is provided in an MI format.  Does this clear
up your concern?

>  (ii) If it's not in a device-dependent format, but an MI format,
>       then I really, really really don't understand your claims to be
>       saving space by eliminating ``unecessary'' duplicated maps
>       (one in devive-dependent format, another in MI format).

The space is saved because the MI S/G list is not provided to the client
in static storage.  The FreeBSD implementation constructs it on the stack,
calls the callback function with a pointer to the data, and it "agoes away"
when the stack unwinds.

>This is making me less certain about what the gains from your approach
>are.  Are we talking past each other yet again?

Yes.

>>  The client always asks for a dma map to be allocated on it's
>>behalf.  What the dma map represents is up to the implentation.
>
>Justin, exactly *what* is "the client" in this context?
>Is it the chip-driver layer?
>The bus-dma layer?
>The host-specific portions of the bus-DMA layer?  

Who allocates dma maps?  The device driver does (ahc, bt, fxp, de, etc.).
This is what I mean by the client.

>Who can get access to the dmamap, if the only way to get hold of it is
>via a callback which the *driver* passes in to the bus-dma API?

The callback doesn't provide the map.  It provides the S/G list for this
particular load operation that occurred on a map that was created by the
client with a call to bus_dmamap_create().

>How can an arbitrary piece of bus-support function translate the
>"implementation dependent" dmamap to something device-independent that
>the bus-specific DMA layer can handle?  

I can't parse this.  The device driver needed care about the contents
of the dma map.  The implementation can use tricks such as indirecting 
through function pointers in the dma tag to get to functions that know
how to deal with a specific map.  The implementation is the only one
that should care about the tag or map contents.

>I sincerely don't see how this would work on a number of NetBSD
>platforms.  Perhaps that's my failing, but if this really is workable,
>the description so far is doing a good, if unintended, job at hiding
>the fact ;)

Maybe you should go look at the code?  Or perhaps someone who has can
better describe how the interface is different.

>I also genuinely don't understand what you mean when you talk about
>NetBSD's dmamaps being "static".

I was talking about static storage of the MI S/G list. My complaint is that
a large portion of what is in the dma map (the MI S/G list) doesn't need to
be in the map at all and that by including it there, the implementation is
forced to consume more memory space than it might otherwise have to.

>Are you saying the problem that
>they're allocated once and for all, rather than allocated on the fly
>for each request and freed after each request?  Is that really the
>issue?  If so, I certainly don't think it _has_ to work that way, or
>that we'd need to change the API to make it work that way. Am I
>mistaken?

Your missing what I'm talking about.  The interface really works much the
same as the NetBSD interface, so it might be best to simply start from
that perspective and then add this change:

The S/G list for a load operation is not stored in the dma map object.
It is provided to the client of bus_dmamap_load() via a callback function.
The lifetime of a mapping is the same as in NetBSD, but the client must
assume that the S/G list passed into it's callback function is only valid
during the lifetime of the callback (i.e. you can't squirrel away a pointer
to the S/G list, you must either use it or copy it to private storage).

That's really most of what is different.

--
Justin