Subject: Re: Machine-independent bus DMA interface proposal
To: Jason Thorpe <thorpej@nas.nasa.gov>
From: Justin T. Gibbs <gibbs@freefall.freebsd.org>
List: tech-kern
Date: 09/23/1996 12:01:17
>Actually, I think you're completely missing the point of "portable driver".
>I think what you're suggesting this this:
>
>	- Define a name for a scatter/gather format used by a bunch
>	  of cards, call it "foo".
>
>	- Create a machine-dependent {alpha,i386}/isa/foo_machdep.c.
>
>	- Place the DMA mapping and scatter/gather list packing into
>	  a function in foo_machdep.c, call it foo_sg_pack().
>
>I'm sorry, but from a software engineering standpoint, that is simply
>_completely_ unacceptable.

If an SG format cannot be created in an MI fashion, then you would
need MD versions for each SG format.  Fortunately, I don't believe that
any of the current formats would require an MD implementation.  If they
did, you still have to provide those several MD routines down at the
driver level to make the driver portable.  The code is the same, its
just at a different level.

>In my mind, a portable driver is one that Just Works without any extra
>goop when the basic bus functionality is implemented for a given platform.
>If you have to create machine-dependent modules for each driver, you're
>simply ignoring portability.

I'm not ignoring portability at all.  Consider the SG format routines as
output filters for your current interface and I think you'll see what
I'm trying to get a here.

>I think you're focusing on the ISA/EISA/PCI systems... I'm trying to
>create a DMA mapping interface that is immediately scalable to Sun DVMA
>(this would have the side-effect of making the SPARC and Sun 3 ports'
>DVMA interfaces match, thus making it much easier to share _several_
>drivers between those ports), DEC TurboChannel DMA, ISA, EISA, PCI,
>and any other wacky thing you might throw at it.

There is nothing in my proposal that prevents this.  Show me how it
does.

>You primary point seems to be that it's wasteful to have a "transient"
>copy of the scatter/gather descriptors.  Ignoring the fact that
>the bus_dma_segment_t's are potentially much more than scatter/gather
>descriptors, let's do a little math:
>
>	In the case of a SCSI driver, the most that will ever be
>	transfered in a single transaction is MAXPHYS, which is
>	currently 64k (in Real Life Situations, the amount
>	actually transfered per-transaction is often much less).
>	In a worst case scenario (all pages in the buffer are
>	sparse) you need a maximum of:
>
>		(MAXPHYS / NBPG) + 1
>
>	scatter/gather descriptors (+1 for the slop page, in case the
>	buffer doesn't start on a page boundary).  On the i386, this
>	evaluates to 17.  In the case of each hardware scatter/gather
>	descriptor being a 32-bit address and a 32-bit length,
>	that comes out to a grand total of 136 bytes.

Your math doesn't scale.  If I want to stream 1M long video segments I
should be able to (there are actually people who do this *today* with
FreeBSD in an embedded system).  1M of data is 2k of SG information for an
aic78X0 on an x86.  2k is an unacceptable penalty especially if I have to
copy it in some for loop.  Other filesystems already in existance use
largertransaction sizes too, for example SGI's new filesystem or LFS.

>In my world, adding code complexity (where code complexity is defined
>as many machine-dependent modules for each Joe Random Scatter/Gather
>Format) is simply not worth optimizing a (corner case) 136-byte copy.

As I said before, MD implementations may not be necessary for all formats.

>Now, if we take a look are the more real work situations, transferring
>a file system block (typically 4k or 8k), on the i386 we're talking
>about a max of 2 scatter/gather descriptors, which in the case of
>32-bit address/32-bit legth, totals out to 16 bytes.  On some
>architectures, 16 bytes is below the optimization threshold for
>bcopy().  I.e. you literally gain _nothing_.

Depends on the size of the bus_dma_segment_t's fields and other factors.
Do the size of the length and address members vary from bus to bus, arch to
arch?  If not, I may have to access a 64bit (or larger) quantity on the x86
even though the bus I'm dealing with will never use those upper 32bits.

> > I haven't been convinced that there is an architectural comprimize.  Right
> > now I'm looking at more code, more memory usage (multiple K), and an extra
> > copy of usually 16 u_int32_t per transaction in an inefficient for
> > loop at the driver level.  If my device doesn't need bouncing, what
> > have I gained?
>
>The architectural compromise is machine-dependent modules for
>device-dependent scatter/gather list formats.  Given that you need to
>have device-, bus-, and machine-independent DMA mapping anyway, your
>suggestion actually adds more code.

Its the same amount of code just in a different location.

>The _only_ place where a device's scatter/gather list format should
>be dealt with is the in the driver for that device.  Period.

More code duplication.

> > I don't see why allowing the user to specify the target SG format and
> > provide a target buffer makes this interface any harder for the programmer
> > to deal with.  You can still put whatever information you need in order
> > to be "sane" in the dma_handle since it is an opaque type.
>
>...uhh, has it occured to you that every time someone invents a new
>scatter/gather list format, your interface would require updating?
>That is, unless you specify some meta-language used to describe
>a format, which is a ball of hair I don't think anyone wants to deal
>with.

If this were true, your interface would have to change each time a new bus
type was added.  You simply create a new instance of the sg_format_t
identifier.

> > I would guess that there are quite a few more devices that use the
> > 32bit addr, 32bit count format.  They also are not PC scsi cards,
> > they are PCI, EISA, and ISA SCSI cards that can be used on multiple
> > architechtures.  Furthermore, the interface is not device, bus or
> > machine dependant, it is simply SG format dependant.  The SG format
> > dependencies have to be dealt with *somewhere*.  My point is only
> > that by handling them in the interface itself, you gain efficiency
> > and code reuse.
>
>I believe your assertion is false, given that you'd have to mostly
>duplicate and then tweak the scatter/gather list packing interface
>for each architecture you port it to.

If this isn't happening in the driver's now, the SG format code can
be completly MI.

> -- save the ancient forests - http://www.bayarea.net/~thorpej/forest/ -- 
>Jason R. Thorpe                                       thorpej@nas.nasa.gov
>NASA Ames Research Center                               Home: 408.866.1912
>NAS: M/S 258-6                                          Work: 415.604.0935
>Moffett Field, CA 94035                                Pager: 415.428.6939

--
Justin T. Gibbs
===========================================
  FreeBSD: Turning PCs into workstations
===========================================