> 
> There are still the issues of buffers that could not be allocated from the
> DMAable memory -- is it worth it to move pages around just to be able to
> do I/O to/from them? (probably not, but it might be fun to write the page
> migration code).  Or what about physical I/O that is done directly to/from the
> user's address space?  Those pages would not be DMA preferred pages either.
> 
And there is always swapping/paging too... mmapped files etc.
As a matter of fact, iv'e implemented page shuffeling once just to improve
on i/o performance by having contigous i/o areas in programs doing physical
i/o. The performance gain was most noticable when doing tape i/o. Streaming
tape drives really did 'stream'. Even if it wasn't a dmaable page issue it
showed that doing something that looked 'heavy' gained in the long run.
Not saying that a complicated bounce scheme would..

> It would be likely on a 20MB machine or even a 32MB machine that you would
> not be able to measure a striking perf difference between a fully optimal
> scheme and a simple, straightforward bouncing scheme.  I do suggest a

What is a fully optimal scheme? Would be hard to identify one. It's likely
that a few things may be found that might help performance, for example
migrating pages that stays in memory for long time to non dmaable pages.
Perhaps that would be more effective than trying to 'guess' the usage for
a page when allocating it... hmmm... preferably ronly pages... sooo maybe
it's not worth moving pages around to be able to _do_ i/o on them but
rather move pages around that are likely to _not do_ io on?

It really depends on your hardware. While some hardware is good on doing
mem to mem copy other's isn't. For instance when i changed the scsi driver
for my mips system to use 'direct' dma the performance gain was about 30%
on read bandwidth compared to 'bouncing'.

I think we both talk about the same solution, a simple straightforward
bounce scheme for pages not dmaable, and direct dma to pages that is
dmaable, right?