bus_dma without bouncing

[Taylor R Campbell <riastradh%NetBSD.org@localhost>]

NIC drivers allocate rx buffers for the device to read network data
into through DMA.  In some devices the device's DMA has constraints,
specified through the DMA tag and/or bus_dmamap_create parameters, and
enforced in bus_dmamap_load by bouncing.

To avoid bouncing, or to satisfy alignment requirements imposed by the
device[1], drivers will often allocate rx buffers themselves with
bus_dmamem_alloc/map.

But, given constraints, bus_dmamap_create will _also_ allocate bounce
buffers with bus_dmamem_alloc/map -- even if the driver will never
bounce because it already allocated rx buffers satisfying the
constraint.  This wastes physical pages and kva that will never be
used -- except if there's a driver bug causing every rx buffer to be
bounced unintentionally, which quietly gives bad performance!

Should we have a way to avoid this?

Here are a couple options that come to mind (with 30sec of thought put
into each name, and alternate bike shed paint suggestions welcome):

1. An option BUS_DMA_BOUNCELESS to bus_dmamap_create that causes
   bus_dmamap_load to fail instead of bouncing if constraints are not
   met.

   => This simple API change generally obviates the need to allocate
      bounce buffers in the scenario above.  However, some
      architectural constraints such as confidential computing
      architecture (AMD SEV, Intel TDX) may still require bouncing for
      other reasons (must selectively expose page unencrypted to the
      hypervisor).

2. (a) An option BUS_DMA_ALLOCBUF to bus_dmamap_create that _forces_
       it to allocate a bounce buffer with bus_dmamem_alloc/map
       meeting the constraints.
   (b) Another operation, such as bus_dmamap_bufmem, to get at the
       bounce buffer so a driver can work directly with it instead of
       calling bus_dmamem_alloc/map itself.

   => This would allow us to avoid the duplicate bounce buffer in the
      confidential computing case.  But it's a more complicated change
      to the API, and some bus_dma implementations -- such as sparc
      and sparc64 -- appear not to bounce at all, ever, so they would
      have to be adapted to add logic for allocating bounce bufers.



[1] It seems there is no MI way with bus_dma tags or bus_dmamap_create
    to make bus_dmamap_load enforce alignment by bouncing into an
    aligned bounce buffer if the given address is misaligned.  But
    there are plenty of devices that require aligned addresses for
    DMA, such as Intel wireless cards which require 8- or 12-bit
    alignment.  Am I missing something?  Is there just no way to
    express this constraint in bus_dmamap_create so that
    bus_dmamap_load will _either_ fail noisily or bounce on misaligned
    inputs?