uvm objects and bus_dma

[Taylor R Campbell <riastradh%NetBSD.org@localhost>]

Several graphics drivers use swap-backed buffers that sometimes need
to be wired into physical memory and made available for DMA by the
device.  For buffers that are always wired and always mapped into KVA,
we can use bus_dmamem.  But these buffers are numerous and large, so
they should not be wired or mapped into KVA all the time.

I propose to add the following MI APIs to bus_dma(9):

int     bus_dmamem_pgfl(bus_dma_tag_t dmat);

   Return a page freelist number fit for use with uao_set_pgfl or
   uvm_pagealloc_strat(UVM_PGA_STRAT_ONLY).  Pages allocated from this
   freelist are guaranteed to have physical addresses fit for DMA with
   dmat.

int     bus_dmamap_load_pglist(bus_dma_tag_t dmat, bus_dmamap_t map,
            struct pglist *pglist, bus_size_t size, int flags)

   Load map for DMA to the pages in pglist.  The pages must have been
   allocated from the freelist returned by bus_dmamem_pgfl(dmat), and
   are typically obtained by uvm_obj_wirepages.

Typical usage:

        /* Limit to 36-bit paddrs for DMA.  */
        if (bus_dmatag_subregion(pa->pa_dmat64, 0, __BITS(0,35), &dmat,
                BUS_DMA_WAITOK))
                goto fail0;

        /* Create an object.  */
        uobj = uao_create(4*1024*1024, false);
        uao_set_pgfl(uobj, bus_dmamem_pgfl(dmat));

        /* Create a DMA map for sharing the object.  */
        error = bus_dmamap_create(dmat, 4*1024*1024, 4*1024*1024/PAGE_SIZE,
            PAGE_SIZE, 0, BUS_DMA_WAITOK|BUS_DMA_ALLOCNOW, &map);
        if (error)
                goto fail1;

        ...

        /* Need to do DMA now.  Wire the pages.  */
        error = uvm_obj_wirepages(uobj, 0, 4*1024*1024, &pglist);
        if (error)
                goto fail2;

        /* Load the DMA map.  */
        error = bus_dmamap_load_pglist(dmat, map, pglist, 4*1024*1024,
            BUS_DMA_NOWAIT);
        if (error)
                goto fail3;

        /* Program the device for DMA.  */
        for (i = 0; i < map->dm_nsegs; i++)
                DWRITE_8(sc, DMASEG_0 + 8*i, map->dm_segs[i].ds_addr);

There is a provisional implementation for x86 for the purposes of DRM
in sys/external/bsd/drm2/include/drm/bus_dma_hacks.h.  (Ignore
bus_dmamem_wire_uvm_object; it was an earlier incarnation of this
approach.)

Questions:

- Should bus_dmamem_pgfl return -1 or panic in case there is no
  workable freelist?  (Currently -- via x86_select_freelist -- it
  panics.)

- Should bus_dmamem_pgfl be called bus_dmamem_pgfreelist or something
  a little less murky?  Likewise uao_set_pgfl.  I parroted the
  abbreviation I found used in uvm, but I found it obscure and
  wouldn't mind if it changed.

Other questions or thoughts?

It would be nice if we could adopt this at least partially in netbsd-7
so that pullups can be made to implement it on architectures where we
want to use devices that need it, e.g. sparc64 radeon.