Subject: DVMA
To: None <port-sparc@NetBSD.ORG>
From: Chuck Cranor <chuck@maria.wustl.edu>
List: port-sparc
Date: 06/06/1995 00:28:55
hi-

   i am looking for ideas on how to handle DVMA with VME devices under
NetBSD/sparc.  here is the problem i am facing:

   the hardware maps the lower addresses of the VME bus into the kernel
virtual address space.   VME address zero maps to DVMA_BASE (0xfff00000),
and the mappings end at DVMA_END (0xfffc0000).   in order to do DMA
to a VME device, the kernel's buffer must be mapped in that range
of virtual addresses.   the address given to the hardware is simply
the kernel virtual address minus DVMA_BASE.  for example, if there was 
a kernel buffer at KVA 0xfffc8000 then the hardware would see it at
VME address 0x8000.

   in NetBSD/sparc, DVMA space is mapped by "phys_map" (a vm_map_t
declared in vm/vm_kern.h).

   i have written a device driver for the xylogics 7053 VME/SMD disk
controller ("xd").   all I/O requests come in via the xdstragegy() routine
in "struct buf *"s.   there are two types of "struct buf *"s that my
driver understands.    if (bp->b_flags & B_PHYS) != 0, then the buffer
is from an I/O operation on the raw disk device (e.g. /dev/rxd0a) and
the buffer is already mapped into DVMA space due to the vmapbuf()
function in sparc/vm_machdep.c.    if (bp->b_flags & B_PHYS) == 0, then
the buffer is part of the kernel's "struct buf" array, and is not
mapped in DVMA space (but is in kernel space).   if that is the case, 
then it must be mapped into DVMA space before I/O can begin.   also, 
once I/O is done this mapping must be removed.

   what I wanted to do was to establish the mapping for the non-B_PHYS
case before I started I/O and the remove it in the xdintr() routine.
to test this, I ran a "while (1) ; fsck /dev/rxd0a ; end" in one window
while doing "iozone" on /dev/xrd1g in another.    I discovered that my 
scheme didn't work.   the problem was that the "fsck" would do an I/O 
on the raw device and it would call vmapbuf().   this would cause phys_map 
to be locked.  in the mean time an xd interrupt would happen (due to 
the "iozone") and the interrupt routine would try and unmap a buffer 
while the phys_map was locked by the vmapbuf() call.   this would cause 
the interrupt to try and go to sleep in lock_write() by calling 
thread_sleep() [vm/kern_lock.c].   this would cause my system to crash.

   i solved this problem by protecting all accesses to "phys_map"
with splbio().   this works just fine.   however, it is unclear to
me if phys_map can be handled this way.   in fact, pk and theo say 
that it is unwise to call kmem_free_wakeup() at interrupt time and
that I should try and to all my memory mappings in the context of a
process rather than in an interrupt.   for the mapin, this is easy enough
to do... you just map it in in the top of the strategy routine
(although this is somewhat wasteful of DVMA space).   however, I am
not sure what to do with the map out operation.   the map out is triggered
by an interrupt.   one thing that was suggested was keeping a queue
of buffers to map out in the device driver's softc structure and then wait
until a non-interrupt time to do it (e.g. hope that someone calls xdstragegy
soon?).   that seems pretty ugly to me.  what else can I do?   i'm
looking for ideas or insight that can help me.

   i would like to get this driver out the door and in the tree as the
controller specific parts of xd.c seem to work just fine.   it is just
a matter of getting the sparc glue right.   [it should also pop in
the sun3 port without too much trouble...]

thanks,
chuck