So I finally figured out my main problem.  Between rev 1.31 and
1.32 of uvm_swap.c, Someone removed the call to brelvp(), in
order to disassociate the soon-to-be-freed buf from the vnode.
Contrary to the comment in r1.31, b_vp is NOT set to NULLVP -- it
is initialized to NULLVP, but then twiddled by the bgetvp() call.
So, we end up with a freed buf that is still on the vnode's list
of busy buffers.

  Next time we do I/O to this swap file, we add the new buf to
the list of bufs associated with the vnode, which means
changing the previous head (i.e., the freed buf)'s prev
pointers.  Corruption of buf on the free list, and panic
ensues at the next pool_get, if we're lucky, or at the second
pool_get if we are unlucky.

  Note that only the brelvp() in the swap-to-file routine
sw_reg_iodone() was removed.  Thus, ordinary swapping to a
device was not broken.

  So uvm_swap.c should be modified.  Chuck and Chuck?

  HOWEVER, once I add back the call to brelvp(), the system does
fine for a while, but eventually panics with:
pmap_change_attrs: found pager VA on pv_listpmap_change_attrs: found pager VA on pv_listpmap_change_attrs: found pager VA on pv_listpmap_change_attrs: found pager VA on pv_listpagedaemon:
deadlock detected!
panic: pagedaemon deadlock

  I don't know if this is related or independent.

  Brian