On Fri, Feb 24, 2006 at 09:56:25AM -0500, Thor Lancelot Simon wrote:
> 
> Even for NetBSD running in domain 0, Xen seems to impose quite a large
> disk latency penalty, though throughput is not seriously effected.  Jed
> did some measurements of this a while ago.  I think most of the cost,
> particularly for domU, is due to the extensive remapping of data via
> the MMU, some done at our behest by the hypervisor.  (Jed, does this
> sound right to you?)

Reading this again, I think I have to correct it: for a dom0, where
there's only the one domain talking to a physical disk, the only thing I
could think of in the critical path is the need to make a hypercall to
update page tables.  (Well, that and syscall overhead, but I distinctly
recall benchmarking repeated calls to getpid() or something trivial like
that and not noticing a significant difference.)

Now that I've been looking at the clock code, I notice that the timer
event will be asserted whenever the domain becomes scheduled.  So, if
the dom0 makes itself blocked in anticipation of a disk interrupt, it'll
get a timer event as well, and (assuming I'm not missing anything)
process that first, because splclock > splbio.  I don't know if this is
anything like a significant factor.

Also, Xen presumbaly adds some latency to getting the interrupt
delivered; that's less easy for NetBSD to do anything about.

A domU is a different matter, because the hypervisor has to switch
context to and from the backend domain, which means changing the
mappings of all the OS-visible memory (i.e., including KVA, which
normally never changes), which seems to me like it'd be expensive in
terms of invalidated cache/TLB.

-- 
(let ((C call-with-current-continuation)) (apply (lambda (x y) (x y)) (map
((lambda (r) ((C C) (lambda (s) (r (lambda l (apply (s s) l))))))  (lambda
(f) (lambda (l) (if (null? l) C (lambda (k) (display (car l)) ((f (cdr l))
(C k)))))))    '((#\J #\d #\D #\v #\s) (#\e #\space #\a #\i #\newline)))))