Re: Interrupt time inflation on Xen

[Thor Lancelot Simon <tls%panix.com@localhost>]

On Wed, Mar 30, 2016 at 03:34:42PM +0200, Christoph Badura wrote:
> 
> Maybe there are some very expensive hypervisor operations being called
> that lead to all "System" and "Interrupt" time.  Perhaps waiting for the
> hypervisor to complete the operations.
> 
> Just dd'ing from disk to /dev/null doesn't show increased "Interrupt" time
> under xen.  (dd if=/dev/rwd0d of=/dev/null bs=64k count=10000)
> It's about 16% "System" time while the dd runs, otherwise idle.  I get
> about 125 MB/s throughput.

It definitely does for me (that same dd, but from /dev/rsd0d, goes to
60% "Interrupt" time on pkgbuild).  I can't help noticing everyone who
doesn't see the problem is using wd, while I see it with ciss or arcmsr.

I have trouble seeing how the SCSI code itself could be to blame, but I
wonder if these two drivers have something in common (how they use bus_dma
perhaps)?

Now that Manuel fixed profiling, I can confirm at least part of your
suspicion:

index % time    self  children    called     name
                                                 <spontaneous>
[1]     85.7   38.10    0.00                 hypercall_page [1]
----------------------------------------------- 

This is of course not a terribly useful profiling record since it cannot
find any other functions in the call graph, so we cannot see which hypercall
might be to blame.  I *think* the use of static inlines in hypercall.h is
causing that problem, though I don't understand why the "callers" of those
inline functions are missing from the call graph.

Still puzzling about how to work through this further.

Thor