Re: xen interrupt system

[Christoph Egger <Christoph_Egger%gmx.de@localhost>]

On Monday 31 August 2009 11:02:07 Manuel Bouyer wrote:
> On Mon, Aug 31, 2009 at 10:52:43AM +0200, Christoph Egger wrote:
> > Intel sent me a new patch to test which maps the value passed
> > PHYSDEVOP_alloc_irq_vector with EVTCHN_bind_pirq internally, so that the
> > value passed to EVTCHN_bind_pirq is "recognized" by Xen when it allocates
> > the real interrupt "on demand" on IOAPIC writes.
> >
> > Now this makes the NetBSD dom0 fail to boot in a different way:
> > Our way to initialize ioapics fails.
> >
> > ioapic0 at mainbus0 apid 8, virtual wire mode
> > (XEN) io_apic.c:2196:
> > (XEN) ioapic_guest_write: apic=0, pin=0, irq=0
> > (XEN) ioapic_guest_write: new_entry=00010900
> > (XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
> > (XEN) io_apic.c:2196:
> > (XEN) ioapic_guest_write: apic=0, pin=2, irq=0
> > (XEN) ioapic_guest_write: new_entry=00010900
> > (XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
> > (XEN) irq.c:1297: dom0: pirq 0 or irq 3 already mapped
>
> This is when we clear the ioapic. It used to fail before too, but it
> was not fatal. E.g. with xen 3.1.4:
> (XEN) io_apic.c:2135:
> (XEN) ioapic_guest_write: apic=0, pin=2, old_irq=0, new_irq=-1
> (XEN) ioapic_guest_write: old_entry=000009f0, new_entry=00010900
> (XEN) ioapic_guest_write: Attempt to remove IO-APIC pin of in-use IRQ!
> (XEN) io_apic.c:2135:
> (XEN) ioapic_guest_write: apic=0, pin=4, old_irq=4, new_irq=-1
> (XEN) ioapic_guest_write: old_entry=000009f1, new_entry=00010900
> (XEN) ioapic_guest_write: Attempt to remove IO-APIC pin of in-use IRQ!
>
> > PHYSDEVOP_APIC_WRITE ret -22
> > panic: PHYSDEVOP_APIC_WRITE
>
> we could change the prototype of ioapic_write_ul to return an error
> number instead of void, and let the caller handle the error (n
> ioapic_attach we would just ignore the error). the native ioapic_write_ul
> would always return 0 of course.

Tried that and makes NetBSD dom0 boot again and I don't see any
lost interrupt issues.

Intel sent me another patch which makes the PHYSDEVOP_APIC_WRITE
hypercall return 0 and prints a warning when an internal mapping is detected.
With Intel's latest patch, NetBSD dom0 boots again w/o any changes.

Now Xen prints:

ioapic0 at mainbus0 apid 8, virtual wire mode
(XEN) io_apic.c:2196: 
(XEN) ioapic_guest_write: apic=0, pin=0, irq=0
(XEN) ioapic_guest_write: new_entry=00010900
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) io_apic.c:2196: 
(XEN) ioapic_guest_write: apic=0, pin=2, irq=0
(XEN) ioapic_guest_write: new_entry=00010900
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) irq.c:1297: dom0: pirq 0 or irq 3 already mapped
(XEN) io_apic.c:2196: 
(XEN) ioapic_guest_write: apic=0, pin=4, irq=4
(XEN) ioapic_guest_write: new_entry=00010900
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) irq.c:1297: dom0: pirq 0 or irq 5 already mapped
(XEN) irq.c:1297: dom0: pirq 0 or irq 6 already mapped
(XEN) irq.c:1297: dom0: pirq 0 or irq 7 already mapped
(XEN) irq.c:1297: dom0: pirq 0 or irq 8 already mapped
(XEN) irq.c:1297: dom0: pirq 0 or irq 9 already mapped
(XEN) irq.c:1297: dom0: pirq 0 or irq 10 already mapped
(XEN) irq.c:1297: dom0: pirq 0 or irq 11 already mapped
(XEN) irq.c:1297: dom0: pirq 0 or irq 12 already mapped
(XEN) irq.c:1297: dom0: pirq 0 or irq 13 already mapped
(XEN) irq.c:1297: dom0: pirq 0 or irq 14 already mapped
(XEN) irq.c:1297: dom0: pirq 0 or irq 15 already mapped
ioapic1 at mainbus0 apid 9, virtual wire mode
(XEN) allocated vector for irq:16
(XEN) irq.c:1297: dom0: pirq 0 or irq 16 already mapped
(XEN) allocated vector for irq:17
(XEN) irq.c:1297: dom0: pirq 0 or irq 17 already mapped
(XEN) allocated vector for irq:18
(XEN) irq.c:1297: dom0: pirq 0 or irq 18 already mapped
(XEN) allocated vector for irq:19
(XEN) irq.c:1297: dom0: pirq 0 or irq 19 already mapped
(XEN) allocated vector for irq:20
(XEN) irq.c:1297: dom0: pirq 0 or irq 20 already mapped
(XEN) allocated vector for irq:21
(XEN) irq.c:1297: dom0: pirq 0 or irq 21 already mapped
(XEN) allocated vector for irq:22
(XEN) irq.c:1297: dom0: pirq 0 or irq 22 already mapped
(XEN) allocated vector for irq:23
(XEN) irq.c:1297: dom0: pirq 0 or irq 23 already mapped
(XEN) allocated vector for irq:24
(XEN) irq.c:1297: dom0: pirq 0 or irq 24 already mapped
(XEN) allocated vector for irq:25
(XEN) irq.c:1297: dom0: pirq 0 or irq 25 already mapped
(XEN) allocated vector for irq:26
(XEN) irq.c:1297: dom0: pirq 0 or irq 26 already mapped
(XEN) allocated vector for irq:27
(XEN) irq.c:1297: dom0: pirq 0 or irq 27 already mapped
(XEN) allocated vector for irq:28
(XEN) irq.c:1297: dom0: pirq 0 or irq 28 already mapped
(XEN) allocated vector for irq:29
(XEN) irq.c:1297: dom0: pirq 0 or irq 29 already mapped
(XEN) allocated vector for irq:30
(XEN) irq.c:1297: dom0: pirq 0 or irq 30 already mapped
(XEN) allocated vector for irq:31
(XEN) irq.c:1297: dom0: pirq 0 or irq 31 already mapped
ioapic2 at mainbus0 apid 10, virtual wire mode
(XEN) allocated vector for irq:32
(XEN) irq.c:1297: dom0: pirq 0 or irq 32 already mapped
(XEN) allocated vector for irq:33
(XEN) irq.c:1297: dom0: pirq 0 or irq 33 already mapped
(XEN) allocated vector for irq:34
(XEN) irq.c:1297: dom0: pirq 0 or irq 34 already mapped
(XEN) allocated vector for irq:35
(XEN) irq.c:1297: dom0: pirq 0 or irq 35 already mapped
(XEN) allocated vector for irq:36
(XEN) irq.c:1297: dom0: pirq 0 or irq 36 already mapped
(XEN) allocated vector for irq:37
(XEN) irq.c:1297: dom0: pirq 0 or irq 37 already mapped
(XEN) allocated vector for irq:38
(XEN) irq.c:1297: dom0: pirq 0 or irq 38 already mapped
(XEN) allocated vector for irq:39
(XEN) irq.c:1297: dom0: pirq 0 or irq 39 already mapped
(XEN) allocated vector for irq:40
(XEN) irq.c:1297: dom0: pirq 0 or irq 40 already mapped
(XEN) allocated vector for irq:41
(XEN) irq.c:1297: dom0: pirq 0 or irq 41 already mapped
(XEN) allocated vector for irq:42
(XEN) irq.c:1297: dom0: pirq 0 or irq 42 already mapped
(XEN) allocated vector for irq:43
(XEN) irq.c:1297: dom0: pirq 0 or irq 43 already mapped
(XEN) allocated vector for irq:44
(XEN) irq.c:1297: dom0: pirq 0 or irq 44 already mapped
(XEN) allocated vector for irq:45
(XEN) irq.c:1297: dom0: pirq 0 or irq 45 already mapped
(XEN) allocated vector for irq:46
(XEN) irq.c:1297: dom0: pirq 0 or irq 46 already mapped
(XEN) allocated vector for irq:47
(XEN) irq.c:1297: dom0: pirq 0 or irq 47 already mapped


> > Intel and XenSource also stated:
> >
> > Intel:
> > "For MSI, we need to solve the pirq conflict issue, since the irq passed
> > by PHYSDEVOP_alloc_irq_vector is recognized as pirq number in the
> > EVTCHN_bind_pirq."
> >
> > XenSource (in response to Intel's statement):
> > "My guess is that NetBSD does not have support for MSI on Xen so that's a
> > non-issue. And Linux provides GSI to alloc_irq_vector and to bind_pirq
> > (when setting up legacy INTx irqs) so obviously avoids the possible
> > conflict. If NetBSD supports MSI-on-Xen in future, it will have to clean
> > up its act regarding the irq numbers it hands to the abovementioned
> > hypercalls. That's a reasonable constraint though."
>
> We don't support MSI for now, even in native mode.

Are there any plans to support MSI's ?

Christoph