Re: attaching cpu via lapic

[Cherry G. Mathew <cherry%zyx.in@localhost>]

Roger Pau Monné <roger.pau%citrix.com@localhost> writes:

> On Tue, Aug 22, 2017 at 06:01:42PM +0200, Manuel Bouyer wrote:
>> On Sat, Aug 19, 2017 at 01:31:09AM +0530, Cherry G. Mathew wrote:
>> > Hello,
>> > 
>> > I'm trying to improve the semantics around x86 lapic vs. cpu, with a
>> > view to wedging in the concept of "vcpu"s.
>> > 
>> > TLDR: please review this patch:
>> > http://ftp.netbsd.org/pub/NetBSD/misc/cherry/tmp/attach-cpu-with-lapic.diff
>> > 
>> > Essentially, the idea is that when the kernel runs under a hypervisor
>> > which it can detect, and this hypervisory exports a "virtual cpu" to the
>> > guest OS, this can be understood in the context of NetBSD's cpu device.
>> > 
>> > The current attachment is as follows:
>> > 
>> > 'attach cpu at cpubus'
>> > 
>> > which is then constrained via:
>> > 
>> > 'cpu* at mainbus'
>> > 
>> > similarly in the virtualised case:
>> > 'attach vcpu at xendevbus'
>> > 
>> > and,
>> > 
>> > 'vcpu* at hypervisor'
>> > 
>> > I've thought of a couple of schemes, but my current thought is as
>> > follows:
>> > 
>> > attach cpu at cpubus with lapic
>> > attach cpu at xendevbus with xvcpu
>> > 
>> > cpu* at mainbus #(via x86/(mpacpi.c|mpbios.c) as usual)
>> > cpu* at hypervisor #(via xen/hypervisor.c as usual)
>> > 
>> > The idea is that the attachment is mediated by the lapic/xvcpu
>> > respectively which can then decide how to go forward (vcpus on xen pvhvm
>> > don't need to fully initialise the underlying cpu, for eg: whereas
>> > physical cpus on xen dom0 cannot fully initialise, since xen disallows
>> > full access to the corresponding lapic).
>> 
>> I may be wrong (it's from memory and I didn't check the sources)
>> but I think Xen Dom0 support needs the physical CPUs, for proper
>> physical interrupt routing. The dom0 is doing the physical APIC configurations
>

Hi Manuel,

Clearly you have forgotten :-)

From the xen/include/i82093var.h Changelog:

revision 1.1
date: 2006-09-29 00:23:15 +0530;  author: bouyer;  state: Exp;
branches:  1.1.2;  1.1.4;  1.1.8;  1.1.58;
Add Xen3 support for ACPI and/or MPBIOS + IOAPIC. To help with this, physical
CPUs are now configured on mainbus only in dom0, and only to know about
their APIC id. virtual CPUs are attached to hypervisor as:
vcpu* at hypervisor?
and this is what's used as curcpu(). The kernel config files needs to be
updated for this, see XEN3_DOM0 or XEN3_DOMU for examples.
XEN3_DOM0 now has acpi, MPBIOS and ioapic by default.
Note that a Xen dom0 kernel doens't have access to the lapic.

> On classic PV Dom0 the IO APIC interrupt routing is done using
> hypercalls (the PHYSDEV ops [0]). It seems like there are also PHYSDEV
> ops to write to the IO APIC entries directly, but I've never used
> those.
>

Hi Roger,

The destination lapic id is explicitly ignored by Xen.

> Both current Linux and FreeBSD (for it's PVH implementation) use
> PHYSDEVOP_setup_gsi, PHYSDEVOP_map_pirq and PHYSDEVOP_eoi to deal with
> IO APIC and MSI interrupts.
>

Is there a document (other than email on the xen devel ML) somewhere
that explains various scenarios for PVH(VM) interrupt servicing - 'cause
I can't find one and it's slowing me down to trial and error.

-- 
~cherry