Re: pae MP on cherry-xenmp

[Cherry G. Mathew <cherry%zyx.in@localhost>]

Hi Michael,

>>>>> "mhitch" == Michael L Hitch <mhitch%lightning.msu.montana.edu@localhost> 
>>>>> writes:

    mhitch> On Wed, 24 Aug 2011, Cherry G. Mathew wrote:
    >> i386/pae domU, which is slightly different from an i386 domU. You
    >> can run this on a 64bit xen hypervisor, and unless you're running
    >> really va hungry (> 3G) applications, an i386 userland would
    >> happily run on it.

    mhitch>   Jeff Rizzo and I had a problem trying the pae MP kernel
    mhitch> with an amd64 hypervisor.  I think I have tracked down that
    mhitch> problem, and I suspect it's because Jeff and I have > 4GB on
    mhitch> our machines.

    mhitch>   The problem is in sys/arch/xen/x86/cpu.c where it's

Good catch! I did go after this hypothesis, but got distracted with
other things.

    mhitch> 
    mhitch> setting
    initctx-> ctrlreg[3] to xpmap_ptom(ci->ci_pae_l3_pdirpa).  The
    mhitch> ctrlreg[] array is an unsigned long type, which is 32 bits
    mhitch> on i386.  While the code is careful to make sure that
    mhitch> ci_pae_l3_pdirpa is belown 4GB on a PAE kernel, that only
    mhitch> reflects the physical address for the VM.  the xpmap_ptom()
    mhitch> macro converts the VM physical address to a "machine"
    mhitch> physical address, which is 64 bits and as best I can tell,
    mhitch> is the real hardware physical address, which doesn't fit in
    mhitch> 32 bits when it's above 4GB.  

I'm not so sure that Xen exports all 64bits to the VM, since
xpmap_ptom() returns a (paddr_t) casted entry in the array,

unsigned long *xpmap_phys_to_machine_mapping;

which is 32bits on i386, which is probably why this wraparound shim is
required. 


    mhitch> Xen handles this by passing the 64 bit address with the high
    mhitch> bits stored in the low 12 bits of ctrlreg[3].  With the
    mhitch> following change, I am able to boot with 2 cpus.  [I didn't
    mhitch> look to see if the xen code had a macro to do this like the
    mhitch> xen kernel does.]

I don't think so, and I'm not sure it's worth it, as nothing else uses
this. 

Your fix looks correct. A small comment in explanation would be great
too.

    mhitch> Index: sys/arch/xen/x86/cpu.c
    mhitch> ===================================================================

Please feel free to check this into the branch, and thanks a lot for
catching this!

I'm working to remove the global xpq lock on the per-cpu mmu queues, so
perhaps if you could try it once I check those changes in, I'd be keen
to know if your panic recurs.

Cheers!
-- 
Cherry