Port-xen archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: pae MP on cherry-xenmp

On Wed, 24 Aug 2011, Cherry G. Mathew wrote:

i386/pae domU, which is slightly different from an i386 domU. You can
run this on a 64bit xen hypervisor, and unless you're running really
va hungry (> 3G) applications, an i386 userland would happily run on

Jeff Rizzo and I had a problem trying the pae MP kernel with an amd64 hypervisor. I think I have tracked down that problem, and I suspect it's because Jeff and I have > 4GB on our machines.

The problem is in sys/arch/xen/x86/cpu.c where it's setting initctx->ctrlreg[3] to xpmap_ptom(ci->ci_pae_l3_pdirpa). The ctrlreg[] array is an unsigned long type, which is 32 bits on i386. While the code is careful to make sure that ci_pae_l3_pdirpa is belown 4GB on a PAE kernel, that only reflects the physical address for the VM. the xpmap_ptom() macro converts the VM physical address to a "machine" physical address, which is 64 bits and as best I can tell, is the real hardware physical address, which doesn't fit in 32 bits when it's above 4GB. Xen handles this by passing the 64 bit address with the high bits stored in the low 12 bits of ctrlreg[3]. With the following change, I am able to boot with 2 cpus. [I didn't look to see if the xen code had a macro to do this like the xen kernel does.]

Index: sys/arch/xen/x86/cpu.c
RCS file: /cvsroot/src/sys/arch/xen/x86/cpu.c,v
retrieving revision
diff -u -p -r1.56.2.8 cpu.c
--- sys/arch/xen/x86/cpu.c      26 Aug 2011 13:33:34 -0000
+++ sys/arch/xen/x86/cpu.c      30 Aug 2011 14:27:42 -0000
@@ -1023,7 +1023,7 @@ xen_init_i386_vcpuctxt(struct cpu_info *
         * per-cpu L4 PD in pmap_cpu_init_late()
 #ifdef PAE
-       initctx->ctrlreg[3] = xpmap_ptom(ci->ci_pae_l3_pdirpa);
+       initctx->ctrlreg[3] = (xpmap_ptom(ci->ci_pae_l3_pdirpa) | 
(xpmap_ptom(ci->ci_pae_l3_pdirpa) >> 32));
 #else /* PAE */
        initctx->ctrlreg[3] = xpmap_ptom(pcb->pcb_cr3);
 #endif /* PAE */

I have been able to run this for a short while, but then it crashed a couple of times with:

panic: kernel diagnostic assertion "gnt_entries[last_gnt_entry] == XENGNT_NO_ENTRY" 
failed: file "/home/mhitch/NetBSD-xenmp/src/sys/arch/xen/xen/xengnt.c", line 208
fatal breakpoint trap in supervisor mode
trap type 1 code 0 eip c0136da4 cs 9 eflags 246 cr2 b6701000 ilevel 6
Stopped in pid 2526.20 (spine-0.8.7g) at        netbsd:breakpoint+0x4:  popl    
551864) at netbsd:breakpoint+0x4
,c04bcd00) at netbsd:panic+0x20c
0) at netbsd:kern_assert+0x39
00) at netbsd:xengnt_get_entry+0x168
xengnt_grant_access(0,1b13d000,1,0,cba1fb0c,b6701000,10e5e,5d,6,0) at netbsd:xen
00,c1b49430) at netbsd:xennet_alloc_rx_buffer+0x216
c48c2,c1c3d000,0) at netbsd:xennet_rx_mbuf_free+0xb0
 at netbsd:m_ext_free+0x90
 000) at netbsd:m_ext_free+0xf2
 soreceive(c1b49430,cc870cd0,cc870c50,0,0,cc870cc4,2,cc870c10,cc870c10,0) at 
 cba8) at netbsd:do_sys_recvmsg+0x160
 0,10000) at netbsd:sys_recvfrom+0x69
 4c,b8400000) at netbsd:syscall+0xc4
 ds          cba10011
 es          c04b0011    copyright+0x346b1
 fs          31
 gs          3f830011
 edi         cba1fb0c
 esi         c04bfd44    copyright+0x443e4
 ebp         cc8709cc
 ebx         104
 edx         c04bfd44    copyright+0x443e4
 ecx         6
 eax         1
 eip         c0136da4    breakpoint+0x4
 cs          9
 eflags      246
 esp         cc8709cc
 ss          11
 netbsd:breakpoint+0x4:  popl    %ebp

I haven't looked into this traceback yet. The VM can get fairly busy (it's running cacti and cacti-spine with about 50 hosts, 9000+ datasources, and 3000+ RRDs.


Michael L. Hitch                        mhitch%montana.edu@localhost
Computer Consultant
Information Technology Center
Montana State University        Bozeman, MT     USA

Home | Main Index | Thread Index | Old Index