Re: pae MP on cherry-xenmp

["Michael L. Hitch" <mhitch%lightning.msu.montana.edu@localhost>]

On Wed, 24 Aug 2011, Cherry G. Mathew wrote:

> i386/pae domU, which is slightly different from an i386 domU. You can
> run this on a 64bit xen hypervisor, and unless you're running really
> va hungry (> 3G) applications, an i386 userland would happily run on
> it.

  Jeff Rizzo and I had a problem trying the pae MP kernel with an amd64 
hypervisor.  I think I have tracked down that problem, and I suspect it's 
because Jeff and I have > 4GB on our machines.

  The problem is in sys/arch/xen/x86/cpu.c where it's setting 
initctx->ctrlreg[3] to xpmap_ptom(ci->ci_pae_l3_pdirpa).  The ctrlreg[] 
array is an unsigned long type, which is 32 bits on i386.  While the code 
is careful to make sure that ci_pae_l3_pdirpa is belown 4GB on a PAE 
kernel, that only reflects the physical address for the VM.  the 
xpmap_ptom() macro converts the VM physical address to a "machine" 
physical address, which is 64 bits and as best I can tell, is the real 
hardware physical address, which doesn't fit in 32 bits when it's above 
4GB.  Xen handles this by passing the 64 bit address with the high bits 
stored in the low 12 bits of ctrlreg[3].  With the following change, I am 
able to boot with 2 cpus.  [I didn't look to see if the xen code had a 
macro to do this like the xen kernel does.]

Index: sys/arch/xen/x86/cpu.c
===================================================================
RCS file: /cvsroot/src/sys/arch/xen/x86/cpu.c,v
retrieving revision 1.56.2.8
diff -u -p -r1.56.2.8 cpu.c
--- sys/arch/xen/x86/cpu.c      26 Aug 2011 13:33:34 -0000      1.56.2.8
+++ sys/arch/xen/x86/cpu.c      30 Aug 2011 14:27:42 -0000
@@ -1023,7 +1023,7 @@ xen_init_i386_vcpuctxt(struct cpu_info *
         * per-cpu L4 PD in pmap_cpu_init_late()
         */
 #ifdef PAE
-       initctx->ctrlreg[3] = xpmap_ptom(ci->ci_pae_l3_pdirpa);
+       initctx->ctrlreg[3] = (xpmap_ptom(ci->ci_pae_l3_pdirpa) | 
(xpmap_ptom(ci->ci_pae_l3_pdirpa) >> 32));
 #else /* PAE */
        initctx->ctrlreg[3] = xpmap_ptom(pcb->pcb_cr3);
 #endif /* PAE */

  I have been able to run this for a short while, but then it crashed 
a couple of times with:

panic: kernel diagnostic assertion "gnt_entries[last_gnt_entry] == XENGNT_NO_ENTRY" 
failed: file "/home/mhitch/NetBSD-xenmp/src/sys/arch/xen/xen/xengnt.c", line 208
fatal breakpoint trap in supervisor mode
trap type 1 code 0 eip c0136da4 cs 9 eflags 246 cr2 b6701000 ilevel 6
Stopped in pid 2526.20 (spine-0.8.7g) at        netbsd:breakpoint+0x4:  popl    
%
ebp
breakpoint(c0482987,cc870a18,cb070440,c03a7bde,cd74cba8,b6701000,2,0,c1457c70,c1
551864) at netbsd:breakpoint+0x4
panic(c04bfd44,c047bd9d,c04bccd0,c04bcd00,d0,cba1fdc8,cc870a4c,c042a348,c047bd9d
,c04bcd00) at netbsd:panic+0x20c
kern_assert(c047bd9d,c04bcd00,d0,c04bccd0,1b13d000,1,cc870a6c,c042a3d4,c47b8000,
0) at netbsd:kern_assert+0x39
xengnt_get_entry(c47b8000,0,0,cba1fb04,10ebb,cba1d004,cc870acc,c01c2176,0,1b13d0
00) at netbsd:xengnt_get_entry+0x168
xengnt_grant_access(0,1b13d000,1,0,cba1fb0c,b6701000,10e5e,5d,6,0) at netbsd:xen
gnt_grant_access+0x24
xennet_alloc_rx_buffer(6,0,cc870b0c,c0222008,ffffffff,cbb2e004,4,c01c2290,c1c3d0
00,c1b49430) at netbsd:xennet_alloc_rx_buffer+0x216
xennet_rx_mbuf_free(c1c3d000,cbb1a000,76,cba1fd50,c1c62d00,c1c62d00,cc870b3c,c03
c48c2,c1c3d000,0) at netbsd:xennet_rx_mbuf_free+0xb0
m_ext_free(c1c3d000,0,0,c0424d37,c1c62d00,c1b494ec,cc870bdc,c03cabbc,c1c62d00,0)
 at netbsd:m_ext_free+0x90
 
m_ext_free(c1c62d00,0,cc870c50,c03e9991,cd97d550,bf800000,cd89c338,0,c1b494ec,10
 000) at netbsd:m_ext_free+0xf2
 soreceive(c1b49430,cc870cd0,cc870c50,0,0,cc870cc4,2,cc870c10,cc870c10,0) at 
netb
 sd:soreceive+0xbfc
 
do_sys_recvmsg(cc7ad2a0,29,cc870cac,cc870cd0,0,cc870d28,b85fb5b0,0,cc7ad2a0,cd74
 cba8) at netbsd:do_sys_recvmsg+0x160
 
sys_recvfrom(cc7ad2a0,cc870d00,cc870d28,b6700000,b6700000,cd938534,1d,29,b670100
 0,10000) at netbsd:sys_recvfrom+0x69
 
syscall(cc870d48,ba7400b3,ba70001f,1f,b85f001f,b85fb40c,ba706040,b85fb41c,bb7c85
 4c,b8400000) at netbsd:syscall+0xc4
 ds          cba10011
 es          c04b0011    copyright+0x346b1
 fs          31
 gs          3f830011
 edi         cba1fb0c
 esi         c04bfd44    copyright+0x443e4
 ebp         cc8709cc
 ebx         104
 edx         c04bfd44    copyright+0x443e4
 ecx         6
 eax         1
 eip         c0136da4    breakpoint+0x4
 cs          9
 eflags      246
 esp         cc8709cc
 ss          11
 netbsd:breakpoint+0x4:  popl    %ebp

  I haven't looked into this traceback yet.  The VM can get fairly busy 
(it's running cacti and cacti-spine with about 50 hosts, 9000+ 
datasources, and 3000+ RRDs.

Mike

--
Michael L. Hitch                        mhitch%montana.edu@localhost
Computer Consultant
Information Technology Center
Montana State University        Bozeman, MT     USA