Re: Alloc userspace process to access mapped grant memory regions

[Roger Pau Monné <roger.pau%citrix.com@localhost>]

On 24/11/12 13:16, Manuel Bouyer wrote:
> On Sat, Nov 24, 2012 at 09:35:47AM +0100, Roger Pau Monné wrote:
>> Thanks for the help, I've changed the code a little bit and tried doing
>> something similar to what is done in the mmap function of xenevt:
>>
>> (inside device mmap handler):
>> xen_shm_map(nentries, domid, grefp, &va, handlep, flags);
>> pmap_extract_ma(pmap_kernel(), map->va, &ma)
>> printf("Kernel MA: %p\n", (void *)ma);
>> return x86_btop(xpmap_mtop(ma));
>>
>> This device also has an ioctl where the user-space process can pass a VA
>> (in the user space) and get the grant associated with this VA (basically
>> the passed VA is the return value of mmap(gntdev...)). I do the
>> following in the ioctl handler:
>>
>> (passed_va is the user provided VA through the ioctl)
>>
>> paddr_t u_pa, k_pa;
>> paddr_t u_ma, k_ma;
>> pmap_t user_pmap = vm_map_pmap(&user_lwp->l_proc->p_vmspace->vm_map);
>>
>> pmap_extract(pmap, passed_va, &u_pa);
>> foreach(kernel_va, list_of_mapped_grants) {
>>      /* kernel_va is map->va in the upper example */
>>      pmap_extract(pmap_kernel(), kernel_va, &k_pa);
>>      if (k_pa == u_pa) {
>>              printf("Find matching PA\n");
>>              pmap_extract_ma(pmap, passed_va, &process_ma);
>>              pmap_extract_ma(pmap_kernel(), kernel_va, &kernel_ma);
>>              printf("Kernel MA: %p - Userspace MA: %p\n",
>>                      (void *)k_ma, (void *)u_ma);
>>      }
>> }
>>
>> The output when performing the mmap is the following:
>>
>> Kernel MA: 0xb24c9000
>>
>> And the output when the user performs the ioctl to find the gref of the
>> user-space VA is:
>>
>> Find matching PA
>> Kernel MA: 0xb24c9000 - Userspace MA: 0xbda71000
>>
>> So it seems like I return the correct MA in the mmap handler, but the
>> userspace process gets a wrong MA associated with it's VA?
> 
> Are the p2m and m2p maps consistent ? I suspect they're not ...

No, they are not. I've tried making them cosistent by adding:

for (i = 0; i < map->count; i++) {
        va = map->va + (i * PAGE_SIZE);
        if (pmap_extract_ma(pmap_kernel(), va, &ma) == false)
                continue;
        xpmap_ptom_map(xpmap_mtop(ma), ma);
}

After the xen_shm_map call, but doing this doesn't seem ok, it triggers
and error in the hypervisor later on when xpq_flush_queue is called,
because we change the owner of the mfn from it's original domain to Dom0
(this mfn doesn't belong to Dom0, but to the guest):

Returning VA: 0xffffa000e565c000
pmap_extract_ma: 0xb24c9000 - pa: 0x6241000
pmap_extract: 0x6241000 - ma: 0xb24c9000
Mapping gntdev index: 0
Returning VA: 0xffffa000e565c000
pmap_extract_ma: 0xb24c9000 - pa: 0x6241000
pmap_extract: 0x6241000 - ma: 0xb24c9000
(XEN) mm.c:793:d0 pg_owner 0 l1e_owner 0, but real_pg_owner 1
(XEN) mm.c:864:d0 Error getting mfn b24c9 (pfn 6241) from L1 entry
80000000b24c9067 for l1e_owner=0, pg_owner=0
xpq_flush_queue: 1 entries (0 successful) on cpu0 (0)
cpu0 (0):
  0x00000001097aef28: 0x80000000b24c9067
 kpm_pdir[0]: 0x1086a4027
 kpm_pdir[254]: 0x108694027
panic: HYPERVISOR_mmu_update failed, ret: -16

fatal breakpoint trap in supervisor mode
trap type 1 code 0 rip ffffffff8020d915 cs e030 rflags 246 cr2
7f7ff7735008 ilevel 8 rsp ffffa000e74bb440
curlwp 0xffffa0000b7176a0 pid 635 lid 1 lowest kstack 0xffffa000e74b8000
Stopped in pid 635.1 (xenstored) at     netbsd:breakpoint+0x5:  leave
breakpoint() at netbsd:breakpoint+0x5
vpanic() at netbsd:vpanic+0x1f2
printf_nolog() at netbsd:printf_nolog
xpq_queue_machphys_update() at netbsd:xpq_queue_machphys_update
pmap_enter_ma() at netbsd:pmap_enter_ma+0x567
pmap_enter() at netbsd:pmap_enter+0x35
udv_fault() at netbsd:udv_fault+0x141
uvm_fault_internal() at netbsd:uvm_fault_internal+0x5d6
uvm_fault_wire() at netbsd:uvm_fault_wire+0x53
uvm_map_pageable() at netbsd:uvm_map_pageable+0x1c6
uvm_mmap() at netbsd:uvm_mmap+0x59c
sys_mmap() at netbsd:sys_mmap+0x20f
syscall() at netbsd:syscall+0x94
--- syscall (number 197) ---
7f7ff74d96ba:
ds          0
es          b480
fs          100
gs          6a80
rdi         0
rsi         deadbeef
rbp         ffffa000e74bb440
rbx         104
rdx         ffffffff
rcx         8
rax         1
r8          ffffffff80c3e240    cpu_info_primary
r9          1
r10         0
r11         ffffa0000b70da48
r12         ffffffff80b21298
r13         ffffa000e74bb480
r14         0
r15         1
rip         ffffffff8020d915    breakpoint+0x5
cs          e030
rflags      246
rsp         ffffa000e74bb440
ss          e02b
netbsd:breakpoint+0x5:  leave

Is there anyway to make the pfn <-> mfn map consistent without marking
the remote mfn as local?

Thanks, Roger.