Re: problem with OEA pmap_procwr()

[Jason Thorpe <thorpej%shagadelic.org@localhost>]

On May 12, 2006, at 3:05 PM, Nathan J. Williams wrote:

> It's not powerpc-general, since a ibm440 evbppc system does much
> better. The symptoms generally point to an icache synchronization
> issue.

Interesting.  Since when does NetBSD run on the 440?

> GDB sets a breakpoint with ptrace(PT_WRITE_D) or ptrace(PT_IO), which
> invokes process_domem(). In turn, that does its data move with
> uvm_io(), which maps the relevant section of the target process into
> the kernel's address space, writes to it, and then unmaps
> it. process_domem() then calls pmap_procwr() to do any writeback/cache
> flushing necessary, passing the virtual address and length. The OEA
> pmap looks up that VA in the target process's map, but finds that it's
> not there - gdb is writing to the page before it's been executed by
> the process. So it punts, the icache is left with the old contents,
> and the breakpoint isn't hit.

If the process hasn't executed it before, then it shouldn't be in the  
I$.  Ah, right, but we have a physically indexed/tagged cache on OEA,  
and since there is another copy of ls that has run, we have a live  
cache line because of that.  Cute.

> The problem can be fixed in a pretty clunky way by calling:
>
>    uvm_fault(&vm->vm_map, trunc_page(addr), VM_PROT_EXECUTE);
>
> before pmap_procwr() in process_domem(), thus ensuring that the page
> is present in the target pmap, but this seems like the wrong layer; I
> think pmap_procwr() should do something different to get at the
> underlying physical page even if it's not currently in the PTEGs.
>
> Suggestions?

I would probably do something like this:

- Add an iterator function:

        uvm_map_foreach_page(map, start, end, callback_fn);

  that walks the VM map and invokes the callback for each page that  
is resident in the
  specified range.  Callback:

        void callback_fn(pg, pgva, v_context);

- In your callback, you can then flush the necessary cache lines for  
that page, and get invoked for as many pages as necessary.

-- thorpej