Re: Patch: significanly reduce TLB shootdowns in x86 pmap

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

On Sun, Jan 11, 2009 at 02:29:47PM +0000, Andrew Doran wrote:
> The below patch does a few different things.
> 
> [...]
> - It eliminates the APTE space which is expensive because its use causes
>   many global TLB shootdows. The APTE space is mainly used during exit()
>   when we are destroying the caller's pmap. Instead of using the APTE space,
>   it temporarily switches the target pmap onto the CPU. During exit, the
>   APTE space is used when we switch to the kernel's pmap in order to tear
>   down the caller's pmap. However due to lazy pmap switching the dying pmap
>   is ~always still installed on the CPU, so with the patch this means no
>   pmap switch or shootdowns are required during exit().
> 
> [...]
> - I have only enabled the APTE trick for the native i386 and amd64 ports.
>   I do not know if it can easily be done for xen; xen/amd64 looks tricky.
>   Manuel, do you have time to take a look at some point if I commit it
>   as-is?

In the way we use xen/amd64, the kernel runs in ring 3, just like user
processes; and the kernel uses its own pmap (when a user->kernel switch
occurs, the hypervisor switches the page table before calling
the domain's kernel, and switches back when the kernel returns
to userland). Because of this we of course can't have kernel mappings
in user pmaps, and access to user PTE always occurs though the APTE.
So we can't avoid the APTE for xen/amd64.

The hypervisor offers other interfaces to page table than the one we're
using right now. It seems one of them allows to run the kernel in
ring0, and this may brings us closer to how a native pmap works. But
I've not looked at this more than seeing options about this.
Also, it's possible it's a compile-time option for the hypervisor, so
using it may make us incompatible with the xen provided by some linux
distros.

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference
--