Re: Fixing excessive shootdowns during FFS read (kern/53124) on x86 - emap, direct map

[Maxime Villard <max%m00nbsd.net@localhost>]

Le 02/04/2018 à 21:28, Jaromír Doleček a écrit :
> 2018-03-31 13:42 GMT+02:00 Jaromír Doleček <jaromir.dolecek%gmail.com@localhost>:
> > 2018-03-25 17:27 GMT+02:00 Joerg Sonnenberger <joerg%bec.de@localhost>:
> > > Yeah, that's what ephemeral mappings where supposed to be for. The other
> > > question is whether we can't just use the direct map for this on amd64
> > > and similar platforms?
> >
> > Right, we could/should use emap. I haven't realized emap is actually already
> > implemented. It's currently used for pipe for the loan/"direct" write.
> >
> > I don't know anything about emap thought. Are there any known issues,
> > do you reckon it's ready to be used for general I/O handling?
>
> Okay, so I've hacked to gether a patch to switch uvm_bio.c to ephemeral
> mapping:
>
> http://www.netbsd.org/~jdolecek/uvm_bio_emap.diff

-		pmap_emap_enter(va, pa, VM_PROT_READ);
+		pmap_emap_enter(va, pa, VM_PROT_READ | VM_PROT_WRITE);

Mmh no, sys_pipe wanted it read-only, we shouldn't make it writable by default.
Adding a prot argument to uvm_emap_enter would be better.

> Looking at the state of usage though, the emap is only used for disabled code
> path for sys_pipe and nowhere else. That code had several on-and-off passes
> for being enabled in 2009, and no further use since then. Doesn't give too much
> confidence.
>
> The only port actually having optimization for emap is x86. Since amd64
> is also the only one supporting direct map, we are really at liberty to pick
> either one. I'd lean towards direct map, since that doesn't require
> adding/removing any mapping in pmap_kernel() at all. From looking on the code,
> I gather direct map is quite easy to implement for other archs like
> sparc64. I'd say significantly easier than adding the necessary emap hooks
> into MD pmaps.

There is a good number of architectures where implementing a direct map is not
possible, because of KVA consumption. With a direct map we consume more than
once the physical space. If you have 4GB of ram, the direct map will consume
4GB of KVA; and in addition pmap_kernel will consume some KVA too. i386 for
example has only 4GB of ram and 4GB of KVA, so we'll just never add a direct
map there.

Direct maps are good when the architecture has much, much more KVA than it has
physical space.

I saw some low-KVA architectures have a "partial direct map", where only a
(tiny) area of the physical space is direct-mapped. There, we would have to
adapt uvm_bio to use pmap_kernel instead, which seems ugly.

As opposed to that, emaps can be implemented everywhere with no constraint on
the arch. I think they are better than the direct map for uvm_bio.