Re: Heads up: UVM changes

[Andrew Doran <ad%netbsd.org@localhost>]

On Sun, Mar 22, 2020 at 12:40:00PM -0700, Jason Thorpe wrote:

> > On Mar 22, 2020, at 12:34 PM, Andrew Doran <ad%netbsd.org@localhost> wrote:
> > 
> > This works well for me on amd64, but I haven't tried it on other machines. 
> > From code inspection I can see that some architectures need more work so
> > this is disabled on them until I can make the needed changes: hppa, mips,
> > powerpc and riscv.  It's enabled everywhere else.
> 
> Can you provide a summary of what's needed to fix these other pmap
> modules?  Or, at least, what the issue is that you're avoiding with the
> switch?

Sure, I'll try to give an outline.  Before this the pmap module could work
with the assumption that for a given page "pg", at all times there was an
exclusive lock held by the VM system on the owning object, and so there
would never be any concurrent pmap operations on "pg".

The pmap module could also assume that within a single pmap, there would
never be concurrent operations on the same set of VAs.

Both of those things have changed now.  For example you can have one process
doing pmap_enter() on a page while a different process does pmap_remove()
for the same page.  You can also have N threads in a multi-threaded process
racing to do pmap_enter() on the exactly the same page and same VA because
they faulted on it at the same time and both think it's unentered (those
threads will never try to do something conflicting though, i.e. the new
mapping will always have the same attributes).

In the pmap module there are situations that can be excluded by making
assumptions about what locks are held in the VM system, but I don't think
the middle ground between "always strongly locked" and "never locked" is a
happy place to be - so for this concurrent handling of faults I think the
pmap should be able to handle anything thrown at it.  The only assumption I
think is reasonable to make is that a page won't change identity while being
worked on in the pmap module, and won't change identity between a call to
e.g. pmap_remove() -> pmap_update().

With that in mind for each pmap it's a matter of code inspection while
thinking about the "what if" scenarios.  For example the races above, or
like what if pmap_is_modified(pg) is called right in the middle of another
thread doing pmap_remove() for the same - will the right answer be produced.

I think the approach to solving it depends on the machine and how much
effort we can put into testing / maintaining it.  For vax I've gone with a
global pmap_lock because it's a good compromise there.  Not taxing on the
machine or the programmer at all.  Obviously that wouldn't make the grade
for something modern with many CPUs.

The other two basic approaches we have to solving it are firstly per-pmap +
per-page locking like x86 or aarch64, and secondly the alpha (old x86)
approach of a global RW lock + per pmap locks (benchmarked extensively back
in 2007 - it works for sure, but is worse than a global mutex from a
performance POV).

Andrew