Re: Changes to reduce pressure on uvm_pageqlock

[Andrew Doran <ad%netbsd.org@localhost>]

On Thu, Dec 12, 2019 at 06:30:49PM +0900, Takashi YAMAMOTO wrote:

> > I do want to add more fields.  It turns out the first two fit into the
> > unused lower bits of pg->phys_addr without too much inconvenience for
> > anybody:
> >
> > - freelist: 'cos uvm_page_lookup_freelist() turns up prominently in traces
> >
> 
> how about having physseg id instead?
> i think it isn't too expensive to get freelist and phys_addr using it.
> i guess numa things have good uses of physseg-equivalent too.

It looks doable, and still better than using the rbtree in uvm_pagefree(),
but would require changes to the allocation of physsegs for UVM_HOTPLUG.
I will go with freelist # to begin with, but it will be easy to change to
seg later with some small changes to the allocation.  I will include that in
the comments.
 
> > - bucket: which cpu->ci_package_id I am from, for L2/L3 locality or NUMA
> >
> > The third looks like it'll fit in the bottom bits of pg->offset but I need
> > to investigate what dificulties it might cause:
> >
> > - waiter count for PG_BUSY, replacing PG_WANTED, to avoid thundering herd
> >
> 
> i guess sooner or later we will need a major surgery to allow concurrent
> faults on a page. at least for easy/common cases.
> (well, it has been "sooner or later" for a decade.)

Yup a waiters count only treats the symptom.  I looked at FreeBSD's vm_page
out of interest and it seems they have turned PG_BUSY into a kind of
reader/writer lock.  Hmm.

Andrew