CVS commit: src/sys

["Taylor R Campbell" <riastradh%netbsd.org@localhost>]

Module Name:    src
Committed By:   riastradh
Date:           Sun Aug 23 22:08:41 UTC 2026

Modified Files:
        src/sys/arch/aarch64/aarch64: pmap.c
        src/sys/arch/aarch64/include: asan.h
        src/sys/arch/amd64/amd64: gdt.c
        src/sys/arch/arm/include: asan.h
        src/sys/arch/i386/i386: gdt.c
        src/sys/arch/m68k/m68k: pmap_68k.c pmap_motorola.c
        src/sys/arch/powerpc/oea: pmap.c
        src/sys/arch/sparc/sparc: pmap.c
        src/sys/arch/sparc64/sparc64: pmap.c
        src/sys/rump/librump/rumpkern: vm.c
        src/sys/uvm: uvm_amap.c uvm_aobj.c uvm_bio.c uvm_fault.c uvm_glue.c
            uvm_km.c uvm_loan.c uvm_object.c uvm_pdaemon.c uvm_pdaemon.h
            uvm_pglist.c uvm_vnode.c
        src/sys/uvm/pmap: pmap_segtab.c

Log Message:
uvm: Fix missed wakeups and reduce lock contention a little bit.

This addresses two problems under heavy load:

(a) uvm_wait would sometimes miss wakeups, causing various processes
    system to hang but then recover from `call wakeup(uvmexp+0x10)'
    in ddb:

    PR kern/58964: uvm: missing wakeup on uvmexp.free

(b) Contention on uvmpd_lock would lead to so much time spent at
    IPL_SOFTBIO softint context in uvm_pageout_done spinning for
    uvmpd_lock that it would trip heartbeat panics:

    PR kern/60029: panic: cpu0: softints stuck for 16 seconds

There are three intertwined parts to this, which I tried to split
into separate commits, but I eventually decided it wasn't worth the
trouble:

1. New ticket idiom for uvm_wait is needed to avoid missing a wakeup
   in the face of multiprocessing or kernel preemption:

top:    ticket = uvm_wait_prepare();
        pg = uvm_pagealloc(...);
        if (pg == NULL) {
                uvm_wait("foo", ticket);
                goto top;
        }

   Without this, a wakeup by the page daemon that arrives after
   uvm_pagealloc has failed and before we have called uvm_wait will
   be lost, so we might hang indefinitely.

   Internally, the ticket is a 64-bit generation number, so it can't
   possibly overflow, that is advanced on any wakeup(&uvmexp.free).
   For LP32 platforms, we manage it with a seqlock/Lamport-type
   algorithm, so that reading it is cheap and unlocked.

   This may also help reduce contention on uvmpd_lock under heavy
   paging load by having uvm_wait skip the lock if there has been
   concurrent paging since uvm_wait_prepare.

2. Split uvmpd_lock into three different locks for different
   purposes:

   - uvmpd_kick_lock to coordinate threads or interrupt handlers
     wanting the page daemon to work with the page daemon wanting to
     sleep (poor beast, woken by any thread or interrupt handler,
     hence IPL_VM),

   - uvmpd_waiter_lock to coordinate the page daemon thread or paging
     completion soft interrupt handler threads waiting for memory to
     be freed up (thread/softint, hence IPL_SOFTBIO), and

   - uvmpd_pool_drain_lock to coordinate the page daemon thread with
     the pool drain thread (thread-to-thread, hence IPL_NONE).

   This way, anyone trying to kick the page daemon, and the page
   daemon's internal coordination with the pool drainer, won't incur
   any contention on the same lock that uvm_pageout_done is trying to
   take on paging completion to notify threads that they can try
   allocation again.

3. Use Dekker-synchronized atomic_swap fast paths to reduce time
   spent spinning on these locks when they are under contention.

   This way, if multiple callers (such as uvm_pageout_done and a
   thread trying to allocate memory) want to wake the page daemon,
   only one of them needs to take the lock most of the time.

   And if there's a lot of pageout activity because many threads want
   to allocate memory, uvm_pageout_done need only take the lock once
   to trigger a wakeup until the threads that have been waiting have
   noticed it.

Additionally, move one call to uvm_availmem out from under any locks,
to match the other calls to it -- this should help reduce the time
anyone spends holding the locks, and thus the time everyone else
spends waiting for it.

Empirically, this seems to enable a heavily loaded bulk build to
complete when before the softbio completions would starve softclk
leading to heartbeat panics.  Even without the heartbeat panics, we
measured over 8sec running time for softint_dispatch for softbio, and
extremely heavy contention on uvmpd_lock in uvm_pageout_done and
uvm_wait.

This isn't really a solution to the contention, however -- it just
mitigates the contention.  To scale to more CPUs in parallel, we'll
probably need to devise a per-CPU or per-NUMA-cluster page daemon
system, with each parallel daemon splitting up the work in some
reasonable way that doesn't add more contention.

Note: Before, all access to uvm_pagedaemon_waiters was serialized by
uvmpd_lock.  Now, all _writes_ are serialized by uvmpd_lock, but
_reads_ are allowed unlocked.  So the writes have to be issued with
atomic_store_*, but there is no need for atomic_r/m/w_*; hence we can
increment it with

        atomic_store_relaxed(&uvm_pagedaemon_waiters,
            atomic_load_relaxed(&uvm_pagedaemon_waiters) + 1);

rather than needing the often costlier

        atomic_inc_uint(&uvm_pagedaemon_waiters);


To generate a diff of this commit:
cvs rdiff -u -r1.158 -r1.159 src/sys/arch/aarch64/aarch64/pmap.c
cvs rdiff -u -r1.19 -r1.20 src/sys/arch/aarch64/include/asan.h
cvs rdiff -u -r1.48 -r1.49 src/sys/arch/amd64/amd64/gdt.c
cvs rdiff -u -r1.8 -r1.9 src/sys/arch/arm/include/asan.h
cvs rdiff -u -r1.74 -r1.75 src/sys/arch/i386/i386/gdt.c
cvs rdiff -u -r1.78 -r1.79 src/sys/arch/m68k/m68k/pmap_68k.c
cvs rdiff -u -r1.110 -r1.111 src/sys/arch/m68k/m68k/pmap_motorola.c
cvs rdiff -u -r1.124 -r1.125 src/sys/arch/powerpc/oea/pmap.c
cvs rdiff -u -r1.379 -r1.380 src/sys/arch/sparc/sparc/pmap.c
cvs rdiff -u -r1.320 -r1.321 src/sys/arch/sparc64/sparc64/pmap.c
cvs rdiff -u -r1.199 -r1.200 src/sys/rump/librump/rumpkern/vm.c
cvs rdiff -u -r1.129 -r1.130 src/sys/uvm/uvm_amap.c
cvs rdiff -u -r1.157 -r1.158 src/sys/uvm/uvm_aobj.c
cvs rdiff -u -r1.128 -r1.129 src/sys/uvm/uvm_bio.c
cvs rdiff -u -r1.239 -r1.240 src/sys/uvm/uvm_fault.c
cvs rdiff -u -r1.183 -r1.184 src/sys/uvm/uvm_glue.c
cvs rdiff -u -r1.168 -r1.169 src/sys/uvm/uvm_km.c
cvs rdiff -u -r1.104 -r1.105 src/sys/uvm/uvm_loan.c
cvs rdiff -u -r1.25 -r1.26 src/sys/uvm/uvm_object.c
cvs rdiff -u -r1.140 -r1.141 src/sys/uvm/uvm_pdaemon.c
cvs rdiff -u -r1.20 -r1.21 src/sys/uvm/uvm_pdaemon.h
cvs rdiff -u -r1.92 -r1.93 src/sys/uvm/uvm_pglist.c
cvs rdiff -u -r1.122 -r1.123 src/sys/uvm/uvm_vnode.c
cvs rdiff -u -r1.37 -r1.38 src/sys/uvm/pmap/pmap_segtab.c

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.