CVS commit: src/sys/dev/raidframe

["Thor Lancelot Simon" <tls%netbsd.org@localhost>]

Module Name:    src
Committed By:   tls
Date:           Sat Oct 24 00:20:24 UTC 2026

Modified Files:
        src/sys/dev/raidframe: rf_alloclist.c rf_alloclist.h rf_callback.c
            rf_dagutils.c rf_dagutils.h rf_diskqueue.c rf_driver.c rf_map.c
            rf_mcpair.c rf_netbsdkintf.c rf_psstatus.c rf_raid.h
            rf_reconbuffer.c rf_reconbuffer.h rf_reconstruct.c rf_reconstruct.h
            rf_reconutil.c rf_revent.c rf_stripelocks.c

Log Message:
Be more conservative with pool and freelist sizes and address potential deadlocks discovered while working on PR kern/60396.

Note that with this change we track the total amount of memory allocated to
both reserved buffers and pools on a per-array basis, and constrain
"openings" to keep it under control on small-memory machines.

1) Per suggestion from oster@, rename the "emergency" freelists that are now
   used before dynamic allocation to "reserved" freelists.

2) Eliminate static pool sizes and compute dynamically instead.  For now,
   most are larger to avoid potential deadlocks discussed below.  With
   some restructuring of the I/O dispatch model they can probably get smaller.

3) Address an M_WAITOK allocation with PSS_MUTEX held in "forced"
   reconstruction.  It appears this could deadlock rf_RaidIOThread
   and the recon thread.  Added another small set of emergency
   buffers to cover this case.

4) Reorganize rf_Configure to initialize pools _after_ disk,
   queue, and openings are determined, so each array's pools can
   be sized according to the actual array geometry.  Made all
   sizes dynamic; but see below as to the actual constraint on
   minimum pool sizes.

5) Address a potential deadlock between dispatch
   and completion for the I/O and Engine threads per array if any
   pool has less than "openings" minimum elements.  This is because
   am outbound dispatch can block in pool_get(..., PR_WAITOK) while
   completed I/Os accumulate in raidPtr->iodone or term_nq.

   The options are to either restructure these as two threads per
   (dispatch, completion) or to grit our teeth and accept that we
   must have "openings" minimum elements in each pool and on small
   memory machines "openings" will thus be constrained.  Elsewise,
   some pools would need as little as a single page of memory.

   Went for the "grit our teeth" option; a shame, but tearing the
   threads apart feels like too much for now.

6) Compute bytes per opening (which can be quite large due to
   emergency buffers, remembering array geometries are possible
   where a stripe is much smaller than MAXPHYS) and use that to
   constrain "openings" so no array allocates more than 1% of
   physical memory.  Comments in the reorganized rf_Configure
   explain how it's done and there are some verbose and debug
   printfs to let the user know what happened.  With a wide array
   on a low-memory machine, the bottom line is that you may be
   all the way back down to openings = 6; but on big machines with
   narrow arrays we can do much better.

7) Each time we configure an array, grow the global
   rf_alloclist_pool to ensure it has enough space according
   to array geometry and openings, so we do not block on this
   pool either.


To generate a diff of this commit:
cvs rdiff -u -r1.29 -r1.30 src/sys/dev/raidframe/rf_alloclist.c \
    src/sys/dev/raidframe/rf_revent.c
cvs rdiff -u -r1.6 -r1.7 src/sys/dev/raidframe/rf_alloclist.h
cvs rdiff -u -r1.25 -r1.26 src/sys/dev/raidframe/rf_callback.c \
    src/sys/dev/raidframe/rf_mcpair.c
cvs rdiff -u -r1.59 -r1.60 src/sys/dev/raidframe/rf_dagutils.c
cvs rdiff -u -r1.22 -r1.23 src/sys/dev/raidframe/rf_dagutils.h
cvs rdiff -u -r1.66 -r1.67 src/sys/dev/raidframe/rf_diskqueue.c
cvs rdiff -u -r1.145 -r1.146 src/sys/dev/raidframe/rf_driver.c
cvs rdiff -u -r1.52 -r1.53 src/sys/dev/raidframe/rf_map.c
cvs rdiff -u -r1.421 -r1.422 src/sys/dev/raidframe/rf_netbsdkintf.c
cvs rdiff -u -r1.39 -r1.40 src/sys/dev/raidframe/rf_psstatus.c
cvs rdiff -u -r1.54 -r1.55 src/sys/dev/raidframe/rf_raid.h
cvs rdiff -u -r1.27 -r1.28 src/sys/dev/raidframe/rf_reconbuffer.c
cvs rdiff -u -r1.8 -r1.9 src/sys/dev/raidframe/rf_reconbuffer.h
cvs rdiff -u -r1.130 -r1.131 src/sys/dev/raidframe/rf_reconstruct.c
cvs rdiff -u -r1.32 -r1.33 src/sys/dev/raidframe/rf_reconstruct.h
cvs rdiff -u -r1.38 -r1.39 src/sys/dev/raidframe/rf_reconutil.c
cvs rdiff -u -r1.35 -r1.36 src/sys/dev/raidframe/rf_stripelocks.c

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.