CVS commit: src/sys/dev/raidframe

["Thor Lancelot Simon" <tls%netbsd.org@localhost>]

Module Name:    src
Committed By:   tls
Date:           Sun Oct 18 01:39:05 UTC 2026

Modified Files:
        src/sys/dev/raidframe: rf_dagutils.c rf_diskqueue.c rf_diskqueue.h
            rf_driver.c rf_map.c rf_netbsdkintf.c rf_paritymap.c rf_paritymap.h
            rf_raid.h rf_states.c

Log Message:
Eliminate very low concurrency limit in RAIDframe, improving write throughput as much as 25%.

RAIDframe maintains a per-set freelist of "emergency buffers" to ensure it can perform I/O
under heavy memory pressure.  The freelist size was 10, and each I/O request can require
up to 2 buffers in some recovery conditions, so raidPtr->openings was limited to RAIDOUTSTANDING,
which was set to 6 (this was a bug; it should have been 5).

There is no need at all for this limit for RAID 0 or RAID 1 sets; they do not buffer I/O in
the relevant way.  Eliminate the allocation entirely in this case; openings is set as the
configured queue length for the RAID set.

For parity RAID sets, we dynamically size two freelists of pre-allocated buffers and the
openings value so that we consume no more than 1% of system RAM per set:
        openings = MIN(maxQueueDepth, MAX(RAIDOUTSTANDING, (1% of physmem), bytes_per_opening))
        stripe buffers: 2 * openings
        I/O buffers: openings * max_stripes_per_bp * numCol

        RAIDOUTSTANDING (now used as a floor) remains 6.

Now we use the pre-allocated buffers first, resorting to malloc() only if we run out.

This can use about 2.8MB of additional memory on very small systems with parity RAID.  It saves
about 1.6MB if they are using mirroring or striping, which is more likely, e.g. a router with
mirrored root flash.  It can use a great deal more memory on systems that happen to already have
a great deal more memory - up to 1% of system RAM per parity RAID set, of course.

Double the RAID metadata pool high-water marks to accomodate additional concurrency; widen
the pending write counter in the in-core parity map (consuming a whopping 4KB) to accommodate
queue lengths > 127.  Fix a couple of buglets.

The punchline: sequential 128K write performance goes from 750MB/sec to 950MB/sec on a RAID1
set of NVMe SSDs (a 25% improvement!); random write performance on a RAID5 set of 4 virtio
disks (backed by a single SSD; host is a Mac Studio M3 Ultra) on an aarch64 QEMU VM goes from
2800tps to 3300tps and throughput increases from 25MB/sec to 31MB/sec (workload is untarring
src.tar.gz from tmpfs to FFS on the RAID1 set).


To generate a diff of this commit:
cvs rdiff -u -r1.58 -r1.59 src/sys/dev/raidframe/rf_dagutils.c
cvs rdiff -u -r1.64 -r1.65 src/sys/dev/raidframe/rf_diskqueue.c
cvs rdiff -u -r1.30 -r1.31 src/sys/dev/raidframe/rf_diskqueue.h
cvs rdiff -u -r1.144 -r1.145 src/sys/dev/raidframe/rf_driver.c
cvs rdiff -u -r1.51 -r1.52 src/sys/dev/raidframe/rf_map.c
cvs rdiff -u -r1.418 -r1.419 src/sys/dev/raidframe/rf_netbsdkintf.c
cvs rdiff -u -r1.11 -r1.12 src/sys/dev/raidframe/rf_paritymap.c
cvs rdiff -u -r1.2 -r1.3 src/sys/dev/raidframe/rf_paritymap.h
cvs rdiff -u -r1.53 -r1.54 src/sys/dev/raidframe/rf_raid.h \
    src/sys/dev/raidframe/rf_states.c

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.