Re: RAIDOUTSTANDING (was: high load, no bottleneck)

[Greg Oster <oster%cs.usask.ca@localhost>]

On Thu, 19 Sep 2013 20:14:33 +0200
Edgar Fuß <ef%math.uni-bonn.de@localhost> wrote:

> > options  RAIDOUTSTANDING=40 #try and enhance raid performance.
> Is there any downside to this other than memory usage?
> How much does one unit cost?

This is from the comment in src/sys/dev/raidframe/rf_netbsdkintf.c :

/*
 * Allow RAIDOUTSTANDING number of simultaneous IO's to this RAID
device.
 * Be aware that large numbers can allow the driver to consume a lot of
 * kernel memory, especially on writes, and in degraded mode reads.
 *
 * For example: with a stripe width of 64 blocks (32k) and 5 disks,
 * a single 64K write will typically require 64K for the old data,
 * 64K for the old parity, and 64K for the new parity, for a total
 * of 192K (if the parity buffer is not re-used immediately).
 * Even it if is used immediately, that's still 128K, which when
multiplied
 * by say 10 requests, is 1280K, *on top* of the 640K of incoming data.
 *
 * Now in degraded mode, for example, a 64K read on the above setup may
 * require data reconstruction, which will require *all* of the 4
remaining
 * disks to participate -- 4 * 32K/disk == 128K again.
 */

The amount of memory used is actually more than this, but the buffers
are the biggest consumer, and the easiest way to get a ball-park
estimate.... So if you have a RAID 5 set with 12 disks and 32K/disk for
the stripe width, then for a *single* degraded write of 32K you'd need:

 11*32K (for reads) + 32K (for the write) + 32K (for the parity write)

which is 416K.  If you want to allow 40 simultaneous requests, then you
need ~16MB of kernel memory.  Maybe not a big deal on a 32GB machine,
but 40 probably isn't a good default for a machine with 128MB RAM.
Also: remember that this is just one RAID set, and each additional RAID
set on the same machine could use that much memory too....
Also2: this really only matters the most in degraded operation.

I can't think, off-hand, of any downsides other than memory usage..

Later...

Greg Oster