Re: Help with low raid5 performance

[Stephen Borrill <netbsd%precedence.co.uk@localhost>]

On Mon, 10 Jan 2011, Greg Oster wrote:
[snip]
> > > If you re-align that partition to be 128 blocks from the start of
> > > the RAID set, it should perform significantly better.
> >
> > OK, I get this, but I'm having problems complying with the stripesize
> > heuristic in:
> >
> > http://mail-index.netbsd.org/current-users/2008/08/29/msg004215.html
> >
> > Namely, (($stripewidth / 2) * ($disks - 1)) <= MAXPHYS
> >
> > I'm using a 4-disk RAID 5 with the partition starting at 45416448
> > (which is very well aligned (on a 65536 boundary!).
> >
> > What's the best stripe size to use here? 64k/3*2=42.7k, so use the
> > next lowest 'nice' number, i.e. 32k?
>
> The problem is that with a 4-disk RAID 5 set you have (effectively) 3
> data disks.  As you can't get a nice power-of-two number to divide
> evenly by 3, so it doesn't matter too much what you pick -- performance
> is going to be greatly hindered by the small-write problem (as for a
> MAXPHYS write, the best you can do is one full-stripe write, and one
> partial-stripe write.)
>
> I'd try both:
>
> 32 1 1 5
>
> and
>
> 16 1 1 5
>
> and see which one gives you better peformance with the filesystem you
> put on it....  You might even try '8 1 1 5' or '64 1 1 5' (yes, larger
> than MAXPHYS, but I think it'll work) just to see how those perform
> too.

I was using 42 1 1 5 (based on an automated calculation in a script) which 
gave:
      -------Sequential Output-------- ---Sequential Input-- --Random--
      -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
   MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
 2048  2757  2.9  2709  1.2  1910  0.9 51110 68.0 122426 29.0 145.7  1.6


Changing to 32 1 1 5 gave:
      -------Sequential Output-------- ---Sequential Input-- --Random--
      -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
   MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
 2048  9137  9.6  7440  3.3  3045  1.3 61801 82.3 149558 37.1 228.2  2.4

These were formatted with: newfs -O 2 -f 4096 -b 32768

I'll add other results later (array is 5.5TB in size at the moment, so 
takes around 15 hours to build - I'll shrink it for further tests).

By comparison a RAID-1 gives:
      -------Sequential Output-------- ---Sequential Input-- --Random--
      -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
   MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
 2048 74416 75.3 109501 30.6 20875  8.6 64641 86.4 99279 26.9 297.7  2.9

--
Stephen