Re: Help with low raid5 performance

[Greg Oster <oster%cs.usask.ca@localhost>]

On Mon, 15 Nov 2010 21:07:07 +0100
Peter Kerwien <peter%kerwien.homeip.net@localhost> wrote:

> 
> 
> On 11/15/2010 08:50 PM, Thor Lancelot Simon wrote:
> > On Mon, Nov 15, 2010 at 06:40:52PM +0100, Peter Kerwien wrote:
> >> Interesting! So just moving the wedge a whole stripe was the trick?
> > You really want filesystem data blocks to be stripe-aligned.  Play
> > with it a little and see how it works for you.
> >
> > Thor
> Thanks, but I don't want to play with it. I would like to understand
> what I perhaps did wrong, and how to correct it.
> 
> /Peter

Sorry that I'm chiming in a bit late at this point, after Ian
Clark already pointed out what is most likely the culprit...

In your initial config:

> START layout
> 64 1 1 5

this says that the stripe width is 64 blocks...  With 2 data blocks and
1 parity block in each stripe, that gives you a total of 128 blocks of
data in a stripe.

When you did this:
>  # gpt show raid0
...
>          64  1953546015      1  GPT part - NetBSD UFS/UFS2

You basically aligned the partition on the half-stripe which, I
believe, ends up in having a whole bunch of the filesystem aligned on
half-stripes.  E.g. every 64K write you do ends up straddling two stripes, 
causing the read-modify-write small-write penalty for every 64K write. 

If you re-align that partition to be 128 blocks from the start of the
RAID set, it should perform significantly better.

I hope this helps some, and thanks to Ian Clark for catching the issue
that I'd missed when I first glanced at your post.

Later...

Greg Oster