Re: Help with low raid5 performance

[Peter Kerwien <peter%kerwien.homeip.net@localhost>]

On 11/09/2010 06:44 PM, Juergen Hannken-Illjes wrote:
> On Tue, Nov 09, 2010 at 04:38:59PM +0100, Peter Kerwien wrote:
>> Hi,
>>
>> I'm testing a RAID-5 configuration with RAIDframe, but cannot get 
>> decent performance. I have read the following thread:
>>
>> http://mail-index.netbsd.org/netbsd-users/2010/04/09/msg006043.html
>>
>> But cannot understand what I'm doing wrong. I'm experimenting with 3 x 
>> WD 500GB RE2 drives, which give approx. 55-60MB/s when using them as 
>> separate disks (measured with iostat during a file copy to the drive). 
>> But when setting up a RAID-5 configuration with all three drives, I'm 
>> seeing something between 10-20 MB/s. Never above 20MB/s. The latest 
>> setup I have tried is the following:
>>
>> Raid configuration:
>>
>> START array
>> 1 3 0
>>
>> START disks
>> /dev/wd1e
>> /dev/wd2e
>> /dev/wd3e
>>
>> START layout
>> 64 1 1 5
>>
>> START queue
>> fifo 100
>>
>> So what I understand gives a stripe size of 64 sectors = 32kBytes. So 
>> now it is important that the partition on raid0 (my raid-5 device) is 
>> aligned with this. I have then created the following gpt:
>>
>> # gpt show raid0
>>        start        size  index  contents
>>            0           1         PMBR
>>            1           1         Pri GPT header
>>            2          32         Pri GPT table
>>           34          30
>>           64  1953546015      1  GPT part - NetBSD UFS/UFS2
>>   1953546079          32         Sec GPT table
>>   1953546111           1         Sec GPT header
>>
>> and the corresponding wedge:
>>
>> # dkctl raid0 addwedge dk0 64 1953546015 raid0
>>
>> and finally a filesystem with:
>>
>> # newfs -O2 -s -64m /dev/rdk0
>>
>> (In my case: block size = 16kB => fragment size = 2kB).
>>
>> What important thing have I missed or done wrong??
> You should try "block size == stripe size" to avoid read-modify-write.
> What throughput do you get from "newfs -b 32k ..."?
Thanks, I have now reformatted with:

# newfs -O2 -b 32k -s -64m /dev/rdk0

I'm now seeing speeds between 20-25MB/s per disk, i.e. approx 40-50MB/s
to the raid. Still much less than I expected. I was hoping for
~50-60MB/s per disk => >100MB/s to the raid.

I think I need more help.

/Peter