Re: Help with low raid5 performance

[Peter Kerwien <peter%kerwien.homeip.net@localhost>]

Any comments on the following:

Same disks, HW but different OS and filesystem: Linux Mint 10 (amd64)
with mdadm created raid-5 and ext4. Chunk size = 32KB, block size = 4KB,
stride size = 8 blocks = 32KB.

# dd if=/mnt/vol1/pub/foo.mkv of=/dev/null bs=100M count=100
100+0 records in
100+0 records out
10485760000 bytes (10 GB) copied, 100.641 s, 104 MB/s

# dd if=/dev/zero of=/mnt/vol1/pub/zero bs=100M count=100
100+0 records in
100+0 records out
10485760000 bytes (10 GB) copied, 102.646 s, 102 MB/s

Note the *slightly* higher write speed. Comments?

/Peter

On 11/11/2010 12:08 AM, Chuck Swiger wrote:
> On Nov 10, 2010, at 11:22 AM, Peter Kerwien wrote:
>> [ ... ]
>> # dd if=/dev/zero of=/mnt/vol1/pub/zero bs=100m count=1k
>> 1024+0 records in
>> 1024+0 records out
>> 107374182400 bytes transferred in 4056.655 secs (26468650 bytes/sec)
>> => 25MB/s
>>
>> I still not understand why I cannot reach higher write performance. I
>> saw that the [system] process consumed like 10% (have a C2D @ 2.4GHz)
>> CPU. Is the write speed limited due to CPU bound? Is there a limit how
>> much CPU the process that calculates the xor checksums is allowed to use?
> No, your CPU is sufficiently fast to keep up with the XOR calculations, and 
> very probably is not the bottleneck.
>
> One problem with a 3-disk RAID-5 setup is that writing 100MB of data to a 
> bare drive will end up sending at least 150MB across your three disks.  The 
> overhead of parity goes down as you add more drives, up until you reach a 
> saturation point for the chipset (ie, limitation on the bandwidth and # of 
> SATA available, or PCIe channels if you have RAID hardware).  However, that 
> is a best-case situation, where you are writing blocks at least as big as the 
> RAID stripe size; if you are doing smaller writes, you have to do a 
> read/recalculate XOR/write cycle, which will be very slow.
>
> It's normal for a RAID-5 volume to exhibit performance which is a factor of 
> ten worse than a bare drive when doing lots of very small writes (ie, some 
> database access patterns, revision control system changes from CVS/SVN/etc).  
> You can improve this somewhat by using RAID-50 instead of just RAID-5, and/or 
> by using a real hardware RAID controller with plenty of onboard cache memory 
> and battery backup.
>
> If you value performance more than cost, RAID-1 or RAID-10 gives much better 
> performance than RAID-5.
>
> Regards,