Re: Why is a wedge on RAID 10x slower?

[Stephen Borrill <netbsd%precedence.co.uk@localhost>]

mlelstv pointed out that because of the 64 sectPerSU RAIDframe setting 
(i.e. 128 sectors of data with 3 components, or 64k), the stripe unit is 
32k, but the whole stripe is 64k. By starting the wedge at 64 sectors,
this is 32k, but I'm attempting to do 64k I/O. Therefore the wedge should 
start at 128, not 64

# dkctl raid0 addwedge bigdata 128 15628073823 ffs
dk3 created successfully.
# dd if=/dev/zero of=/dev/rdk3 bs=1m
^C5296+0 records in
5295+0 records out
5552209920 bytes transferred in 18.395 secs (301832558 bytes/sec)

Bingo.

The final piece of the jigsaw is to ensure block and fragment sizes in 
the filesystem are suitable. I'd originally used a 4k fragment size 
(because of the disks having 4k sectors internally), but this means a 32k 
block size. By using a 64k block size (8k fragments), performance is as it 
should be:

With newfs -O2 -V2 -b 32768 -f 4096 /dev/rdk3

# dd if=/dev/zero of=/data/block bs=64k
^C19891+0 records in
19890+0 records out
1303511040 bytes transferred in 39.746 secs (32796030 bytes/sec)

With newfs -O2 -V2 -b 65536 -f 8192 /dev/rdk3
# dd if=/dev/zero of=/data/block bs=64k
^C51478+0 records in
51477+0 records out
3373596672 bytes transferred in 11.520 secs (292846933 bytes/sec)

On Wed, 30 Nov 2016, Stephen Borrill wrote:

> I've a RAIDframe RAID-5 on 3x WD40ERFX drives. There's a well-aligned gpt and 
> wedge on each:
>
> # gpt show wd0
>       start        size  index  contents
>           0           1         PMBR
>           1           1         Pri GPT header
>           2          32         Pri GPT table
>          34          30
>          64  7814037071      1  GPT part - NetBSD RAIDFrame component
>  7814037135          32         Sec GPT table
>  7814037167           1         Sec GPT header
>
> # raidctl -G raid0
> # raidctl config file for /dev/rraid0d
>
> START array
> # numRow numCol numSpare
> 1 3 0
>
> START disks
> /dev/dk0
> /dev/dk1
> /dev/dk2
>
> START layout
> # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_5
> 64 1 1 5
>
> START queue
> fifo 100
>
> Underlying write performance on the raw RAID device is great:
> # dd if=/dev/zero of=/dev/rraid0d bs=1m
> ^C19476+0 records in
> 19475+0 records out
> 20421017600 bytes transferred in 67.089 secs (304386972 bytes/sec)
>
> If I create a similarly well-aligned GPT and wedge on the RAID device, write 
> performance is horrible:
> # gpt show raid0
>        start         size  index  contents
>            0            1         PMBR
>            1            1         Pri GPT header
>            2           32         Pri GPT table
>           34           30
>           64  15628073887      1  GPT part - NetBSD FFSv1/FFSv2
>  15628073951           32         Sec GPT table
>  15628073983            1         Sec GPT header
> backup 50# dkctl raid0 listwedges
> /dev/rraid0d: 1 wedge:
> dk3: bigdata, 15628073887 blocks at 64, type: ffs
>
> backup 51# dd if=/dev/zero of=/dev/rdk3 bs=1m
> ^C388+0 records in
> 387+0 records out
> 405798912 bytes transferred in 24.320 secs (16685810 bytes/sec)
>
> I don't understand the disparity in performance.
>
> --
> Stephen