NetBSD-Users archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: NetBSD Raid5, slow write speeds, using big disks?!

On 6/30/26 11:10 PM, Robert Elz wrote

That's pointless as you are just going to use those filesystems again
(I presume) in your raid0.conf file (if had unique names for the two,
you could use NAME=island_1 and NAME=island_2 - or whatever you choose,
so you're not tied to dkN names, which might be different tomorrow)
Thank you, good point! Will apply for my raid5 now also. Had used /dev/dk7 in my fstab file. 
:
   | gpt add -a 1m -b 1m -t ffs -l island raid0

   | dkctl raid0 addwedge island3 1048576 4191152128 ffs
   | newfs -O2 -a 1m -b 64k -s -1g /dev/dk9
   |
   | Where for both raid1 and raid0 i used
   | # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_0
   | 128 1 1 0/1 depeding on raid type

Do you mean raid1 and raid0 or raid1 and raid2 ?   Or all 3?

It shouldn't matter for mirrors, using a 64KB block size though.
And everything appears aligned, so it isn't surprising that:

   | 4194304000 bytes transferred in 8.255 secs (508092549 bytes/sec) CHAZAAAM!

goes fairly fast, but that's too fast I suspect, you might want to try
that with a bigger than 4GB transfer, the chances are that lots of that
might be just writing into your buffer cache (it isn't clear there whether
you're doing a write onto the bare drive, or a file in a filesystem on the
drive).   If it is the latter, just repeat that several times, with different
output file names (irrelevant what the input is).  Just to make sure it
is really sustaining that rate when you're well beyond what the kernel can
just buffer.   Your 1TB (almost, 1000GB) spaces should make a filesystem
big enough for quite a lot of those files!

   | The best i got for raid5 so far, is:
   | 4194304000 bytes transferred in 107.170 secs (39136922 bytes/sec)

That looks a bit slower than I would expect for locally connected drives
(your drives might not be quite as fast as mine, bigger ones tend to
transfer faster as they squeeze more bytes into each revolution, but not
that much slower).

   | In raid5.conf you suggest: 16 1 1 5 that would have me use these
   | commands while doing the disks and raid
   | gpt add -t raid -l raid5@wd0 -b 64k -a 32k -s 1000g wd0 ****
   | gpt add -a 32k -b 64k -t ffs -l island raid5
   | newfs -O2 -a 32k -b 64k -s -1g /dev/dk7

All looks OK to me.  Show me the gpt labels for the raid5@wdN drives
(or at least 1 of them, if they are all the same, at least all the same
block numbers, they should have different labels, but if they don't, that's
harmless for now) - just "gpt show -t wd0" (etc) should be enough.

And then the label for the raid5 (your dk7 there).

You should drop the "-a 32k" on the newfs, that doesn't do what you think
it does, I believe it probably does nothing at all - but there's an outside
chance that it might be the problem if it really is limiting I/O to being 32K
(which was once its purpose, I think it should be obsolete now - it certainly
has nothing to do with alignment).

kre



CASE CLOSED! :)
i think the 16 sectpersu and 32k align was the secret sauce. Now my small 3 drive raid5 is speedy, like 400mb/s writes! So i will skip raid10 for now, and see if adding the 4th drive to the raid5 makes difference. I had no -t option to gpt show, and remove the align for newfs... then a last mail with the final config of the whole disks... 
Writes below, were done right after eachother...

Soooo many thanks! atleast the ai was correct in that, i was not crazy :D

nas1# gpt show wd0
        start         size  index  contents
            0            1         PMBR (active)
            1            1         Pri GPT header
            2           32         Pri GPT table
           34        65502         Unused
        65536   2097152000      1  GPT part - NetBSD RAIDFrame component
   2097217536  13530835599         Unused
  15628053135           32         Sec GPT table
  15628053167            1         Sec GPT header


nas1# gpt create -Af raid5
nas1# gpt add -a 32k -b 64k -t ffs -l island raid5
/dev/rraid5d: Partition 1 added: 49f48d5a-b10e-11dc-b99b-0019d1879648 65536 4194238272 
nas1# dkctl raid5 addwedge island 65536 4194238272 ffs
dk7 created successfully.
nas1# newfs -O2 -a 32k -b 64k -s -1g /dev/dk7
/dev/rdk7: 2046943.9MB (4192141120 sectors) block size 65536, fragment size 8192 
        using 689 cylinder groups of 2970.94MB, 47535 blks, 94464 inodes.
super-block backups (for fsck_ffs -b #) at:
256, 6084736, 12169216, 18253696, 24338176, 30422656, 36507136, 42591616, 48676096, 54760576, 60845056, 66929536, 73014016, 79098496, 85182976, 91267456, 97351936, 103436416, 109520896, 115605376, 121689856, 127774336, 133858816, 139943296, 146027776, 152112256, 158196736, 
.......................................................................................................................................................................................................................................................................................
nas1# mount /dev/dk7 /mnt/island
nas1# dd if=/dev/zero bs=1024k count=1000 of=/mnt/island/test.txt
1048576000 bytes transferred in 1.805 secs (580928531 bytes/sec)
nas1# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test.txt
4194304000 bytes transferred in 8.762 secs (478692535 bytes/sec)
nas1# dd if=/dev/zero bs=64k count=100000 of=/mnt/island/test.txt
6553600000 bytes transferred in 14.928 secs (439013933 bytes/sec)
nas1# dd if=/dev/zero bs=64k count=100000 of=/mnt/island/test1.txt
6553600000 bytes transferred in 15.075 secs (434733001 bytes/sec)
nas1# dd if=/dev/zero bs=64k count=100000 of=/mnt/island/test2.txt
6553600000 bytes transferred in 14.967 secs (437869980 bytes/sec)
nas1# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test3.txt
4194304000 bytes transferred in 9.063 secs (462794218 bytes/sec)
nas1# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test4.txt
4194304000 bytes transferred in 9.267 secs (452606453 bytes/sec)
nas1# dd if=/dev/zero bs=8196k count=1000 of=/mnt/island/test5.txt
8392704000 bytes transferred in 19.717 secs (425658264 bytes/sec)
nas1# dd if=/dev/zero bs=10m count=1000 of=/mnt/island/test6.txt
10485760000 bytes transferred in 24.011 secs (436706509 bytes/sec)
Home | Main Index | Thread Index | Old Index