NetBSD-Users archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: NetBSD Raid5, slow write speeds, using big disks?! - Resolved
On 7/1/26 1:07 AM, smurfd wrote:
On 6/30/26 11:10 PM, Robert Elz wrote
That's pointless as you are just going to use those filesystems again
(I presume) in your raid0.conf file (if had unique names for the two,
you could use NAME=island_1 and NAME=island_2 - or whatever you choose,
so you're not tied to dkN names, which might be different tomorrow)
Thank you, good point! Will apply for my raid5 now also. Had used
/dev/dk7 in my fstab file.
:
| gpt add -a 1m -b 1m -t ffs -l island raid0
| dkctl raid0 addwedge island3 1048576 4191152128 ffs
| newfs -O2 -a 1m -b 64k -s -1g /dev/dk9
|
| Where for both raid1 and raid0 i used
| # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_0
| 128 1 1 0/1 depeding on raid type
Do you mean raid1 and raid0 or raid1 and raid2 ? Or all 3?
It shouldn't matter for mirrors, using a 64KB block size though.
And everything appears aligned, so it isn't surprising that:
| 4194304000 bytes transferred in 8.255 secs (508092549 bytes/sec)
CHAZAAAM!
goes fairly fast, but that's too fast I suspect, you might want to try
that with a bigger than 4GB transfer, the chances are that lots of that
might be just writing into your buffer cache (it isn't clear there
whether
you're doing a write onto the bare drive, or a file in a filesystem
on the
drive). If it is the latter, just repeat that several times, with
different
output file names (irrelevant what the input is). Just to make sure it
is really sustaining that rate when you're well beyond what the
kernel can
just buffer. Your 1TB (almost, 1000GB) spaces should make a filesystem
big enough for quite a lot of those files!
| The best i got for raid5 so far, is:
| 4194304000 bytes transferred in 107.170 secs (39136922 bytes/sec)
That looks a bit slower than I would expect for locally connected drives
(your drives might not be quite as fast as mine, bigger ones tend to
transfer faster as they squeeze more bytes into each revolution, but not
that much slower).
| In raid5.conf you suggest: 16 1 1 5 that would have me use these
| commands while doing the disks and raid
| gpt add -t raid -l raid5@wd0 -b 64k -a 32k -s 1000g wd0 ****
| gpt add -a 32k -b 64k -t ffs -l island raid5
| newfs -O2 -a 32k -b 64k -s -1g /dev/dk7
All looks OK to me. Show me the gpt labels for the raid5@wdN drives
(or at least 1 of them, if they are all the same, at least all the same
block numbers, they should have different labels, but if they don't,
that's
harmless for now) - just "gpt show -t wd0" (etc) should be enough.
And then the label for the raid5 (your dk7 there).
You should drop the "-a 32k" on the newfs, that doesn't do what you
think
it does, I believe it probably does nothing at all - but there's an
outside
chance that it might be the problem if it really is limiting I/O to
being 32K
(which was once its purpose, I think it should be obsolete now - it
certainly
has nothing to do with alignment).
kre
CASE CLOSED! :)
i think the 16 sectpersu and 32k align was the secret sauce. Now my
small 3 drive raid5 is speedy, like 400mb/s writes!
So i will skip raid10 for now, and see if adding the 4th drive to the
raid5 makes difference. I had no -t option to gpt show, and remove the
align for newfs... then a last mail with the final config of the whole
disks...
Writes below, were done right after eachother...
Soooo many thanks! atleast the ai was correct in that, i was not crazy :D
nas1# gpt show wd0
start size index contents
0 1 PMBR (active)
1 1 Pri GPT header
2 32 Pri GPT table
34 65502 Unused
65536 2097152000 1 GPT part - NetBSD RAIDFrame component
2097217536 13530835599 Unused
15628053135 32 Sec GPT table
15628053167 1 Sec GPT header
nas1# gpt create -Af raid5
nas1# gpt add -a 32k -b 64k -t ffs -l island raid5
/dev/rraid5d: Partition 1 added: 49f48d5a-b10e-11dc-b99b-0019d1879648
65536 4194238272
nas1# dkctl raid5 addwedge island 65536 4194238272 ffs
dk7 created successfully.
nas1# newfs -O2 -a 32k -b 64k -s -1g /dev/dk7
/dev/rdk7: 2046943.9MB (4192141120 sectors) block size 65536, fragment
size 8192
using 689 cylinder groups of 2970.94MB, 47535 blks, 94464 inodes.
super-block backups (for fsck_ffs -b #) at:
256, 6084736, 12169216, 18253696, 24338176, 30422656, 36507136,
42591616, 48676096, 54760576, 60845056, 66929536, 73014016, 79098496,
85182976, 91267456, 97351936, 103436416, 109520896, 115605376,
121689856, 127774336, 133858816, 139943296, 146027776, 152112256,
158196736,
.......................................................................................................................................................................................................................................................................................
nas1# mount /dev/dk7 /mnt/island
nas1# dd if=/dev/zero bs=1024k count=1000 of=/mnt/island/test.txt
1048576000 bytes transferred in 1.805 secs (580928531 bytes/sec)
nas1# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test.txt
4194304000 bytes transferred in 8.762 secs (478692535 bytes/sec)
nas1# dd if=/dev/zero bs=64k count=100000 of=/mnt/island/test.txt
6553600000 bytes transferred in 14.928 secs (439013933 bytes/sec)
nas1# dd if=/dev/zero bs=64k count=100000 of=/mnt/island/test1.txt
6553600000 bytes transferred in 15.075 secs (434733001 bytes/sec)
nas1# dd if=/dev/zero bs=64k count=100000 of=/mnt/island/test2.txt
6553600000 bytes transferred in 14.967 secs (437869980 bytes/sec)
nas1# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test3.txt
4194304000 bytes transferred in 9.063 secs (462794218 bytes/sec)
nas1# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test4.txt
4194304000 bytes transferred in 9.267 secs (452606453 bytes/sec)
nas1# dd if=/dev/zero bs=8196k count=1000 of=/mnt/island/test5.txt
8392704000 bytes transferred in 19.717 secs (425658264 bytes/sec)
nas1# dd if=/dev/zero bs=10m count=1000 of=/mnt/island/test6.txt
10485760000 bytes transferred in 24.011 secs (436706509 bytes/sec)
This problem is now resolved!!
If you have arrived here, with a similar problem (should be applicable
for softraid for *BSD).
This is what, after alot of help in this mailthread, i did.
The secret i believe was sectPerSU=16 and a=32k. In the man page for
raidctl, it states that
for raid5, use sectPerSU=16, but me thinking that this was outdated info
or for smaller disks
was incorrect. Another thing that i believe bite me, was that the gpt
destroy and raidctl -u
commands did not really clear the previous raid config. Im not sure
about this, but, its a
feeling i have... so i started doing a dd zero to /dev/rraid5 before
recreating a raid.
This is the /etc/raid5.conf file i use. While trying i had it as
/etc/r5.conf so if i needed
to reboot, it would not start...
/etc/raid5.conf
START array
# numRow numCol numSpare
1 3 1
START disks
/dev/dk3
/dev/dk4
/dev/dk5
START spare
/dev/dk6
START layout
# sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_5
16 1 1 5
START queue
fifo 127
These are the commands i ran:
---
# destroy possible previous raid5 config, also important while testing
gpt destroy raid5
raidctl -u raid5
dd if=/dev/zero of=/dev/rraid5 bs=1024k count=1 # i saw this was needed
to really clear its parameters
gpt destroy wd0 **** # **** means for each of the 4 disks, wd0,wd1,wd2,wd3
gpt create -Af wd0 ***
gpt add -t raid -l raid5@wd0 -a 32k -b 64k wd0 **** # start with small
disks until you get the speed (-s 1000g), then remove the size
raidctl -C /etc/raid5.conf raid5
raidctl -I 13371337 raid5
raidctl -iv raid5. # this takes many hours, maby between 8 and 72+
hours, depending on size
gpt create -Af raid5
gpt add -a 32k -b 64k -t ffs -l island raid5
dkctl raid5 addwedge island <last two values from above commands output> ffs
newfs -O2 -a 32k -b 64k -s -1g /dev/dk7 # reserve 1g for journal files
mount /dev/dk7 /mnt/island
---
Tried the above configs, but instead of 3 disk + 1 spare, i used 4
disks. Speeds were alot slower, so i stuck to 3 + 1 disks
4 disks
# dd if=/dev/zero bs=1024k count=1000 of=/mnt/island/test.txt
1048576000 bytes transferred in 17.498 secs (59925477 bytes/sec)
# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test.txt
4194304000 bytes transferred in 122.339 secs (34284275 bytes/sec)
3 disks
# dd if=/dev/zero bs=1024k count=1000 of=/mnt/island/test.txt
1048576000 bytes transferred in 1.695 secs (618628908 bytes/sec)
# dd if=/dev/zero bs=4096k count=1000 of=/mnt/island/test.txt
4194304000 bytes transferred in 8.912 secs (470635547 bytes/sec)
# dd if=/dev/zero bs=4096k count=3000 of=/mnt/island/test.txt
12582912000 bytes transferred in 28.801 secs (436891496 bytes/sec)
Tried the above dd commands a couple of times to verify that the speeds
were similar, which they were.
Used different out-filenames
/etc/fstab add row, once done, since /dev/dk7 might change at reboot.
Where NAME= is the same value you
use in the -l parameter for the raid.
NAME=island /mnt/island ffs rw,-m1666,log 1 1
# df -h |grep dk7
/dev/dk7 14T 27G 14T 0% /mnt/island
Tips:
Check dmesg after each command. When you create the raid with the
raidctl -C,
it might complain with fatal errors, but they are ignored. that seems to
be okey!
Raidctl -iv should take time. If instant, something is wrong!
To save time on the raidctl -iv, start with smaller disks, like 1TB.
If you get this error, if i remember correctly when you create the gpt
for raid5,
you can clear that with: dd if=/dev/zero of=/dev/rraid5 bs=1024k count=1
gpt: /dev/rraid5: map entry doesn't fit media: new start + new size <
start + size
Again, super thanks!
ps. no commentary needed! :)
Br
- References:
- Re: NetBSD Raid5, slow write speeds, using big disks?!
- Re: NetBSD Raid5, slow write speeds, using big disks?!
- Re: NetBSD Raid5, slow write speeds, using big disks?!
- Re: NetBSD Raid5, slow write speeds, using big disks?!
- Re: NetBSD Raid5, slow write speeds, using big disks?!
- Re: NetBSD Raid5, slow write speeds, using big disks?!
- Re: NetBSD Raid5, slow write speeds, using big disks?!
Home |
Main Index |
Thread Index |
Old Index