To answer your most important question first: Are your expectations wrong? Yes, slightly.
When you tested ZFS on FreeBSD using dd if=/dev/zero, you weren't actually testing your hard drives. You were testing your RAM and CPU.
* ZFS is a highly intelligent, Copy-on-Write filesystem. By default, it uses compression. When you feed it a stream of zeroes, ZFS compresses that data into virtually nothing and caches it in RAM (the ARC).
* The 4.03 GB/sec write speed you saw is physically impossible for four Seagate IronWolf NAS drives. Those drives max out at around 210–250 MB/s each. Even in a perfect RAID0 (no parity), the theoretical maximum physical write speed would be around 1 GB/s.
What should you expect?On a perfectly tuned 4-disk RAID5 array of modern 7TB spinning drives, you should expect sustained sequential write speeds somewhere between 300 MB/s and 500 MB/s.
Your NetBSD speeds are genuinely terrible, and there are two main culprits working together to sabotage your performance: Stripe Misalignmentand the Read-Modify-Write (RMW) penalty.
When you configure RAID5, data is striped across the disks in chunks, with one block reserved for parity. If you write a file that is smaller than the stripe size, or misaligned with the stripe, the RAID controller (or RAIDframe in this case) cannot just write the data. It has to:
1. Read the old data and the old parity from the disks.
2. Modify the data in memory.
3. Write the new data and the new parity back to the disks.
This turns one write operation into four separate I/O operations.
In your /etc/raid5.conf, you defined:
sectPerSU 32
RAIDframe calculates this in 512-byte sectors.
32 \times 512 \text{ bytes} = 16\text{ KB} per disk.
Because you have 4 disks in RAID5, 3 are used for data and 1 for parity.
Your total data stripe width is:
3 \text{ data disks} \times 16\text{ KB} = 48\text{ KB stripe width}
However, when you created your filesystem, you used:
newfs -b 65536 (which dictates a 64 KB block size).
Because your filesystem blocks (64 KB) don't match your RAID stripes (48 KB), every single write you make overlaps multiple stripes. This guarantees a massive Read-Modify-Write penalty for every single block of data. The initial 1.6 GB/s burst you saw was NetBSD writing to your system RAM (buffer cache). The 1.3 MB/s drop happened when the RAM filled up and the system desperately tried to flush those misaligned blocks to the physical disks.
To fix this, we need to ensure your filesystem block size fits neatly into your RAID stripe units.
Since you have 4 disks (3 data), it is mathematically impossible to make the total stripe width a perfect power-of-two (like 64KB) because 3 is not a power-of-two. The best workaround is to make the *individual disk stripe unit* match your filesystem block size.
Change your /etc/raid5.conf to use
128 sectors per SU:
START array
1 4 0
START disks
/dev/dk3
/dev/dk4
/dev/dk5
/dev/dk6
START layout
# sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_5
128 1 1 5
START queue
fifo 256
* Why 128? 128 \times 512 \text{ bytes} = 64\text{ KB}.
* Now, a 64 KB filesystem block from FFS will fit perfectly onto a *single* disk's chunk. RAIDframe will only need to calculate parity against that specific chunk, rather than spanning multiple chunks and fragmenting the I/O.
* I also bumped your queue size to 256 to give the disks a bit more breathing room for optimization.
You enabled WAPBL using tunefs -l 1g. WAPBL is great for crash consistency, but it works by writing small, synchronous metadata updates to the disk. On RAID5, small synchronous writes trigger the Read-Modify-Write penalty constantly.
Once you rebuild the RAID with sectPerSU 128, test the write speeds before enabling the WAPBL log. You'll likely see a massive improvement. If enabling the log slows things down again, consider placing the WAPBL log on your SSD instead of the RAID array.
You aren't crazy, and NetBSD isn't inherently 200x slower. ZFS was just tricking you with RAM speeds, and your NetBSD array was mathematically fighting against itself due to a stripe-size mismatch. And yes, the raidctl man page absolutely needs a modern "large disk" section!
Regards,
Arya