The adventure of building a 4TB raid 5 under NetBSD 5.1

[Robert Vargason <robert%archon.ifoer.tuwien.ac.at@localhost>]

Hi all,

I'm writing down this story not because there are open questions but
so that it can be googled and others may eventually profit from it.
Comments are welcome.

Got a new system with 4 2TB wd harddisks. Installed NetBSD 5.1 (had
a few troubles with this, but that's a different story) on wd0.
System up and running. Now I made a raid 5 out of the remaining 3
harddisks. Everything went fine until 'raidctl -i raid0'.
The raid wouldn't construct because one of the harddisks had
'wd1: (uncorrectable data error)' and had to be replaced. I took
it from the machine and sent it in to be replaced. The remaining
2 disks formed a raid 1 for test purposes and everthing went fine.
Then the replaced harddisk arrived, I built it in and wanted
to contruct my raid 5 again with the three harddisks wd1 wd2 wd3.
According to handbook:

Nuking the first few sektors:
install# dd if=/dev/zero of=/dev/rwd1d bs=8k count=1
install# dd if=/dev/zero of=/dev/rwd2d bs=8k count=1
install# dd if=/dev/zero of=/dev/rwd3d bs=8k count=1

Disklabel contruction:
install# fdisk -0ua /dev/rwd1d
install# fdisk -0ua /dev/rwd2d
install# fdisk -0ua /dev/rwd3d

install# disklabel -r -e -I wd1
disklabel: could not read existing label

What? After some desperate attempts (nuking more sektors,
rebooting, trying different parameters) I read a posting where
somebody had (in a different situation but with a not so
different problem) deleted the devices.

install# cd /dev
install# rm wd1*
install# ./MAKEDEV wd1

Now disklabel worked. Hurray!

install# disklabel -r -e -I wd1
install# disklabel -r -e -I wd2
install# disklabel -r -e -I wd3

/root/raid0.conf:
raid0.conf
START array
1 3 0
START disks
/dev/wd1a
/dev/wd2a
/dev/wd3a 
START spare
START layout
# sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level
128 1 1 5 
START queue
fifo 100

install# raidctl -C /root/raid0.conf raid0
install# raidctl -I 231742 raid0
install# raidctl -i raid0
install# raidctl -S raid0

I had a raid. Trying to get a filesystem on it:

But disklabel wouldn't let me. Could be a 2TB limit?

raid0: total sector size in disklabel (3519090688) != the size of raid 
(7814057984)

Read a posting from a guy called David 'Setting up an 8TB 
NetBSD file server' (Thanks for the documentation!).
So according to this I had to use gpt:

install# gpt create raid0
gpt: unable to open device 'd'

Here I was slightly stuck. After reading man pages and a few postings I
went on to compile my own gpt from the NetBSD 5.1 source so that I could
eventually insert my own debug statements (to find out which device was
really meant).

install# cd /usr/src/sbin/gpt/  
install# gcc -lprop -lutil add.c create.c destroy.c gpt.c label.c map.c 
migrate.c recover.c remove.c show.c -o gpt

install# ./gpt create raid0

This worked. Without any debugging necessary ... strange but
who am I to complain: another job well done.

install# ./gpt add -b 128 raid0

using the output:
install# dkctl raid0 addwedge dk0 128 7814057823 4.2BSD
install# ./gpt show raid0
looks good.

reboot

install# newfs -O2 -b64k -I dk0
install# mount_ffs -o async /dev/dk0 /mnt

Now it seems I have a working 4TB raid on my system. Yeah!

Thanks to all the girls and boys providing source, man pages and
postings for solving this adventure. Keep up the good work!

Cheers
Robert Vargason