Re: raidframe performance question

[Paul Goyette <paul%whooppee.com@localhost>]

After posting lots of questions, and getting lots of expert help, I
finally got my new server up and running with raid!  Since I asked
so many questions, it was suggested that I recap the process so that
someone else might be able to make it work without having to ask!

So ...

Here's what I did recently to get raidframe(4) up and running on my
new server.

0. This is really important!  Read the man page for raidctl(8).  It
   is a long man page, but it is important to understand how things
   work.  And there are some actual examples towards the end, which
   will only make sense if you've read the intro material.  Don't be
   intimidated by all the detail, just RTFM before you start!

1. Use dd(1) to completely erase the drives.  This is useful to make
   sure that any existing labels/wedges/etc. get removed, and also
   makes it easier to initialize the raidset parity.  On Western
   Digital SATA-II 500GB drive, it took less than two hours (each)
   for this step.  (The operation starts off at ~110GB/sec, but will
   get slower as the heads get closer to the center of the drive
   where there are fewer sectors per cylinder.)

        dd if=/dev/zero bs=32k of=/dev/rwd2a
        dd if=/dev/zero bs=32k of=/dev/rwd3a

2. (This step might not be necessary but I wasn't taking any chances!)
   Use fdisk(8) to set up DOS/Windows partition data.  Make sure you
   use the "-A 2048" option for alignment.

        fdisk -uai -A 2048 wd2a
        fdisk -uai -A 2048 wd3a

   (If ytou are using an older version of fdisk and get the error

        fdisk: Bad argument to the -A flag.

   you can specify "-A 2048/2048" instead.)

   Set the partition 0 type to 169 (NetBSD), and let it occupy the
   entire drive.  If you're going to boot from this raidset, you
   should also make this the active partition, and update the boot
   blocks when prompted.

3. Use disklabel(8) to provide a NetBSD disklabel on each member of
   the raidset.  Create an 'e' partition that describes the whole
   disk - so it is essentially a copy of the 'd' partition.  Here's
   my partition info:

   5 partitions:
   #        size  offset  fstype [fsize bsize cpg/sgs]
    c: 976771120    2048  unused      0     0        # (Cyl.    2*- 969020)
    d: 976773168       0  unused      0     0        # (Cyl.    0 - 969020)
    e: 976771120    2048    RAID                     # (Cyl.    2*- 969020)

4: Now create a raid.conf file to describe the raidset.  Mine is a
   simple RAID-1 (mirror):

        START array
        #numrow numcol numspare
        1 2 0

        # Identify physical disks
        START disks
        /dev/wd2e
        /dev/wd3e

        # Layout is simple - 64 sectors per stripe
        START layout
        #Sect/StripeUnit StripeUnit/ParityUnit StripeUnit/ReconUnit RaidLevel
        128 1 1 1

        # No spares
        #START spare

        # Command queueing
        START queue
        fifo 100

5. Initialize the raidset using raidctl(8).  Until you do this, there
   are no raid "component labels" on the member drives, so you need to
   use the "-C" option;  if the raidset has been previously initialized,
   you should use "-c" instead (this is what /etc/rc.d/raidframe uses
   at system startup).

        raidctl -C /etc/raid1.conf raid1

6. Next, initialize the component labels with raidctl.

        raidctl -I 201105061405 raid1

   (For the serial number, I just used the current date/time; any 64-bit
   integer will work.)

7. Now use raidctl to initialize the parity of the raidset.  On my 500GB
   drives, this took about 1.5 hours.

        raidctl -i raid1

8. Now you can treat the raidset as a normal disk.  First you need to
   write your NetBSD disklabel (this label is separate from the label
   on the physical drive).  Make certain that all the offsets are
   multiples of the drive's native block size; otherwise you will have
   unaligned transfers and performance will be very bad!  (This is
   also why, in step X above, we use "-A2048" for fdisk rather than
   letting it default to 63!")

   Here is what my drive looks like:

   7 partitions:
   #        size    offset  fstype [fsize bsize cpg/sgs]
    c: 976770944         0  unused      0     0       # (Cyl.      0 - 953877*)
    d: 976770944         0  unused      0     0       # (Cyl.      0 - 953877*)
    e: 536870912         0  4.2BSD   2048 16384    0  # (Cyl.      0 - 524287)
    f: 104857600 536870912  4.2BSD   2048 16384    0  # (Cyl. 524288 - 626687)
    g: 335042432 641728512  4.2BSD   2048 16384    0  # (Cyl. 626688 - 953877*)

9. Now, just as you would for a real disk, use newfs to initialize the
   filesystems.  Then mount them and start using your new raidset!

        newfs /dev/raid1e
        mount /dev/raid1e /mnt

10.If your kernel includes the RAID_AUTOCONFIG option, you can use
   raidctl to let your system automatically reconfigure the raidset at
   every boot.  (If you don't do this, system startup will still take
   care of this, as long as your configuration file is properly named
   "raid[0-9].conf" or "raid[1-9][0-9].conf" and is located in /etc
   directory.)

        raidctl -A yes raid1

11.If you're going to boot from the raidset, you probably want to have
   an 'a' partition.  Here's the partition from my other, bootable,
   raidset:

   6 partitions:
   #        size    offset  fstype [fsize bsize cpg/sgs]
    a:  41943040         0  4.2BSD   2048 16384    0  # (Cyl.      0 -  40959)
    b:  62914560  41943040    swap                    # (Cyl.  40960 - 102399)
    c: 488395008         0  unused      0     0       # (Cyl.      0 - 476948*)
    d: 488395008         0  unused      0     0       # (Cyl.      0 - 476948*)
    e: 125829120 104857600  4.2BSD   2048 16384    0  # (Cyl. 102400 - 225279)
    f: 257708288 230686720  4.2BSD   2048 16384    0  # (Cyl. 225280 - 476948*)

12.You will also need to run installboot(8) to make the raidset bootable.
   Do this for each member drive:

        mount /dev/raid0a /mnt
        installboot /dev/wd0a /usr/mdec/bootxx_ffsv1 /mnt/boot
        installboot /dev/wd1a /usr/mdec/bootxx_ffsv1 /mnt/boot

13.Finally, you need to make sure your kernel has the RAID_AUTOCONFIG
   option, and enable the raidset as the root device:

        raidctl -A root raid0


-------------------------------------------------------------------------
| Paul Goyette     | PGP Key fingerprint:     | E-mail addresses:       |
| Customer Service | FA29 0E3B 35AF E8AE 6651 | paul at whooppee.com    |
| Network Engineer | 0786 F758 55DE 53BA 7731 | pgoyette at juniper.net |
| Kernel Developer |                          | pgoyette at netbsd.org  |
-------------------------------------------------------------------------