Installing 10.0_RC1 using GPT/UEFI/RaidFrame

[Havard Eidnes <he%NetBSD.org@localhost>]

Hi,

I recently had reason to install NetBSD on a new (to me) server.
This server had previously ran Linux/Debian with a software RAID
setup over two drives.

The dmesg of the server is visible at

  https://dmesgd.nycbug.org/index.cgi?do=view&id=7403

I wanted to continue to use software RAID with this host, but
also wanted to install NetBSD.  I also wanted to try out the new
features in sysinst, which allows you to configure a raidframe
setup without having to escape to the shell to do so.

The partitioning was already:

sd0 sd0: 558 GB, 241845 cyl, 3 head, 1615 sec, 512 bytes/sect x 1172123568 sector
sd0: GPT GUID: 498d796a-a7cd-4fd4-85cc-0b825e15a291
  dk0 at sd0: "....", 524288 blocks at 2048, type: msdos
  dk1 at sd0: "....", 1171595264 blocks at 526336, type: raidframe
sd1: 558 GB, 241845 cyl, 3 head, 1615 sec, 512 bytes/sect x 1172123568 sectors
sd1: GPT GUID: df5880c8-f9b8-42f2-90e9-430371502ee5
  dk2 at sd1: "....", 524288 blocks at 2048, type: msdos
  dk3 at sd1: "....", 1171595264 blocks at 526336, type: raidframe

Well, this was after I ran the installer, of course, the type for
dk1 and dk3 was different from "raidframe" when I started.  The
dk0 and dk2 drives are UEFI boot partitions, and dk1 and dk3 are
(here) available for data / raidframe components.

Using sysinst's "Utility Menu" -> "Partition a disk", I could
first change the "type" of dk1 and dk3 to "RAID", and could then
use the menu entry "Create software RAID" to create a raid set,
adding the two drives and configure it as raid1 (mirror).

Somewhat unfortunately, when that's done, the raidframe set is
initialized "synchronously" in sysinst.  In my case I had to
leave the machine sitting for about 60 minutes before the next
step could be performed.  It could be argued that the
initialization could continue in the background (it does that
anyway) -- it's only the display of the progress which is
synchronous.

When it then came time to do the installation, "raid0" was now
visible as a possible choice for installation disk (which I
picked).  The installation of NetBSD itself was largely
uneventful, and followed a familiar and well-trodden path.

It then became time to try to boot from the newly installed
system.  This was when the first surprise hit.  Sysinst had not
dealt with the UEFI boot partitions, so the Debian boot bits were
still left in dk0 and dk2, and of course it brought up Grub, and
not NetBSD.  So I had to boot up using the USB key with the UEFI
image again, escape to the shell, and do

# mount -t msdos /dev/dk0 /mnt
# cd /mnt/EFI
# mkdir boot
# rm -rf debian
# cp /usr/mdec/bootx64.efi /mnt/EFI/boot/
# cd /
# umount /mnt

and repeat for dk2.

This gets us booting NetBSD.  However, the kernel when it comes
up says

WARNING: findroot: double match for boot device (bootinfo/bootwedge:sd0 bootinfo/bootwedge:sd1)

and the kernel then proceeds to interactively ask for the root
file system partition, swap partition, and root file system type.
Obviously that is undesireable.

Thinking that the raidframe disk had a normal disk label, I
replied "raid0a", "raid0b" and defaulted the last, but that got
me errno=16 == EBUSY.  The kernel therefore asked once more.
If I insisted, and replied "raid0a", "raid0b" and default once
more, I got a kernel panic with "locking against myself". Fun.

However, what I had not noticed was that raid0 had been equipped
with a GPT partition table, so new dk-devices had been created
for the GPT partitions, and presumbly that caused the "EBUSY"
error.  Live and learn.

However, we still want to avoid having to interactively respond
to which file system to use for root.  At this point, several
paths are available, some of them not without hurdles.

1) It turns out that the raidframe set is not set to either of
   "softroot" or "forceroot".  It turns out that setting it to
   "softroot" via

   # raidctl -A softroot raid0

   is sufficient to get us booting directly into multi-user
   without console interaction.

   (Forceroot is best avoided, as it will force the root file
   system to the raid set even when you boot e.g. from a USB
   key for recovery, should the need arise.)

2) The above manual installation of NetBSD UEFI boot bits does
   not install a boot.cfg file, which could be done.  That file
   is apparently supposed to be in /EFI/NetBSD/boot.cfg in the
   UEFI boot file system.

   There is a possibility to label the wedges with a more
   user-friendly name (as I've done) via

   # gpt show raid0

   Pick up the 'index' value of the FFS partition, and label it
   and the swap partition -- in my case the FFS is '2' and swap
   is '3':

   # gpt label -l NetBSD-root -i 2 raid0
   # gpt label -l NetBSD-swap -i 3 raid0

   Verify with "gpt show -l raid0".

   These labels can apparently be used in /boot.cfg, like so:

   menu=Boot normally:rndseed /var/db/entropy-file;boot NAME=NetBSD-root:/netbsd

   and ... also in /etc/fstab.

   However, when you set your own labels, you *must* also edit
   /etc/fstab which sysinst has prepared to use the new names
   instead of using NAME=UUID, because sysinst has set up to
   refer to the root and swap partitions via NAME=UUID, and
   putting your own name in the label will overwrite that field.
   (Ask me how I know...)

Of these two, obviously #1 is easiest, and if you so choose, you
could leave the labeling as it is and let /etc/fstab be as it is.

One could possibly argue that when sysinst installs NetBSD to a
disk matching "raid*", it should set the "softroot" flag itself,
so that the additional manual post-install futzing can be
reduced.

Not sure how the boot situation could be improved with a setup
such as this -- the boot partition which I think is actuall being
used is outside of the root disk.  I also see traces of sysinst
having left an UEFI boot partition inside raid0 (but I don't
think it's being recognized or used by the BIOS when booting).

Regards,

- Håvard