port-xen/59451: XEN3_DOM0 kernel finds the wrong root device

[frchuckz%gmail.com@localhost]

>Number:         59451
>Category:       port-xen
>Synopsis:       XEN3_DOM0 kernel finds the wrong root device
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    port-xen-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat May 31 19:10:00 +0000 2025
>Originator:     Chuck Zmudzinski
>Release:        NetBSD 10.1 Release
>Organization:
Home User
>Environment:
NetBSD netbsd 10.1 NetBSD 10.1 (XEN3_DOM0) #0: Mon Dec 16 13:08:11 UTC 2024  mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/xen/compile/XEN3_DOM0 amd64
>Description:
On a system with UEFI/GPT for booting and disk partitioning, the XEN3_DOM0 kernel can spectacularly fail in its attempt to find the correct root device even when the correct dk wedge is passed to the kernel in boot.cfg.

This problem was discussed extensively in a thread on netbsd-users, especially starting with this message: https://mail-index.netbsd.org/netbsd-users/2025/05/29/msg032694.html

I credit Greg Woods, the author of the aforementioned message on netbsd-users, for correctly identifying the problem.

The tl;dr version of the problem:

This code from sys/arch/xen/xen/xen_machdep.c is not strict enough in
its sanity checks and in some cases picks the wrong boot device and
try to find a "booted_partition" when there is no "boot_partition" to find:

		if (strncmp(xcp.xcp_bootdev, devname, strlen(devname)))
			continue;

		if (is_disk && strlen(xcp.xcp_bootdev) > strlen(devname)) {
			/* XXX check device_cfdata as in x86_autoconf.c? */
			booted_partition = toupper(
				xcp.xcp_bootdev[strlen(devname)]) - 'A';
			DPRINTF(("%s: booted_partition: %d\n", __func__, booted_partition));
		}
>How-To-Repeat:
1. It is most likely necessary to have a UEFI/GPT system that can boot NetBSD/xen to reproduce this problem. It might be possible to reproduce it on a system that uses BIOS booting and MBR partitioning, but I would recommend only trying to reproduce this problem on a UEFI/GPT system that relies on dk wedges rather than a NetBSD disklabel to find the root partition.

2. It is also necessary to have more than 10 dk wedges on the system and to install the NetBSD root/boot device on a dk wedge with a double digit index. On my box, NetBSD was installed on dk12. If NetBSD is installed on any wedge with a single digit index, I doubt the problem is reproducible. So if NetBSD is installed on dk9, for example, the problem will not be reproducible unless you do another install of NetBSD on dk10 or dk11, for example. The important thing is to have a NetBSD/xen PV DOM0 system installed on a wedge with a double digit wedge index.

3. Write a boot.cfg file to boot the NetBSD/xen system installed on the wedge with a double digit index and try to boot it. In such a case, you will need to set bootdev=dk12 (or whatever wedge NetBSD is installed on). But remember the problem will not be reproducible if you set bootdev=dk9, dk8, etc. It must be dk10 or higher. And NetBSD/xen PV DOM0 must be installed on the wedge that bootdev points to.

4. Also, it is probably necessary to setup Xen to use a serial console so you can interact with Xen and DOM0 at the serial console during the boot process to actually observe the details of this problem.

5. You should see the problem. In my case, with bootdev=dk12, the kernel tried to boot dk1 instead which, on my box, happened to have a Linux distro installed on it. So of course it failed to boot and the boot failed when the kernel tried to load the Linux distro's /sbin/init which in that case was systemd. This should up on the serial console like this:

snip ...
[   5.1699079] boot device: dk1

[   5.1699079] root on dk1 dumps on dk11

[   5.1799090] Your machine does not initialize mem_clusters; sparse_dumps disabled

[   5.1799090] root file system type: ext2fs

[   5.1799090] kern.module.path=/stand/amd64/10.1/modules

[   5.1826204] exec /sbin/init: error 8

[   5.1826204] init: trying /sbin/oinit

[   5.1826204] exec /sbin/oinit: error 2

[   5.1826204] init: trying /sbin/init.bak

[   5.1826204] exec /sbin/init.bak: error 2

[   5.1826204] init: trying /rescue/init

[   5.1826204] exec /rescue/init: error 2

[   5.1826204] init path (default /sbin/init):

Note the error 8 in response to exec /sbin/init in the output above. That is when the NetBSD DOM0 kernel tried to execute systemd on the Linux distro that was installed on dk1. It did this even though I gave the kernel the correct parameter of bootdev=dk12 in boot.cfg!

It was impossible to recover from this situation except by rebooting without Xen, setting bootdev=<something that will cause the kernel to give me a "root device" prompt> in the Xen menu item of boot.cfg, and reboot again with Xen and interactively enter the correct root device and dump device at the serial console when the kernel prompts for those. Then, finally, you get a successful boot.
>Fix:
I wrote a very simple 3-line proof of concept patch to fix it for the netbsd-10 branch. I don't endorse this patch as the final fix, though, because with this patch we still have garbage for boot_partition which could throw an assertion later on during the boot process (but it didn't in my test because the code where the assertion is never executed):

--- sys/arch/xen/xen/xen_machdep.c.orig 2023-10-18 12:53:03.000000000 -0400
+++ sys/arch/xen/xen/xen_machdep.c      2025-05-30 20:42:39.936253878 -0400
@@ -553,7 +553,10 @@
                        /* XXX check device_cfdata as in x86_autoconf.c? */
                        booted_partition = toupper(
                                xcp.xcp_bootdev[strlen(devname)]) - 'A';
+                       /* Check that the value of booted_partition is sane */
+                       if (booted_partition & 0xfffffff0)
+                               continue;
                        DPRINTF(("%s: booted_partition: %d\n", __func__, booted_partition));
                }

                booted_device = dv;

With this patch, the DOM0 kernel correctly detects dk12 as the root device when I set bootdev=dk12 in boot.cfg, and the kernel successfully loads /sbin/init from the NetBSD root partition and the boot proceeds as normal to completion:

snip ...
[    13.924443] boot device: dk12
[    14.114443] root on dk12 dumps on dk11
...