OVMF, lvm devices, and hvm guests

[Brad Spencer <brad%anduin.eldar.org@localhost>]

This is a set of notes and observations I acquired when looking into
trying to use OVMF (uefi bios) with Xen HVM guests.  Sorry for the
length.

The DOM0 I am using is a 9.2_STABLE from 2022-06-16 or so and Xen
4.15.2nb2 from my own build of pkgsrc/sysutils/xen* from 2022Q2 from
2022-06-30 or so.

----
I tried to get Xen 4.15 to build OVMF on its own with the option
--enable-ovmf.  This didn't work.  The attempt tried to pull down a
crapton of stuff using git, which I allowed, had one error where a "||
defined(__NetBSD__)" was needed (for uuid.h) to be added and, more or
less, added the patches from pkgsrc/sysutils/ovmf for build.sh.
However, the result wouldn't build and the python builder didn't really
tell me why.

Gave up on that.

(Old fart rant: It looks like someone tried to recreate make as a python
script..  It is all very odd and pretty hard to follow)

----
Built pkgsrc/sysutils/ovmf.  This is an older OVMF version than the one
Xen may be able to build, but appears to have built just fine on
9.2_STABLE using PKGSRC 2022Q2.

Then I modified a test guest and added:

bios = "ovmf"
bios_path_override = "/usr/pkg/share/ovmf/OVMFX64.fd"

to the guest config.  The guest was set up to use a disk that is the
install image for 9.99.99 (NetBSD-9.99.99-amd64-install.img from the
CDN).

This mostly works.  The OVMF uefi firmware is used and does start
running.  However, NetBSD has problems.  NetBSD will notice that it is
running under a Xen hypervisor and try to be a PVH or PVHVM guest and
around the time it tries to use the disk devices the guest hangs up and
does not proceed any further.  This happens after it probes for balloon
and just before it prints which disk devices it found.

A custom GENERIC kernel was compiled without Xen guest support at all
and put in place of the GENERIC kernel on the install image.  This boots
up just fine with the OVMF bios, finding the re0 nic and wd0 disk
drives.  I used this installer to install on another disk.  I then
replaced the GENERIC kernel on the second disk, with the No Xen GENERIC
kernel and reconfigured the guest to use the just installed drive.  The
result all booted up and worked as expected.  So, Xen using UEFI booting
on a HVM guest did work.

(Brief aside... I tried making minimal changes to the GENERIC config,
leaving in some Xen guest support and that didn't work either..  It had
to be all removed to make a functional kernel)

Please note that if I remove the 'bios = "ovmf"' stuff and let the guest
use SeaBIOS (and MBR probably) I do not have to use the custom kernel
and the normal GENERIC kernel on the installer images works and the
system becomes a PVHVM guest just fine.

At this point all disks that were used were file based disks.

-------
(Use the clutch, switching gears)

Usually I like to use LVM raw devices for Xen guests rather than files.
So, I created a LVM LV that was exactly the same size as the file based
disk used in the previous section.  I then dd'ed the contents of the
file into the raw LVM LV and adjusted the guest config to the LVM LV
block device.  This does not work.  One of a couple of things happen:

1) If the kernel actually boots up, it will not be able to find it root
filesystem and nothing that I put in telling it what that was helped.  I
had actually noticed before that I was unable to use the new qemu-dm
with old HVM guests and had to use use qemu traditional (and I think
RomBIOS) instead.  Those HVM guests had the same problem in that the
SeaBIOS could not find a bootable disk while the older qemu could.

2) There was more than one DOM0 kernel panic / reboot that I didn't
manage to catch even though I have ddb.onpanic set to 1.  This doesn't
happen all of the time, but at least twice.  I don't know what the panic
was, if there was one, as it happened too quickly and I wasn't present
at the console.


In all cases, using a raw LVM LV produced these on the console (dozens
and dozens of these lines):

Aug 24 13:37:23 dom0system /netbsd: [ 266.8875858] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x70343e1e7000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 266.8975885] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x70343e1e7000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 266.8975885] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x70343e1e7000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 266.8975885] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x70343e1e7000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 266.9075878] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x703433bc4000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 266.9075878] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x703433bc4000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 266.9075878] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x703433bc4000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 266.9075878] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x703433bc4000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 304.9071596] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x70343e1f1000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 304.9071596] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x70343e1f1000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 304.9071596] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x70343e1f1000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 304.9171584] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x70343668e000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 304.9171584] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x703436693000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 304.9171584] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x70343668c000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 304.9171584] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x70343668c000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 304.9171584] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x703436691000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 304.9171584] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x70343668a000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 304.9171584] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x70343668a000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 304.9171584] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x70343668f000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 304.9171584] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x703436688000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 304.9271568] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x703436688000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 304.9271568] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x70343668d000 did not change!
Aug 24 13:37:23 dom0system /netbsd: [ 304.9271568] pmap_unwire: wiring for pmap 0xffffbf000535dba8 va 0x703436686000 did not change!

The exact same data if used from a file based drive does not produce any
of this.

------
Final thing...  Since I had access to it, I pulled the OVMF firmware out
of an ArchLinux system I had that had the OVMF package installed and
tried to use that instead of the one built from pkgsrc/sysutils/ovmf.
Although it booted up the system, NetBSD didn't like it and panic'ed
when trying to look at the ACPI tables very early in the kernel boot.
I assume that the ArchLinux version of the OVMF package is much newer
than the one in pkgsrc.






-- 
Brad Spencer - brad%anduin.eldar.org@localhost - KC8VKS - http://anduin.eldar.org