Re: QEMU booting of non-NetBSD CDs and virtual HDs

[Robert Nestor <rnestor%mac.com@localhost>]

On Aug 24, 2020, at 1:32 PM, Chavdar Ivanov <ci4ic4%gmail.com@localhost> wrote:

> On Mon, 24 Aug 2020 at 15:08, Robert Nestor <rnestor%mac.com@localhost> wrote:
>> 
>> 
>> On Aug 23, 2020, at 1:57 PM, Chavdar Ivanov <ci4ic4%gmail.com@localhost> wrote:
>> 
>>> On Sun, 23 Aug 2020 at 15:41, Robert Nestor <rnestor%mac.com@localhost> wrote:
>>>> 
>>>> I received a couple of messages off list that suggested a few things and it prompted me to try investigating further with just components found in NetBSD.
>>>> 
>>>> This test was run on a fairly recent NetBSD build of 9.99.70.  I downloaded the amd64 images for 9.99.71 (the ISO and IMG files), and tried booting them with qemu using -nvmm and the OVMF binaries currently in pkgsrc with the following:
>>>> 
>>>> qemu-system-x86_64 -m 4096 -machine q35 -accel=nvmm -boot menu=on \
>>> 
>>> -accel nvmm
>>> 
>>>>   -device qemu-xhci -device usb-tablet -device usb-mouse -smbios type=2 \
>>>>   -drive if=pflash,format=raw,readonly,file=/usr/pkg/share/OVMFX64.fd \
>>> 
>>> the OVMFX64.fd file is actually in /usrpkg/share/ovmf directory, but
>>> perhaps this is a typo. Anyway. I have no idea about this particular
>>> way of specifying the bios; anyway, with
>>> 
>>> 
>>> -bios /usr/pkg/share/ovmf/OVMFX64.fd \
>>> 
>>> it boots just fine. Otherwise I get the same crash as you.
>>> 
>>> 
>>>>   -device ich9-ahci,id=sata \
>>>>   -device ide-cd,bus=sata.0,drive=disk \
>>>>   -drive id=disk,if=none,media=cdrom,format=raw,file=NetBSD-9.99.71-amd64.iso
>>>> 
>>>> This produces an immediate “failed to start VCPU” and results in a core dump. Also tried the NetNSD-9.99.71-amd64-install.img file with:
>>> 
>>>> 
>>>> qemu-system-x86_64 -m 4096 -machine q35 -accel=nvmm -boot menu=on \
>>>>   -device qemu-xhci -device usb-tablet -device usb-mouse -smbios type=2 \
>>>>   -drive if=pflash,format=raw,readonly,file=/usr/pkg/share/OVMFX64.fd \
>>>>   -device ich9-ahci,id=sata \
>>>>   -device ide-hd,bus=sata.0,drive=disk \
>>>>   -drive id=disk,if=none,media=disk,format=raw,file=NetBSD-9.99.71-amd64-install.img
>>>> 
>>>> And it provides the same results - “failed to start VCPU” and a core dump.
>>>> 
>>>> Removing the “-accel=nvmm” from both of the scripts allows the boot to proceed, but the OVMF code fails to find the CD or HD image and boot falls back to attempting to boot over the network.  This appears to be a bug in the version of OVMF found in pkgsrc which is based on stable2018.  Replacing the OVMF with binaries obtained from a build of stable202005 fixes the disk access issue and the boot then succeeds brining up the NetBSD installer.
>>>> 
>>>> I then proceeded to do two installations of NetBSD under qem; one using the defaults for an MBR setup and one for a GPT setup.  The resulting MBR disk doesn’t boot under qemu; the GPT disk does boot however.  In the case of the MBR disk it appears the problem is that OVMF can’t find the disk or anything bootable on it.
>>>> 
>>>> I’ve opened two PRs for these issues.  PR-55582 for the NVMM issues and PR-55582 for the OVMF issue.
>>>> 
>> 
>> Indeed, using the OVMFX64.fd file as a -bios parameter does eliminate the NVMM crash and it does boot up the NetBSD CD ISO file.  From there I was able to do two installations of NetBSD, one specifying an MBR partition setup and one with GPT taking the defaults for both.  However, neither of the created disk image files will boot with the OVMFX64.fd file.  Using a newer version from a stable202005 OVMF build and one from a build done in the last week or so does boot up the GPT created disk image but not the newly created MBR disk image.
> 
> I haven't tried an MBR installation, but my GPT one boots just fine
> with OVMFX86 -
> 
> qemu-system-x86_64 \
>                -m 3072 \
>                -machine q35 \
>                -accel nvmm \
>                -device qemu-xhci \
>                -device usb-tablet \
>                -device usb-mouse \
>                -k en-gb \
>                -smp 2 \
>                -net tap,fd=3 3<>/dev/tap1 \
>                -boot menu=on \
>                -vnc :1 \
>                -net nic \
>                -bios /usr/pkg/share/ovmf/OVMFX64.fd \
>                -drive format=raw,file=/dev/zvol/rdsk/tank/ztest
> 
> This is the system I installed yesterday on a fresh zvol, it boots
> fine. As I have said previously, the same system can't boot if I use X
> server / gtk output - it boots only if I use vnc. So there is a
> problem, but I can't figure out who to  blame in this case

Did a bit more digging on this and came up with a couple of things.  It appears the MBR boot failure may be caused by not having the CSM code enabled in the version of OVMF being used. I had some limited success using a version of OVMF that was built with CSM support which builds in a copy of SeaBIOS.  This type of build for OVMF doesn’t appear to be recommended though as it requires a lot of mode switching between 16 bit and 32/64 bit mode that is fairly complex.

Second, while the original boot issue with NVMM that I reported can be circumvented by specifying the OVMF file as a -bios option rather than as a pflash that is considered an illegal or invalid approach according to this:

	https://lists.gnu.org/archive/html/qemu-discuss/2018-04/msg00045.html

Basically when the OVMF is attached with the -bios option the address space is mapped R/O and there’s no way for it to retrieve or set any UEFI variables. If the OVMF is attached via the pflash as a single file, it is mapped R/O but it can access and store UEFI variables in the NvVars file in the ESP Parittion.  And if the OVMF file is mapped via pflash with the code in one entry it is mapped R/O and the second entry specifies a R/W area that is synch’d to the corresponding disk file.

Finally, the fault causing NVMM to dump core may be related to some callback condition involving switching in and out of ROMD mode.  "Namely, in the default ROMD mode, the guest-physical address range backed by the flash device reads and executes as ROM (it does not trap from NVMM to QEMU). The first write access in this mode traps to QEMU, and flips the device out of ROMD mode.”