Re: PVH boot with qemu

[Emile `iMil' Heitor <imil%home.imil.net@localhost>]

It's nearly been a year since I started working in this topic, and
while it's still waiting for review before integration in the source
tree, I'd like it to be as battle tested as possible.
The whole set of patches touches a lot of ground, from new features to
performance improvements of existing code, and I probably haven't
tested every possible corner case.

This set of patches allow the NetBSD/amd64 kernel to boot directly in
PVH mode, i.e. without BIOS initialization: all needed bits are
provided by the hypervisor (tested with qemu/kvm, qemu/nvmm and AWS's
firecracker).

In order to allow fast boot, I borrowed the idea from OpenBSD to
implement a dummy bus, pv(4), where MMIO and pvclock(4) attach.

pvclock(4) is a port of OpenBSD's pvclock(4)[1], which uses (for now
only) KVM paravirtual clock which we can use instead of lapic, thus
avoiding calibration and gaining ~100ms in the boot process.

MMIO permits the use of memory-mapped devices, attaching it to pv(4)
allows to disable PCI support. Again, size and speed gains.

Also to avoid TSC calibration and gain another ~100ms, I borrowed
various CPU frequency detection from both FreeBSD and OpenBSD.

In order to witness the performance gains, I also ported FreeBSD's
tslog(4)[2] boot-time event tracing facility.

This kernel boots in about 15ms on an AMD Ryzen 5 5600X using
qemu/kvm.

The clean patch set is available here:
https://github.com/NetBSD/src/compare/trunk...NetBSDfr:NetBSD-src:nbfr_rebase

Patched current NetBSD code is available here:
https://github.com/NetBSDfr/NetBSD-src/tree/nbfr_master
I keep this in sync as much as possible to avoid conflicts.

A pre-built kernel with all the cited features is available here:
https://smolbsd.org/assets/netbsd-SMOL

Here's the kernel configuration file
https://raw.githubusercontent.com/NetBSD/src/2d93cbac9b741481c48d89278288a4392f68417d/sys/arch/amd64/conf/MICROVM

Feedback welcome,

iMil

[1]: https://man.openbsd.org/pvclock.4
[2]: https://man.freebsd.org/cgi/man.cgi?query=tslog

------------------------------------------------------------------------
Emile `iMil' Heitor <imil@{home.imil.net,NetBSD.org}> | https://imil.net