Re: Problems booting NetBSD/Xen on Intel NUC 12 Extreme w/ i9-12900

[Johan Stenstam <johani%johani.org@localhost>]

Hi all,

New day, new efforts. I’ve now installed a modern Linux (Ubuntu 23.04) with Xen 4.17.1 to use for comparison.

> 1. Around line 250 there is this:
> 
> (XEN) Enabled directed EOI with ioapic_ack_old on!
> (XEN) ENABLING IO-APIC IRQs
> (XEN)  -> Using old ACK method
> (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
> (XEN) ..no 8254 timer found - trying HPET Legacy Replacement Mode
> (XEN) TSC deadline timer enabled
> (XEN) Defaulting to alternative key handling; send 'A' to switch to normal mode.
> (XEN) Allocated console ring of 256 KiB.
> (XEN) mwait-idle: does not run on family 6 model 151

I see the exact same messages from Xen 4.17, so this is neither in any way NetBSD nor 4.15 specific, but rather just as it seems, statements about how the hardware in my machine really looks (to Xen).

> 2. At the tail end there is some output that clearly relates to the failed creation of a DOMU. The DOM0 is just rebooted, so the first DOMU will be “domain 1”:
> 
> ...
> (XEN) d1v0 Unhandled general protection fault fault/trap [#13, ec=0000]
> (XEN) domain_crash_sync called from entry.S: fault at ffff82d040300e78 x86_64/entry.S#create_bounce_frame+0x14f/0x167
> (XEN) Domain 1 (vcpu#0) crashed on cpu#22:
> (XEN) ----[ Xen-4.15.4nb0  x86_64  debug=n  Not tainted ]----
> (XEN) CPU:    22
> (XEN) RIP:    e033:[<ffffffff8021b18d>]
> (XEN) RFLAGS: 0000000000000202   EM: 1   CONTEXT: pv guest (d1v0)
> (XEN) rax: 0000000000070106   rbx: 0000000000ae4000   rcx: 0000000000000277
> (XEN) rdx: 0000000000070106   rsi: 0000000000000002   rdi: ffffffff80642780
> (XEN) rbp: ffffffff80ae2eb0   rsp: ffffffff80ae2e70   r8:  000000001fc9cbf5
> (XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000001
> (XEN) r12: 0000000000000000   r13: ffffffff80adf000   r14: 0000000000000000
> (XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 0000000000b426e0
> (XEN) cr3: 000000103c0aa000   cr2: 0000000000000000
> (XEN) fsb: 0000000000000000   gsb: ffffffff80642780   gss: 0000000000000000
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
> (XEN) Guest stack trace from rsp=ffffffff80ae2e70:
> (XEN)    0000000000000277 0000000000000001 0000000000000000 ffffffff8021b18d
> (XEN)    000000010000e030 0000000000010002 ffffffff80ae2eb0 000000000000e02b

Here is the same thing happening under Linux (using the exact same DOMU image and kernel:

(XEN) d77v0 Unhandled: vec 13, #GP[0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d040203bc8 x86_64/entry.S#create_bounce_frame+0x14f/0x167
(XEN) Domain 77 (vcpu#0) crashed on cpu#16:
(XEN) ----[ Xen-4.17.1-pre  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    16
(XEN) RIP:    e033:[<ffffffff8021b18d>]
(XEN) RFLAGS: 0000000000000202   EM: 1   CONTEXT: pv guest (d77v0)
(XEN) rax: 0000000000070106   rbx: 0000000000964000   rcx: 0000000000000277
(XEN) rdx: 0000000000070106   rsi: 0000000000000002   rdi: ffffffff80642780

So, again, I was quite clearly wrong and this is, indeed, some sort of problem with my DOMU. Or at least I cannot find any other explanation. So I created a trivial Linux PV DOMU and replaced the pvgrub boot method with direct kernel booting like I use for NetBSD. Here are the xen configs for my NetBSD 9.2 DOMU (not working) and the Linux DOMU (working fine):

name        = 'master.axfr.net'
kernel      = "/usr/store/xen/kernels/netbsd.XEN3_DOMU.amd64.9.2_REL"
root        = '/dev/xbd0a'
vcpus       = '1'
memory      = '256'
vif         = [ 'ip=172.16.128.200 ,mac=00:16:3E:69:1F:6F' ]
disk        = [ 'file:/usr/store/xen/lab/vm/master.img,0x3,w' ]


name        = 'lunix1.axfr.net'
kernel      = "/usr/store/lunix/kernels/vmlinuz-6.2.0-20-generic"
root        = '/dev/xvda2 ro'
vcpus       = '1'
memory      = '256'
vif         = [ 'ip=172.16.128.199 ,mac=00:16:3E:69:1F:6E' ]
disk        = [ 'file:/usr/store/lunix/lunix1.axfr.net/disk.img,xvda2,w',
                'file:/usr/store/lunix/lunix1.axfr.net/swap.img,xvda1,w' ]

I’ve rearranged the NetBSD config to make it as close to the Linux config as possible and I must say that they look very similar. The only thing I can figure out that could be wrong is the “0x3” parameter at the end of the disk variable for the NetBSD DOMU. Not sure what the right value is there, but 0x3 is what I use on all my NetBSD machines and I also tried various Linux-like alternatives (“xvda2”, “xvda1”, etc), with no difference.

I said before that I’m confident that the DOMU image is fine but at this point I’m not in a position to claim anything whatsoever. But I will say that just now I copied that exact DOMU image across the local network to a NetBSD 9.2/xen 4.11 DOM0 and that image with the above config booted with zero problems there.

So it looks to me that the DOMU crash above is:
- not H/W (because I can boot a Linux PV DOM0 on the same H/W)
- not the xen hypervisor (as it can boot other PV DOMUs)
- not NetBSD as DOM0 (because I see the exact same crash with a Linux DOM0)
- not the xen config (as the NetBSD and Linux configs are close to identical)

That would clearly indicate that my DOMU image is broken somehow… except that it works flawlessly on a NetBSD 9.2/xen 4.11 DOM0.

I cannot be the only one confused at this stage.

Regards,
Johan