Re: Xen 4.18.5_20250521nb0 not ELF binary (Was: Re: EFI and Xen)

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

On Sat, May 24, 2025 at 07:00:55AM -0400, Chuck Zmudzinski wrote:
> On 5/24/2025 3:40 AM, Manuel Bouyer wrote:
> > On Fri, May 23, 2025 at 10:06:05PM -0400, Chuck Zmudzinski wrote:
> >> On 5/23/2025 5:18 PM, Chuck Zmudzinski wrote:
> >> > On 5/23/2025 4:22 PM, Manuel Bouyer wrote:
> >> >> On Fri, May 23, 2025 at 04:19:46PM -0400, Chuck Zmudzinski wrote:
> >> >>> On 5/23/2025 4:15 PM, Manuel Bouyer wrote:
> >> >>> > On Fri, May 23, 2025 at 04:13:01PM -0400, Chuck Zmudzinski wrote:
> >> >>> >> [...]
> >> >>> >> I needed to acquire a PCI serial card to get some useful messages because the motherboard does not have a built-in serial port.
> >> >>> >> 
> >> >>> >> It appears Xen 4.18.4_20241221nb0 from pkgsrc crashes on my box before the NetBSD dom0 starts:
> >> >>> >> 
> >> >>> >> -------- snip ------
> >> >>> >> (XEN) Xen call trace:
> >> >>> >> (XEN)    [<ffff82d040261a00>] R intel_iommu_enable_eim+0/0xe0
> >> >>> >> (XEN)    [<ffff82d040268ac4>] S iommu_enable_x2apic+0x24/0x70
> >> >>> >> (XEN)    [<ffff82d0403f579d>] S x2apic_bsp_setup+0x12d/0x220
> >> >>> >> (XEN)    [<ffff82d0403ff0e4>] S __start_xen+0x1af4/0x2620
> >> >>> >> (XEN)    [<ffff82d040204434>] S __high_start+0x94/0xa0
> >> >>> >> (XEN) 
> >> >>> >> (XEN) 
> >> >>> >> (XEN) ****************************************
> >> >>> >> (XEN) Panic on CPU 0:
> >> >>> >> (XEN) CONTROL-FLOW PROTECTION FAULT: #CP[0003] endbranch
> >> >>> >> (XEN) ****************************************
> >> >>> >> (XEN) 
> >> >>> >> (XEN) Reboot in five seconds...
> >> >>> >> (XEN) Resetting with ACPI MEMORY or I/O RESET_REG.
> >> >>> >> -------- end messages ------
> >> >>> > 
> >> >>> > Do you have a way to disable x2apic in the BIOS ?
> >> >>> 
> >> >>> I can check later this evening but I have an appointment now.
> >> >>> 
> >> >>> > 
> >> >>> >> 
> >> >>> >> Please note only the pkgsrc version of Xen crashes. I can boot Xen with no problems using Xen packaged by Fedora, AlmaLinux, etc. on this box.
> >> >>> > 
> >> >>> > Is it the same Xen version ?
> >> >>> 
> >> >>> More or less - I know Xen 4.18.4 worked, now Fedora is at Xen 4.19.
> >> >> 
> >> >> strange; maybe they had a patch.
> >> >> Note that I just updated xen in pkgsrc to 4.18.5
> >> >> 
> >> > 
> >> > Actually, I do remember that Xen 4.19.1 from Fedora did not work on my box but a patch that was added in 4.19.2 fixed it. I see the patch I needed on the Xen 4.19 branch was not added in Xen 4.18.4 but it is added in Xen 4.18.5, so maybe it will be fixed in Xen 4.18.5. I will try it when I can download Xen 4.18.5.
> >> > 
> >> > Thanks.
> >> 
> >> I rebuilt the latest Xen from pkgsrc but no help. Then I searched
> >> Google for information about the error I am getting which has to
> >> do with Control-flow Enforcement Technology (cet) and tried experimenting
> >> with the Xen cet= options.
> >> 
> >> Now I am not getting the crash, but just when it is ready to build the
> >> PV Dom0, it complains that it does not have an ELF binary.
> >> 
> >> Note the magic incantation to get past the crash is to add the option
> >> cet=no-ibt to the Xen command line.
> >> 
> >> With this configuration, I get:
> >> 
> >> (XEN) Xen version 4.18.5_20250521nb0 (chuckz@) (gcc (nb3 20231008) 10.5.0) debug=n Fri May 23 18:59:50 EDT 2025
> >> (XEN) Latest ChangeSet: 
> >> (XEN) build-id: 86afac8b52e2d124001208bb6261d17088a2f26f
> >> (XEN) Bootloader: NetBSD/x86 EFI Boot (x64), Revision 1.2 (Mon Dec 16 13:08:11 UTC 2024) (from NetBSD 10.1)
> >> (XEN) Command line: dom0_mem=2G com2=9600,8n1,0x40c0,16,1:0.0 console=com2 cet=no-ibt pv-l1tf=false
> >> ... snip many messages
> >> (XEN)  First cpu on runqueue, activating
> >> (XEN) Adding cpu 17 to runqueue 2
> >> (XEN) Adding cpu 18 to runqueue 2
> >> (XEN) Adding cpu 19 to runqueue 2
> >> (XEN) mcheck_poll: Machine check polling timer started.
> >> (XEN) NX (Execute Disable) protection active
> >> (XEN) d0 has maximum 4160 PIRQs
> >> (XEN) *** Building a PV Dom0 ***
> >> (XEN) ELF: not an ELF binary
> >> (XEN) 
> >> (XEN) ****************************************
> >> (XEN) Panic on CPU 0:
> >> (XEN) Could not construct domain 0
> >> (XEN) ****************************************
> >> (XEN) 
> >> (XEN) Reboot in five seconds...
> >> (XEN) Resetting with ACPI MEMORY or I/O RESET_REG.
> >> 
> >> I tried uncompressing (with gunzip) the netbsd-XEN3_DOM0.gz file, and I
> >> tried omitting the dom0_mem setting, but neither change helped.
> > 
> > This one I don't know; what boot loader command are you using ?
> > 
> 
> menu=Boot normally with Xen:dev hd2d:;rndseed /var/db/entropy-file;load /netbsd-XEN3_DOM0 console=pc;multiboot /xen.gz dom0_mem=2G com2=9600,8n1,0x40c0,16,1:0.0 console=com2 cet=no-ibt pv-l1tf=false

I don't have the rndseed entry in my settings; maybe that's the problem

BTW, as you're using a serial console you probably want
console=com0
for NetBSD

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference
--