Re: Xen 4.18.5_20250521nb0 not ELF binary (Was: Re: EFI and Xen)

[Chuck Zmudzinski <frchuckz%gmail.com@localhost>]

On 5/23/2025 5:18 PM, Chuck Zmudzinski wrote:
> On 5/23/2025 4:22 PM, Manuel Bouyer wrote:
>> On Fri, May 23, 2025 at 04:19:46PM -0400, Chuck Zmudzinski wrote:
>>> On 5/23/2025 4:15 PM, Manuel Bouyer wrote:
>>> > On Fri, May 23, 2025 at 04:13:01PM -0400, Chuck Zmudzinski wrote:
>>> >> [...]
>>> >> I needed to acquire a PCI serial card to get some useful messages because the motherboard does not have a built-in serial port.
>>> >> 
>>> >> It appears Xen 4.18.4_20241221nb0 from pkgsrc crashes on my box before the NetBSD dom0 starts:
>>> >> 
>>> >> -------- snip ------
>>> >> (XEN) Xen call trace:
>>> >> (XEN)    [<ffff82d040261a00>] R intel_iommu_enable_eim+0/0xe0
>>> >> (XEN)    [<ffff82d040268ac4>] S iommu_enable_x2apic+0x24/0x70
>>> >> (XEN)    [<ffff82d0403f579d>] S x2apic_bsp_setup+0x12d/0x220
>>> >> (XEN)    [<ffff82d0403ff0e4>] S __start_xen+0x1af4/0x2620
>>> >> (XEN)    [<ffff82d040204434>] S __high_start+0x94/0xa0
>>> >> (XEN) 
>>> >> (XEN) 
>>> >> (XEN) ****************************************
>>> >> (XEN) Panic on CPU 0:
>>> >> (XEN) CONTROL-FLOW PROTECTION FAULT: #CP[0003] endbranch
>>> >> (XEN) ****************************************
>>> >> (XEN) 
>>> >> (XEN) Reboot in five seconds...
>>> >> (XEN) Resetting with ACPI MEMORY or I/O RESET_REG.
>>> >> -------- end messages ------
>>> > 
>>> > Do you have a way to disable x2apic in the BIOS ?
>>> 
>>> I can check later this evening but I have an appointment now.
>>> 
>>> > 
>>> >> 
>>> >> Please note only the pkgsrc version of Xen crashes. I can boot Xen with no problems using Xen packaged by Fedora, AlmaLinux, etc. on this box.
>>> > 
>>> > Is it the same Xen version ?
>>> 
>>> More or less - I know Xen 4.18.4 worked, now Fedora is at Xen 4.19.
>> 
>> strange; maybe they had a patch.
>> Note that I just updated xen in pkgsrc to 4.18.5
>> 
> 
> Actually, I do remember that Xen 4.19.1 from Fedora did not work on my box but a patch that was added in 4.19.2 fixed it. I see the patch I needed on the Xen 4.19 branch was not added in Xen 4.18.4 but it is added in Xen 4.18.5, so maybe it will be fixed in Xen 4.18.5. I will try it when I can download Xen 4.18.5.
> 
> Thanks.

I rebuilt the latest Xen from pkgsrc but no help. Then I searched
Google for information about the error I am getting which has to
do with Control-flow Enforcement Technology (cet) and tried experimenting
with the Xen cet= options.

Now I am not getting the crash, but just when it is ready to build the
PV Dom0, it complains that it does not have an ELF binary.

Note the magic incantation to get past the crash is to add the option
cet=no-ibt to the Xen command line.

With this configuration, I get:

(XEN) Xen version 4.18.5_20250521nb0 (chuckz@) (gcc (nb3 20231008) 10.5.0) debug=n Fri May 23 18:59:50 EDT 2025
(XEN) Latest ChangeSet: 
(XEN) build-id: 86afac8b52e2d124001208bb6261d17088a2f26f
(XEN) Bootloader: NetBSD/x86 EFI Boot (x64), Revision 1.2 (Mon Dec 16 13:08:11 UTC 2024) (from NetBSD 10.1)
(XEN) Command line: dom0_mem=2G com2=9600,8n1,0x40c0,16,1:0.0 console=com2 cet=no-ibt pv-l1tf=false
... snip many messages
(XEN)  First cpu on runqueue, activating
(XEN) Adding cpu 17 to runqueue 2
(XEN) Adding cpu 18 to runqueue 2
(XEN) Adding cpu 19 to runqueue 2
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) NX (Execute Disable) protection active
(XEN) d0 has maximum 4160 PIRQs
(XEN) *** Building a PV Dom0 ***
(XEN) ELF: not an ELF binary
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Could not construct domain 0
(XEN) ****************************************
(XEN) 
(XEN) Reboot in five seconds...
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.

I tried uncompressing (with gunzip) the netbsd-XEN3_DOM0.gz file, and I
tried omitting the dom0_mem setting, but neither change helped.