Re: Xen boot strangeness (Was: Re: [SOLVED] Re: Xen 4.18.5_20250521nb0 not ELF binary (Was: Re: EFI and Xen))

[Chuck Zmudzinski <frchuckz%gmail.com@localhost>]

On 5/29/2025 10:24 PM, Chuck Zmudzinski wrote:
> On 5/29/2025 7:39 PM, Manuel Bouyer wrote:
>> On Thu, May 29, 2025 at 05:20:53PM -0400, Chuck Zmudzinski wrote:
>>> On 5/29/2025 3:23 PM, Manuel Bouyer wrote:
>>> > On Thu, May 29, 2025 at 03:01:50PM -0400, Chuck Zmudzinski wrote:
>>> >> [...]
>>> 
>>> > 
>>> > This is because the userconf command from the boot loader doens't make it
>>> > to the dom0 kernel.
>>> > Remember that when booting Xen; Xen is the kernel and XEN3_DOM0 a module.
>>> > I don't know if multiboot allows passing extended informations to a module.
>>> 
>>> So can you confirm the only way to invoke userconf when booting NetBSD DOM0
>>> as a module is by passing -c on the NetBSD command line and that it is
>>> not possible to invoke userconf from a setting in boot.cfg? 
>> 
>> Correct
> 
> Then the Xen Howto Wiki page needs to be updated. It makes no mention of the
> -c option. Instead it has this bit of information:
> 
>> "userconf" statements intended for NetBSD should be attached to the load statement, not the multiboot statement. (\todo Validate and test an example.)
> 
> I tried attaching the userconf statement to the load statement but it did not work.
> 
> So to correct this, the Wiki should be changed to say something like:
> 
> "userconf" statements intended for NetBSD can only be configured interactively by passing the -c option to the NetBSD DOM0 command line. This is required, for example, when Xen uses a com device as its console other than what NetBSD detects as com0 with the default XEN3_DOM0 kernel. In that case, the userconf command that must be entered interactively during boot from the console at the uc> prompt is disable com* to get a successful boot.
> 
> In general, I think the Wiki is not well-suited to help someone
> boot NetBSD/Xen PV Dom0 on a modern EFI system with multiple disks
> and multiple cpus.
> 
> For example, in addition to the above mentioned problem about userconf
> in the Wiki:
> 
> 1. On my system with 20 cpus, I could only find out the trick to
>    set dom0_max_vcpus=4 to overcome the gsi < NR_EVENT_CHANNELS failed
>    assertion by trial and error. The Wiki could be updated with such
>    hints to help users boot NetBSD/Xen on modern EFI systems like mine
>    with many cpus/vcpus that can cause gsi to exceed NR_EVENT_CHANNELS
>    if one is not careful about setting dom0_max_vpus correctly on
>    such a system so it can boot the NetBSD XEN3_DOM0 kernel.
> 
> 2. The Wiki does not explain clearly enough the difference between a
>    bootdev= directive in the arguments to the command to load the
>    NetBSD DOM0 kernel and the root device that is entered interactively
>    when the bootloader or the NetBSD kernel (I am not sure which one)
>    fails to find a valid root device it can mount. For example, the
>    Wiki states that if using GPT partitioning, the root device is likely
>    to be dk0 but does not explain how boot.cfg should be changed in
>    that case. Would you say to set bootdev=dk0 in boot.cfg in that
>    case even though dk0 is usually an EFI system partition, not a
>    NetBSD root device, when using GPT partitioning? What the Wiki
>    actually says now is not clear and does not make much sense to me
>    and was not really very helpful in helping me figure out how to
>    write the boot.cfg file to get my system to boot NetBSD/Xen.
> 
> 3. The Wiki does not clearly say you can pass a *root* device such as
>    dk12 as the bootdev option in boot.cfg instead of a *boot* device
>    such as wd1 and then either the NetBSD kernel or the bootloader (I
>    am not sure which one) deduces that wd1a is the root device and wd1b
>    is the dump device, which is the behavior I saw, but this is not
>    explained at all in the Wiki.

To complete my suggestions for the Xen Howto Wiki, I would add:

4. The Wiki does not explain that boot.cfg must be located in the
   EFI/NetBSD directory on the FAT formatted EFI system partition. This
   information is in the x86 boot(8) man page near the very bottom in the
   FILES section where the /usr/mdec/bootx64.efi file is documented, but is
   not in either the Xen Howto Wiki or the 'Installation on UEFI systems
   Wiki' page. I did need to hunt around the documentation to find this
   bit of essential information so I could actually setup my UEFI-only
   system to have the nice boot menu that boot.cfg offers. It would be
   good to also include this bit of information about the location of
   boot.cfg on the 'Installation on UEFI systems Wiki' page as well as
   on the Xen Howto Wiki page, perhaps in a new section dedicated to how
   to boot NetBSD/xen on UEFI systems, and also probably on the x86
   boot.cfg(5) man page which also lacks this essential information.

> 
> I would be willing to help update the Wiki with lessons learned while
> trying to boot NetBSD/Xen on this recent Raptor Lake i5-14500 CPU with
> 6 P cores and 8 E cores for a total of 20 cpus. The Wiki is woefully
> inadequate to explain how to boot NetBSD/xen on such a system. 
> 
> And thanks for your work in maintaining the Xen port!
> 
> Kind regards,
> 
> Chuck Zmudzinski