Re: Xen boot strangeness (Was: Re: [SOLVED] Re: Xen 4.18.5_20250521nb0 not ELF binary (Was: Re: EFI and Xen))

[Chuck Zmudzinski <frchuckz%gmail.com@localhost>]

On 5/29/2025 7:39 PM, Manuel Bouyer wrote:
> On Thu, May 29, 2025 at 05:20:53PM -0400, Chuck Zmudzinski wrote:
>> On 5/29/2025 3:23 PM, Manuel Bouyer wrote:
>> > On Thu, May 29, 2025 at 03:01:50PM -0400, Chuck Zmudzinski wrote:
>> >> [...]
>> 
>> > 
>> > This is because the userconf command from the boot loader doens't make it
>> > to the dom0 kernel.
>> > Remember that when booting Xen; Xen is the kernel and XEN3_DOM0 a module.
>> > I don't know if multiboot allows passing extended informations to a module.
>> 
>> So can you confirm the only way to invoke userconf when booting NetBSD DOM0
>> as a module is by passing -c on the NetBSD command line and that it is
>> not possible to invoke userconf from a setting in boot.cfg? 
> 
> Correct

Then the Xen Howto Wiki page needs to be updated. It makes no mention of the
-c option. Instead it has this bit of information:

> "userconf" statements intended for NetBSD should be attached to the load statement, not the multiboot statement. (\todo Validate and test an example.)

I tried attaching the userconf statement to the load statement but it did not work.

So to correct this, the Wiki should be changed to say something like:

"userconf" statements intended for NetBSD can only be configured interactively by passing the -c option to the NetBSD DOM0 command line. This is required, for example, when Xen uses a com device as its console other than what NetBSD detects as com0 with the default XEN3_DOM0 kernel. In that case, the userconf command that must be entered interactively during boot from the console at the uc> prompt is disable com* to get a successful boot.

In general, I think the Wiki is not well-suited to help someone
boot NetBSD/Xen PV Dom0 on a modern EFI system with multiple disks
and multiple cpus.

For example, in addition to the above mentioned problem about userconf
in the Wiki:

1. On my system with 20 cpus, I could only find out the trick to
   set dom0_max_vcpus=4 to overcome the gsi < NR_EVENT_CHANNELS failed
   assertion by trial and error. The Wiki could be updated with such
   hints to help users boot NetBSD/Xen on modern EFI systems like mine
   with many cpus/vcpus that can cause gsi to exceed NR_EVENT_CHANNELS
   if one is not careful about setting dom0_max_vpus correctly on
   such a system so it can boot the NetBSD XEN3_DOM0 kernel.

2. The Wiki does not explain clearly enough the difference between a
   bootdev= directive in the arguments to the command to load the
   NetBSD DOM0 kernel and the root device that is entered interactively
   when the bootloader or the NetBSD kernel (I am not sure which one)
   fails to find a valid root device it can mount. For example, the
   Wiki states that if using GPT partitioning, the root device is likely
   to be dk0 but does not explain how boot.cfg should be changed in
   that case. Would you say to set bootdev=dk0 in boot.cfg in that
   case even though dk0 is usually an EFI system partition, not a
   NetBSD root device, when using GPT partitioning? What the Wiki
   actually says now is not clear and does not make much sense to me
   and was not really very helpful in helping me figure out how to
   write the boot.cfg file to get my system to boot NetBSD/Xen.

3. The Wiki does not clearly say you can pass a *root* device such as
   dk12 as the bootdev option in boot.cfg instead of a *boot* device
   such as wd1 and then either the NetBSD kernel or the bootloader (I
   am not sure which one) deduces that wd1a is the root device and wd1b
   is the dump device, which is the behavior I saw, but this is not
   explained at all in the Wiki.

I would be willing to help update the Wiki with lessons learned while
trying to boot NetBSD/Xen on this recent Raptor Lake i5-14500 CPU with
6 P cores and 8 E cores for a total of 20 cpus. The Wiki is woefully
inadequate to explain how to boot NetBSD/xen on such a system. 

And thanks for your work in maintaining the Xen port!

Kind regards,

Chuck Zmudzinski