Re: port-amd64/57661: Crash when booting on Xeon Silver 4416+ in KVM/Qemu

["Taylor R Campbell via gnats" <gnats-admin%NetBSD.org@localhost>]

The following reply was made to PR port-amd64/57661; it has been noted by GNATS.

From: Taylor R Campbell <riastradh%NetBSD.org@localhost>
To: netbsd-bugs%NetBSD.org@localhost, gnats-bugs%NetBSD.org@localhost
Cc: 
Subject: Re: port-amd64/57661: Crash when booting on Xeon Silver 4416+ in KVM/Qemu
Date: Fri, 11 Apr 2025 14:25:46 +0000

 This is a multi-part message in MIME format.
 --=_tVq9BOFzAnpXS9N5Qy8XBZQoi+hFOgDk
 
 Followup from thread on tech-kern:
 
 https://mail-index.netbsd.org/tech-kern/2025/04/11/msg030349.html
 
 --=_tVq9BOFzAnpXS9N5Qy8XBZQoi+hFOgDk
 Content-Type: message/rfc822
 Content-Disposition: inline
 
 Date: Fri, 11 Apr 2025 14:24:47 +0000
 From: Taylor R Campbell <riastradh%NetBSD.org@localhost>
 To: Emmanuel Dreyfus <manu%netbsd.org@localhost>
 CC: tech-kern%netbsd.org@localhost
 Subject: Re: Page fault early in boot
 Message-Id: <20250411142453.75A4284D3F%mail.netbsd.org@localhost>
 In-reply-to: <Z_jRY4rZhMk4WALO%homeworld.netbsd.org@localhost> (manu%netbsd.org@localhost)
 MIME-Version: 1.0
 Content-Type: text/plain; charset=us-ascii
 Content-Transfer-Encoding: quoted-printable
 
 > Date: Fri, 11 Apr 2025 08:22:59 +0000
 > From: Emmanuel Dreyfus <manu%netbsd.org@localhost>
 >=20
 > I tried to boot NetBSD-current (i386 and amd6), BIOS and UEFI modes) on a=
 =20
 > Dell server, it always panic on a "fatal page fault in supervisor mode"
 > early during the boot. The backtrace says we went there through
 > main/fork1/lwp_create/memset
 >=20
 > A screenshot is available there:
 > https://dl.espci.fr/ticket/2a34048a0a9749c68cfb0dc380cc2259
 >=20
 > Any idea how to debug that? I expected NetBSD to lack some hardware
 > support on this recent machine, not to be unable to reach single user=20
 > mode.
 
 This looks like the same issue as:
 
 PR kern/57661: Crash when booting on Xeon Silver 4416+ in KVM/Qemu
 https://gnats.netbsd.org/57661
 
 Some additional diagnostics from the PR:
 
 > The stack trace is a bit of a red herring, I traced down the memset to li=
 ne 343 of src/sys/arch/x86/x86/fpu.c, so we actually have:
 > lwp_create() -> uvm_lwp_fork() -> cpu_lwp_fork() -> fpu_lwp_fork() -> mem=
 set()
 >=20
 > Adding
 >   printf("DEBUG: sizeof(pcb2->pcb_savefpu)=3D=3D%ld\n", sizeof(pcb2->pcb_=
 savefpu));
 >   printf("DEBUG: x86_fpu_save_size=3D=3D%d\n", x86_fpu_save_size);
 >=20
 > before the memset() call prints
 >=20
 >   [   1.8432366] DEBUG: sizeof(pcb2->pcb_savefpu)=3D=3D576
 >   [   1.8432366] DEBUG: x86_fpu_save_size=3D=3D11008
 >=20
 > Changing the VM's CPU to Sandy Bridge prints
 >=20
 >   [  1.8897648] DEBUG: sizeof(pcb2->pcb_savefpu)=3D=3D576
 >   [  1.8897648] DEBUG: x86_fpu_save_size=3D=3D832
 
 So, the x86_fpu_save_size is larger than we can handle and we don't
 report the problem gracefully either -- my initial attempt to put a
 more obvious panic in x86/identcpu.c when we initially determine
 x86_fpu_save_size hit the problem that it happens too early and sends
 the machine into a reboot loop.
 
 Unfortunately, simply expanding the space for pcb_savefpu is not
 trivial:
 
      80 /*
      81  * IMPORTANT NOTE: this structure, including the variable-sized FPU=
  state at
      82  * the end, must fit within one page.
      83  */
      84 struct pcb {
 ...
     103 	union savefpu	pcb_savefpu __aligned(64); /* floating point state */
     104 	/* **** DO NOT ADD ANYTHING HERE **** */
     105 };
 
 https://nxr.netbsd.org/xref/src/sys/arch/amd64/include/pcb.h?r=3D1.32#80
 
 The top comment came from this commit:
 
 > Module Name:    src
 > Committed By:   maxv
 > Date:           Tue Mar 17 17:18:49 UTC 2020
 >=20
 > Modified Files:
 >         src/sys/arch/amd64/include: cpu.h param.h pcb.h types.h
 >         src/sys/arch/x86/x86: vm_machdep.c
 >=20
 > Log Message:
 > Add a redzone between the pcb and the stack. Sent to port-amd64@.
 
 https://mail-index.netbsd.org/source-changes/2020/03/17/msg115178.html
 
 So, the immediate issue is that the memset in fpu_lwp_fork is hitting
 the guard page.  Perhaps resolving that is simply a matter of teaching
 cpu_uarea_alloc to advance by round_page(offsetof(struct pcb,
 pcb_savefpu) + x86_fpu_save_size) rather than by PAGE_SIZE when
 deciding where to set the guard page:
 
     364 void *
     365 cpu_uarea_alloc(bool system)
     366 {
     367 	vaddr_t base, va;
     368 	paddr_t pa;
     369=20
     370 	base =3D uvm_km_alloc(kernel_map, USPACE + PAGE_SIZE, 0,
     371 	    UVM_KMF_WIRED|UVM_KMF_WAITVA);
     372=20
     373 	/* Page[1] =3D RedZone */
     374 	va =3D base + PAGE_SIZE;
     375 	if (!pmap_extract(pmap_kernel(), va, &pa)) {
     376 		panic("%s: impossible, Page[1] unmapped", __func__);
     377 	}
     378 	pmap_kremove(va, PAGE_SIZE);
     379 	uvm_pagefree(PHYS_TO_VM_PAGE(pa));
 
 https://nxr.netbsd.org/xref/src/sys/arch/x86/x86/vm_machdep.c?r=3D1.46#352
 
 But the bottom comment, /* **** DO NOT ADD ANYTHING HERE **** */, is
 much older:
 
 > Module Name:    src
 > Committed By:   dsl
 > Date:           Thu Feb 20 18:19:10 UTC 2014
 >=20
 > Modified Files:
 >         src/sys/arch/amd64/amd64: genassym.cf machdep.c
 >         src/sys/arch/amd64/include: pcb.h proc.h
 >         src/sys/arch/i386/i386: locore.S machdep.c
 >         src/sys/arch/i386/include: pcb.h proc.h
 >         src/sys/arch/x86/x86: vm_machdep.c
 >         src/sys/sys: param.h
 >=20
 > Log Message:
 > Move the amd64 and i386 pcb to the bottom of the uarea, and move the
 >   kernel stack to the top.
 > Change the pcb layouts so that fpu save area is at the end and is
 >   64byte aligned ready for xsave (saving the ymm registers).
 > Welcome to 6.99.32
 
 https://mail-index.netbsd.org/source-changes/2014/02/20/msg051841.html
 
 It's not obvious to me whether the comment affects anything at hand.
 Perhaps it's just admonishing future developers not to add things
 _other than fpu state_ because the fpu state is actually variable size
 and may extend past the size of a union savefpu object.
 
 --=_tVq9BOFzAnpXS9N5Qy8XBZQoi+hFOgDk--