kern/60887: nvmm(4): x86 VMX CR0 guest/host mask does not trap CR0.PE; guest mode switches leave VMCS state stale

["david@crap.solutions via gnats" <gnats-admin%NetBSD.org@localhost>]

>Number:         60887
>Category:       kern
>Synopsis:       nvmm(4): x86 VMX CR0 guest/host mask does not trap CR0.PE; guest mode switches leave VMCS state stale
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Oct 16 04:55:00 +0000 2026
>Originator:     David Bidner
>Release:        
>Organization:
-
>Environment:
>Description:
nvmm_x86_vmx.c sets the CR0 guest/host mask to CR0_STATIC_MASK (CR0_ET | CR0_NW | CR0_CD)
and reconstructs the previous guest CR0 from VMCS_CR0_SHADOW and VMCS_GUEST_CR0 using the
same mask. A CR0 write that changes only unmasked bits therefore does not cause a VM exit,
and the hardware applies CR0.PE directly while the VMCS guest state (and entry controls)
are not reconfigured by vmx_inkernel_handle_cr0().

A guest whose AP start-up enters protected mode by writing CR0 with PE set from real mode,
keeping the cache-disable bits at the values it reads back, hits exactly this: the AP faults
back into firmware instead of entering the guest kernel. GNU Mach does this (observed on
DragonFly's nvmm port as "Waiting for AP 1"; fixed there with the same change, where 32-bit
and 64-bit guests then boot with -smp 2). The same code, masks and shadow logic are present
in NetBSD-current.
>How-To-Repeat:
Run a guest whose firmware/AP start-up writes CR0 with only CR0.PE
changing relative to the shadowed value (e.g. GNU Mach under an nvmm build on the same
code); the AP does not enter the guest kernel. Alternatively audit vmx_inkernel_handle_cr0():
no VM exit occurs for PE-only writes, so the VMCS entry state update there is unreachable
for that transition.

>Fix:
Add CR0_PE to the trapped bits: define CR0_TRAP_MASK as (CR0_STATIC_MASK | CR0_PE), write
it into VMCS_CR0_MASK, and use it for the CR0 shadow computations in
vmx_inkernel_handle_cr0(), vmx_vcpu_setstate() and vmx_vcpu_getstate(). CR0.PG is
deliberately not trapped: that would break 32-bit PAE guests, whose PDPTEs this handler
does not load.

Found while fixing the same issue in DragonFly's nvmm port; the DragonFly change
was validated with 32-bit and 64-bit guests booting to login at -smp 2. The NetBSD side is
the same audit and the same change; not runtime-tested on NetBSD.

--- a/sys/dev/nvmm/x86/nvmm_x86_vmx.c
+++ b/sys/dev/nvmm/x86/nvmm_x86_vmx.c
@@ -730,6 +730,9 @@
 #define CR0_STATIC_MASK \
 	(CR0_ET | CR0_NW | CR0_CD)
 
+#define CR0_TRAP_MASK \
+	(CR0_STATIC_MASK | CR0_PE)
+
 #define CR4_VALID \
 	(CR4_VME |			\
 	 CR4_PVI |			\
@@ -1694,8 +1697,8 @@
 		vmx_vmwrite(VMCS_ENTRY_CTLS, ctls1);
 	}
 
-	oldcr0 = (vmx_vmread(VMCS_CR0_SHADOW) & CR0_STATIC_MASK) |
-	    (vmx_vmread(VMCS_GUEST_CR0) & ~CR0_STATIC_MASK);
+	oldcr0 = (vmx_vmread(VMCS_CR0_SHADOW) & CR0_TRAP_MASK) |
+	    (vmx_vmread(VMCS_GUEST_CR0) & ~CR0_TRAP_MASK);
 	if ((oldcr0 ^ fakecr0) & CR0_TLB_FLUSH) {
 		cpudata->gtlb_want_flush = true;
 	}
@@ -2794,7 +2797,7 @@
 		 * CR0_NW and CR0_CD must be 0 in the real register.
 		 */
 		vmx_vmwrite(VMCS_CR0_SHADOW,
-		    (state->crs[NVMM_X64_CR_CR0] & CR0_STATIC_MASK) |
+		    (state->crs[NVMM_X64_CR_CR0] & CR0_TRAP_MASK) |
 		    CR0_ET);
 		vmx_vmwrite(VMCS_GUEST_CR0,
 		    (state->crs[NVMM_X64_CR_CR0] & ~CR0_STATIC_MASK) |
@@ -2945,8 +2948,8 @@
 
 	if (flags & NVMM_X64_STATE_CRS) {
 		state->crs[NVMM_X64_CR_CR0] =
-		    (vmx_vmread(VMCS_CR0_SHADOW) & CR0_STATIC_MASK) |
-		    (vmx_vmread(VMCS_GUEST_CR0) & ~CR0_STATIC_MASK);
+		    (vmx_vmread(VMCS_CR0_SHADOW) & CR0_TRAP_MASK) |
+		    (vmx_vmread(VMCS_GUEST_CR0) & ~CR0_TRAP_MASK);
 		state->crs[NVMM_X64_CR_CR2] = cpudata->gcr2;
 		state->crs[NVMM_X64_CR_CR3] = vmx_vmread(VMCS_GUEST_CR3);
 		state->crs[NVMM_X64_CR_CR4] = vmx_vmread(VMCS_GUEST_CR4);
@@ -3140,7 +3143,7 @@
 	vmx_vmwrite(VMCS_EXIT_MSR_STORE_COUNT, VMX_MSRLIST_EXIT_NMSR);
 
 	/* Set the CR0 mask. Any change of these bits causes a VMEXIT. */
-	vmx_vmwrite(VMCS_CR0_MASK, CR0_STATIC_MASK);
+	vmx_vmwrite(VMCS_CR0_MASK, CR0_TRAP_MASK);
 
 	/* Force unsupported CR4 fields to zero. */
 	vmx_vmwrite(VMCS_CR4_MASK, CR4_INVALID);