port-i386/55363: NetBSD/i386 PAE has issues

[john%ziaspace.com@localhost]

>Number:         55363
>Category:       port-i386
>Synopsis:       NetBSD/i386 PAE has issues with memory and NVMe
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    port-i386-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Jun 10 22:05:00 +0000 2020
>Originator:     John Klos
>Release:        NetBSD 9.0_STABLE
>Organization:
	
>Environment:
	
	
System: NetBSD doozer.zia.io 9.0_STABLE NetBSD 9.0_STABLE (GENERIC_PAE) #0: Fri May 22 04:28:49 UTC 2020  john%helena.zia.io@localhost:/usr/obj-i386/sys/arch/i386/compile/GENERIC_PAE i386
Architecture: i386
Machine: i386
>Description:
	
NetBSD 9.0, NetBSD 9 as of 10-June-2020 has two noticeable problems:

On a Ryzen 3600 system with 64 gigs and a PAE kernel, only 3.5 gigs is usable:

[     1.000000] total memory = 3527 MB
[     1.000000] avail memory = 3445 MB

When memory is removed so the system has just 32 gigs, all 32 gigs is usable:

[     1.000000] total memory = 32699 MB
[     1.000000] avail memory = 32112 MB

With 64 gigs (3.5 usable), a PCIe NVMe card is usable as expected:

[     1.046309] nvme0 at pci3 dev 0 function 0: vendor 8086 product 0953 (rev. 0x01)
[     1.046309] nvme0: NVMe 1.0
[     1.046309] nvme0: for admin queue interrupting at msix0 vec 0
[     1.046309] nvme0: INTEL SSDPEDMW400G4, firmware 8EV101H0, serial CVCQ542200BT400AGN
[     1.046309] nvme0: for io queue 1 interrupting at msix0 vec 1 affinity to cpu0
[     1.046309] nvme0: for io queue 2 interrupting at msix0 vec 2 affinity to cpu1
[     1.046309] nvme0: for io queue 3 interrupting at msix0 vec 3 affinity to cpu2
[     1.046309] nvme0: for io queue 4 interrupting at msix0 vec 4 affinity to cpu3
[     1.046309] nvme0: for io queue 5 interrupting at msix0 vec 5 affinity to cpu4
[     1.046309] nvme0: for io queue 6 interrupting at msix0 vec 6 affinity to cpu5
[     1.046309] nvme0: for io queue 7 interrupting at msix0 vec 7 affinity to cpu6
[     1.046309] nvme0: for io queue 8 interrupting at msix0 vec 8 affinity to cpu7
[     1.046309] nvme0: for io queue 9 interrupting at msix0 vec 9 affinity to cpu8
[     1.046309] nvme0: for io queue 10 interrupting at msix0 vec 10 affinity to cpu9
[     1.046309] nvme0: for io queue 11 interrupting at msix0 vec 11 affinity to cpu10
[     1.046309] nvme0: for io queue 12 interrupting at msix0 vec 12 affinity to cpu11
[     1.046309] ld0 at nvme0 nsid 1

However, with 32 gigs (32 usable), the NVMe card stops working:

[     1.038776] nvme0 at pci3 dev 0 function 0: vendor 8086 product 0953 (rev. 0x01)
[     1.038776] nvme0: NVMe 1.0
[     1.038776] nvme0: for admin queue interrupting at msix0 vec 0
[     1.038776] nvme0: INTEL SSDPEDMW400G4, firmware 8EV101H0, serial CVCQ542200BT400AGN
[     1.038776] nvme0: autoconfiguration error: unable to allocate ccbs
[     1.038776] nvme0: autoconfiguration error: unable to allocate io queue

>How-To-Repeat:
	
>Fix:
	

>Unformatted: