port-i386/57078: Occasional crash in pmap/uvm on load, on 2xVIA Nehemiah based board

[vezhlys%gmail.com@localhost]

>Number:         57078
>Category:       port-i386
>Synopsis:       Occasional crash in pmap/uvm on load, on 2xVIA Nehemiah based board
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    port-i386-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Oct 31 11:20:00 +0000 2022
>Originator:     Andrius V
>Release:        
>Organization:
>Environment:
NetBSD agraphic-nas 9.99.104 NetBSD 9.99.104 (VT-310DP) #5: Sun Oct 30 13:46:50 EET 2022  andriusv@agraphic-nas:/home/andriusv/netbsd-src/sys/arch/i386/compile/VT-310DP i386
>Description:
I have occasional crashes under load in pmap/uvm code code (mainly while building pkgsrc packages) on my old VIA Epia VT-310DP board. It has 2xVIA Nehemiah based CPUs (VIA Eden-N 1GHz). I also suspect it is  easier to trigger while building on xterm (with Xorg running) compared to terminal.

Backtrace can be different between crashes or the system may just hang or just reboot without producing any backtrace. It may be some assertion failure in pmap or crash in mutex/pmap code. Will copy few last backtraces below. Since hardware is old, I partially suspect it may be hardware related, however memtest+ tests are passing without errors in multiple iterations, HDD seems to be fine too. In case system is idle/lightly loaded, no crashes are observed in days of uptime. Crashes may not occur for long enough to finish hours of builds without issues as well, but at the same time, they can occur quite constantly in a short period time. Unfortunately, I don't have a well defined steps to reproduce it all the time.

latest backtrace (9.99.104, build on October 30):
uvm_fault(0xc126d1e0, 0xafec1000, 1) -> 0xe
fatal page fault in supervisor mode
trap type 6 code 0 eip 0xc0a387cd cs 0x8 eflags 0x10286 cr2 0xafec1000 ilevel 0 esp 0xc312f4b8
curlwp 0xcd1ba940 pid 23993 lid 23993 lowest kstack 0xebdf12c0
panic: trap
cpu1: Begin traceback...
vpanic(c0ee69ce,ebdf3c18,ebdf3cd4,c012d882,c0ee69ce,ebdf3ce0,ebdf3ce0,5db9,ebdf12c0,10286) at netbsd:vpanic+0x196
panic(c0ee69ce,ebdf3ce0,ebdf3ce0,5db9,ebdf12c0,10286,afec1000,0,c312f4b8,c126d1e0) at netbsd:panic+0x18
trap() at netbsd:trap+0xd7c
--- trap (number 6) ---
mutex_oncpu(c09d9934,0,c6057c7c,ffffffff,ebdf3d70,c09f30dc,cd21edc4,c6057c7c,0,0) at netbsd:mutex_oncpu+0x19
mutex_vector_enter(c301c49c,0,0,cd2760c0,c488bc14,c488bc10,ebdf3dd0,c60a4bc8,c301c470,cb20a580) at netbsd:mutex_vector_enter+0xbf
uvm_pagefree(c301c470,2,c120ad80,ccc148c0,ffffffff,c60a4bc8,0,cd21edc4,ebdf3e20,c09d6ae1) at netbsd:uvm_pagefree+0xa3
uvm_anfree(c60a4bc8,1,c3f4e400,0,0,ccc86a40,0,ccc86a40,ebdf3e48,c09e8d66) at netbsd:uvm_anfree+0x7d
amap_wipeout(cd21edc4,0,1,0,c5e38f00,c40b58bc,2000000,ebdf3e9c,ebdf3e70,c09eb546) at netbsd:amap_wipeout+0x93
uvm_unmap_detach(c5e38f00,0,bfbff000,ebdf3e64,2000000,c5e38f00,cd200680,cd1ba940,ebdf3f1c,c0a26161) at netbsd:uvm_unmap_detach+0x50
uvmspace_free(c40b58bc,0,ebdf3e9c,ebdf3ee4,c0ecb808,c9626000,0,cd2008c8,ebdf3ec0,0) at netbsd:uvmspace_free+0xf7
exit1(cd1ba940,0,0,cd1ba940,ebdf3fa8,ebdf3f9c,c040a5bb,cd1ba940,ebdf3f68,ebdf3f60) at netbsd:exit1+0x1ae
sys_exit(cd1ba940,ebdf3f68,ebdf3f60,cd1de648,1,1,ebdf3f60,ebdf3f68,0,0) at netbsd:sys_exit+0x3b
syscall() at netbsd:syscall+0x17c
--- syscall (number 1) ---
afdf3f67:
cpu1: End traceback...

(uvm_amap.c:790 -> kern/kern_mutex.c:558 -> kern/kern_mutex.c:428

the one before it (9.99.101 build on October 21):
[ 47494.463552] panic: kernel diagnostic assertion "pp->pp_pte.pte_ptp == ptp" failed: file "/home/andriusv/netbsd-src/sys/arch/x86/x86/pmap.c", line 3200 
panic: kernel diagnostic assertion "pp->pp_pte.pte_ptp == ptp" failed: file "/home/andriusv/netbsd-src/sys/arch/x86/x86/pmap.c", line 3200 
cpu1: Begin traceback...
vpanic(c0ede394,e3f92d5c,e3f92e48,c0407350,c0ede394,c0ede2e7,c0f2e3e4,c0f2d170,c80,0) at netbsd:vpanic+0x196
kern_assert(c0ede394,c0ede2e7,c0f2e3e4,c0f2d170,c80,0,1,0,c66062d0,c6606380) at netbsd:kern_assert+0x23
pmap_remove_all(c66062c0,c6848c00,c41820cc,c0a4a1ec,60b7ac2c,cca2f680,cca2f680,c6848c00,e3f92f1c,c0a20a71) at netbsd:pmap_remove_all+0x69a
uvmspace_free(cca39414,0,e3f92e9c,e3f92ee4,c0ec4408,cb00a600,0,cca2f8c8,e3f92ec0,0) at netbsd:uvmspace_free+0x3c
exit1(c6848c00,0,0,c6848c00,e3f92fa8,e3f92f9c,c0409feb,c6848c00,e3f92f68,e3f92f60) at netbsd:exit1+0x1ae
sys_exit(c6848c00,e3f92f68,e3f92f60,c665d788,0,1,e3f92f60,e3f92f68,0,0) at netbsd:sys_exit+0x3b
syscall() at netbsd:syscall+0x17c
syscall (number 1) ---
add40f67:
cpu1: End traceback..

Back in June (9.99.97):
panic: kernel diagnostic assertion "(new_pp)->pp_lock.mtx_ipl._ipl == IPL_VM" failed: file "/usr/src/sys/arch/x86/x86/pmap.c", line 4886 bad pmap_page 0xc2d00d68
cpu0: Begin traceback...
vpanic(c1311d74,dc8f9ba4,dc8f9c14,c04a97e5,c1311d74,c12b1a5b,c13128cc,c1311b54,1316,c2d00d68) at netbsd:vpanic+0x196
kern_assert(c1311d74,c12b1a5b,c13128cc,c1311b54,1316,c2d00d68,0,dc8f9d0c,c1296868,c4466a00) at netbsd:kern_assert+0x23
pmap_enter_ma(c4b7e880,ac771000,427da000,427da000,5,20,0,0,20,ac771000) at netbsd:pmap_enter_ma+0xc46
pmap_enter_default(c4b7e880,ac771000,427da000,5,20,3,1,0,40,dc8f9ca0) at netbsd:pmap_enter_default+0x43
uvm_fault_internal(c4ce7348,ac774000,1,0,3,cd6f1058,0,0,ffffffff,c4ce7348) at netbsd:uvm_fault_internal+0x17b8
trap() at netbsd:trap+0x661
--- trap (number 6) ---
copyinstr(ac774480,dc8f9ecc,c0dbcac7,cd2d9740,0,106,106,cd2d9740,dc8f9f9c,c012280c) at netbsd:copyinstr+0x32
do_sys_readlinkat(bfb303e7,400,dc8f9f60,0,0,3a,dc8f9f60,dc8f9f68,0,0) at netbsd:do_sys_readlinkat+0x1b
syscall() at netbsd:syscall+0x17c
--- syscall (number 58) ---
ac5fe727:
cpu0: End traceback...

In the beginning of 2021 it crashed on this line (don't have backtrace anymore): LIST_INSERT_HEAD(&pp->pp_pvlist, pve, pve_list); pmap.c#2317 currently.

Latest dmesg:
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017,
    2018, 2019, 2020, 2021, 2022
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 9.99.104 (VT-310DP) #5: Sun Oct 30 13:46:50 EET 2022
	andriusv@agraphic-nas:/home/andriusv/netbsd-src/sys/arch/i386/compile/VT-310DP
total memory = 1981 MB
avail memory = 1925 MB
timecounter: Timecounters tick every 10.000 msec
Kernelized RAIDframe activated
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
mainbus0 (root)
ACPI: RSDP 0x00000000000F8940 000014 (v00 CN400 )
ACPI: RSDT 0x000000007BDE3040 00002C (v01 CN400  AWRDACPI 42302E31 AWRD 00000000)
ACPI: FACP 0x000000007BDE30C0 000074 (v01 CN400  AWRDACPI 42302E31 AWRD 00000000)
ACPI: DSDT 0x000000007BDE3180 004D8C (v01 CN400  AWRDACPI 00001000 MSFT 0100000E)
ACPI: FACS 0x000000007BDE0000 000040
ACPI: APIC 0x000000007BDE7F80 000068 (v01 CN400  AWRDACPI 42302E31 AWRD 00000000)
ACPI: 1 ACPI AML tables successfully acquired and loaded
ioapic0 at mainbus0 apid 2: pa 0xfec00000, version 0x3, 24 pins
cpu0 at mainbus0 apid 0
cpu0: Use cpuid to serialize rdtsc
cpu0: VIA Nehemiah, id 0x69a
cpu0: node 0, package 0, core 0, smt 0
cpu1 at mainbus0 apid 1
cpu1: VIA Nehemiah, id 0x69a
cpu1: node 0, package 0, core 0, smt 0
acpi0 at mainbus0: Intel ACPICA 20220331
acpi0: X/RSDT: OemId <CN400 ,AWRDACPI,42302e31>, AslId <AWRD,00000000>
acpi0: SCI interrupting at int 9
acpi0: fixed power button present
timecounter: Timecounter "ACPI-Fast" frequency 3579545 Hz quality 1000
acpibut0 at acpi0 (PWRB, PNP0C0C): ACPI Power Button
MEM (PNP0C01) at acpi0 not configured
attimer1 at acpi0 (TMR, PNP0100): io 0x40-0x43 irq 0
pcppi1 at acpi0 (SPKR, PNP0800): io 0x61
midi0 at pcppi1: PC speaker
sysbeep0 at pcppi1
COPR (PNP0C04) at acpi0 not configured
UAR1 (PNP0501) at acpi0 not configured
UAR2 (PNP0501) at acpi0 not configured
acpifan0 at acpi0 (FAN, PNP0C0B): ACPI Fan
acpitz0 at acpi0 (THRM): cpu0
acpitz0: active cooling level 0: 70.0C
acpitz0: levels: critical 75.0 C, passive 70.0 C
apm0 at acpi0: Power Management spec V1.2
ACPI: Enabled 1 GPEs in block 00 to 0F
attimer1: attached to pcppi1
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0: VIA Technologies product 0259 (rev. 0x00)
agp0 at pchb0: aperture at 0xe8000000, size 0x8000000
pchb1 at pci0 dev 0 function 1: VIA Technologies product 1259 (rev. 0x00)
pchb2 at pci0 dev 0 function 2: VIA Technologies product 2259 (rev. 0x00)
pchb3 at pci0 dev 0 function 3: VIA Technologies product 3259 (rev. 0x00)
pchb4 at pci0 dev 0 function 4: VIA Technologies product 4259 (rev. 0x00)
pchb5 at pci0 dev 0 function 7: VIA Technologies product 7259 (rev. 0x00)
ppb0 at pci0 dev 1 function 0: VIA Technologies VT8377CE CPU-AGP Bridge (rev. 0x00)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled
vga0 at pci1 dev 0 function 0: VIA Technologies product 3118 (rev. 0x02)
wsdisplay0 at vga0 kbdmux 1: console (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
viadrmums0 at vga0
[drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[drm] No driver support for vblank timestamp query.
[drm] Initialized via 2.11.1 20070202 for viadrmums0 on minor 0
fxp0 at pci0 dev 9 function 0: i82551 Ethernet (rev. 0x10)
fxp0: interrupting at ioapic0 pin 17
fxp0: Ethernet address
inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 4
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vge0 at pci0 dev 10 function 0: VIA VT612X Gigabit Ethernet (rev. 0x11)
vge0: interrupting at ioapic0 pin 18
vge0: Ethernet address
ciphy0 at vge0 phy 1: Cicada CIS8201 10/100/1000TX PHY, rev. 2
ciphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
viaide0 at pci0 dev 15 function 0: VIA Technologies VT8237 SATA Controller (rev. 0x80)
viaide0: bus-master DMA support present
viaide0: primary channel wired to native-PCI mode
viaide0: using ioapic0 pin 20 for native-PCI interrupt
atabus0 at viaide0 channel 0
viaide0: secondary channel wired to native-PCI mode
atabus1 at viaide0 channel 1
viaide1 at pci0 dev 15 function 1
viaide1: VIA Technologies VT8237 ATA133 controller
viaide1: bus-master DMA support present
viaide1: primary channel configured to compatibility mode
viaide1: primary channel interrupting at ioapic0 pin 14
atabus2 at viaide1 channel 0
viaide1: secondary channel configured to compatibility mode
viaide1: secondary channel ignored (disabled)
uhci0 at pci0 dev 16 function 0: VIA Technologies VT83C572 USB Controller (rev. 0x81)
uhci0: interrupting at ioapic0 pin 21
usb0 at uhci0: USB revision 1.0
uhci1 at pci0 dev 16 function 1: VIA Technologies VT83C572 USB Controller (rev. 0x81)
uhci1: interrupting at ioapic0 pin 21
usb1 at uhci1: USB revision 1.0
uhci2 at pci0 dev 16 function 2: VIA Technologies VT83C572 USB Controller (rev. 0x81)
uhci2: interrupting at ioapic0 pin 21
usb2 at uhci2: USB revision 1.0
uhci3 at pci0 dev 16 function 3: VIA Technologies VT83C572 USB Controller (rev. 0x81)
uhci3: interrupting at ioapic0 pin 21
usb3 at uhci3: USB revision 1.0
ehci0 at pci0 dev 16 function 4: VIA Technologies VT8237 EHCI USB Controller (rev. 0x86)
ehci0: 32-bit DMA
ehci0: interrupting at ioapic0 pin 21
ehci0: dropped intr workaround enabled
ehci0: BIOS refuses to give up ownership, using force
ehci0: EHCI version 1.0
ehci0: 4 companion controllers, 2 ports each: uhci0 uhci1 uhci2 uhci3
ehci0: Using DMA subregion for control data structures
usb4 at ehci0: USB revision 2.0
viapcib0 at pci0 dev 17 function 0: VIA Technologies VT8237 PCI-LPC Bridge (rev. 0x00)
viapcib0: SMBus found at 0x500 (revision 0x0)
iic0 at viapcib0: I2C bus
auvia0 at pci0 dev 17 function 5: VIA Technologies VT8237 AC'97 Audio (rev 0x60)
auvia0: interrupting at ioapic0 pin 22
auvia0: ac97: VIA Technologies VT1616i codec; headphone, 18 bit DAC, 18 bit ADC, KS Waves 3D
auvia0: ac97: ext id 0x1c1<LDAC,SDAC,CDAC,VRA>
audio0 at auvia0: playback, capture, full duplex, independent
audio0: slinear_le:16 2ch 48000Hz, blk 1920 bytes (10ms) for playback
audio0: slinear_le:16 2ch 48000Hz, blk 1920 bytes (10ms) for recording
vr0 at pci0 dev 18 function 0: VIA Technologies VT6102 (Rhine II) 10/100 Ethernet (rev. 0x78)
vr0: interrupting at ioapic0 pin 23
vr0: Ethernet address
ukphy0 at vr0 phy 1: Data Track Technology VT6103 10/100 PHY (OUI 0x0002c6, model 0x0032), rev. 10
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
isa0 at viapcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, 16-byte FIFO
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, 16-byte FIFO
pckbc0 at isa0 port 0x60-0x64
acpicpu0 at cpu0: ACPI CPU
acpicpu0: C1: HLT, lat   0 us, pow     0 mW
padlock0 at cpu0: VIA PadLock
acpicpu1 at cpu1: ACPI CPU
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
padlock0: RNG ACE
IPsec: Initialized Security Association Processing.
aes: VIA ACE
chacha: Portable C ChaCha
uhub0 at usb0: NetBSD (0x0000) UHCI root hub (0x0000), class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhub1 at usb1: NetBSD (0x0000) UHCI root hub (0x0000), class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhub2 at usb2: NetBSD (0x0000) UHCI root hub (0x0000), class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
uhub3 at usb3: NetBSD (0x0000) UHCI root hub (0x0000), class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
uhub4 at usb4: NetBSD (0x0000) EHCI root hub (0x0000), class 9/0, rev 2.00/1.00, addr 1
uhub4: 8 ports with 8 removable, self powered
viaide0 port 1: device present, speed: 1.5Gb/s
ehci0: handing over low speed device on port 2 to uhci0
wd0 at atabus1 drive 0
wd0: <SAMSUNG HD160JJ>
wd0: drive supports 16-sector PIO transfers, LBA48 addressing
wd0: 149 GB, 310098 cyl, 16 head, 63 sec, 512 bytes/sect x 312579695 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 7, NCQ (32 tags)
wd0(viaide0:1:0): using PIO mode 4, Ultra-DMA mode 6 (Ultra/133) (using DMA)
atapibus0 at atabus2: 2 targets
cd0 at atapibus0 drive 0: <TSSTcorp CDDVDW SH-S202H, , SB02> cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 4 (Ultra/66)
cd0(viaide1:0:0): using PIO mode 4, Ultra-DMA mode 4 (Ultra/66) (using DMA)
umass0 at uhub4 port 3 configuration 1 interface 0
umass0: SanDisk (0x0781) Cruzer Blade (0x5567), rev 2.00/1.27, addr 2
umass0: using SCSI over Bulk-Only
scsibus0 at umass0: 2 targets, 1 lun per target
sd0 at scsibus0 target 0 lun 0: <SanDisk, Cruzer Blade, 1.27> disk removable
sd0: 7485 MB, 15208 cyl, 16 head, 63 sec, 512 bytes/sect x 15330304 sectors
uhidev0 at uhub0 port 2 configuration 1 interface 0
uhidev0: CASUE (0x2a7a) CASUE USB Keyboard (0x9a18), rev 1.10/0.01, addr 2, iclass 3/1
ukbd0 at uhidev0
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub0 port 2 configuration 1 interface 1
uhidev1: CASUE (0x2a7a) CASUE USB Keyboard (0x9a18), rev 1.10/0.01, addr 2, iclass 3/0
uhidev1: 2 report ids
uhid0 at uhidev1 reportid 1: input=1, output=0, feature=0
uhid1 at uhidev1 reportid 2: input=3, output=0, feature=0
ehci0: handing over full speed device on port 4 to uhci1
swwdog0: software watchdog initialized
boot device: wd0
root on wd0a dumps on wd0b
root file system type: ffs
kern.module.path=/stand/i386/9.99.104/modules
uhidev2 at uhub1 port 2 configuration 1 interface 0
uhidev2: Logitech (0x046d) USB Receiver (0xc52b), rev 2.00/12.10, addr 2, iclass 3/1
ukbd1 at uhidev2
wskbd1 at ukbd1 mux 1
wskbd1: connecting to wsdisplay0
uhidev3 at uhub1 port 2 configuration 1 interface 1
uhidev3: Logitech (0x046d) USB Receiver (0xc52b), rev 2.00/12.10, addr 2, iclass 3/1
uhidev3: 8 report ids
ums0 at uhidev3 reportid 2: 16 buttons, W and Z dirs
wsmouse0 at ums0 mux 0
uhid2 at uhidev3 reportid 3: input=4, output=0, feature=0
uhid3 at uhidev3 reportid 4: input=1, output=0, feature=0
uhid4 at uhidev3 reportid 8: input=1, output=0, feature=0
uhidev4 at uhub1 port 2 configuration 1 interface 2
uhidev4: Logitech (0x046d) USB Receiver (0xc52b), rev 2.00/12.10, addr 2, iclass 3/0
uhidev4: 33 report ids
uhid5 at uhidev4 reportid 16: input=6, output=6, feature=0
uhid6 at uhidev4 reportid 17: input=19, output=19, feature=0
uhid7 at uhidev4 reportid 32: input=14, output=14, feature=0
uhid8 at uhidev4 reportid 33: input=31, output=31, feature=0
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)


Regards,
Andrius V
>How-To-Repeat:
build pkgsrc packages, observe crashes/hangs.
>Fix: