Re: kern/57580: evbarm/earmv7hf RPI2 scheduler/cpu stall

[Frank Kardel <kardel%netbsd.org@localhost>]

The following reply was made to PR kern/57580; it has been noted by GNATS.

From: Frank Kardel <kardel%netbsd.org@localhost>
To: gnats-bugs%netbsd.org@localhost
Cc: 
Subject: Re: kern/57580: evbarm/earmv7hf RPI2 scheduler/cpu stall
Date: Fri, 25 Aug 2023 11:27:09 +0200

 I now ran -current with option HEARTBEAT for 635730 seconds.
 Then HEARTBEAT checks triggered:
 
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0017400] cpu0: found cpu1 heart 
 stopped beating after 16 seconds
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0068278] panic: cpu1[29860 perl]: 
 heart stopped beating
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0125393] cpu1: Begin traceback...
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0163590] 0xbb29b71c: 
 netbsd:db_panic+0x14
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0208674] 0xbb29b73c: 
 netbsd:vpanic+0x114
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0252987] 0xbb29b754: netbsd:panic+0x24
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0295481] 0xbb29b774: 
 netbsd:defibrillator+0x54
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0360377] 0xbb29b7bc: 
 netbsd:ipi_msg_cpu_handler+0x9c
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0416531] 0xbb29b804: 
 netbsd:ipi_cpu_handler+0xdc
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0467708] 0xbb29b814: 
 netbsd:pic_ipi_generic+0x10
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0518910] 0xbb29b83c: 
 netbsd:bcm2836mp_ipi_handler+0xbc
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0575454] 0xbb29b88c: 
 netbsd:pic_dispatch+0x54
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0624438] 0xbb29b914: 
 netbsd:pic_do_pending_ints+0x434
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0679535] 0xbb29ba64: 
 netbsd:irq_entry+0x94
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0725627] cpu1: End traceback...
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0764817] Mutex error: 
 mutex_vector_enter,515: assertion failed: !cpu_intr_p()
 Aug 24 19:42:22 rpi /netbsd:
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0840143] lock address : 917cf07c
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0877541] current cpu 
 :                  1
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0923754] current lwp  : 
 0x00000000918dd300
 Aug 24 19:42:22 rpi /netbsd: [ 635730.0969675] owner field  : 
 000000000000000000 wait/spin:                0/0
 Aug 24 19:42:22 rpi /netbsd:
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1043366] Skipping crash dump on 
 recursive panic
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1093675] panic: lock error: Mutex: 
 mutex_vector_enter,515: assertion failed: !cpu_intr_p(): lock 0x917cf07c 
 cpu 1 lwp 0x918dd300
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1214484] cpu1: Begin traceback...
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1252763] 0xbb29b414: 
 netbsd:db_panic+0x14
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1297859] 0xbb29b434: 
 netbsd:vpanic+0x114
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1341996] 0xbb29b44c: netbsd:panic+0x24
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1384551] 0xbb29b514: 
 netbsd:lockdebug_abort+0xe8
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1443758] 0xbb29b52c: 
 netbsd:mutex_abort+0x30
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1492993] 0xbb29b59c: 
 netbsd:mutex_enter+0x4a0
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1542221] 0xbb29b5d4: 
 netbsd:dk_open+0x3c
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1586363] 0xbb29b604: 
 netbsd:ldopen+0x78
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1632125] 0xbb29b634: 
 netbsd:dk_size+0x64
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1676452] 0xbb29b674: 
 netbsd:bdev_size+0xac
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1722500] 0xbb29b684: 
 netbsd:cpu_dumpconf+0x28
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1771197] 0xbb29b6d4: 
 netbsd:dodumpsys+0x1fc
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1817926] 0xbb29b704: 
 netbsd:dumpsys+0x18
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1862678] 0xbb29b71c: 
 netbsd:kern_reboot+0x80
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1910014] 0xbb29b73c: 
 netbsd:vpanic+0x120
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1954316] 0xbb29b754: netbsd:panic+0x24
 Aug 24 19:42:22 rpi /netbsd: [ 635730.1996777] 0xbb29b774: 
 netbsd:defibrillator+0x54
 ... stack continues as in first stack dump
 
 So current status is
      It happens on evbarm RPI2.
      It is not too frequent
      It happens with NetBSD-10.0_BETA
      It happens with -current (seems to take longer)
      It looks like perl always the victim that is set runnable and then 
 cpu1 gets a silent heart attack