re: port-evbarm/57614: Kernel panic rebooting a Pine64 RockPro64 with netbsd10.0_beta; successful after panic reboot.

[joelp%sdf.org@localhost]

The following reply was made to PR port-evbarm/57614; it has been noted by GNATS.

From: joelp%sdf.org@localhost
To: "matthew green" <mrg%eterna.com.au@localhost>
Cc: gnats-bugs%netbsd.org@localhost, joelp%sdf.org@localhost, port-evbarm-maintainer%netbsd.org@localhost,
        gnats-admin%netbsd.org@localhost, netbsd-bugs%netbsd.org@localhost
Subject: re: port-evbarm/57614: Kernel panic rebooting a Pine64 RockPro64
 with netbsd10.0_beta; successful after panic reboot.
Date: Tue, 12 Sep 2023 23:19:15 -0000

 I have an APC UPS attached to a USB2 port (and running apcupsd) when I see
 the panic on boot.
 
 I do not see a panic when I attach the UPS to the TypeA USB3 port. I
 rebooted several times to be sure -- no panic.
 
 I've been unable to get any more information from the debugger. I first
 tried without DDB and DDB_ONPANIC=1, and about every 2nd or 3rd time, the
 debugger would break, but 'show kernhist usbhist' did not give any more
 details.
 
 Looking at the docs, I tried increasing the value of hw.usb.debug, etc,
 sysctl settings but that didn't help either.
 
 I then added DDB and DDB_ONPANIC=1 options and that kernel never landed in
 the debugger after several manual reboot attempts. It does still panic and
 reboots (see below).
 
 Also, per the docs, it mentions I can use boot netbsd -d to start in ddb;
 however I'm never able to continue running under ddb:
 
 [   1.0000000] total memory = 3909 MB
 [   1.0000000] avail memory = 3768 MB
 [   1.0000000] entropy: ready
 [   1.0000000] Entering DDB...
 Stopped in pid 0.0 (system) at  netbsd:cpu_Debugger+0xc:        ldp    
 x29, x30
 , [sp],#16
 db{0}> continue
 
 After that continue, there are several more panics, then it freezes. I'm
 probably missing something here, sorry, my kernel debugging background is
 Windows unfortunately :)
 
 Is there any more config or kernel options I can include? Is there
 something more I need to do after boot netbsd -d?
 
 Here's the panic that won't break into debugger with DDB set:
 
 Waiting for duplicate address detection to finish...
 Starting dhcpcd.
 [   5.0229996] panic: kernel debugging assertion
 "usb_valid_block_p(f->ufd_block, &usb_blk_fraglist)" failed: file
 "/lab/src-netbsd10/src/sys/dev/usb/usb_mem.c", line 303 usb_allocmem: usb
 frag 0xffffc001553cf080: unknown block pointer 0x0
 [   5.0229996] cpu2: Begin traceback...
 [   5.0229996] trace fp ffffc001552cca60
 [   5.0229996] fp ffffc001552cca90 vpanic() at ffffc000005b8768
 netbsd:vpanic+0x178
 [   5.0229996] fp ffffc001552ccaf0 kern_assert() at ffffc00000851588
 netbsd:kern_assert+0x58
 [   5.0330044] fp ffffc001552ccb80 usb_allocmem() at ffffc00000197a4c
 netbsd:usb_allocmem+0x5ac
 [   5.0330044] fp ffffc001552ccc00 ohci_open() at ffffc0000023f668
 netbsd:ohci_open+0x198
 [   5.0330044] fp ffffc001552ccc50 usbd_setup_pipe_flags() at
 ffffc00000193670 netbsd:usbd_setup_pipe_flags+0x1a0
 [   5.0330044] fp ffffc001552cccc0 usbd_new_device() at ffffc000001956a4
 netbsd:usbd_new_device+0x744
 [   5.0430010] fp ffffc001552ccd60 uhub_explore() at ffffc0000019b200
 netbsd:uhub_explore+0x2d0
 [   5.0430010] fp ffffc001552cce20 usb_discover() at ffffc000001846a0
 netbsd:usb_discover+0x70
 [   5.0430010] fp ffffc001552cce70 usb_event_thread() at ffffc00000184988
 netbsd:usb_event_thread+0xc8
 [   5.0430010] tf ffffc001552cced0 el0_trap() at ffffc000000b6ff0
 netbsd:el1_trap_exit+0x68
 [   5.0530002] cpu2: End traceback...
 [   5.0530002] rebooting...
 
 Here are my changes to the kernel config (copied from GENERIC64):
 
 #options    DIAGNOSTIC  # internal consistency checks
 options     DDB
 options     DDB_ONPANIC=1
 options     DEBUG
 options     USB_DEBUG
 options     UHUB_DEBUG
 options     XHCI_DEBUG
 options     EHCI_DEBUG
 #options    LOCKDEBUG
 
 Thanks - Joel
 
 > do you have devices plugged in to the usb2 (ehci) port?  if possible,
 > can you use the usb3 (xhci) port only and see if the problem occurs?
 >
 > i've seen this problem (at least the syncmem version of it) irregularly
 > for a while on rk3399 (pinebookpro mostly, not very often on rockpro64),
 > but i haven't yet figured out what causes it.
 >
 > i don't think there's a race condition related fsck, while it normally
 > happens at reboot, i've seen it after being up, and then later inserting
 > a usb device and it crashed.
 >
 > (i thought i'd logged a PR about it, but i can't find it.)
 >
 > can you build a kernel with USB_DEBUG, UHUB_DEBUG, XHCI_DEBUG, and
 > EHCI_DEBUG debug options enabled, and with set these sysctl values
 > in /etc sysctl.conf:
 >
 >    hw.usb.debug=1
 >    hw.uhub.debug=1
 >    hw.xhci.debug=1
 >    hw.ehci.debug=1
 >
 > and then when it crashes, you can either collect a crash dump or if
 > the serial console works properly, simply run this:
 >
 >   db{1}> show kernhist usbhist
 >
 > which will output a bunch debugging events from before the crash.
 >
 > thanks
 >
 >
 > .mrg.
 >