Re: evbarm hang

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

Overnight lockdebug did find something:
login: [ 1908.3939406] Mutex error: mutex_vector_enter,504: spinout
                                                             
[ 1908.3939406] lock address : 0x0000000090b79074 type     :               spin
[ 1908.3939406] initialized  : 0x000000008041601c     
[ 1908.3939406] shared holds :                  0 exclusive:                  1
[ 1908.3939406] shares wanted:                  0 exclusive:                  1
[ 1908.3939406] current cpu  :                  0 last held:                  1
[ 1908.3939406] current lwp  : 0x0000000091fc3760 last held: 0x0000000091fc26e0
[ 1908.3939406] last locked* : 0x0000000080416668 unlocked : 0x00000000804169e8
[ 1908.3939406] owner field  : 0x0000000000010500 wait/spin:                0/1
                                                 
[ 1908.4626458] panic: LOCKDEBUG: Mutex error: mutex_vector_enter,504: spinout
[ 1908.4626458] cpu0: Begin traceback...         
[ 1908.4626458] 0x9e4a192c: netbsd:db_panic+0x14
[ 1908.4626458] 0x9e4a1944: netbsd:vpanic+0x194
[ 1908.4626458] 0x9e4a195c: netbsd:snprintf                                
[ 1908.4626458] 0x9e4a199c: netbsd:lockdebug_more
[ 1908.4626458] 0x9e4a19d4: netbsd:lockdebug_abort+0xc0
[ 1908.4626458] 0x9e4a19f4: netbsd:mutex_abort+0x34
[ 1908.4626458] 0x9e4a1a64: netbsd:mutex_enter+0x580
[ 1908.4626458] 0x9e4a1abc: netbsd:pool_get+0x70
[ 1908.4626458] 0x9e4a1b0c: netbsd:pool_cache_get_slow+0x1f4
[ 1908.4626458] 0x9e4a1b5c: netbsd:pool_cache_get_paddr+0x288
[ 1908.4626458] 0x9e4a1b7c: netbsd:m_clget+0x34
[ 1908.4626458] 0x9e4a1bdc: netbsd:dwc_gmac_intr+0x194
[ 1908.4626458] 0x9e4a1bf4: netbsd:gic_fdt_intr+0x2c
[ 1908.4626458] 0x9e4a1c1c: netbsd:pic_dispatch+0x110
[ 1908.4626458] 0x9e4a1c7c: netbsd:armgic_irq_handler+0xf4
[ 1908.4626458] 0x9e4a1db4: netbsd:irq_entry+0x68
[ 1908.4626458] 0x9e4a1dec: netbsd:tcp_send_wrapper+0x9c
[ 1908.4626458] 0x9e4a1e84: netbsd:sosend+0x6fc
[ 1908.4626458] 0x9e4a1eac: netbsd:soo_write+0x3c
[ 1908.4626458] 0x9e4a1f04: netbsd:dofilewrite+0x7c
[ 1908.4626458] 0x9e4a1f34: netbsd:sys_write+0x5c
[ 1908.4626458] 0x9e4a1fac: netbsd:syscall+0x12c
[ 1908.4626458] cpu0: End traceback...

db{0}> tr/a 0x0000000091fc3760
trace: pid 21381 lid 1 at 0x9e4a1dbc
0x9e4a1dbc: 914a1730
Bad frame pointer: 0x801f884c
db{0}> tr/a 0x0000000091fc26e0
trace: pid 21532 lid 1 at 0x9cf97dbc
0x9cf97dbc: 91dfcc18

0x0000000080416668 is
	mutex_enter(&pp->pr_lock);
in pool_get().

PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
21562    1 2   0         0           92d177c0                gcc
21381>   1 7   0         0           91fc3760                gcc
21532>   1 7   1         0           91fc26e0                gcc
23218    1 2   1         0           92968f60                gcc


owner field  : 0x0000000000010500
if I decode this properly, mtx_ipl is 5 and mtx_lock is 1.
5 would be IPL_VM on evbarm.

db{0}> mach cpu 1
kdb_trap: switching to cpu1
Stopped in pid 21532.1 (gcc) at netbsd:_kernel_lock+0x19c:<hang>

So cpu 1 is indeed running the LWP hodling the spin lock, and it looks
like it's itself waiting for a mutex.
Now I have to find why "mach cpu 1" hangs, and how to avoid it ...

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference
--