Re: A20 hang

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

On Sat, Sep 08, 2018 at 07:26:13AM +0100, Nick Hudson wrote:
> On 07/09/18 15:29, Manuel Bouyer wrote:
> > Hello
> > on a olimex lime2 (A20 CPU) running up to date -current, I see hard hangs
> > while building packages. Nothing on serial console and when I try to
> > enter ddb it hangs too:
> > login: Sep  7 15:51:07 chartplotter-dd su: bouyer to root on /dev/pts/1
> > ~Stopped in pid 2778.1 (g++) at  netbsd:cpu_Debugger+0x4:<hang>
> > 
> > filesystems are on SATA disk, network is used for ssh and distcc.
> > Any idea what could be wrong, or how to debug this ?
> > 
> 
> Do you have any/all of DIAGNOSTIC/DEBUG/LOCKDEBUG enabled?

OK, with all that I get more informations:

[ 5931.0429800] Mutex error: mutex_vector_enter,504: spinout

[ 5931.0429800] lock address : 0x0000000090b3c2f4 type     :               spin
[ 5931.0429800] initialized  : 0x00000000803803a0
[ 5931.0429800] shared holds :                  0 exclusive:                  1
[ 5931.0429800] shares wanted:                  0 exclusive:                  1
[ 5931.0429800] current cpu  :                  0 last held:                  1
[ 5931.0429800] current lwp  : 0x0000000091dd23c0 last held: 0x0000000091f07460
[ 5931.0429800] last locked* : 0x00000000803809c8 unlocked : 0x0000000080380c40
[ 5931.0429800] owner field  : 0x0000000000010500 wait/spin:                0/1

[ 5931.1044194] panic: LOCKDEBUG: Mutex error: mutex_vector_enter,504: spinout
[ 5931.1044194] cpu0: Begin traceback...
[ 5931.1044194] 0x9def7964: netbsd:db_panic+0x14
[ 5931.1044194] 0x9def797c: netbsd:vpanic+0x194
[ 5931.1044194] 0x9def7994: netbsd:snprintf
[ 5931.1044194] 0x9def79d4: netbsd:lockdebug_more
[ 5931.1044194] 0x9def7a0c: netbsd:lockdebug_abort+0xc0
[ 5931.1044194] 0x9def7a2c: netbsd:mutex_abort+0x34
[ 5931.1044194] 0x9def7a9c: netbsd:mutex_enter+0x614
[ 5931.1044194] 0x9def7af4: netbsd:pool_get+0x7c
[ 5931.1044194] 0x9def7b3c: netbsd:pool_cache_get_slow+0x1ec
[ 5931.1044194] 0x9def7b8c: netbsd:pool_cache_get_paddr+0x29c
[ 5931.1044194] 0x9def7bac: netbsd:m_clget+0x34
[ 5931.1044194] 0x9def7c0c: netbsd:dwc_gmac_intr+0x3b4
[ 5931.1044194] 0x9def7c24: netbsd:gic_fdt_intr+0x28
[ 5931.1044194] 0x9def7c4c: netbsd:pic_dispatch+0x58
[ 5931.1044194] 0x9def7cb4: netbsd:armgic_irq_handler+0x100
[ 5931.1044194] 0x9def7d34: netbsd:irq_entry+0x68
[ 5931.1044194] 0x9def7d7c: netbsd:pool_cache_get_slow+0x1e0
[ 5931.1044194] 0x9def7dcc: netbsd:pool_cache_get_paddr+0x29c
[ 5931.1044194] 0x9def7dec: netbsd:m_clget+0x34
[ 5931.1044194] 0x9def7e84: netbsd:sosend+0x51c
[ 5931.1044194] 0x9def7eac: netbsd:soo_write+0x3c
[ 5931.1044194] 0x9def7f04: netbsd:dofilewrite+0x98
[ 5931.1044194] 0x9def7f34: netbsd:sys_write+0x70
[ 5931.1044194] 0x9def7fac: netbsd:syscall+0x18c
[ 5931.1044194] cpu0: End traceback...
Stopped in pid 15903.1 (gcc) at netbsd:cpu_Debugger+0x4:        bx      r14
db{0}> tr/a 0x0000000091f07460
trace: pid 29212 lid 1 at 0x9e033dbc
0x9e033dbc: 800
--- tf 0x9e033fb0 ---
db{0}> tr/a 0x0000000091dd23c0
trace: pid 15903 lid 1 at 0x9def7dbc
0x9def7dbc: 1800
--- tf 0x9def7fb0 ---


15903>   1 7   0         0           91dd23c0                gcc
29212>   1 7   1         0           91f07460                gcc

Unfortunably I couldn't get the trace of the second cpu:
db{0}> mach cpu 1
kdb_trap: switching to cpu1
Stopped in pid 29212.1 (gcc) at netbsd:_kernel_lock+0x190:<hang>

It's not out of ram:
Current UVM status:
  pagesize=8192 (0x2000), pagemask=0x1fff, pageshift=13, ncolors=1
  128078 VM pages: 53437 active, 86 inactive, 1053 wired, 60512 free

Nor reached a pool limit:
POOL CACHE mbpl: size 256, align 8, ioff 0, roflags 0x00000000
        alloc 0x8065616c                                     
        minitems 16, minpages 1, maxpages 4294967295, npages 14
        itemsperpage 32, nitems 35, nout 413, hardlimit 4294967295
        nget 413, nfail 0, nput 0                  
        npagealloc 14, npagefree 0, hiwat 14, nidle 1
        cpu layer hits 669534 misses 4917
        cache layer hits 4504 misses 413
        cache layer entry uncontended 4887 contended 30
        cache layer empty groups 4 full groups 0
POOL CACHE mclpl: size 2048, align 8, ioff 0, roflags 0x00000000
        alloc 0x8065616c
        minitems 8, minpages 2, maxpages 4096, npages 91
        itemsperpage 4, nitems 7, nout 357, hardlimit 16384
        nget 357, nfail 0, nput 0
        npagealloc 91, npagefree 0, hiwat 91, nidle 1
        cpu layer hits 183460 misses 1633
        cache layer hits 1274 misses 359
        cache layer entry uncontended 1633 contended 0
        cache layer empty groups 3 full groups 0

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference
--