port-hpcarm/38591: port-hpcarm kernels unbootable after merge of matt-armv6 branch

[rafal%netbsd.org@localhost]

>Number:         38591
>Category:       port-hpcarm
>Synopsis:       port-hpcarm kernels unbootable after merge of matt-armv6 branch
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    port-hpcarm-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue May 06 00:55:00 +0000 2008
>Originator:     rafal%netbsd.org@localhost
>Release:        NetBSD 4.99.62
>Organization:
TNF
>Environment:
System: NetBSD j720 4.99.62 NetBSD 4.99.62 (J720) #0: Mon May  5 15:57:08 EDT 
2008 
rafal%fearless-vampire-killer.waterside.net@localhost:/extra/netbsd-src/hpcarm-bughunt.head/src/sys/arch/hpcarm/compile/J720
 hpcarm
Architecture: arm
Machine: hpcarm
>Description:
        Boot a stock kernel (e.g. JORNADA720), watch it hang after autoconfig.
        Boot a LOCKDEBUG/DIAGNOSTIC kernel and watch it die early in the
        boot with a uninitialized lock panic.

        Here's a boot log from non-LOCKDEBUG / non-DIAGNOSTIC kernel with a
        bit of extra debug goo and some of the issues noted below fixed; for
        one it disables interrupts in hpcarm's initarm(), else it would not
        have gotten that far either.  However, it still has bad interrupt
        masks:

--------HPCBOOT--------
FileManager: FAT
hpcboot build number: 15
HP Jornada 720 (US/UK) (cpu=0x0c108000 machine=0x02c20201)
[progress] 2
[0] 0xc0000000 size 0x08000000
[1] 0xc8000000 size 0x08000000
[2] 0xd0000000 size 0x08000000
[3] 0xd8000000 size 0x08000000
_WIN32_WCE = 400
GetVersionEx
Windows CE 3.0
GetSystemInfo:
wProcessorArchitecture      0x5
wProcessorLevel             0x4
wProcessorRevision          0x9
dwPageSize                  0x1000
dwAllocationGranularity     0x00010000
dwProcessorType             0xa11
Display: 640x240 16bpp
Reg0 :6901b119
Reg1 :c002327f
Reg2 :c002327f
Reg3 :00000001
Reg5 :c0023007
Reg6 :0004f014
Reg13:0a000000
Reg14:cc8443ad
CPSR :400000df
[progress] 3
[progress] 4
open file "\Storage Card\netbsd-4.99.61.gz"(1833675 byte).
[progress] 5
Loader: ELF
[progress] 6
file size: 0x2f13d4+0xbf94+[ksyms: header 0x5d0, symtab 0x5f1a0, strtab 0x32610 
= 0x91d80]+[extra: 0x4fb0] = 0x394d30 bytes
address translation table 928 pages. (0x1d00 bytes)
allocated 928 page. mapped 928 page.
[progress] 7
2nd bootloader vaddr=0x0048f000 paddr=0xc1d24000
2nd bootloader copy done.
[progress] 8
seg[0] paddr 0xc0040000 file size 0x2f13d4 mem size 0x2f13d4
        ->load 0xc0040000+0x002f13d4=0xc03313d4 ofs=0x00008000+0x2f13d4
seg[1] paddr 0xc03393e0 file size 0xbf94 mem size 0x6d144
        ->load 0xc03393e0+0x0006d144=0xc03a6524 ofs=0x002f93e0+0xbf94
        ->zero 0xc0345374+0x000611b0=0xc03a6524
ksyms
        ->load 0xc03a6524+0x000005d0=0xc03a6af4
        ->load 0xc03a6af4+0x0005f1a0=0xc0405c94 ofs=0x00322378+0x5f1a0
        ->load 0xc0405c94+0x00032610=0xc04382a4 ofs=0x00381518+0x32610
[progress] 9
load link 918, zero clear link 1
kernel entry address: 0xc0040000
framebuffer: 640x240 type=5 linebytes=1280 addr=0x48200000
console = 2
[progress] 10
sp for bootloader = c1d22000 + 00001000 = c1d23000
kernsize=0x366524
Allocating page tables
IRQ stack: p0xc0010000 v0xc0010000
ABT stack: p0xc0011000 v0xc0011000
UND stack: p0xc0012000 v0xc0012000
SVC stack: p0xc0013000 v0xc0013000
Creating L1 page table
Mapping kernel
pmap_map_chunk: pa=0xc0040000 va=0xc0040000 size=0x400000 resid=0x400000 
prot=0x3 cache=1
LLLLLLLLLLLLSSSLLLL
Constructing L2 page tables
pmap_map_chunk: pa=0xc0010000 va=0xc0010000 size=0x1000 resid=0x1000 prot=0x3 
cache=1
P
pmap_map_chunk: pa=0xc0011000 va=0xc0011000 size=0x1000 resid=0x1000 prot=0x3 
cache=1
P
pmap_map_chunk: pa=0xc0012000 va=0xc0012000 size=0x1000 resid=0x1000 prot=0x3 
cache=1
P
pmap_map_chunk: pa=0xc0013000 va=0xc0013000 size=0x2000 resid=0x2000 prot=0x3 
cache=1
PP
pmap_map_chunk: pa=0xc0000000 va=0xc0000000 size=0x4000 resid=0x4000 prot=0x3 
cache=2
PPPP
pmap_map_chunk: pa=0xc0000000 va=0xc0000000 size=0x10000 resid=0x10000 prot=0x3 
cache=2
L
devmap: 80050000 -> 80050023 @ d000d000
pmap_map_chunk: pa=0x80050000 va=0xd000d000 size=0x24 resid=0x1000 prot=0x3 
cache=0
P
pmap_map_chunk: pa=0xe0000000 va=0xc0018000 size=0x8000 resid=0x8000 prot=0x3 
cache=1
PPPPPPPP
done.
init subsystems: stacks vectors c028018c c0280778 c027fd88
undefined freemempos=c0021000
MMU enabled. control=c000107d
kernsize=0x400000 (including 0x91d80 symbols)
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 4.99.61 (J720) #5: Sun May  4 18:16:17 EDT 2008
        
rafal%fearless-vampire-killer.waterside.net@localhost:/extra/netbsd-src/hpcarm-bughunt.61/src/sys/arch/hpcarm/compile/J720
total memory = 32768 KB
avail memory = 27164 KB
timecounter: Timecounters tick every 10.000 msec
mainbus0 (root)
cpu0 at mainbus0: SA-1110 step B-5 (SA-1 core)
cpu0: DC enabled IC enabled WB enabled LABT
cpu0: 16KB/32B 32-way Instruction cache
cpu0: 8KB/32B 32-way write-back Data cache
saip0 at mainbus0
saost0 at saip0 addr 0x90000000-0x9000001f
saost0: SA-11x0 OS Timer
sacom0 at saip0 addr 0x80050000-0x80050023 intr 17
sacom0: SA-11x0 UART3
sacom0: console
sacc0 at saip0 addr 0x40000000-0x40001fff
sacc0: SA-1111 rev 1.1
sacpcic0 at sacc0
pcmcia0 at sacpcic0
pcmcia1 at sacpcic0
sed0 at saip0
sed0: Epson SED1356
sed0: framebuffer address: 0x48200000
sed0: WARNING: powerhook_establish is deprecated
hpcfb0 at sed0
hpcfb0: WARNING: powerhook_establish is deprecated
wsdisplay0 at hpcfb0 kbdmux 1
wsmux1: connecting to wsdisplay0
hpcfb: 640x240 pixels, 65536 colors, 80x24 chars
hpcfb: 640x240 pixels, 65536 colors, 80x24 chars
wsdisplay0: screen 0-1 added (std, vt100 emulation)
j720ssp0 at saip0 addr 0x80070000-0x800700ff
j720kbd0 at j720ssp0
hpckbd0 at j720kbd0
wskbd0 at hpckbd0 mux 1
wskbd0: connecting to wsdisplay0
j720tp0 at j720ssp0
wsmouse0 at j720tp0 mux 0
wskbd at j720tp0 not configured
j720lcd0 at j720ssp0: brightness 38, contrast 135
j720pwr0 at j720ssp0
hpcapm0 at j720pwr0: pseudo power management module
apmdev0 at hpcapm0: Power Management spec V1.2
ipl_none=00000000 ipl_bio=00020203 ipl_net=00020203 ipl_tty=00020203
ipl_vm=00020203 ipl_audio=00020203 ipl_imp=00020002 
ipl_high=00020002ipl_serial=00020002
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
clock: hz=100 stathz=64
timecounter: Timecounter "saost_count" frequency 3686400 Hz quality 100
sacpcic0: card present
IPsec: Initialized Security Association Processing.
ne0 at pcmcia0 function 0: <Ethernet, Adapter, 2.0>
ne0: Ethernet address 00:e0:98:04:2d:ca
ne0: WARNING: powerhook_establish is deprecated
sacpcic0: card present
wdc0 at pcmcia1 function 0: <SanDisk, SDP, 5/3 0.6>
wdc0: i/o mapped mode
atabus0 at wdc0 channel 0
wd0 at atabus0 drive 0: <SanDisk SDCFB-2048>
wd0: drive supports 4-sector PIO transfers, LBA addressing
wd0: 1953 MB, 3970 cyl, 16 head, 63 sec, 512 bytes/sect x 4001760 sectors
wd0: drive supports PIO mode 4
boot device: wd0
root on wd0a dumps on wd0b
root file system type: ffs
WARNING: no TOD clock present
WARNING: using filesystem time
WARNING: CHECK AND RESET THE DATE!
Enter pathname of shell or RETURN for /bin/sh: 
# fsck -p
/dev/rwd0a: file system is clean; not checking
# fsck -fp
/dev/rwd0a: 22535 files, 
*HANG*

        Here's a boot log from LOCKDEBUG / DIAGNOSTIC kernel from sources
        earlier today (a totally clean tree):

--------HPCBOOT--------
FileManager: FAT
hpcboot build number: 15
HP Jornada 720 (US/UK) (cpu=0x0c108000 machine=0x02c20201)
[progress] 2
[0] 0xc0000000 size 0x08000000
[1] 0xc8000000 size 0x08000000
[2] 0xd0000000 size 0x08000000
[3] 0xd8000000 size 0x08000000
_WIN32_WCE = 400
GetVersionEx
Windows CE 3.0
GetSystemInfo:
wProcessorArchitecture      0x5
wProcessorLevel             0x4
wProcessorRevision          0x9
dwPageSize                  0x1000
dwAllocationGranularity     0x00010000
dwProcessorType             0xa11
Display: 640x240 16bpp
Reg0 :6901b119
Reg1 :c002327f
Reg2 :c002327f
Reg3 :00000001
Reg5 :c0023007
Reg6 :000277f0
Reg13:0a000000
Reg14:ccc443ad
CPSR :400000df
[progress] 3
[progress] 4
open file "\Storage Card\netbsd.gz"(1961945 byte).
[progress] 5
Loader: ELF
[progress] 6
file size: 0x33177c+0xc0c8+[ksyms: header 0x5d0, symtab 0x63680, strtab 0x32cc3 
= 0x96913]+[extra: 0x4fb0] = 0x3da8c3 bytes
address translation table 1008 pages. (0x1f80 bytes)
allocated 1008 page. mapped 1008 page.
[progress] 7
2nd bootloader vaddr=0x004df000 paddr=0xc1d27000
2nd bootloader copy done.
[progress] 8
seg[0] paddr 0xc0040000 file size 0x33177c mem size 0x33177c
        ->load 0xc0040000+0x0033177c=0xc037177c ofs=0x00008000+0x33177c
seg[1] paddr 0xc0379780 file size 0xc0c8 mem size 0x7745c
        ->load 0xc0379780+0x0007745c=0xc03f0bdc ofs=0x00339780+0xc0c8
        ->zero 0xc0385848+0x0006b394=0xc03f0bdc
ksyms
        ->load 0xc03f0bdc+0x000005d0=0xc03f11ac
        ->load 0xc03f11ac+0x00063680=0xc045482c ofs=0x00362c88+0x63680
        ->load 0xc045482c+0x00032cc3=0xc04874ef ofs=0x003c6308+0x32cc3
[progress] 9
load link 986, zero clear link 1
kernel entry address: 0xc0040000
framebuffer: 640x240 type=5 linebytes=1280 addr=0x48200000
console = 2
[progress] 10
sp for bootloader = c1d25000 + 00001000 = c1d26000
kernsize=0x3b0bdc
Allocating page tables
IRQ stack: p0xc0010000 v0xc0010000
ABT stack: p0xc0011000 v0xc0011000
UND stack: p0xc0012000 v0xc0012000
SVC stack: p0xc0013000 v0xc0013000
Creating L1 page table
Mapping kernel
pmap_map_chunk: pa=0xc0040000 va=0xc0040000 size=0x44c000 resid=0x44c000 
prot=0x3 cache=1
LLLLLLLLLLLLSSSLLLLLLLLPPPPPPPPPPPP
Constructing L2 page tables
pmap_map_chunk: pa=0xc0010000 va=0xc0010000 size=0x1000 resid=0x1000 prot=0x3 
cache=1
P
pmap_map_chunk: pa=0xc0011000 va=0xc0011000 size=0x1000 resid=0x1000 prot=0x3 
cache=1
P
pmap_map_chunk: pa=0xc0012000 va=0xc0012000 size=0x1000 resid=0x1000 prot=0x3 
cache=1
P
pmap_map_chunk: pa=0xc0013000 va=0xc0013000 size=0x2000 resid=0x2000 prot=0x3 
cache=1
PP
pmap_map_chunk: pa=0xc0000000 va=0xc0000000 size=0x4000 resid=0x4000 prot=0x3 
cache=2
PPPP
pmap_map_chunk: pa=0xc0000000 va=0xc0000000 size=0x10000 resid=0x10000 prot=0x3 
cache=2
L
devmap: 80050000 -> 80050023 @ d000d000
pmap_map_chunk: pa=0x80050000 va=0xd000d000 size=0x24 resid=0x1000 prot=0x3 
cache=0
P
pmap_map_chunk: pa=0xe0000000 va=0xc0018000 size=0x8000 resid=0x8000 prot=0x3 
cache=1
PPPPPPPP
done.
init subsystems: stacks vectors c02a9fec c02aa5fc c02a9bf0
undefined freemempos=c0021000
MMU enabled. control=c000107d
kernsize=0x44c000 (including 0x96913 symbols)
panic: lockdebug_lookup: uninitialized lock (lock=0x14)
Begin traceback...
0xc0014bd0
        scp=0xc0014bd0 rlv=0xc0014ba0 (0xc0014ba0)
        rsp=0xc0014ba8 rfp=0xc022a0fc
0xe1a0c00d
        scp=0xe1a0c00d rlv=0xc03455a0 (netbsd:__kernassert+0x39ad4)
        rsp=0xc03bfbe0 rfp=0xc0382218
*HANG*

        It looks like there are a few problems:

        (1) Interrupts are not disabled early in startup, so interrupts
            can sneak in earlier than expected (I suspect that's the cause
            of the LOCKDEBUG assertion firing above).

        (2) Interrupt masks don't make sense -- IPL_HIGH doesn't block all
            interrupts, IPL_NONE does, etc.

        (3) It looks like interrupts are enabled in arch/arm/sa11x0/sa11x0.c
            but interrupt masks aren't set up beforehand (maybe a non-issue).

        (4) Mutex init of the SA-1111 PCMCIA controller mutex should be moved
            to arch/arm/sa11x0/sa11x1_pcic.c (sacpcic_attach_common) from
            sapcic_kthread_create in arch/arm/sa11x0/sa11xx_pcic.c (otherwise
            we init it twice, per DIAGNOSTIC/LOCKDEBUG). 

        Others?  This is a catch-all bug since I'm not sure what else might
        have been broken by the transition to using the arm/arm32 interrupt
        code (on top of pre-existing broken-ness that seemed to somehow not
        surface).

>How-To-Repeat:
        Try to boot the JORNADA720 kernel on a Jornada 720; try to boot a
        LOCKDEBUG / DIAGNOSTIC version of said kernel.

#
# Customized (debug-added) kernel config file for the Jornada 720
#
#       $NetBSD: J720,v 1.3 2006/10/02 03:28:30 chs Exp $
#

include         "arch/hpcarm/conf/JORNADA720"

options         DEBUG_BEFOREMMU
#options        INTR_DEBUG
options         DIAGNOSTIC
options         LOCKDEBUG
options         VERBOSE_INIT_ARM
options         DDB_ONPANIC=2

>Fix:
        None yet.. I'm working on it as time allows.  The big issue to sort
        out is fixing the interrupt masks; the rest is hopefully just small
        fries after that, but at least the interrupts-enabled-early issue
        made it harder than necessary to even get there.

>Unformatted:
                NetBSD 4.99.62 (J720) #0: Mon May  5 15:57:08 EDT 2008
                Clean source from May 5th, ~14:00 EDT