port-evbarm/54557: Booting fails using U-Boot TPL on ROCK64

[simon%simonsouth.net@localhost]

>Number:         54557
>Category:       port-evbarm
>Synopsis:       Booting fails using U-Boot TPL on ROCK64
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    port-evbarm-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Sep 18 11:25:00 +0000 2019
>Originator:     Simon South
>Release:        CURRENT (9.99.11) 2019-09-15 12:10 UTC
>Organization:
>Environment:
NetBSD arm64 9.99.11 NetBSD 9.99.11 (GENERIC64) #8: Mon Sep 16 17:09:14 EDT 2019
>Description:
Booting NetBSD on a PINE64 ROCK64 fails using a U-Boot image built
with U-Boot's own tertiary program loader (TPL). The same NetBSD image
boots fine using the binary-only TPL supplied by Rockchip (at
https://github.com/rockchip-linux/rkbin).

(Note this is using the latest revision of U-Boot in its git
repository; the most recent release isn't new enough to include the
code for the TPL.)

The exact failure varies. At times the kernel fails quickly with

    >> NetBSD/evbarm EFI Boot (aarch64), Revision 1.11 (Wed Sep 18 00:35:33 UTC 2019) (from NetBSD 9.99.11)
    Press return to boot now, any other key for boot prompt
    booting netbsd - starting in 0 seconds.
    6099992+2730496+1985652+1823764 [701640+490607]=0xec5af0
    fdt_open_into failed: -9
    resetting ...

The "-9" error code corresponds to FDT_ERR_BADMAGIC
(sys/external/bsd/libfdt/dist/libfdt.h:97), indicating the FDT image
is corrupt or can't be read properly. And in fact U-Boot itself will
sometimes (but not always) report

    libfdt fdt_check_header(): FDT_ERR_BADMAGIC

during startup, suggesting the problem may lie with it and not NetBSD.

Other times the kernel will start to boot but fail a moment later with
either

    panic: LIST_* forw 0xffff00000936a7a8 /usr/src/sys/uvm/uvm_page.c:897

or

    panic: kernel diagnostic assertion "ucpu == VM_FREE_PAGE_TO_CPU(pg)" failed: file "/usr/src/sys/uvm/uvm_page.c", line 852

I've pasted a complete sample of the output below.

All of this suggests to me U-Boot's TPL may be failing to set up the
RK3328 to read from memory reliably. That, or there is some extra
setup work the kernel is expected to do that is currently missing from
NetBSD.

----------

    >> NetBSD/evbarm EFI Boot (aarch64), Revision 1.11 (Thu Sep 12 13:07:08 UTC 2019) (from NetBSD 9.99.11)
    Press return to boot now, any other key for boot prompt
    booting netbsd - starting in 0 seconds.
    6098264+2729992+1985140+1824276 [701496+490387]=0xec5988
    [   1.0000000] NetBSD/evbarm (fdt) booting ...
    [   1.0000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    [   1.0000000]     2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017,
    [   1.0000000]     2018, 2019 The NetBSD Foundation, Inc.  All rights reserved.
    [   1.0000000] Copyright (c) 1982, 1986, 1989, 1991, 1993
    [   1.0000000]     The Regents of the University of California.  All rights reserved.

    [   1.0000000] NetBSD 9.99.11 (GENERIC64) #4: Thu Sep 12 09:24:05 EDT 2019
    [   1.0000000]  ssouth@laptop:/usr/src/sys/arch/evbarm/compile/obj/GENERIC64
    [   1.0000000] total memory = 4064 MB
    [   1.0000000] avail memory = 3918 MB
    [   1.0000000] panic: LIST_* forw 0xffff00000936a7a8 /usr/src/sys/uvm/uvm_page.c:897
    [   1.0000000] cpu0: Begin traceback...
    [   1.0000000] trace fp ffffffc000bec560
    [   1.0000000] fp ffffffc000bec580 vpanic() at ffffffc000491df0 netbsd:vpanic+0x198
    [   1.0000000] fp ffffffc000bec5e0 panic() at ffffffc000491ec0 netbsd:panic+0x48
    [   1.0000000] fp ffffffc000bec670 uvm_pagealloc_pgfl() at ffffffc00040f024 netbsd:uvm_pagealloc_pgfl+0x664
    [   1.0000000] fp ffffffc000bec6d0 uvm_pagealloc_strat() at ffffffc00040fc34 netbsd:uvm_pagealloc_strat+0x18c
    [   1.0000000] fp ffffffc000bec760 uvm_km_kmem_alloc() at ffffffc000403740 netbsd:uvm_km_kmem_alloc+0x60
    [   1.0000000] fp ffffffc000bec7d0 pool_page_alloc() at ffffffc00048a9f8 netbsd:pool_page_alloc+0x28
    [   1.0000000] fp ffffffc000bec7f0 pool_grow() at ffffffc00048cbc4 netbsd:pool_grow+0x35c
    [   1.0000000] fp ffffffc000bec860 pool_get() at ffffffc00048bf18 netbsd:pool_get+0xa0
    [   1.0000000] fp ffffffc000bec8c0 pool_cache_get_slow() at ffffffc00048e1d4 netbsd:pool_cache_get_slow+0x1dc
    [   1.0000000] fp ffffffc000bec910 pool_cache_get_paddr() at ffffffc00048fb18 netbsd:pool_cache_get_paddr+0x258
    [   1.0000000] fp ffffffc000bec970 kmem_intr_alloc() at ffffffc000485328 netbsd:kmem_intr_alloc+0x60
    [   1.0000000] fp ffffffc000bec9a0 kern_malloc() at ffffffc000441a94 netbsd:kern_malloc+0x4c
    [   1.0000000] fp ffffffc000bec9d0 sysctl_alloc() at ffffffc0005d0b3c netbsd:sysctl_alloc+0x70
    [   1.0000000] fp ffffffc000beca00 sysctl_create() at ffffffc0004651d8 netbsd:sysctl_create+0x800
    [   1.0000000] fp ffffffc000becba0 sysctl_createv() at ffffffc0004666ec netbsd:sysctl_createv+0x234
    [   1.0000000] fp ffffffc000becdb0 sysctl_security_pax_setup() at ffffffc000447ce4 netbsd:sysctl_security_pax_setup+0x384
    [   1.0000000] fp ffffffc000bece20 sysctl_init() at ffffffc000465928 netbsd:sysctl_init+0x68
    [   1.0000000] fp ffffffc000bece50 main() at ffffffc0005d03f0 netbsd:main+0xf0
    [   1.0000000] fp 0000000000000000 aarch64_start() at ffffffc00000183c netbsd:aarch64_start+0x103c
    [   1.0000000] cpu0: End traceback...
    Stopped in pid 0.1 (system) at  netbsd:cpu_Debugger+0x4:        ret
    db{0}>

At other times:

    ...
    [   1.0000000] total memory = 4064 MB
    [   1.0000000] avail memory = 3918 MB
    [   1.0000000] panic: kernel diagnostic assertion "ucpu == VM_FREE_PAGE_TO_CPU(pg)" failed: file "/usr/src/sys/uvm/uvm_page.c", line 852
    [   1.0000000] cpu0: Begin traceback...
    [   1.0000000] trace fp ffffffc000bec730
    [   1.0000000] fp ffffffc000bec750 vpanic() at ffffffc000491df0 netbsd:vpanic+0x198
    [   1.0000000] fp ffffffc000bec7b0 kern_assert() at ffffffc0005cd824 netbsd:kern_assert+0x5c
    [   1.0000000] fp ffffffc000bec840 uvm_pagealloc_pgfl() at ffffffc00040ef88 netbsd:uvm_pagealloc_pgfl+0x5c8
    [   1.0000000] fp ffffffc000bec8a0 uvm_pagealloc_strat() at ffffffc00040fc34 netbsd:uvm_pagealloc_strat+0x18c
    ...

>How-To-Repeat:
1. Build a NetBSD evbarm64 image as usual and write this to a microSD
   card.

2. Build a U-Boot image for the ROCK64 that incorporates U-Boot's own
   TPL (done by default in the current revision of the code from git,
   following the instructions in doc/README.rockchip) and write this
   to the microSD card as well.

3. Place the microSD card in the ROCK64 and attempt to boot it.

>Fix:
The workaround at present is to build U-Boot using Rockchip's
(closed-source) TPL, using a sequence of commands like the following
(as described in doc/README.rockchip in older versions of U-Boot):

    make distclean; rm -f idbloader.img
    make rock64-rk3328_defconfig all u-boot.itb
    tools/mkimage -n rk3328 -T rksd -d ~/src/rockchip-linux/rkbin/bin/rk33/rk3328_ddr_333MHz_v1.16.bin idbloader.img
    cat spl/u-boot-spl.bin >> idbloader.img