kern/54395: earmv7hf binaries trigger kernel panic on aarch64

[rokuyama.rk%gmail.com@localhost]

>Number:         54395
>Category:       kern
>Synopsis:       earmv7hf binaries trigger kernel panic on aarch64
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Jul 22 04:05:00 +0000 2019
>Originator:     Rin Okuyama
>Release:        8.99.51
>Organization:
Department of Physics, Meiji University
>Environment:
NetBSD  8.99.51 NetBSD 8.99.51 (GENERIC64_UVMHIST) #0: Mon Jul 22 12:12:23 JST 2019  rin@latipes:/build/src/sys/arch/evbarm/compile/GENERIC64_UVMHIST evbarm
>Description:
KASSERT fires when running earmv7hf binary on aarch64:

# file /emul/netbsd32/bin/sh
/emul/netbsd32/bin/sh: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /libexec/ld.elf_so, for NetBSD 8.99.50, compiled for: earmv7hf, not stripped
# chroot /emul/netbsd32 su -
panic: kernel diagnostic assertion "!topdown || hint <= orig_hint" failed: file "../../../../uvm/uvm_map.c", line 2164 hint: fbefc000, orig_hint: f2c2f000
cpu1: Begin traceback...
trace fp ffffffc044447950
fp ffffffc044447970 vpanic() at ffffffc0004a4508 netbsd:vpanic+0x198
fp ffffffc0444479d0 kern_assert() at ffffffc0005e1344 netbsd:kern_assert+0x5c
fp ffffffc044447a60 uvm_map_findspace() at ffffffc0004139a4 netbsd:uvm_map_findspace+0x714
fp ffffffc044447b20 uvm_map_prepare() at ffffffc000414528 netbsd:uvm_map_prepare+0x3f0
fp ffffffc044447bd0 uvm_map() at ffffffc000418eac netbsd:uvm_map+0x64
fp ffffffc044447c60 uvm_mmap.part.0() at ffffffc00041c1a4 netbsd:uvm_mmap.part.0+0x174
fp ffffffc044447cd0 sys_mmap() at ffffffc00041ca58 netbsd:sys_mmap+0x378
fp ffffffc044447d80 netbsd32_mmap() at ffffffc0001b30c8 netbsd:netbsd32_mmap+0x40
fp ffffffc044447de0 netbsd32_syscall() at ffffffc00008020c netbsd:netbsd32_syscall+-0x127ff4
tf ffffffc044447ed0 el0_trap() at ffffffc0000744c0 netbsd:el0_trap
---- trapframe 0xffffffc044447ed0 (304 bytes) ----
    pc=00000000f2b33560,   spsr=0000000060080010 (AArch32)
   esr=00000000460000c5,    far=00000000f2b25f4c
    r0=0000000000000000,     r1=0000000000002000
    r2=0000000000000000,     r3=000000000d001002
    r4=0000000000000000,     r5=0000000000002000
    r6=0000000000000000,     r7=00000000fff21084
    r8=00000000f2b41543,     r9=0000000001000000
   r10=0000000000000000,    r11=00000000fff20ed4
   r12=00000000f2b3355c, sp=r13=00000000fff20ea8
lr=r14=00000000f2a7be9c, pc=r15=00000000f2b33560
------------------------------------------------
cpu1: End traceback...
Stopped in pid 8.1 (su) at      netbsd:cpu_Debugger+0x4:        ret

Kernel timestamps are omitted here for readability. uvmhist reads:

db{1}> show kernhist
...
1563765238.656282 uvm_map_prepare#695@1: called!
1563765238.656283 uvm_map_prepare#695@1: (map=0xffff0000bed65a10, start=0xf2c2f000, size=8192, flags=0x80010)
1563765238.656283 uvm_map_prepare#695@1:   uobj/offset 0/-1
1563765238.656284 uvm_map_findspace#665@1: called!
1563765238.656285 uvm_map_findspace#665@1: (map=0xffff0000bed65a10, hint=0xf2c2f000, len=8192, flags=0x80010)
1563765238.656286 uvm_map_lookup_entry#3431@1: called!
1563765238.656287 uvm_map_lookup_entry#3431@1: (map=0xffff0000bed65a10,addr=0xf2c2f000,ent=0xffffffc044447b18)
1563765238.656289 uvm_map_lookup_entry#3431@1: <- failed!
1563765238.656290 uvm_map_findspace#665@1: <- got it!  (result=0xfbefc000)
db{1}>

The failure began after this commit:

http://cvsweb.netbsd.org/bsdweb.cgi/src/external/bsd/jemalloc/dist/src/pages.c#rev1.5

by which jemalloc for earmv7hf became to mmap 8KB-aligned anonymous
meomory (Note that pagesizes are 4KB and 8KB for aarch64 and earmv7hf,
respectively).

Dirty printf debug reveals the scenario below:

(1) An earmv7hf binary requires 8KB-aligned anonymous memory mapped at
any address.

(2) Since native pagesie is 4KB, uvm_map() requires address lower than
0xf2c2f000, which is 4KB-aligned but not 8KB-aligned.

(3) uvm_findspace() tries to find entry of hint = 0xf2c2f000:

https://nxr.netbsd.org/xref/src/sys/uvm/uvm_map.c#1919

(4) Since entry is not available, and UVM_FLAG_FIXED is not specified,
uvm_map_space_avail() is called for hint = 0xf2c2f000:

https://nxr.netbsd.org/xref/src/sys/uvm/uvm_map.c#1942

(5) Since 8KB alignment is requires but 0xf2c2f000 (= *start, start =
&hint in uvm_findspace()) is not aligned, uvm_map_space_avail()
truncate it to 0xf2c2e000:

https://nxr.netbsd.org/xref/src/sys/uvm/uvm_map.c#1809

Then, fails to find a space and returns 0 to uvm_findspace().

(6) uvm_findspace() consider
"Still there is a chance to fit if hint > entry->end.":

https://nxr.netbsd.org/xref/src/sys/uvm/uvm_map.c#1950

However, this assumption of hint > entry->end is no longer valid,
since hint is truncated in step (5).

(7) Then, hint is updated based on a wrong assumption:

https://nxr.netbsd.org/xref/src/sys/uvm/uvm_map.c#1973

This results in KASSERT falure for hint <= orig_hint:

https://nxr.netbsd.org/xref/src/sys/uvm/uvm_map.c#2163
>How-To-Repeat:
Execute some 32bit binaries on aarch64 kernel.

"cd /usr/tests && atf-run | atf-report" in chroot environment
definitely causes panic.
>Fix:
As described above, this is not MD, but MI bug. uvm_findspace() assumes
that hint is not modified by uvm_map_space_avail(). But this is not true
when alignment is required but hint is not aligned properly.

To fix this, align hint at the beginning of uvm_findspace(). Then,
inconsistent does not occur:

http://www.netbsd.org/~rin/uvm_map_20190722.patch

With this patch, I can run earmv7hf userland on aarch64 kernel more than
a week without problems.