kern/50469: PaX ASLR breaks netbsd32 emulation

[Pierre Pronchery <khorben%defora.org@localhost>]

>Number:         50469
>Category:       kern
>Synopsis:       PaX ASLR breaks netbsd32 emulation
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Nov 25 00:30:00 +0000 2015
>Originator:     Pierre Pronchery
>Release:        NetBSD 7.0
>Organization:
The DeforaOS Project
>Environment:
System: NetBSD kwarx.station.defora.lan 7.0 NetBSD 7.0 (GENERIC) amd64
Architecture: x86_64
Machine: amd64
>Description:
NetBSD fails to execute 32-bit binaries when using netbsd32 emulation.
The mmap(2) syscall seems to be the culprint, and the kernel logs the
following message:

netbsd32_mmap: retval out of range: 0xfffffdb0f4aa9000
(without end-of-line character, I might add)
>How-To-Repeat:
$ tar xzf $OBJDIR/releasedir/i386/binary/sets/base.tgz
$ ./bin/sh 
Cannot map anonymous memoryCannot allocate memory: Cannot allocate memory
$ ktruss ./bin/sh
Cannot map anonymous memoryCannot allocate memory: Cannot allocate memory
 13411      1 ktruss   fcntl(0x4, 0x3, 0)          = 4194305
 13411      1 ktruss   emul(netbsd)
 13411      1 ktruss   fcntl(0x4, 0x4, 0x400001)   = 0
 13411      1 sh       execve("./bin/sh", 0x7f7fffb3c7a8, 0x7f7fffb3c7b8) JUSTRETURN
 13411      1 sh       emul(netbsd32)
 13411      1 sh       netbsd32_mmap(0, 0x8000, 0x3, 0x1002, 0xffffffffffffffff, 0, 0, 0) Err#12 ENOMEM
 13411      1 sh       netbsd32_write(0x2, 0xffffffffff7aa6b0, 0x1b) = 27
       "Cannot map anonymous memory"
 13411      1 sh       netbsd32_write(0x2, 0xffffffffff7aa6b8, 0x16) = 22
       "Cannot allocate memory"
 13411      1 sh       netbsd32_write(0x2, 0xffffffffff7aa6ac, 0x19) = 25
       ": Cannot allocate memory\n"
 13411      1 sh       netbsd32_exit(0x1)
>Fix:
The following commit works around the problem, but is wrong and should
not be applied as-is. With this change, netbsd32 emulation works again,
but mmap(2) no longer returns randomized values - thus breaking ASLR.

commit a2ac44cb29bb9c65cc7436bd5dc74eba7db54034
Author: Pierre Pronchery <khorben%EdgeBSD.org@localhost>
Date:   Tue Oct 27 00:39:12 2015 +0100

    Tentative fix for netbsd32_mmap() with PAX_ASLR

diff --git a/sys/uvm/uvm_mmap.c b/sys/uvm/uvm_mmap.c
index cbb45be..ed4684d 100644
--- a/sys/uvm/uvm_mmap.c
+++ b/sys/uvm/uvm_mmap.c
@@ -342,6 +342,10 @@ sys_mmap(struct lwp *l, const struct sys_mmap_args *uap, register_t *retval)
 	size += pageoff;			/* add offset */
 	size = (vsize_t)round_page(size);	/* round up */
 
+#ifdef PAX_ASLR
+	pax_aslr(l, &addr, orig_addr, flags);
+#endif /* PAX_ASLR */
+
 	/*
 	 * now check (MAP_FIXED) or get (!MAP_FIXED) the "addr"
 	 */
@@ -417,10 +421,6 @@ sys_mmap(struct lwp *l, const struct sys_mmap_args *uap, register_t *retval)
 	pax_mprotect(l, &prot, &maxprot);
 #endif /* PAX_MPROTECT */
 
-#ifdef PAX_ASLR
-	pax_aslr(l, &addr, orig_addr, flags);
-#endif /* PAX_ASLR */
-
 	/*
 	 * now let kernel internal function uvm_mmap do the work.
 	 */

>Unformatted:
 netbsd-7 from beginning of October 2015
 $NetBSD: uvm_mmap.c,v 1.148.4.2 2015/01/11 06:27:40 snj Exp $