kern/50986: pax_aslr_mmap() may produce out of bounds results

[martin%NetBSD.org@localhost]

>Number:         50986
>Category:       kern
>Synopsis:       pax_aslr_mmap() may produce out of bounds results
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Mar 20 12:55:00 +0000 2016
>Originator:     Martin Husemann
>Release:        NetBSD 7.99.26
>Organization:
The NetBSD foundation, Inc.
>Environment:
System: NetBSD night-owl.duskware.de 7.99.26 NetBSD 7.99.26 (NIGHT-OWL) #393: Fri Mar 11 14:19:47 CET 2016 martin%night-owl.duskware.de@localhost:/usr/src/sys/arch/amd64/compile/NIGHT-OWL amd64
Architecture: x86_64
Machine: amd64
>Description:

When ASLR is enabled, sys_mmap() will invoke pax_aslr_mmap() to randomize
the mapping VA. This, however, may move the VA outside of the vmspaces valid
range (as pax_aslr_mmap is completely agnostic to any limits) and no further
checking is done (like a call to range_test).

This may lead to lock ups later in uvm_mmap() when pmap_enter() fails for
the mapping (due to an invalid address) and the process goes into an
infinite wait on "flt_pmfail2".

>How-To-Repeat:

on sparc64:
cp /bin/hostname /tmp
paxctl +A /tmp/hostname
/tmp/hostname

and be lucky if you only get a segfault.

>Fix:
Make the randomization inside ASLR aware of vmpspace limits.

Unrelated to that, the "flt_pmfail2" should probably not happen for this
kind of inherently persistent failures (depending on pmap_enter() return
value).