kern/41877: PaX ASLR reduces available stack size

[Manuel.Bouyer%lip6.fr@localhost]

>Number:         41877
>Category:       kern
>Synopsis:       PaX ASLR reduces available stack size
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Aug 12 12:05:00 +0000 2009
>Originator:     Manuel Bouyer
>Release:        NetBSD 5.0_STABLE
>Organization:
>Environment:
System: NetBSD disco 5.0_STABLE NetBSD 5.0_STABLE (DISCO) #3: Fri Aug 7 
08:59:42 MEST 2009 
bouyer@disco:/home/bouyer/src-5/src/sys/arch/i386/compile/obj/DISCO i386
Architecture: i386
Machine: i386
>Description:
        Enabling PaX ASLR on a server, some programs started to
        randomly dump core. gdb revealed that they did get a
        segmentation fault trying to write to the stack, and indeed
        the address at %esp is not mapped. The stack size is the default
        for i386 (2Mb).
        So I wrote this test program:
#define ALLOCASZ (2048 * 1024 - 4096 * 3) /* largest size that doesn't segv */
main()
{
int i;
char *c = alloca(ALLOCASZ);
printf("0x%lx 0x%lx 0x%lx\n", (long)(&i), (long)(&c[0]), (long)(&c[ALLOCASZ]));
c[0]=1;
c[ALLOCASZ]=1;
exit(0);
}
        Without ASLR it executes as:
# ./test
0xbf7fee4c 0xbf601e30 0xbf7fee30
        (with of course repeteable results).
        With ASLR (and kernel build with PAX_DEBUG):
# paxctl +A ./test
# ./test
stack 0xbf800000 d=0xce1000 0xbeb1f000
applying to 0xbb7f8000 orig_addr=0x0 f=1002
result 0xb714c000
applying to 0xbb7ff000 orig_addr=0x0 f=1
result 0xb7153000
applying to 0xbb710000 orig_addr=0x0 f=2
result 0xb7064000
not applying to 0xb7134000 orig_addr=0xb7134000 f=12
not applying to 0xb713c000 orig_addr=0xb713c000 f=1012
applying to 0xbb700000 orig_addr=0x0 f=14001002
result 0xb7054000
0xbeb1d70c 0xbe9206f0 0xbeb1d6f0
# ./test
stack 0xbf800000 d=0xa1a000 0xbede6000
applying to 0xbb7f8000 orig_addr=0x0 f=1002
result 0xb1e10000
applying to 0xbb7ff000 orig_addr=0x0 f=1
result 0xb1e17000
applying to 0xbb710000 orig_addr=0x0 f=2
result 0xb1d28000
not applying to 0xb1df8000 orig_addr=0xb1df8000 f=12
not applying to 0xb1e00000 orig_addr=0xb1e00000 f=1012
applying to 0xbb700000 orig_addr=0x0 f=14001002
result 0xb1d18000
0xbede4b6c 0xbebe7b50 0xbede4b50
[...]
# ./test
stack 0xbf800000 d=0x971000 0xbee8f000
applying to 0xbb7f8000 orig_addr=0x0 f=1002
result 0xba4ac000
applying to 0xbb7ff000 orig_addr=0x0 f=1
result 0xba4b3000
applying to 0xbb710000 orig_addr=0x0 f=2
result 0xba3c4000
not applying to 0xba494000 orig_addr=0xba494000 f=12
not applying to 0xba49c000 orig_addr=0xba49c000 f=1012
pid 57 (test), uid 0: exited on signal 11 (core dumped)
Memory fault (core dumped)
# gdb test test.core
Core was generated by `test'.
Program terminated with signal 11, Segmentation fault.
#0  0xbb7f6dc3 in _rtld_find_symdef () from /usr/libexec/ld.elf_so
(gdb) info reg
eax            0xa77    2679
ecx            0xba4ae400       -1169497088
edx            0xba3d9b90       -1170367600
ebx            0xbb7fee60       -1149243808
esp            0xbec8effc       0xbec8effc
ebp            0xbec8f4e8       0xbec8f4e8
esi            0xdd828  907304
edi            0xba3bc000       -1170489344
eip            0xbb7f6dc3       0xbb7f6dc3 <_rtld_find_symdef+7>
eflags         0x10296  [ PF AF SF IF RF ]
cs             0x17     23
ss             0x1f     31
ds             0x1f     31
es             0x1f     31
fs             0x1f     31
gs             0x1f     31
(gdb) disas 0xbb7f6dc3
Dump of assembler code for function _rtld_find_symdef:
0xbb7f6dbc <_rtld_find_symdef+0>:       push   %ebp
0xbb7f6dbd <_rtld_find_symdef+1>:       push   %edi
0xbb7f6dbe <_rtld_find_symdef+2>:       push   %esi
0xbb7f6dbf <_rtld_find_symdef+3>:       push   %ebx
0xbb7f6dc0 <_rtld_find_symdef+4>:       sub    $0x1c,%esp
0xbb7f6dc3 <_rtld_find_symdef+7>:       call   0xbb7f6dc8 <_rtld_find_symdef+12>

So, even though pax_aslr_stack() always returns a pointer aligned on
page size, ALSR causes something to consume a variable amount of
stack size before main() is called, which causes the executed program to
have a smaller stack size than expected and eventually core dump.

>How-To-Repeat:
        See above
>Fix:
        unknow