Subject: kern/1439: stacksize rlimit value not multiple of pagesize causes execve to fail.
To: None <gnats-bugs@NetBSD.ORG>
From: None <>
List: netbsd-bugs
Date: 09/05/1995 18:33:58
>Number:         1439
>Category:       kern
>Synopsis:       stacksize rlimit value not multiple of pagesize causes execve to fail.
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    kern-bug-people (Kernel Bug People)
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Sep  5 13:35:00 1995
Free Hardware Foundation, UnLtd.
>Release:        1.0
System: NetBSD 1.0 NetBSD 1.0 (HONEY) #3: Mon May 8 00:07:00 MET DST 1995 i386

If the value of p->p_rlimit[RLIMIT_STACK] is set to a value that is not a
multiple of the pagesize, a process' children may (will?) fail when trying
to do an execve.  As far as I can see the vm subsystem will react
negatively to such a setting when trying to set up the new program's
stack. (execve() -> vmcmd_map_zero() -> vm->allocate() -> ??).
vm_allocate() will return an KERN_NO_SPACE error code.

Run csh and do a 'limit stacksize 512'.  Due to the way csh parses this
input it will actually set the stacksize to 0x80200, i.e. 512k + 512
bytes.  All attempts at starting new programs will now fail with an IOT
trap.  Doing a 'limit stacksize 512k' will set the stacksize to 0x80000,
i.e. 512k exactly, and starting other programs will work again.  This
behavior was observed with gdb.
There are at least three ways to fix this:  
* To make sure that the stacksize is always rounded (or truncated) to a
proper pagealigned value wherever it's being used. 
* Round/truncate the stacksize to a pagealigned value where the rlimit
is being set.  This second fix is the `quickest', but I'm not really sure
which one is the most correct one.... Among other things it will make
`limit stacksize 512' set the stacksize to 516k.
* Make the exec- and vm-subsystem deal with non-aligned address and size
values properly.

Second fix:
*** kern_resource.c.orig        Tue Sep  5 12:37:55 1995
--- kern_resource.c     Tue Sep  5 18:10:11 1995
*** 254,257 ****
--- 254,258 ----
        case RLIMIT_STACK:
+               limp->rlim_cur = round_page(limp->rlim_cur);
                if (limp->rlim_cur > maxsmap)
                        limp->rlim_cur = maxsmap;

trigraph ??!??! die