kern/52239: Changing protections of already mmap'ed region can fail

[gson%gson.org@localhost (Andreas Gustafsson)]

>Number:         52239
>Category:       kern
>Synopsis:       Changing protections of already mmap'ed region can fail
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed May 17 17:25:00 +0000 2017
>Originator:     Andreas Gustafsson
>Release:        NetBSD 7.0.2
>Organization:

>Environment:
System: NetBSD
Architecture: x86_64
Machine: amd64
>Description:

As reported in https://github.com/jemalloc/jemalloc/issues/837
recent versions of jemalloc fail on NetBSD.  I have now tracked down
the cause of this problem to an issue with NetBSD's mmap().

With its default setting of "os_overcommits = false", jemalloc will
allocate regions of anonymous memory using

  p = mmap(0, ...PROT_NONE, MAP_ANON|MAP_PRIVATE...)

and then selectively enable read and write access on parts that are
about to actually be used using

  mmap(p, ...PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE|MAP_FIXED...)

where p is the pointer returned by the first mmap call.

The problem is that if a second thread simultaneously does an
unrelated allocation of anonymous memory using mmap(0, ...), the
memory region whose protections are in the process of being changed by
the first thread can end up being returned by the mmap() of the second
thread instead, and the mmap(...MAP_FIXED...) of the first thread then
fails with ENOMEM.

I have now worked around this issue in the pkgsrc jemalloc package by
setting "os_overcommits = true".

>How-To-Repeat:

Here is a test case.  It can be run with

  cc testcase.c  -lpthread -o testcase && ./testcase

and will print "update mmap: Cannot allocate memory" when it fails.

#include <assert.h>
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <pthread.h>

#include <sys/mman.h>

#define SIZE (13*4096)

#define NITER 10000000

void *thread_main(void *arg) {
    int i, r;
    for (i = 0; i < NITER; i++) {
	// Get some unrelated memory
	void *t = mmap(0, SIZE, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0);
	assert(t);
	r = munmap(t, SIZE);
	assert(r == 0);
    }
    return 0;
}

int main(int argc, char **argv) {
    int i, r;

    pthread_t thread;
    r = pthread_create(&thread, 0, thread_main, 0);
    assert(r == 0);

    for (i = 0; i < NITER; i++) {
	// Get a placeholder region
	void *p = mmap(0, SIZE, PROT_NONE, MAP_ANON|MAP_PRIVATE, -1, 0);
	if (p == MAP_FAILED) {
	    printf("mmap: %s\n", strerror(errno));
	}
	assert(p != MAP_FAILED);

	// Upgrade placeholder to read/write access
	void *q = mmap(p, SIZE, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE|MAP_FIXED, -1, 0);
	if (q == MAP_FAILED) {
	    printf("update mmap: %s\n", strerror(errno));
	    exit(1);
	}
	assert(q == p);

	// Free it
	r = munmap(q, SIZE);
	if (r != 0) {
	    printf("munmap: %s\n", strerror(errno));
	    exit(1);
	}
    }
    return 0;
}

>Fix: