Source-Changes-HG archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
[src/trunk]: src Split the implementation details of malloc(3) to a separate ...
details: https://anonhg.NetBSD.org/src/rev/8f819306c85d
branches: trunk
changeset: 754525:8f819306c85d
user: jruoho <jruoho%NetBSD.org@localhost>
date: Mon May 03 05:01:53 2010 +0000
description:
Split the implementation details of malloc(3) to a separate jemalloc(3)
manual page. This way malloc(3) can follow the standard terse format used
elsewhere in the libc, while additional details can be added to jemalloc(3).
Note HISTORY and AUTHORS in jemalloc(3), and add some reading material to
SEE ALSO.
diffstat:
distrib/sets/lists/comp/mi | 5 +-
lib/libc/stdlib/Makefile.inc | 3 +-
lib/libc/stdlib/jemalloc.3 | 377 +++++++++++++++++++++++++++++++++++++++++++
lib/libc/stdlib/malloc.3 | 267 +-----------------------------
4 files changed, 390 insertions(+), 262 deletions(-)
diffs (truncated from 746 to 300 lines):
diff -r b11ef03eff3e -r 8f819306c85d distrib/sets/lists/comp/mi
--- a/distrib/sets/lists/comp/mi Mon May 03 04:36:43 2010 +0000
+++ b/distrib/sets/lists/comp/mi Mon May 03 05:01:53 2010 +0000
@@ -1,4 +1,4 @@
-# $NetBSD: mi,v 1.1440 2010/05/01 23:27:22 pooka Exp $
+# $NetBSD: mi,v 1.1441 2010/05/03 05:01:53 jruoho Exp $
#
# Note: don't delete entries from here - mark them as "obsolete" instead.
#
@@ -6277,6 +6277,7 @@
./usr/share/man/cat3/j0f.0 comp-c-catman .cat
./usr/share/man/cat3/j1.0 comp-c-catman .cat
./usr/share/man/cat3/j1f.0 comp-c-catman .cat
+./usr/share/man/cat3/jemalloc.0 comp-c-catman .cat
./usr/share/man/cat3/jn.0 comp-c-catman .cat
./usr/share/man/cat3/jnf.0 comp-c-catman .cat
./usr/share/man/cat3/jrand48.0 comp-c-catman .cat
@@ -12102,6 +12103,7 @@
./usr/share/man/html3/j0f.html comp-c-htmlman html
./usr/share/man/html3/j1.html comp-c-htmlman html
./usr/share/man/html3/j1f.html comp-c-htmlman html
+./usr/share/man/html3/jemalloc.html comp-c-htmlman html
./usr/share/man/html3/jn.html comp-c-htmlman html
./usr/share/man/html3/jnf.html comp-c-htmlman html
./usr/share/man/html3/jrand48.html comp-c-htmlman html
@@ -17845,6 +17847,7 @@
./usr/share/man/man3/j0f.3 comp-c-man .man
./usr/share/man/man3/j1.3 comp-c-man .man
./usr/share/man/man3/j1f.3 comp-c-man .man
+./usr/share/man/man3/jemalloc.3 comp-c-man .man
./usr/share/man/man3/jn.3 comp-c-man .man
./usr/share/man/man3/jnf.3 comp-c-man .man
./usr/share/man/man3/jrand48.3 comp-c-man .man
diff -r b11ef03eff3e -r 8f819306c85d lib/libc/stdlib/Makefile.inc
--- a/lib/libc/stdlib/Makefile.inc Mon May 03 04:36:43 2010 +0000
+++ b/lib/libc/stdlib/Makefile.inc Mon May 03 05:01:53 2010 +0000
@@ -1,4 +1,4 @@
-# $NetBSD: Makefile.inc,v 1.73 2009/07/31 20:39:59 dsl Exp $
+# $NetBSD: Makefile.inc,v 1.74 2010/05/03 05:01:53 jruoho Exp $
# from: @(#)Makefile.inc 8.3 (Berkeley) 2/4/95
# stdlib sources
@@ -44,6 +44,7 @@
getenv.3 getopt.3 getopt_long.3 getsubopt.3 grantpt.3 \
hcreate.3 \
imaxabs.3 imaxdiv.3 insque.3 \
+ jemalloc.3 \
labs.3 ldiv.3 llabs.3 lldiv.3 lsearch.3 \
malloc.3 memory.3 mi_vector_hash.3 \
posix_memalign.3 posix_openpt.3 ptsname.3 \
diff -r b11ef03eff3e -r 8f819306c85d lib/libc/stdlib/jemalloc.3
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/lib/libc/stdlib/jemalloc.3 Mon May 03 05:01:53 2010 +0000
@@ -0,0 +1,377 @@
+.\" $NetBSD $
+.\"
+.\" Copyright (c) 1980, 1991, 1993
+.\" The Regents of the University of California. All rights reserved.
+.\"
+.\" This code is derived from software contributed to Berkeley by
+.\" the American National Standards Committee X3, on Information
+.\" Processing Systems.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\" notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\" notice, this list of conditions and the following disclaimer in the
+.\" documentation and/or other materials provided with the distribution.
+.\" 3. Neither the name of the University nor the names of its contributors
+.\" may be used to endorse or promote products derived from this software
+.\" without specific prior written permission.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
+.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+.\" ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
+.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+.\" SUCH DAMAGE.
+.\"
+.\" @(#)malloc.3 8.1 (Berkeley) 6/4/93
+.\" $FreeBSD: src/lib/libc/stdlib/malloc.3,v 1.73 2007/06/15 22:32:33 jasone Exp $
+.\"
+.Dd May 2, 2010
+.Os
+.Dt JEMALLOC 3
+.Sh NAME
+.Nm jemalloc
+.Nd the default system allocator
+.Sh DESCRIPTION
+The
+.Nm
+is a general-purpose concurrent
+.Xr malloc 3
+implementation specifically designed to be scalable
+on modern multi-processor systems.
+It is the default user space system allocator in
+.Nx .
+.Sh TUNING
+When the first call is made to one of the memory allocation
+routines such as
+.Fn malloc
+or
+.Fn realloc ,
+various flags that affect the workings of the allocator are set or reset.
+These are described below.
+.Pp
+The
+.Dq name
+of the file referenced by the symbolic link named
+.Pa /etc/malloc.conf ,
+the value of the environment variable
+.Ev MALLOC_OPTIONS ,
+and the string pointed to by the global variable
+.Va _malloc_options
+will be interpreted, in that order, character by character as flags.
+.Pp
+Most flags are single letters.
+Uppercase letters indicate that the behavior is set, or on,
+and lowercase letters mean that the behavior is not set, or off.
+The following options are available.
+.Bl -tag -width "A " -offset 3n
+.It Em A
+All warnings (except for the warning about unknown
+flags being set) become fatal.
+The process will call
+.Xr abort 3
+in these cases.
+.It Em H
+Use
+.Xr madvise 2
+when pages within a chunk are no longer in use, but the chunk as a whole cannot
+yet be deallocated.
+This is primarily of use when swapping is a real possibility, due to the high
+overhead of the
+.Fn madvise
+system call.
+.It Em J
+Each byte of new memory allocated by
+.Fn malloc ,
+.Fn realloc
+will be initialized to 0xa5.
+All memory returned by
+.Fn free ,
+.Fn realloc
+will be initialized to 0x5a.
+This is intended for debugging and will impact performance negatively.
+.It Em K
+Increase/decrease the virtual memory chunk size by a factor of two.
+The default chunk size is 1 MB.
+This option can be specified multiple times.
+.It Em N
+Increase/decrease the number of arenas by a factor of two.
+The default number of arenas is four times the number of CPUs, or one if there
+is a single CPU.
+This option can be specified multiple times.
+.It Em P
+Various statistics are printed at program exit via an
+.Xr atexit 3
+function.
+This has the potential to cause deadlock for a multi-threaded process that exits
+while one or more threads are executing in the memory allocation functions.
+Therefore, this option should only be used with care; it is primarily intended
+as a performance tuning aid during application development.
+.It Em Q
+Increase/decrease the size of the allocation quantum by a factor of two.
+The default quantum is the minimum allowed by the architecture (typically 8 or
+16 bytes).
+This option can be specified multiple times.
+.It Em S
+Increase/decrease the size of the maximum size class that is a multiple of the
+quantum by a factor of two.
+Above this size, power-of-two spacing is used for size classes.
+The default value is 512 bytes.
+This option can be specified multiple times.
+.It Em U
+Generate
+.Dq utrace
+entries for
+.Xr ktrace 1 ,
+for all operations.
+Consult the source for details on this option.
+.It Em V
+Attempting to allocate zero bytes will return a
+.Dv NULL
+pointer instead of a valid pointer.
+(The default behavior is to make a minimal allocation and return a
+pointer to it.)
+This option is provided for System V compatibility.
+This option is incompatible with the
+.Em X
+option.
+.It Em X
+Rather than return failure for any allocation function,
+display a diagnostic message on
+.Dv stderr
+and cause the program to drop
+core (using
+.Xr abort 3 ) .
+This option should be set at compile time by including the following in
+the source code:
+.Bd -literal -offset indent
+_malloc_options = "X";
+.Ed
+.Pp
+.It Em Z
+Each byte of new memory allocated by
+.Fn malloc ,
+.Fn realloc
+will be initialized to 0.
+Note that this initialization only happens once for each byte, so
+.Fn realloc
+does not zero memory that was previously allocated.
+This is intended for debugging and will impact performance negatively.
+.El
+.Pp
+The
+.Em J
+and
+.Em Z
+options are intended for testing and debugging.
+An application which changes its behavior when these options are used
+is flawed.
+.Sh IMPLEMENTATION NOTES
+The
+.Nm
+allocator uses multiple arenas in order to reduce lock
+contention for threaded programs on multi-processor systems.
+This works well with regard to threading scalability, but incurs some costs.
+There is a small fixed per-arena overhead, and additionally, arenas manage
+memory completely independently of each other, which means a small fixed
+increase in overall memory fragmentation.
+These overheads are not generally an issue,
+given the number of arenas normally used.
+Note that using substantially more arenas than the default is not likely to
+improve performance, mainly due to reduced cache performance.
+However, it may make sense to reduce the number of arenas if an application
+does not make much use of the allocation functions.
+.Pp
+Memory is conceptually broken into equal-sized chunks,
+where the chunk size is a power of two that is greater than the page size.
+Chunks are always aligned to multiples of the chunk size.
+This alignment makes it possible to find
+metadata for user objects very quickly.
+.Pp
+User objects are broken into three categories according to size:
+.Bl -enum -offset 3n
+.It
+Small objects are smaller than one page.
+.It
+Large objects are smaller than the chunk size.
+.It
+Huge objects are a multiple of the chunk size.
+.El
+.Pp
+Small and large objects are managed by arenas; huge objects are managed
+separately in a single data structure that is shared by all threads.
+Huge objects are used by applications infrequently enough that this single
+data structure is not a scalability issue.
+.Pp
+Each chunk that is managed by an arena tracks its contents in a page map as
+runs of contiguous pages (unused, backing a set of small objects, or backing
+one large object).
+The combination of chunk alignment and chunk page maps makes it possible to
+determine all metadata regarding small and large allocations in constant time.
+.Pp
+Small objects are managed in groups by page runs.
+Each run maintains a bitmap that tracks which regions are in use.
+Allocation requests can be grouped as follows.
+.Pp
+.Bl -bullet -offset 3n
+.It
+Allocation requests that are no more than half the quantum (see the
+.Em Q
+option) are rounded up to the nearest power of two (typically 2, 4, or 8).
+.It
+Allocation requests that are more than half the quantum, but no more than the
+maximum quantum-multiple size class (see the
+.Em S
+option) are rounded up to the nearest multiple of the quantum.
+.It
+Allocation requests that are larger than the maximum quantum-multiple size
+class, but no larger than one half of a page, are rounded up to the nearest
+power of two.
+.It
+Allocation requests that are larger than half of a page, but small enough to
+fit in an arena-managed chunk (see the
+.Em K
+option), are rounded up to the nearest run size.
+.It
+Allocation requests that are too large to fit in an arena-managed chunk are
+rounded up to the nearest multiple of the chunk size.
+.El
Home |
Main Index |
Thread Index |
Old Index