Subject: Re: pkgsrc on SMP machines
To: Hubert Feyrer <>
From: Geert Hendrickx <>
List: tech-pkg
Date: 12/16/2005 10:49:37
On Fri, Dec 16, 2005 at 10:19:33AM +0100, Hubert Feyrer wrote:
> On Thu, 15 Dec 2005, Geert Hendrickx wrote:
> >>I looked it up: the user can set something like MAKEOPTS="-j4" in his
> >>/etc/make.conf to enable parallel builds, which is then used by default
> >>for all packages.  Only some some individual packages ("ebuilds"), like
> >>mozilla, disable this (when they are known to break otherwise).
> >           \-> by explicitly invoking "make -j1"
> >
> >I asked #gentoo about it.  They told me most packages are thread-safe.
> >
> >We could mark tested packages with THREAD_SAFE=(yes|no), and have those
> >built with -j${THREADS}, where THREADS is user-settable.  THREAD_SAFE would
> >default to "no" (for unverified packages), at least in the beginning.  When
> >it turns out that this works fine, we could change this default to yes (but
> >THREADS would anyway default to 1).
> >
> >Just brain-storming here.
> Please don't call this "thread"safe, that's something different.
> If anything, it's SMP-safe.

Ok.  They're just names.  Let's say MAKE_J_SAFE for now.  

> Else, I guess that's pretty much what I suggested.

This works for me: 

Index: mk/
RCS file: /pub/NetBSD-CVS/pkgsrc/mk/,v
retrieving revision 1.1774
diff -u -r1.1774
--- mk/	5 Dec 2005 22:07:07 -0000	1.1774
+++ mk/	16 Dec 2005 09:38:38 -0000
@@ -839,6 +839,9 @@
 # Unprivileged builds
 .include "../../mk/"
+# Parallel builds
+.include "../../mk/"
 # If NO_BUILD is defined, default to not needing a compiler.
 .if defined(NO_BUILD)
 USE_LANGUAGES?=		# empty
@@ -1896,7 +1899,7 @@
 	cd ${WRKSRC}; cd ${_dir_};					\
 .  endfor

With this mk/ file:

# $NetBSD$
# Makefile fragment for configuring parallel builds (make -jN)
#   Description: 
#     Invididual, tested packages can set this.  Indicates
#     whether they build correctly with "make -jN" or not.
#   Possible values:
#     yes, no
#   Default value:
#     no
#   Description:
#     Package developers can set this in their environment
#     (NOT in mk.conf!) to test their packages.
#   Possible values:
#     defined, undefined
#   Default value:
#     undefined
#   Description:
#     Number of parallel make jobs that will be invoked for
#     packages that are known to work with it.  Users can
#     set this in the environment or in mk.conf.
#   Possible values:
#     Any number >= 1
#   Default value:
#     1



.if ${MAKE_J_SAFE} == "yes" || defined(OVERRIDE_MAKE_J_SAFE)

The intended use is the following: 

A developer (with or without an SMP machine) wants to test www/links.  He
sets MAKE_JOBS=4 in his environment (or /etc/mk.conf), and does: 

cd pkgsrc/www/links

This seems to work fine.  So he adds MAKE_J_SAFE=yes to links' Makefile.  
Users without an SMP machine now continue to build with "make -j1" (the
default), so they are not affected.  Users with an SMP machine (or distcc
setup) can set MAKE_JOBS=anything in their /etc/mk.conf, and from now on,
www/links will be built in parallel for them.  

Packages which are known to break with any -jN will have MAKE_J_SAFE=no in
their Makefile.  This is important if we ever want to change the default
for MAKE_J_SAFE in the future.  Then this would be roughly equivalent to
what Gentoo currently has.