Subject: Re: pkgsrc on SMP machines
To: Hubert Feyrer <feyrer@cs.stevens.edu>
From: None <segv@netctl.net>
List: tech-pkg
Date: 12/19/2005 17:42:44
On Thu, 15 Dec 2005 04:39:51 +0100 (CET)
Hubert Feyrer <feyrer@cs.stevens.edu> wrote:

> On Thu, 15 Dec 2005 segv@netctl.net wrote:
> > Hi, can anyone suggest a good strategy for utilising the most of SMP
> > hardware when building software from pkgsrc??
> ...
> > Fixing all Makefiles to support -j > 1 seems quite unreasonable, I guess the
> > best solution would be to have a multithreaded compiler, which distributes
> > workload to a specified number of threads, i.e.:
> 
> How about compiling the pkgs themselves (i.e. whatever gets extracted from 
> sources) with 'make -j', e.g. as the patch below does. At least that's how 
> Unix is supposed to work...
> 
> Of course that's a crude hack, and a real solution would involving a 
> user-settable variable plus a per-package maximum, and the real value 
> would be the somewhere in between there.
> 
> (There is that other idea to build pkgsrc packages in parallel which is 
> stuck to some people's head, but it grossly violates the KISS principle 
> and involves all sort of nasty things like locking and whatnot; I still 
> suggest not going there)
> 
> 
>   - Hubert
> 
> 
> Index: bsd.pkg.mk
> ===================================================================
> RCS file: /cvsroot/pkgsrc/mk/bsd.pkg.mk,v
> retrieving revision 1.1774
> diff -u -r1.1774 bsd.pkg.mk
> --- bsd.pkg.mk	5 Dec 2005 22:07:07 -0000	1.1774
> +++ bsd.pkg.mk	15 Dec 2005 03:34:35 -0000
> @@ -1895,7 +1895,7 @@
>   .  for _dir_ in ${BUILD_DIRS}
>   	${_PKG_SILENT}${_PKG_DEBUG}${_ULIMIT_CMD}			\
>   	cd ${WRKSRC}; cd ${_dir_};					\
> -	${SETENV} ${MAKE_ENV} ${MAKE_PROGRAM} ${BUILD_MAKE_FLAGS}	\
> +	${SETENV} ${MAKE_ENV} ${MAKE_PROGRAM} -j17
> ${BUILD_MAKE_FLAGS}	\ -f ${MAKEFILE} ${BUILD_TARGET}
>   .  endfor
>   .endif

There seems to be a problem with pkgsrc wrapper scripts. I keep getting the
following errors when I apply your patch to bsd.pkg.mk but with -j6

/opt/pkg.obj/x11/gtk2/work/.wrapper/bin/cc[51]: syntax error at line 51 :
`case' unmatched gmake[3]: *** [gdk-pixbuf-xlib-render.lo] Error 1
gmake[3]: *** Waiting for unfinished jobs....

Any idea what might be causing this?