tech-pkg archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

continuing excessive parallelism building go



I'm building go126, and my system's load average is up to 28.  While I
have 8 cpus, I have set MAKE_JOBS to 2, wanting package builds to be
sort of a background activity.   Interactive performance is seriously
impacted.

I think the basic issue is that the go compiler, without having been
told that it can be parallel, decides to run multiple threads
(hw.ncpu*2?).  pkgsrc has MAKE_JOBS, to control the number of
simultaneous build jobs, but this isn't translated into a limit on the
number of active build threads like it should be.

I have a diff in lang/go124, not brought forward to go126 when I built
it.  This should be via ALL_ENV, but the replace do-build doesn't insert
ALL_ENV (which is IMHO a bug, but that's not important).

Are other people seeing this?  If you build go126, with MAKE_JOBS=2, on
a computer with a highish nw.ncpu, do you find that the load average
impact is only 2, or is it even more than MAKE_JOBS*hw.ncpu?

I see that this MAKE_JOBS_SAFE=no/GOMAXPROCS pattern exists in
lang/go/go-module.mk.  That seems to work; I get one compile process
that seems to max out at 200% cpu when building e.g. hugo.  (It seems
that it belongs in go-package.mk also.)

With this diff applied to go12[56] also, and building go126, I see two
compile processes at times, but I can't say that it isn't handing out
two cpu-use tokens internally.  It's at least into not causing me
problems.


Thus, my patch for go12[456] seems not wrong, mostly sufficient, and a
huge improvement, so I'd like to commit it in a few days, unless someone
would like to fix the excessive concurrency problem differently.  What I
really care about is that a build with MAKE_JOBS=2 does not attempt to
run more than 2 compute-bound threads at once.  For doing lots of
building, go compilers are really the only problem child.


Index: Makefile
===================================================================
RCS file: /cvsroot/pkgsrc/lang/go124/Makefile,v
retrieving revision 1.2
diff -u -p -r1.2 Makefile
--- Makefile	2 Dec 2025 19:24:16 -0000	1.2
+++ Makefile	8 May 2026 13:27:38 -0000
@@ -96,9 +96,12 @@ PRINT_PLIST_AWK+=	/^bin\/gofmt${GOVERSSU
 post-extract:
 	${RM} -r -f ${WRKSRC}/test/fixedbugs/issue27836*
 
+# Respect pkgsrc concurrency limits.
+MAKE_JOBS_SAFE=		no
 do-build:
 	cd ${WRKSRC}/src && \
 		env \
+		GOMAXPROCS=${MAKE_JOBS:U1} \
 		GOROOT_BOOTSTRAP=${GOROOT_BOOTSTRAP:Q} \
 		${GOOPT} \
 		GOCACHE=${WRKDIR}/.cache/go-build \


Home | Main Index | Thread Index | Old Index