tech-pkg archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
continuing excessive parallelism building go
I'm building go126, and my system's load average is up to 28. While I
have 8 cpus, I have set MAKE_JOBS to 2, wanting package builds to be
sort of a background activity. Interactive performance is seriously
impacted.
I think the basic issue is that the go compiler, without having been
told that it can be parallel, decides to run multiple threads
(hw.ncpu*2?). pkgsrc has MAKE_JOBS, to control the number of
simultaneous build jobs, but this isn't translated into a limit on the
number of active build threads like it should be.
I have a diff in lang/go124, not brought forward to go126 when I built
it. This should be via ALL_ENV, but the replace do-build doesn't insert
ALL_ENV (which is IMHO a bug, but that's not important).
Are other people seeing this? If you build go126, with MAKE_JOBS=2, on
a computer with a highish nw.ncpu, do you find that the load average
impact is only 2, or is it even more than MAKE_JOBS*hw.ncpu?
I see that this MAKE_JOBS_SAFE=no/GOMAXPROCS pattern exists in
lang/go/go-module.mk. That seems to work; I get one compile process
that seems to max out at 200% cpu when building e.g. hugo. (It seems
that it belongs in go-package.mk also.)
With this diff applied to go12[56] also, and building go126, I see two
compile processes at times, but I can't say that it isn't handing out
two cpu-use tokens internally. It's at least into not causing me
problems.
Thus, my patch for go12[456] seems not wrong, mostly sufficient, and a
huge improvement, so I'd like to commit it in a few days, unless someone
would like to fix the excessive concurrency problem differently. What I
really care about is that a build with MAKE_JOBS=2 does not attempt to
run more than 2 compute-bound threads at once. For doing lots of
building, go compilers are really the only problem child.
Index: Makefile
===================================================================
RCS file: /cvsroot/pkgsrc/lang/go124/Makefile,v
retrieving revision 1.2
diff -u -p -r1.2 Makefile
--- Makefile 2 Dec 2025 19:24:16 -0000 1.2
+++ Makefile 8 May 2026 13:27:38 -0000
@@ -96,9 +96,12 @@ PRINT_PLIST_AWK+= /^bin\/gofmt${GOVERSSU
post-extract:
${RM} -r -f ${WRKSRC}/test/fixedbugs/issue27836*
+# Respect pkgsrc concurrency limits.
+MAKE_JOBS_SAFE= no
do-build:
cd ${WRKSRC}/src && \
env \
+ GOMAXPROCS=${MAKE_JOBS:U1} \
GOROOT_BOOTSTRAP=${GOROOT_BOOTSTRAP:Q} \
${GOOPT} \
GOCACHE=${WRKDIR}/.cache/go-build \
Home |
Main Index |
Thread Index |
Old Index