misc/54456: Building in a directory whose name contains "-j" may fail

[gson%gson.org@localhost (Andreas Gustafsson)]

>Number:         54456
>Category:       misc
>Synopsis:       Building in a directory whose name contains "-j" may fail
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    misc-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Aug 11 11:35:00 +0000 2019
>Originator:     Andreas Gustafsson
>Release:        NetBSD-current
>Organization:

>Environment:
System: NetBSD
Architecture: x86_64
Machine: amd64
>Description:

Some of my builds have been failing with "vfork: Resource temporarily
unavailable", suggesting that they are running out of processes.
These are "build.sh -j 12 release" builds and "ulimit -p" is set
to 500, which ought to be enough (more than 40 processes per make
job).

To debug this, I set up a script to run "proctree" from pkgsrc once a
second during the build and save any output with more than 300 lines.
This caught a process with a lot of descendants and the following
command line:

   /tmp/bracket/build/2019.08.08.14.00.32-amd64-job-000012/tools/bin/nbgmake -j -e MACHINE= MAKEINFO=/tmp/bracket/build/2019.08.08.14.00.32-amd64-job-000012/tools/bin/nbmakeinfo LIBGCC= LIBGCC1= LIBGCC1_TEST= LIBGCC2= INSTALL_LIBGCC= EXTRA_PARTS= CPPFLAGS=-DNETBSD_TOOLS -DTARGET_SYSTEM_ROOT=0  -DTARGET_SYSTEM_ROOT_RELOCATABLE AR=ar RANLIB=ranlib BISON=true DESTDIR= INSTALL=/tmp/bracket/build/2019.08.08.14.00.32-amd64-job-000012/tools/bin/x86_64--netbsd-install -c  -r all-gcc 

Note the "-j" option which is not followed by an argument.
"man gmake" says:

   If the -j option is given without an argument, make will not limit
   the number of jobs that can run simultaneously.

The argument-less -j comes from this line in src/tools/Makefile.gnuhost:

   GMAKE_J_ARGS?=  ${MAKEFLAGS:[*]:M*-j*:C/.*(-j ?[0-9]*).*/\1/W}

These particular builds are made in a directory like
"/tmp/bracket/build/2019.08.08.14.00.32-amd64-job-000012".
In Makefile.gnuhost, the directory name gets expanded into
MAKEFLAGS, which then looks like:

   -d e -m /tmp/bracket/build/2019.08.08.14.00.32-amd64-job-000014/src/share/mk -j 12 -J 15,16 .MAKE.LEVEL.ENV=MAKELEVEL HOST_OSTYPE=NetBSD-8.1-amd64 MKOBJDIRS=yes _SRC_TOP_=/tmp/bracket/build/2019.08.08.14.00.32-amd64-job-000014/src _SRC_TOP_OBJ_=/tmp/bracket/build/2019.08.08.14.00.32-amd64-job-000014/obj _THISDIR_=tools/

which contains both "-j 12" and "amd64-job".  The presence of the
substring "-j" in the latter appears to be confusing the convoluted
make variable expansion used to set GMAKE_J_ARGS such that it yields
just "-j" rather than "-j 12".

As a minimal test case, running the following as a makefile prints
"-j 12":

MAKEFLAGS=-j 12 saint-mary
GMAKE_J_ARGS?=  ${MAKEFLAGS:[*]:M*-j*:C/.*(-j ?[0-9]*).*/\1/W}
all:
	@echo ${GMAKE_J_ARGS}

but if you change "saint-mary" to "saint-john", it prints just "-j".

>How-To-Repeat:

Set "ulimit -p" to 300 or less, and then attempt a "build.sh -j 12
release" (including the tools build, and perhaps changing -j 12 to
whatever is appropriate for your machine) in a directory whose name
contains the substring "-j".

>Fix: