python nproc vs MAKE_JOBS issues

[Jonathan Perkin <jperkin%mnx.io@localhost>]

I've seen a few similar things to this recently where python modules are 
being built with complete disregard to MAKE_JOBS, and instead are just 
spawning a thread for each CPU they find.  For example:

98644  /opt/tools/bin/pbulk-build -c 192.168.100.48:2008 -b /opt/tools/libexec/pbulk/p
  95104  /usr/bin/bash /opt/tools/libexec/pbulk/pkg-build
    72343  /opt/local/bin/bmake all BATCH=1 DEPENDS_TARGET=/nonexistent PYTHON_VERSION_REQ
      72705  /usr/bin/bash /tmp/makeMJaUuJ
        72708  /opt/local/bin/bmake _MAKE=/opt/local/bin/bmake NATIVE_LOWER_OPSYS=solaris LOWE
          72873  /usr/bin/bash /tmp/make4XaiGJ
            72878  /opt/local/bin/python3.10 -m build --wheel --skip-dependency-check --no-isolati
              73087  /opt/local/bin/python3.10 /opt/local/lib/python3.10/site-packages/pyproject_hoo
                73197  maturin-3.10 pep517 build-wheel -i /opt/local/bin/python3.10 --compatibility of
                  74585  cargo rustc --message-format json-render-diagnostics --manifest-path /home/pbul
                    76760  rustc --crate-name regex_syntax --edition=2021 /home/pbulk/build/devel/py-ruff/
                    84834  rustc --crate-name syn --edition=2021 /home/pbulk/build/devel/py-ruff/work/vend
                    86312  rustc --crate-name regex_automata --edition=2021 /home/pbulk/build/devel/py-ruf
                    87613  rustc --crate-name itertools --edition=2018 /home/pbulk/build/devel/py-ruff/wor
                    88360  rustc --crate-name similar --edition=2018 /home/pbulk/build/devel/py-ruff/work/
                    88431  rustc --crate-name cc --edition=2018 /home/pbulk/build/devel/py-ruff/work/vendo
                    88745  rustc --crate-name regex_syntax --edition=2018 /home/pbulk/build/devel/py-ruff/
                    88832  rustc --crate-name hashbrown --edition=2021 /home/pbulk/build/devel/py-ruff/wor
                    88919  rustc --crate-name hashbrown --edition=2021 /home/pbulk/build/devel/py-ruff/wor
                    88965  rustc --crate-name rand_chacha --edition=2018 /home/pbulk/build/devel/py-ruff/w
                    88974  rustc --crate-name build_script_build --edition=2021 /home/pbulk/build/devel/py
                      89669  /home/pbulk/build/devel/py-ruff/work/.gcc/bin/gcc -fcommon -m64 -fstack-protect
                        89673  /opt/local/gcc13/libexec/gcc/x86_64-sun-solaris2.11/13.3.0/collect2 -Y P,/lib/a
                          89674  /bin/ld -Y P,/lib/amd64:/usr/lib/amd64 -Qy -o /home/pbulk/build/devel/py-ruff/w
                    89232  rustc --crate-name rand_chacha --edition=2018 /home/pbulk/build/devel/py-ruff/w
                    89353  rustc --crate-name unicode_normalization --edition=2018 /home/pbulk/build/devel
                    89379  rustc --crate-name minimal_lexical --edition=2018 /home/pbulk/build/devel/py-ru
                    89389  rustc --crate-name build_script_build --edition=2021 /home/pbulk/build/devel/py
                      89613  /home/pbulk/build/devel/py-ruff/work/.gcc/bin/gcc -fcommon -m64 -fstack-protect
                        89615  /opt/local/gcc13/libexec/gcc/x86_64-sun-solaris2.11/13.3.0/collect2 -Y P,/lib/a
                          89666  /bin/ld -Y P,/lib/amd64:/usr/lib/amd64 -Qy -o /home/pbulk/build/devel/py-ruff/w
                    89454  rustc --crate-name dirs --edition=2015 /home/pbulk/build/devel/py-ruff/work/ven
                    89501  rustc --crate-name thread_local --edition=2021 /home/pbulk/build/devel/py-ruff/
                    89508  rustc --crate-name etcetera --edition=2018 /home/pbulk/build/devel/py-ruff/work
                    89538  rustc --crate-name bitflags --edition=2018 /home/pbulk/build/devel/py-ruff/work
                    89544  cargo rustc --message-format json-render-diagnostics --manifest-path /home/pbul
                    89574  /home/pbulk/build/devel/py-ruff/work/ruff-0.7.2/target/release/build/fs-err-2c6
                      89677  rustc --version --verbose
                    89576  rustc --crate-name clap_builder --edition=2021 /home/pbulk/build/devel/py-ruff/
                    89587  rustc --crate-name sharded_slab --edition=2018 /home/pbulk/build/devel/py-ruff/
                    89590  rustc --crate-name rand --edition=2018 /home/pbulk/build/devel/py-ruff/work/ven
                    89608  rustc --crate-name rand --edition=2018 /home/pbulk/build/devel/py-ruff/work/ven
                    89612  rustc --crate-name chrono --edition=2021 /home/pbulk/build/devel/py-ruff/work/v
                    89618  cargo rustc --message-format json-render-diagnostics --manifest-path /home/pbul
                    89640  rustc --crate-name fnv --edition=2015 /home/pbulk/build/devel/py-ruff/work/vend
                    89642  rustc --crate-name shellexpand --edition=2018 /home/pbulk/build/devel/py-ruff/w
                    89647  cargo rustc --message-format json-render-diagnostics --manifest-path /home/pbul
                    89675  rustc --crate-name peg --edition=2018 /home/pbulk/build/devel/py-ruff/work/vend

This kills performance on my build hosts, especially as right now there 
are three of them running (devel/py-ruff for 3.10, 3.11, 3.12), along 
with all of the other concurrent builds that are correctly observing 
MAKE_JOBS.  So, well over 100 build processes on a 32-core machine...

Do any python folks know what is causing this and are able to fix it?

Thanks,

--
Jonathan Perkin   -   mnx.io   -   pkgsrc.smartos.org
Open Source Complete Cloud   www.tritondatacenter.com