On 1/5/24 09:14, Taylor R Campbell wrote:
Date: Fri, 5 Jan 2024 08:22:13 -0600 From: Jason Bacon <jtocino%gmx.com@localhost> On 1/4/24 23:27, Robert Elz wrote:If we are going to implement something like this, do it in a general way which supports current, and likely future, systems. Not a hack which just seems like it might work.Yes, we can, for the purpose of py-joblib and many other scenarios.How about we just make py-joblib return 1, always? We already have a system of command-line arguments like `make -j42' and make/environment variables like MAKE_JOBS=42 to configure how many jobs a program should try to use. It's better for programs to respect that than to rummage around and try to find extra CPU resources they're not supposed to be using -- maybe it's better to discourage those shenanigans.
I think you're looking at cores used for a program build. How would this affect a python application using joblib at runtime? Joblib, and most parallelization tools, allow the user to control the number of cores using the API and/or environment variables. Typically, the total number of hardware cores/hyperthreads is used as a default. See OMP_NUM_THREADS as an example. Anyway, I don't think it's a good idea for us to decide for every user how they should utilize available resources. This is contrary to the Unix "trust the user" tenet. I would make all potentially useful information available and let users decide for themselves what to do with it. Withholding information that we think they might misuse will only encourage them to go somewhere else.