Re: devel/py-joblib patch

[Taylor R Campbell <campbell+netbsd-tech-pkg%mumble.net@localhost>]

> Date: Wed, 10 Jan 2024 20:26:52 -0600
> From: Jason Bacon <jtocino%gmx.com@localhost>
> 
> On 1/10/24 16:00, Taylor R Campbell wrote:
> >> Date: Wed, 10 Jan 2024 08:29:32 -0600
> >> From: Jason Bacon <jtocino%gmx.com@localhost>
> >>
> >> On 12/29/23 11:14, Jason Bacon wrote:
> >> As per the github discussion, we'll have to stick with subprocesses for
> >> now.  sysconf does not return the desired core count and there is no
> >> portable sysctl interface for python at this time.  The pypi sysctl
> >> package is FreeBSD-specific, while completely different implementations
> >> exist for NetBSD and Linux.
> >
> > I'm confused, what number is wrong and why do you need sysctl at the
> > moment?
> >
> > I understand we might add sysctls in the future to express the full
> > topology, including the gory details of hyperthreading and big.LITTLE
> > and newer finer-grained variants thereof or whatever -- but for now
> > you're just going for the number of configured or online `cores'
> > (i.e., threads), right?
> 
> No, upstream wants the number of physical cores (hw.physicalcpu on
> Darwin, kern.smp.cores on FreeBSD) rather then hyperthreads, to avoid
> oversubscribing the host when the default "use all available cores" is
> invoked.  sysconf cannot provide that, nor can sysctl on NetBSD at the
> moment.  For the moment, py-joblib patched to use hw.ncpu on NetBSD,
> with a comment indicating that this is not ideal.  It's better than
> erroring out, though.

OK, but for NetBSD today, why is running `sysctl hw.ncpu' in a
subprocess better than calling os.sysconf('SC_NPROCESSORS_CONF')?

They are always the same, because sysconf(_SC_NPROCESSORS_CONF) in
libc just queries hw.ncpu (but with substantially less overhead than
running a subprocess).

Maybe tomorrow we'll add hw.ncoresnotcountingthreads or whatever, but
I'm confused why you say you have to stick with subprocesses and not
os.sysconf for now.