Am Fri, 16 Feb 2018 07:40:50 -0600
schrieb Jason Bacon <bacon4000%gmail.com@localhost>:
I think the potential failures you cited when using a multithreaded
library actually speak in favor of the openblas-devel approach.
[…] wonder if the pthread code is still being maintained faithfully.
Please note that in the case of numpy (maybe older pythons only, not
sure), the pthread variant works, while the openmp variant freezes …
under circumstances. See
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=684344
There is no way to make everyone happy …
I did some testing myself, and for our setup, the OpenMP variant as
default would work, partially because we have a default of
OMP_NUM_THREADS=1 in the environment. Users have to think a bit about
their application and then enable the multithreading as desired. With
the thread count of 1, the OpenMP version does not cause trouble. This
is preferrable to us having to build another instance of numpy and R
(and others).
It sounds like the most stable approach would be to install a serial
version of the library under the default name libopenblas.{a,so} and
offer a multithreaded version under a different name for those who want
to explicitly choose it. Under this strategy, you could offer both
openmp and pthreads versions under different names, but I'm not sure
whenther this would actually benefit anyone.
See above … sadly, the openmp version is not clearly superior for
everyone. When we install 2 variants, we could just as well also offer 3.
Only argument against that would well-established usage of the
libopenblasp.so name. I am not sure of that, besides Debian and FreeBSD.
Also, by following the FreeBSD port example, we can leverage examples of
how dependent ports utilize the parallel version. A quick grep on the
FreeBSD ports tree reveals a few ports that are tuned to use openblasp:
lang/julia/Makefile: libopenblasp.so:math/openblas \
math/R-cran-spdep/Makefile:LIB_DEPENDS= libopenblasp.so:math/openblas
math/armadillo/Makefile: libopenblasp.so:math/openblas \
math/py-numpy/Makefile: -e "s|%%BLASLIBS%%|openblasp, gfortran|" \
Of course you can hack them to use libopenblasp, while libopenblas
might be supported as a normal option already. But that is all
optional, right? Instead of offering choice of 'blas openblas', we need
to add options 'blas openblas openblas_pthread openblas_openmp' to all
packages. And the user must decide on each one, as each package can
have its own trouble with a parallel BLAS.
We can put openblas_openmp into PKG_DEFAULT_OPTIONS, right? Any
PKG_OPTIONS.package totally overwrites the default … or does it add to
it? I am not sure of that from reading the docs … testing … so: If I have
PKG_DEFAULT_OPTIONS+=openblas_openmp
I need
PKG_OPTIONS.octave+=-openblas_openmp -openblas_pthread openblas_serial
to ensure that it uses the serial libopenblas? I'm presuming that
multiple occurences of exclusive options like that are an error. I am
trying to test that with my wip/OpenBLAS snapshot that I built
alongside 2017Q2 … how can
bmake show-var varname=PKG_OPTIONS
(also PKG_DEFAULT_OPTIONS, PKG_OPTIONS.openblas) show nothing while
there are clearly options being applied (as shown by show-options)?
This is still too much of a mystery machine to me:-(
Anyway, it seems like the last value in the options list wins. Is this
intentional or an accident?
My point is this: We complicate the situation by adding the choice
of the openblas variant not just to the openblas package, but also to
every dependent package. If you built a program linking to the OpenMP
(or pthread, for that matter) openblas library, you can run it with
OMP_NUM_THREADS=1 in the environment and it will behave like the serial
one. Also it refuses spawning further threads if called inside OpenMP
parallel regions (no nesting).
My tentative assumption is that people who enable openblas at all could
be expected to read up on the possible pitfalls when they choose a
parallel build for pkgsrc. The default should still be a serial build.
I am not opposed to installing multiple libs in principle, but I do not
like what happens with the rest of the installation. Please have a look
at OpenBLAS/PLIST and openblas-devel/PLIST. There are things that
depend on build-time choices. And others that are just rudely dropped.
Do you agree that lapacke.h, cblas.h should be present? They might be
not really specific to OpenBLAS (not sure), but I don't see another
package offering them.
Also, what about
lib/pkgconfig/openblas.pc
lib/cmake/openblas/OpenBLASConfig.cmake
? I guess we'd need to add
lib/pkgconfig/openblas_pthread.pc
lib/pkgconfig/openblas_openmp.pc
lib/pkgconfig/openblasp.pc
[…]
but those probably will be in vain as any configure script most
probably will just look for openblas. Dunno about the cmake stuff, but
I guess it's the same.
There needs to be some brains put in to avoid shipping broken
auxilliary files that assist in using libopenblas when we install
multiple builds. The solution of FreeBSD apparently was to drop
everything besides the bare library. A valid approach, but it should be
discussed. I like my pkgconfig files, for example (need to push my
netcdf.pc/netcdf-fortran.pc upstream).
I see the great divide between people caring about openblas and people
not caring. I see the latter group being the large majority. The
further division into people who would like easy switchery between
libopenblas variants apart from a chosen version during pkg build …
seems rather nitpicky to me. Especially considering that people who
really care about own applications they build can build OpenBLAS easily,
too.
Maybe we can have input from someone else intending to offer
R/numpy/octave with openblas from a common build out of pkgsrc to a set
of users? An individual workstation will have the user just choose one
version and be happy. A multi-user site may have differing preferences,
people needing serial OpenBLAS and others needing parallel. But on our
multi-user site that is catered for by one parallel build and a default
setting of OMP_NUM_THREADS=1 to be changed by people who want to do
parallel BLAS.
Is there an actual prospective user site not happy with deciding once
about OpenBLAS parallelism at build-time? If there is, I could be
convinced to integrate the openblas-devel approach. But lacking that, I
see only hackery around the upstream installation routine and
confguration space bloat for depending packages … more work for me,
more build time for the machines, for questionable gain.
Having pre-built packages only with the serial version is an acceptable
compromise, I guess. In HPC, we build from source. A user on a beefy
workstation might do that, too. And even on end-user PCs, the boost
from the vectorization itself is bigger than just running on 2 or 4
cores, even.
Alrighty then,
Thomas
PS: Interesting entry in OpenBLAS change log (06-Mar-2014, Version
0.2.9.rc2, long time after adding OpenMP):
* Make OpenBLAS thread-pool resilient to fork via pthread_atfork.
So the pthread version appears to be maintained. OpenMP+fork trouble needs
non-existing workaround in GCC.