tech-pkg archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: Deciding on wich variant(s) of OpenBLAS library to install



On 03/11/18 19:00, Greg Troxel wrote:
(I've been not quite following, so take this with a grain of salt.)

The normal approach in pkgsrc is to set a system-wide variable to
specify the implementation, and thus to install only one of the
alternatives.  We do this for pgsql and kerberos.

If the ABI of the variants is really the same, then one can flip the
variable and "make replace OLDNAME=" with the new openblas.  Or just
pkg_delete -f and install the ne one.

If they are made parallel installable, sort of like guile20 and guile22,
then there can be a per-package variable to link against various
versions.  However, if there is a library that links against OpenBLAS,
and then something else links against that, and also directly, or
indirectly via another library that links againat a different version,
then it gets very messy very fast.  It's likely that the files will be
namespaces (/usr/pkg/openblas-{foo,bar}/lib/libopenblas.so), but much
harder to namespace the symbols.  So two in one binary is very likely
not going to work.

An alternative approach is to bootstrap pkgsrc multiple time, with a
single global variable set to different implementations, and build
different packages in the the different pkg instances.  While the astute
reader will note that this does not scale well to multiple difficult
packages, we don't really need scaling, but rather feasible solutions
for the actual problems of interest.  Still, it seems too complicated.

Another approach is to write better tests and fix the various packages.
Without understanding the details, it seems that it is basically a bug
that multiple variants exist.  But I realize coping with bugs is
sometimes easier than fixing them.



I see in wip openblas-devel and OpenBLAS, and they seem to be the same
version.  Where  are the various versions being considered to be
switched?

To summarize as simply as possible, there are a variety of BLAS implementations which are mostly compatible, but will never be completely interchangeable due to ongoing development in different directions.  Trying to make them interchangeable would be somewhat like trying to make all POSIX operating systems interchangeable, but on a smaller scale.

The most popular in my experience are Netlib BLAS (currently the de facto standard and the only implementation in pkgsrc), ATLAS, OpenBLAS, CBLAS, and Intel's closed-source MKL, but there are many others, including additional open source variants that we might want to package someday:

https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms#Implementations

I agree that it would be nice to have a tunable default for BLAS-agnostic dependent packages, but some dependent packages will always rely on extensions or performance-enhancing features (e.g. openmp) of a particular implementation.

There is *A LOT* of scientific software that depends on BLAS. Some works better with OpenBLAS, others work better with ATLAS or MKL.  Most will work OK with Netlib BLAS, but performance will likely be subpar and many HPC users are averse to it.

Allowing only one BLAS implementation to be installed at a time would effectively eliminate pkgsrc as a viable deployment tool for BLAS-dependent software.



Home | Main Index | Thread Index | Old Index