tech-pkg archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: Deciding on wich variant(s) of OpenBLAS library to install

Am Tue, 27 Feb 2018 09:18:05 -0600
schrieb Jason Bacon <>:

> dependent package
> simply select one library (libblas, libatlas, libopenblas, 
> libopenblas-openmp, etc.) that's
> known to work with it.

And what about the possible conflicts/subtle unintended switching of
actually active BLAS at runtime I elaborated on in my other post? I
build the pkgsrc software partly so that users can build their software
using the libraries provided by pkgsrc. Say, their own version of
numpy. Or R packages. I will damn make sure that every package in my
pkgsrc that uses some BLAS is using the _same_ BLAS to avoid any
ambiguity when linking multiple libraries into one end user program.

[parallel BLAS just one variant of BLAS]
> I'm not sure it should be.  As you pointed out before, some dependents 
> work better
> with one library and others with a different one.  Again, this is going 

Regarding implicit parallelizations, yes, some combinations can be
problematic. That's why my favoured variant was to choose _one_ build
of openblas to use for all BLAS needs in my install of pkgsrc, knowing
that it will behave in our local setup. With OMP_NUM_THREADS=1, the
openmp build works nicely with the otherwise problematic uses in R or
numpy, just like the serial build.

For official binary packages, I would choose a serial openblas as a
safe choice after all (for platforms that are supported). Factory
default might stay netlib BLAS, as it's slow (depending on how good the
compiler is), but stable.

But: The BLAS API as such is a standard. It is one set of symbols
provided by multiple implementations of the library. As long as we are
talking about serial implementations, every package should work with
every BLAS. The parallel ones, even if they work, may not be what you
want in all setups. Just observe how openblas actively avoids creating
more parallelism if you call it inside an OpenMP parallel region. As a
packager, you cannot predict the needs of the sites.

> to soak up a lot
> of man hours that we could be using to create and maintain other 
> important packages.

Yes, properly handling several parallel installs of BLAS packages would be a
headache. My point is that installing multiple BLAS packages next to
each other and _not_ managing them in a consistent way would also be a
huge headache, with fun PRs resulting.

> I favor an approach the produces reliable dependent packages first and 
> foremost.

Hence I suggest resisting the urge to install multiple openblas
libraries at the same time. Chose one default for the platform
(serial), offer choice to admin to select a different one at openblas
build time. I'd even go so far as to avoid installing netlib BLAS
alongside openblas.

> Improving performance on a per-package basis would be my next priority, 
> followed
> by interchangeability.

To be honest: I'm tempted to have pkgsrc use generic/external BLAS only
and handle the interchangeable builds of openblas and friends outside
pkgsrc. If at all. I am also looking at EasyBuild, which is a framework
for packaging software that was created to partly do what I do with
pkgsrc, in the HPC context. They group BLAS into the toolchain domain,
along with the compiler and MPI library. Maybe it is worth considering
this functionality as a low-level one, basic as the choice of a compiler
(and at the same time, something you can switch at runtime, at the
highest level, but only because it has a stable ABI).

I'm willing to give up the run-time switchery for simplicity. But only
if that indeed results in getting rid of problems.

Just had a team discussion again … after attempts to explain the
situation and the various options about interchanging BLAS or not, we
settled on the plan to have our install of pkgsrc feature exactly _one_
BLAS variant. If people want to shoot around in the general foot area,
they can play games with other builds of (Open)BLAS and LD_LIBRARY_PATH
or perhaps better LD_PRELOAD. This will be fine if you know what you're
doing with your application. It would be nice if our kind of
installation would be a possible option of a normal pkgsrc tree.
Otherwise I'll carry it as patches like I'm carrying lots of patches
already (with varying success of getting them upstreamed).

> The problem I see is that the various BLAS implementations are not 
> freely interchangeable
> in reality, even though they are meant to be.

You do agree, though, that serial implementations are (should) be

> Leaving it to the 
> end-user to switch between
> them at run-time is going to cause problems and generate a lot of PRs.

But trying to decide for all users at build-time will not always be the
correct guess. Even if they are using the same software, some might
want to employ parallel BLAS, some might require a serial one. In the
case of stuff like numpy or R, this depends on the R script you run,
not on the R binary you use. And you have the possible interference of
combining packages (libraries) that (try to) employ different BLAS. You
will have unhappy users either way.

Avoiding problems means, for me, to avoid a possible mix-up of BLAS
implementations in the tree, in a running application.

There might be cases where openblas-pthread/openmp might not work with
a certain package, but apart from the serial/parallel mess: Do you know
any application that will _not_ work with -lopenblas instead of -lblas

> I think things will go much more smoothly if the choice is left to the 
> packager rather than the end-user.

The packager does not have the information needed for the decision. I
don't see anything sane apart from settling on serial openblas for
supported platforms as a default, leaving the admin/user to decide to
override this for a parallel one in a local build, knowing what the
actual users need.

> This way we don't have to test (blas packages) * (dependent packages) 
> different configurations and retest
> every dependent package every time *any* blas package is updated.

Every platform has one BLAS. Every package shall use that by default
and that is what is tested in bulk builds. What kind of tests will you
do? Any variant of openblas will build/link just fine with any
dependent package. The possible issues arise at runtime in specific
ways of using the packages. Do you want to test that?

> of prioritizing limited manpower.  I have a long list of new scientific 
> packages in the queue that I think would
> be more valuable to the pkgsrc community than interchangeable blas binaries.

OK, then. We're halfway agreeing already. You just need to let go of
wanting to install multiple BLAS libraries next to each other, too;-)

Going back to the details of what to install from openblas. We can set
NO_CBLAS=1 to pull cblas.h and libcblas from the cblas package instead.
We could contemplate setting NO_LAPACK=1, too, relying on the netlib
LAPACK only, but we miss some optimizations, then. So, I'd rather
include it. It boils down to deciding which variants of BLAS support
are supported as a combination. I imagine creating those:


In the first incarnation, they all use a single variable to decide on
netlib/atlas/openblas/serial/parallel. Depending packages use variables
exported by these fragments (BLAS_LIBS, LAPACK_LIBS, etc.) and insert
those in build configuration as needed. The cblas and lapacke ones
would just point to the netlib implementations, which then depend on
mk/blas and mk/lapack.

I really, really do not want to hack each package for building with a
certain BLAS when multiple libraries with differing names are
installed. That would eat my time and nerves.

A second incarnation could try to play games with combining things like
netlib LAPACK and openblas BLAS. Or get cblas from openblas, too. We
might never reach this second stage given developer time constraints,
but the path would be clear and there is no breakage of existing
infrastructure needed.

A third incarnation could introduce interchangeable BLAS and friends
and have the above fragments default BLAS_LIBS=-lblas as there are
wrapper/symlink libraries with that name available. No changes needed
to dependent packages at all. Again, things would still be OK if we
never reach this stage.

Also, I think pkgsrc maintainers should only be concerned with testing
and supporting the default choice for the platform. I for sure won't
build and test atlas just for fun. There might be subtleties once
LAPACK version increases. But as long as you have one defined default
BLAS+LAPACK for a platform, this gets tested. It would only be an issue
for run-time switching.

> I think the important thing is to make all implementations available, 
> without install conflicts.  I'm agnostic
> about how this is done.

There I disagree. Well, it might be OK to install both atlas and
openblas without conflicts (neither of them installing a,
but I strongly oppose packages in pkgsrc sometimes using one or the
other. This is the madness that leads to subtle bugs. As the libraries
heavily overlap in the symbols they offer, we really should ensure they
never get loaded together in package builds.

If the end user decides to use atlas in a build of their own software
where pkgsrc used openblas, then it's on them to sort that out. And
they have the chance to see where it went wrong. But if libraries from
pkgsrc already pull in differing libraries with overlapping symbols,
that's just bad form.

> As long as I can make a dependent package use any of the available blas 
> implementations and I can install
> them all on the same cluster, I'm happy.

Of course you can, by

a) implementing run-time override of the dynamic libs or
b) installing multiple pkgsrc trees with differing choices

You can automate b) of course. You can achieve a) by LD_PRELOADing one
of your other installed BLAS choices in the tree. You can even load to override, except for the symbols specific
to openblas (openblas_set_num_threads()). I even suspect that this
would work where openblas_set_num_threads() is called. It just wouldn't
set the threads for the other implementation. Or it would, by accident.

I want to stress this: Do you really _not_ see multiple different BLAS
libs being pulled in by a dependency chain as a problem?

Alrighty then,


Dr. Thomas Orgis
Universität Hamburg
RRZ / Basis-Infrastruktur / HPC
Schlüterstr. 70
20146 Hamburg
Tel.: 040/42838 8826
Fax: 040/428 38 6270

Attachment: smime.p7s
Description: S/MIME cryptographic signature

Home | Main Index | Thread Index | Old Index