tech-pkg archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: Deciding on wich variant(s) of OpenBLAS library to install

Am Tue, 6 Mar 2018 20:37:58 -0600
schrieb Jason Bacon <>: 

> I'm OK with any setup that allows all BLAS implementations to be 
> installed simultaneously and for dependent packages to select a 
> non-default BLAS.  I think a switchable default is fine, as long as it 
> doesn't lock every dependent package into the same implementation.

> My main concern is the ability to work around inevitable regressions 
> quickly without risk to other dependent packages.

Great, then we're on the same page. Ultimate choice lies with the
person building the package in the end, but all packages default to the
same BLAS.

As I am not full-time pkgsrc expert yet … can you confirm that me
adding a BLAS variable that triggers choice of library in
mk/ will automagically provide individual packages
with the option to use BLAS.packagename instead as value for BLAS?
Additional code needed? Ideally, I would have the packages only include
the mk/ without individual hacking besides inserting
the resulting library names into build scripts.

> If we can show that pkgsrc reduces software deployment man-hours by an 
> order of magnitude *and* allows users a great deal of freedom to control 
> builds, then it will be accepted and become popular in HPC.

I'm going to give a presentation on our software management, which
includes a certain way to set up environment modules and has pkgsrc to
provide infrastructure as well as end-user software. There are still
custom scripts on top that may rely on dependencies in pkgsrc, building
software that is not in pkgsrc or just for getting another version on
top (just built R 3.4.3 on top of pkgsrc 2017Q2). You always need
some ultimate flexibility for the end user/admin.

The recipients of the talk are HPC admins from German universities,
mainly. I guess there will be discussion about using EasyBuild instead,
which is an effort out of the HPC community to achieve reproducable
software installation. Maybe one can reconcile the approaches to have
EasyBuild accept a pkgsrc prefix for providing dependencies and have
the EasyBuild receipe of some  weird (commercial) application on top.
In our case, it's our on little scripting framework to install things
like TURBOMOLE (bad example, as it has basically no dependencies),
which will not be part of pkgsrc. Another candidate for add-on to
pkgsrc is genetics or machine learning stuff that moves so quickly that
pkgsrc is good as a base providing python and some stable modules for
it, but not for providing the version scientists want now.

I hope I can get my native MPI patch and the BLAS stuff (including the
choice to use MKL) at least on a track to get merged into pkgsrc proper
until then (about two weeks;-). Without that, there is not much point
in proposing pkgsrc as an option to the colleagues.

> As BLAS is a centerpiece in much of HPC, how we implement it will be 
> critical to how pkgsrc is perceived.

Agreed. Let's work towards this, all two of us;-)

Alrighty then,


PS: On building R, I observed funny messages from configure regarding
BLAS and LAPACK support … I am not sure what it would have done, but it
reported 'generic' LAPACK when I gave --with-lapack=-lopenblas
(actually, any string would have caused that) and 'integrated in BLAS'
for --with-lapack=. So some packages are well aware of LAPACK being
shipped with BLAS. I _hope_ --with-lapack=-lopenblas it would just work
the same, but the configure output is misleading.

Dr. Thomas Orgis
Universität Hamburg
RRZ / Basisinfrastruktur / HPC
Schlüterstr. 70
20146 Hamburg
Tel.: 040/42838 8826
Fax: 040/428 38 6270

Attachment: smime.p7s
Description: S/MIME cryptographic signature

Home | Main Index | Thread Index | Old Index