[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

**To**:**Jason Bacon <outpaddling%yahoo.com@localhost>****Subject**:**Re: Deciding on wich variant(s) of OpenBLAS library to install****From**:**"Dr. Thomas Orgis" <thomas.orgis%uni-hamburg.de@localhost>**- Date: Wed, 28 Feb 2018 10:29:01 +0100

Am Tue, 27 Feb 2018 14:48:00 -0600 schrieb Jason Bacon <outpaddling%yahoo.com@localhost>: > Currently, there aren't many finished packages linking against blas, and > most of them do not install libraries, so this situation is going to be > rare in the short term: Well, I see most of them as libraries;-) > netbsddev# pkg-grep blas/build|grep -v wip > biology/mpqc/Makefile:.include "../../math/blas/buildlink3.mk" > biology/plink/Makefile:.include "../../math/blas/buildlink3.mk" > lang/lush/Makefile:.include "../../math/blas/buildlink3.mk" > math/R-RandomFields/Makefile:.include "../../math/blas/buildlink3.mk" > math/R-RcppEigen/Makefile:.include "../../math/blas/buildlink3.mk" > math/R-gstat/Makefile:.include "../../math/blas/buildlink3.mk" > math/R-quantreg/Makefile:.include "../../math/blas/buildlink3.mk" > math/R-wle/Makefile:.include "../../math/blas/buildlink3.mk" > math/R/Makefile:.include "../../math/blas/buildlink3.mk" R is a set of libraries with an interpreter on top. Users build R extensions/modules that link against R libs. It is really mandatory that _at_least_ all R-related packages use the same BLAS! An individual choice per package would be an immediate mess. > math/harminv/Makefile:.include "../../math/blas/buildlink3.mk" > math/ipopt/Makefile:.include "../../math/blas/buildlink3.mk" > math/itpp/Makefile:.include "../../math/blas/buildlink3.mk" > math/lapack/Makefile:.include "../../math/blas/buildlink3.mk" Well, lapack is obviously a library;-) If you have something using lapack and also directly using BLAS functions, it better link to the same BLAS as lapack. > math/octave/Makefile:.include "../../math/blas/buildlink3.mk" Octave seems to be more monolithic than R, but I guess the argument about people building extensions (just like Matlab) still applies. If you add a binding to a C/Fortran library that in turn uses BLAS behind the scenes, you get a mix-up with the BLAS that Octave directly links to. > math/py-numpy/Makefile:.include "../../math/blas/buildlink3.mk" > math/py-scikit-learn/Makefile:.include "../../math/blas/buildlink3.mk" Yeah, consider py-numpy and py-scikit-learn _not_ using the same BLAS. I am not sure how loading of these modules (eggs?) in one python process works … do they get separate address spaces? Dlopen and manual symbol mapping? I guess they also could overwrite each other's BLAS, depening on who comes first. > I think it would be sufficient to flag the potential for this situation > somehow, In your list, I see lots of potential in nearly every entry:-/ The big change we are discussing is not exchanging the BLAS implementation, but ending the situation where everyone uses the same one. > On the whole, I still feel like the best approach for the short-term is > for ${PREFIX}/lib/lib{blas,lapack} to be the netlib implementation (the > only one that's well tested under pkgsrc right now, as atlas, cblas, and > openblas exist only in wip) Note: cblas is _not_ a BLAS implementation. It is a C wrapper library over a BLAS implementation. Openblas does also provide cblas if you don't forbid it to. This is another level of decision, the (de-)coupling of cblas/lapack/lapacke (from)to blas lib. This is easier when we decide on one global choice of linear algebra toolchain being used for dependent packages. We can then use one combination and stick to it. If it's all free to decide per package, the mess is inevitable. >, and every other implementation to be > installed under a different name or different path I agree with that. We can do that, make them all installable next to each other. That would probably mean NO_CBLAS=1 for openblas, maybe something similar with atlas to avoid conflict with the cblas package. But I really want to convince you that we should go from > biology/mpqc/Makefile:.include "../../math/blas/buildlink3.mk" > biology/plink/Makefile:.include "../../math/blas/buildlink3.mk" to biology/mpqc/Makefile:.include "../../mk/blas.buildlink3.mk" biology/plink/Makefile:.include "../../mk/blas.buildlink3.mk" just like mpi.buildlink3.mk. In fact … there are parallels to MPI. Would you also rather decide per-pacakge which MPI to use? If you really are sure that a certain package only uses BLAS itself without linking to anything else that does and does offer no libs to link to, you can add set BLAS.package=anotherone to override a global BLAS=openblas-serial, right? I just advise against doing this on a packager level, as it makes the job harder for an admin who wants to ensure consistent BLAS. > As we (all maintainers of dependent packages) gain more experience with > the various blas implementations and work out the bugs in the blas > packages, Well, let's keep netlib blas as default for the forseeable future. No additional testing. Just moving a choice into the blas.buildling3.mk and lapack.buildlink3.mk. Those who add openblas/atlas to those will test those, and users who dare. At some point one can flip the switch on the default implementation if there is enough confidence on a certain platform. I really, really just want to avoid us entering a dependency hell where there is a mix-up of use of differing libraries with the same symbols in packages. I want the differing BLAS availabe for users in their own programs, but I will document in the environment module help that they need to consider any package from pkgsrc being linked against the site-default BLAS, with potential surprises if they choose a different one in their programs using libs from pkgsrc. Alrighty then, Thomas -- Dr. Thomas Orgis Universität Hamburg RRZ / Basisinfrastruktur / HPC Schlüterstr. 70 20146 Hamburg Tel.: 040/42838 8826 Fax: 040/428 38 6270

**Attachment:
smime.p7s**

**Follow-Ups**:**Re: Deciding on wich variant(s) of OpenBLAS library to install***From:*Dr. Thomas Orgis

**References**:**Deciding on wich variant(s) of OpenBLAS library to install***From:*Dr. Thomas Orgis

**Re: Deciding on wich variant(s) of OpenBLAS library to install***From:*Jason Bacon

**Re: Deciding on wich variant(s) of OpenBLAS library to install***From:*Dr. Thomas Orgis

**Re: Deciding on wich variant(s) of OpenBLAS library to install***From:*Jason Bacon

**Re: Deciding on wich variant(s) of OpenBLAS library to install***From:*Dr. Thomas Orgis

**Re: Deciding on wich variant(s) of OpenBLAS library to install***From:*Jason Bacon

**Re: Deciding on wich variant(s) of OpenBLAS library to install***From:*Dr. Thomas Orgis

**Re: Deciding on wich variant(s) of OpenBLAS library to install***From:*Dr. Thomas Orgis

- Prev by Date:
**daily pkgsrc CVS update output** - Next by Date:
**Re: Deciding on wich variant(s) of OpenBLAS library to install** - Previous by Thread:
**Re: Deciding on wich variant(s) of OpenBLAS library to install** - Next by Thread:
**Re: Deciding on wich variant(s) of OpenBLAS library to install** - Indexes: