Re: [PATCH] math/fftw and math/fftwf OpenMP and MPI options

["Dr. Thomas Orgis" <thomas.orgis%uni-hamburg.de@localhost>]

Am Wed, 15 Jul 2020 14:23:59 -0500
schrieb Jason Bacon <outpaddling%yahoo.com@localhost>: 

> We'll also need the ability to install dependent packages under the MPI 
> prefix.  Consider a large HPC cluster with hundreds of users each with 
> their own software needs.

At our HPC site, we had a long discussion about how to organize the
software stacks regarding toolchain choices. The starting point is the
compiler suite, which may be GNU or Intel or Pgi or possibly something
more exotic. Often, the commercial suites come with their own MPI
implementation. One might replace that with one specific for the HPC
system at hand, though. To complicate things further, there's Intel
providing binaries for additional software (a Pythom build, fftw,
whatever is also included in MKL, besides BLAS/LAPACK, even) and the
whole CUDA train.

To get to the point: At out site, we define a software stack as

1. Compiler (+CUDA)
2. MPI
3. pkgsrc
4. special libs outside pkgsrc
5. user application using libs from the above

In this stack, there is one choice of MPI active. We won't enter the
complexity of having multiple MPI implementations supported in the same
pkgsrc installation. For a differing MPI, a new prefix of pkgsrc is
built. We don't branch inside pkgsrc, but below it.

Also, a new release of pkgsrc means a new stack is built. We keep
versions of software for the batch jobs (there might be a series of
jobs running over a year or more needing the same binaries).

From my perspective, support for differing variants of MPI in pkgsrc is
good, but we really do not need to install multiple ones at the same
time in the same pkgsrc prefix. I imagine that other HPC sites also
would not like to mix up things too much.

Putting the MPI-supporting fftw into some separate prefix instead of
$prefix/lib, to support multiple MPI libs, will be a nuisance for us
when we try to build software on top of pkgsrc, but not as package in
pkgsrc itself. No showstopper, but inconvenient. So, before adding such
complexity, it needs to be really established that there are sites that
want it that way.

I do think pkgsrc should sport a current openmpi package, but have to
stress that I would never build it. I only use native MPI (my own patch
for MPI_TYPE=native, which just does nothing and finds MPI compiler
wrappers in $PATH). Same as many sites probably would like to use MKL
for BLAS.

For BLAS, we decided on installing the BLAS libs next to each other,
but having one default choice that packages are linked against. Do we
want a prefix for netlib or openblas, too? This multiplies quickly.

> Leaf packages can have one MPI implementation hard-coded, it's just the 
> MPI "middleware" that will require support for multiple implementations.

Do you have cases where a package really needs a specific MPI and a
common decision for all packages in an installation won't work?


Alrighty then,

Thomas

-- 
Dr. Thomas Orgis
HPC @ Universität Hamburg