tech-pkg archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: [PATCH] math/fftw and math/fftwf OpenMP and MPI options

On 2020-07-15 14:57, Dr. Thomas Orgis wrote:
Am Wed, 15 Jul 2020 14:23:59 -0500
schrieb Jason Bacon <>:

We'll also need the ability to install dependent packages under the MPI
prefix.  Consider a large HPC cluster with hundreds of users each with
their own software needs.
At our HPC site, we had a long discussion about how to organize the
software stacks regarding toolchain choices. The starting point is the
compiler suite, which may be GNU or Intel or Pgi or possibly something
more exotic. Often, the commercial suites come with their own MPI
implementation. One might replace that with one specific for the HPC
system at hand, though. To complicate things further, there's Intel
providing binaries for additional software (a Pythom build, fftw,
whatever is also included in MKL, besides BLAS/LAPACK, even) and the
whole CUDA train.

To get to the point: At out site, we define a software stack as

1. Compiler (+CUDA)
2. MPI
3. pkgsrc
4. special libs outside pkgsrc
5. user application using libs from the above

In this stack, there is one choice of MPI active. We won't enter the
complexity of having multiple MPI implementations supported in the same
pkgsrc installation. For a differing MPI, a new prefix of pkgsrc is
built. We don't branch inside pkgsrc, but below it.

Also, a new release of pkgsrc means a new stack is built. We keep
versions of software for the batch jobs (there might be a series of
jobs running over a year or more needing the same binaries).

 From my perspective, support for differing variants of MPI in pkgsrc is
good, but we really do not need to install multiple ones at the same
time in the same pkgsrc prefix. I imagine that other HPC sites also
would not like to mix up things too much.

Putting the MPI-supporting fftw into some separate prefix instead of
$prefix/lib, to support multiple MPI libs, will be a nuisance for us
when we try to build software on top of pkgsrc, but not as package in
pkgsrc itself. No showstopper, but inconvenient. So, before adding such
complexity, it needs to be really established that there are sites that
want it that way.

I do think pkgsrc should sport a current openmpi package, but have to
stress that I would never build it. I only use native MPI (my own patch
for MPI_TYPE=native, which just does nothing and finds MPI compiler
wrappers in $PATH). Same as many sites probably would like to use MKL
for BLAS.

For BLAS, we decided on installing the BLAS libs next to each other,
but having one default choice that packages are linked against. Do we
want a prefix for netlib or openblas, too? This multiplies quickly.

Leaf packages can have one MPI implementation hard-coded, it's just the
MPI "middleware" that will require support for multiple implementations.
Do you have cases where a package really needs a specific MPI and a
common decision for all packages in an installation won't work?

Alrighty then,


Different MPI implementations have different feature sets and somewhat different user interfaces, so I don't think an interchangeable system like you developed for BLAS is worth attempting.  MPI is orders of magnitude more complex.  Some software may work with one and not others, and some users have personal preferences, or benefit from features that are not universally supported (SLURM integration, CPU affinity, etc).

In my previous work on this issue, I concluded the following:

1. MPI implementations need separate installation prefixes because they install many conflicting files.  ( FreeBSD ports already does this for openmpi versions, though nobody has bothered yet to bring mpich in-line. )
2. Installing each implementation into its own subdirectory is easy.
3. Installing dependent packages into MPI subdirectories is not difficult.  I tested the concept on fftw. 4. Installing a whole separate pkgsrc tree for each MPI implementation would be a redundant time expenditure that I don't think is justified by the modest effort needed to separate implementations within the same tree.

I would probably just set default prefixes for each implementation and allow users who only use one implementation and want their MPI libs in ${PREFIX}/lib to override with an mk variable, e.g. PKGSRC_OPENMPI3_PREFIX.  There's no reason we have to impose one iron-clad standard on everyone and waste time arguing about what it should look like.



Home | Main Index | Thread Index | Old Index