Enable unsupported deve/hdf5 like others?

["Dr. Thomas Orgis" <thomas.orgis%uni-hamburg.de@localhost>]

TLDR; Is it fine to add --enable-unsupported to the hdf5 threads option?

Dear all,

I have a group of physicists using the LOFAR software stack and I am
providing dependencies for them via pkgsrc. One of those is devel/hdf5.
They do need it with --enable-threadsafe, which adds a crude global
lock around the basic C interface of the library.

Officially per upstream, this option is incompatible with the
high-level interface as well as the C++ interface, by extension. Hence,
I modified devel/hdf5/options.mk and PLIST to reflect that.

Also, I added a code bit to devel/hdf5-c++ to tell the user why it
fails to build with the threads option in hdf5.

This seemed like the safe thing to do. But the situation is this:

1. The physicists' software stack does need both threadsafe API _and_
   the high-level and C++ APIs.

2. The 'standard' way of shipping HDF5 (well, in Debian) indeed is using
   --enable-unsupported --enable-threadsafe --enable-cxx. That is what they
   are used to.

3. The HDF5 upstream people are uneasy about this, but do not care
   enough/do not have enough manpower to support the de facto standard
   configuration.

It's not only Debian. The other (probably increasingly) popular
packaging system for HPC that I'm setting pkgsrc up against as a
bundled alternative, Spack, does also enable all the unsupported
setups. They are verbose about this in

	https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/hdf5/package.py#L234

    # There are several officially unsupported combinations of the features:
    # 1. Thread safety is not guaranteed via high-level C-API but in some cases
    #    it works.
    # conflicts('+threadsafe+hl')

    # 2. Thread safety is not guaranteed via Fortran (CXX) API, but it's
    #    possible for a dependency tree to contain a package that uses Fortran
    #    (CXX) API in a single thread and another one that uses low-level C-API
    #    in multiple threads. To allow for such scenarios, we don't specify the
    #    following conflicts.
    # conflicts('+threadsafe+cxx')
    # conflicts('+threadsafe+fortran')

    # 3. Parallel features are not supported via CXX API, but for the reasons
    #    described in #2 we allow for such combination.
    # conflicts('+mpi+cxx')

Btw., they've got a damn fancy package browser with lots of structured
metadata:

	https://packages.spack.io/package.html?name=hdf5

The FreeBSD ports (they got it in science/hdf5, btw;-) don't seem to
have the threadsafe variant.

Upstream themselves acknowledge that they're creating a situation that
just is broken:

	https://forum.hdfgroup.org/t/high-level-thread-safe/902

Question:

When I run hdf5-1.10.1 through CMake it complains that HL and
thread-safety options are not supported. When did this happen? I use
both.

Answer:

The reason the high-level library is not considered thread-safe is
because the global library lock only exists in the C library and is not
hoisted up into the HL library. The thread-safe + high-level library
combination was not explicitly marked as unsafe and unsupported until a
few years ago. You can override this setting in CMake by setting
ENABLE_UNSUPPORTED (edit: ALLOW_UNSUPPORTED) to ON or configuring with
--enable-unsupported in the autotools.

This may work for you or you may encounter problems; we simply haven’t
had the internal resources to investigate this properly. Making
thread-safety and the high-level library mutually exclusive came about
when we were cleaning up the build systems and that combination (as
well as the language wrappers) was flagged as problematic and not
supported, not because we know for sure that there are problems.

We’ve talked about fixing this for some time, but unfortunately it’s
never made it to the front of our queue. If anyone cares deeply about
this and wants to fund the work, that could quickly change.

Also, for what it’s worth, regardless of your file access patterns, the
HDF5 C library contains global state that can mutate in even read-only
cases. You are far, far better off building with thread-safety and the
high-level library (even though it’s technically unsupported) than
attempting to use the non-thread-safe library from multiple threads. In
the former, you at least have the bulk of the code protected by the
global lock.

[end of quote]

I see the real deal being someone just stepping up and fixing
upstream's intolerable situation. It seems like the majority of the
open source users are using binaries in the unsupported configuration.
That doesn't look sane. 'It mostly works' is a great mantra for doing
scientific computing:-(

But, as long as we got that situation, is it fine for pkgsrc to follow
suit and build hdf5 both with thread safety and the high-level
interfaces?

Should we even make it the default build without an option, as it's the
build people on other platforms are used to?


Alrighty then,

Thomas

PS: Note how Spack does indeed attempt to package many versions and setups
of the same package in a big tree of prefixes for each setup. They
solve the most general problem of installing different configurations
of software with lots of even conflicting dependencies. This is
admirable in one respect, but creates a lot of clutter and somehow
encourages upstreams to continue shipping shitty software that does not
monotonically improve with time and keeps stable APIs. IMHO, they're
worsening the situation by solving a problem that should not exist. But
it could well be that their approach wins in the HPC space and my
approach of making/keeping pkgsrc relevant there is getting ridiculous,
maybe even just because it being massively understaffed in comparison.
I like to dream of a world where integration of software into a
consistent package set with one version each at a point in time is
(continuing to be) possible.

-- 
Dr. Thomas Orgis
HPC @ Universität Hamburg