Re: py-tables build failure on Darwin

[Jason Bacon <jtocino%gmx.com@localhost>]

On 11/26/24 10:48, Jason Bacon wrote:
> On 11/26/24 08:38, Jonathan Perkin wrote:
> > * On 2024-11-26 at 14:23 GMT, Jason Bacon wrote:
> >
> > > Anyone interested in math/py-tables on macOS, please test the 
> > > following patch:
> > >
> > > --- ../../math/py-tables/PLIST.Darwin   1969-12-31 18:00:00.000000000 
> > > -0600
> > > +++ ./PLIST.Darwin      2024-11-26 08:21:14.059878353 -0600
> > > @@ -0,0 +1 @@
> > > +${PYSITELIB}/tables/libblosc2.dylib
> > >
> > > This is necessary on my Mac M1, but I don't have the means to test on 
> > > Intel or older macOS versions.
> >
> > This corroborates with the failures in the latest bulk build:
> >
> >    https://us-central.manta.mnx.io/pkgsrc/public/reports/Darwin/12.3/ 
> > arm64/20241125.1313/meta/report.html
> >
> > That said I'd recommend seeing if there's a better fix that is less 
> > likely to break on updates.  Is there a particular reason only macOS 
> > is shipping a shared library for this and not any other platforms?
>
> The 1200+ line setup.py looks like a hack job.  The logic leading to 
> libblosc being copied is here:
>
>          if not rundir:
>              loc = {
>                  "posix": "the default library paths",
>                  "nt": "any of the directories in %%PATH%%",
>              }[os.name]
>
>              if package.name == "blosc2":
>                  # We will copy this into the tables directory
>                  print(
>                      "  * Copying blosc2 runtime library to 'tables' dir"
>                      " because it was not found in standard locations"
>                  )
>                  platform_system = platform.system()
>                  if platform_system == "Linux":
>                      shutil.copy(libdir / "libblosc2.so", ROOT / "tables")
>                      copy_libs += ["libblosc2.so"]
>
>                      dll_dir = "/tmp/hdf5/lib"
>                      # Copy dlls when producing the wheels in CI
>                      if "bdist_wheel" in sys.argv and 
> os.path.exists(dll_dir):
>                          shared_libs = glob.glob(str(libdir) + "/ 
> libblosc2.so*")
>                          for lib in shared_libs:
>                              shutil.copy(lib, dll_dir)
>
>                  elif platform_system == "Darwin":
>                      shutil.copy(libdir / "libblosc2.dylib", ROOT / 
> "tables")
>                      copy_libs += ["libblosc2.dylib"]
>
> So the question is why it's finding libblosc in "standard locations" on 
> Linux, but not on macOS.

I traced the problem to this function in setup.py:

    def find_runtime_path(self, locations=None):
        """
        returns True if the runtime can be found
        returns None otherwise
        """
        # An explicit path can not be provided for runtime libraries.
        # (The argument is accepted for compatibility with previous
        # methods.)

        # dlopen() won't tell us where the file is, just whether
        # success occurred, so this returns True instead of a filename
        for prefix in self._runtime_prefixes:
            for suffix in self._runtime_suffixes:
                print("find_runtime_path() trying ",
                      f"{prefix}{self.runtime_name}{suffix}")
                try:
                    ctypes.CDLL(f"{prefix}{self.runtime_name}{suffix}")
                except OSError:
                    pass
                else:
                    return True

Called via:

    path = find_path(use_locations)

Relevant output on Darwin:

use_locations =  [PosixPath('/Users/bacon/Pkgsrc/pkg/lib'), 
PosixPath('/usr/local/lib64'), PosixPath('/usr/local/lib'), 
PosixPath('/sw/lib64'), PosixPath('/sw/lib'), PosixPath('/opt/lib64'), 
PosixPath('/opt/lib'), PosixPath('/opt/local/lib64'), 
PosixPath('/opt/local/lib'), PosixPath('/usr/lib64'), 
PosixPath('/usr/lib'), PosixPath('/lib64'), PosixPath('/lib')]
find_runtime_path() trying  libblosc2.so
find_runtime_path() trying  libblosc2.dylib
path is not set!!  This is a problem.

On NetBSD:

use_locations =  [PosixPath('/usr/lib'), 
PosixPath('/home/bacon/Pkgsrc/pkg/lib'), PosixPath('/usr/local/lib64'), 
PosixPath('/usr/local/lib'), PosixPath('/sw/lib64'), 
PosixPath('/sw/lib'), PosixPath('/opt/lib64'), PosixPath('/opt/lib'), 
PosixPath('/opt/local/lib64'), PosixPath('/opt/local/lib'), 
PosixPath('/usr/lib64'), PosixPath('/usr/lib'), PosixPath('/lib64'), 
PosixPath('/lib')]
find_runtime_path() trying  libblosc2.so
path =  True

So, ctypes.CDLL() is finding libblosc.so and libblosc2.so on NetBSD and 
Linux, but not libblsoc*.dylib on Darwin.  The correct location in this 
instance is /Users/bacon/Pkgsrc/pkg/lib.  The following is a workaround:

MAKE_ENV+=    DYLD_LIBRARY_PATH=${PREFIX}/lib

LD_LIBRARY_PATH is not set in the NetBSD case.

Now I'm wondering whether this is a py-tables bug, or a pkgsrc 
shortcoming on Darwin.  Why would ctypes.CDLL() find a .so but not a 
.dylib when given the same information?

BTW, there are 3 committed packages using this workaround:

# pkg-grep 'DYLD_LIBRARY.*PREFIX'
/home/bacon/Pkgsrc/pkgsrc/wip/py-tables/Makefile:MAKE_ENV+= 
DYLD_LIBRARY_PATH=${PREFIX}/lib
/home/bacon/Pkgsrc/pkgsrc/security/p5-Crypt-OpenSSL-RSA/Makefile:MAKE_ENV+= 
   DYLD_LIBRARY_PATH=${PREFIX}/lib # build on macOS
/home/bacon/Pkgsrc/pkgsrc/security/p5-Crypt-OpenSSL-Random/Makefile:MAKE_ENV+= 
DYLD_LIBRARY_PATH=${PREFIX}/lib # build on macOS
/home/bacon/Pkgsrc/pkgsrc/security/p5-Net-SSLeay/Makefile:MAKE_ENV+= 
      DYLD_LIBRARY_PATH=${PREFIX}/lib # build on Mac OS X

Maybe this should be unnecessary.

--
Life is a game.  Play hard.  Play fair.  Have fun.