On 11/26/24 10:48, Jason Bacon wrote:
On 11/26/24 08:38, Jonathan Perkin wrote:
* On 2024-11-26 at 14:23 GMT, Jason Bacon wrote:
Anyone interested in math/py-tables on macOS, please test the
following patch:
--- ../../math/py-tables/PLIST.Darwin 1969-12-31
18:00:00.000000000 -0600
+++ ./PLIST.Darwin 2024-11-26 08:21:14.059878353 -0600
@@ -0,0 +1 @@
+${PYSITELIB}/tables/libblosc2.dylib
This is necessary on my Mac M1, but I don't have the means to test
on Intel or older macOS versions.
This corroborates with the failures in the latest bulk build:
https://us-central.manta.mnx.io/pkgsrc/public/reports/Darwin/12.3/
arm64/20241125.1313/meta/report.html
That said I'd recommend seeing if there's a better fix that is less
likely to break on updates. Is there a particular reason only macOS
is shipping a shared library for this and not any other platforms?
The 1200+ line setup.py looks like a hack job. The logic leading to
libblosc being copied is here:
if not rundir:
loc = {
"posix": "the default library paths",
"nt": "any of the directories in %%PATH%%",
}[os.name]
if package.name == "blosc2":
# We will copy this into the tables directory
print(
" * Copying blosc2 runtime library to 'tables' dir"
" because it was not found in standard locations"
)
platform_system = platform.system()
if platform_system == "Linux":
shutil.copy(libdir / "libblosc2.so", ROOT /
"tables")
copy_libs += ["libblosc2.so"]
dll_dir = "/tmp/hdf5/lib"
# Copy dlls when producing the wheels in CI
if "bdist_wheel" in sys.argv and
os.path.exists(dll_dir):
shared_libs = glob.glob(str(libdir) + "/
libblosc2.so*")
for lib in shared_libs:
shutil.copy(lib, dll_dir)
elif platform_system == "Darwin":
shutil.copy(libdir / "libblosc2.dylib", ROOT /
"tables")
copy_libs += ["libblosc2.dylib"]
So the question is why it's finding libblosc in "standard locations"
on Linux, but not on macOS.
I traced the problem to this function in setup.py:
def find_runtime_path(self, locations=None):
"""
returns True if the runtime can be found
returns None otherwise
"""
# An explicit path can not be provided for runtime libraries.
# (The argument is accepted for compatibility with previous
# methods.)
# dlopen() won't tell us where the file is, just whether
# success occurred, so this returns True instead of a filename
for prefix in self._runtime_prefixes:
for suffix in self._runtime_suffixes:
print("find_runtime_path() trying ",
f"{prefix}{self.runtime_name}{suffix}")
try:
ctypes.CDLL(f"{prefix}{self.runtime_name}{suffix}")
except OSError:
pass
else:
return True
Called via:
path = find_path(use_locations)
Relevant output on Darwin:
use_locations = [PosixPath('/Users/bacon/Pkgsrc/pkg/lib'), PosixPath('/
usr/local/lib64'), PosixPath('/usr/local/lib'), PosixPath('/sw/lib64'),
PosixPath('/sw/lib'), PosixPath('/opt/lib64'), PosixPath('/opt/lib'),
PosixPath('/opt/local/lib64'), PosixPath('/opt/local/lib'), PosixPath('/
usr/lib64'), PosixPath('/usr/lib'), PosixPath('/lib64'), PosixPath('/lib')]
find_runtime_path() trying libblosc2.so
find_runtime_path() trying libblosc2.dylib
path is not set!! This is a problem.
On NetBSD:
use_locations = [PosixPath('/usr/lib'), PosixPath('/home/bacon/Pkgsrc/
pkg/lib'), PosixPath('/usr/local/lib64'), PosixPath('/usr/local/lib'),
PosixPath('/sw/lib64'), PosixPath('/sw/lib'), PosixPath('/opt/lib64'),
PosixPath('/opt/lib'), PosixPath('/opt/local/lib64'), PosixPath('/opt/
local/lib'), PosixPath('/usr/lib64'), PosixPath('/usr/lib'),
PosixPath('/lib64'), PosixPath('/lib')]
find_runtime_path() trying libblosc2.so
path = True
So, ctypes.CDLL() is finding libblosc.so and libblosc2.so on NetBSD and
Linux, but not libblsoc*.dylib on Darwin. The correct location in this
instance is /Users/bacon/Pkgsrc/pkg/lib. The following is a workaround:
MAKE_ENV+= DYLD_LIBRARY_PATH=${PREFIX}/lib
LD_LIBRARY_PATH is not set in the NetBSD case.
Now I'm wondering whether this is a py-tables bug, or a pkgsrc
shortcoming on Darwin. Why would ctypes.CDLL() find a .so but not
a .dylib when given the same information?
BTW, there are 3 committed packages using this workaround:
# pkg-grep 'DYLD_LIBRARY.*PREFIX'
/home/bacon/Pkgsrc/pkgsrc/wip/py-tables/Makefile:MAKE_ENV+=
DYLD_LIBRARY_PATH=${PREFIX}/lib
/home/bacon/Pkgsrc/pkgsrc/security/p5-Crypt-OpenSSL-RSA/
Makefile:MAKE_ENV+= DYLD_LIBRARY_PATH=${PREFIX}/lib # build on macOS
/home/bacon/Pkgsrc/pkgsrc/security/p5-Crypt-OpenSSL-Random/
Makefile:MAKE_ENV+= DYLD_LIBRARY_PATH=${PREFIX}/lib # build on macOS
/home/bacon/Pkgsrc/pkgsrc/security/p5-Net-SSLeay/Makefile:MAKE_ENV+=
DYLD_LIBRARY_PATH=${PREFIX}/lib # build on Mac OS X
Maybe this should be unnecessary.