At Thu, 3 Mar 2022 09:49:43 +0000, Jonathan Perkin <jperkin%joyent.com@localhost> wrote: Subject: Re: macOS 12.x (Monterey) vs. pkgsrc > > It's pretty simple, REQUIRES and PROVIDES must be full paths to the > libraries. Your "fix" means that the PROVIDES for libavltree and the > REQUIRES for any packages that depend on it will simply list > "libavltree.so.1.1" rather than e.g. "/opt/pkg/lib/libavltree.so.1.1", > and any attempt to install those packages using pkgin or similar will > result in a "libavltree.so.1.1 is not available on this system" or > similar error. I'm beginning to understand much more about macOS ld(1) and dyld(1), and while my old hack only bypasses the install problem, the current situation is no better and is not actually going to achieve ANY good result. On Darwin/macOS if Mansour's recommendation is more or less followed, along with using the proper ".dylib" naming convention, then the library looks like this: $ otool -L /opt/pkg/lib/libyajl.dylib /opt/pkg/lib/libyajl.dylib: @rpath/libyajl.2.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0) (I haven't got the right version in there yet) Then when you link a program with that library, like this: $ cc -o json_reformat json_reformat.o -L/opt/pkg/lib -Wl,-rpath,/opt/pkg/lib -lyajl You end up with the binary saying this: $ otool -L json_reformat json_reformat: @rpath/libyajl.2.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0) and the secret sauce for finding the library is recorded in the binary's LC_RPATH: $ otool -L json_reformat | grep -A2 LC_RPATH < cmd LC_RPATH cmdsize 32 path /opt/pkg/lib (offset 12) (and just to be clear, the resulting binary does run and work) In any case treating the LC_ID_DYLIB field as a plain pathname requirement of the library itself is just plain and obviously very wrong. In fact the value of LC_ID_DYLIB can be changed in a number of more complex ways at build time, the best of which Mansour showed, but for the purposes of pkgsrc finding _OTHER_ requirements for the library itself, it must simply be ignored by pkgsrc, as more or less as I've shown. Using install_name_tool(1) to bash a library's LC_ID_DYLIB field with the absolute pathname for that library is also arguably wrong (for all package libraries, not just those produced by BSD Makefiles) even though it papers over a number of issues for non-BSD-Makefile packages. In any case my hack was slightly incorrect. The correct "otool -XL" line to ignore is the one that matches the output of "otool -XD": --- check-shlibs-macho.awk.~1.9.~ 2022-02-25 13:29:44.287094249 -0800 +++ check-shlibs-macho.awk 2022-03-03 12:54:02.914508668 -0800 @@ -69,6 +69,9 @@ } function checkshlib(DSO, needed, found) { + cmd = "otool -XD " shquote(DSO) " 2>/dev/null" + cmd | getline instnm + close(cmd) cmd = "otool -XL " shquote(DSO) " 2>/dev/null" while ((cmd | getline) > 0) { if ($0 !~ /^\t/) @@ -85,6 +88,13 @@ ndirs = split(cross_destdir ":" destdir, check_dirs, ":") for (lib in needed) { # + # ignore the "install_name" -- libraries don't normally depend + # upon themselves, and it will likely contain special tokens + # like "@rpath/" + # + if (lib == instnm) + continue; + # # Ensure we don't have any WRKDIR references. # if (lib == wrkdir || Even with all else in pkgsrc untouched, the above fix is now correct and complete. However for effective shared libraries from packages using BSD Makefiles on Darwin/macos, the above fix _and_ both of Mansour's LDFLAGS are all necessary, along with an as-yet implemented hack to link the ELF name to lib${LIB}.dylib (and maybe auto-add that to the PLIST). Adding the right LDFLAGS for setting the version numbers would be good too. > Anyway, rather than more talk, I already committed a fixup of the > library as I suggested earlier, and the package now builds on macOS. That change is going to be effectively useless and actually quite short-sighted and counterproductive in the long run. In actual fact no shared library built by any package using BSD Makefiles within pkgsrc as-is currently can be used as a shared library on Darwin/macOS. That's because they're not installed with a file named like "lib${LIB}.dylib". The only really robust long-term fix will be to update bootstrap-mk-files so that they actually "Do All The Right Things" when creating shared libraries on Darwin/macOS, with one of those necessary things being to use the "lib${LIB}.dylib" naming convention, another very necessary thing being to use "-Wl,-rpath,${PREFIX}/lib" when linking against a shared library. > Just to be clear I'm definitely in favour of someone working on an > update for bootstrap-mk-files. The problem is that it's an absolutely > crazy amount of work to verify that merging over a decade of changes > from upstream won't break any of the 20+ platforms we support, and as > yet nobody has stepped up to be responsible for doing this, mostly > because the risk is monumental and the rewards are microscopic. That's why I suggest moving over to Simon's mk-files. They're already portable to Darwin/macOS, and even to several other interesting platforms, including (mostly) in how they build, install, and use shared libraries for all those target platforms. I suspect, but have not yet confirmed, that Simon's mk-files already cover all pkgsrc targets, and of course adding new targets to them is relatively easy (assuming access for testing). In theory Simon's mk-files can be used directly as a drop-in replacement, though of course they may need some fine tuning for the pkgsrc environment. That will get rid of most of the maintenance problem, at least so long as Simon is willing to take any changes upstream that might be necessary to tweak his mk-files for pkgsrc. It is also sad that devel/bmake is way behind Simon's current release, though perhaps switching to his mk-files would improve this situation for the future. It is also a much bigger win for packages using BSD Makefiles since they can trust pkgsrc bmake for more standard cross-platform support instead of, for example, separately packaging and depending on Simon's mk-files. If I can get enough of my development environment fixed and built on my new machine I may be able to try this soon-ish, but I also have to get on to upgrading non-macOS machines too! -- Greg A. Woods <gwoods%acm.org@localhost> Kelowna, BC +1 250 762-7675 RoboHack <woods%robohack.ca@localhost> Planix, Inc. <woods%planix.com@localhost> Avoncote Farms <woods%avoncote.ca@localhost>
Attachment:
pgpmGUlMJz8GT.pgp
Description: OpenPGP Digital Signature