At Thu, 3 Mar 2022 09:49:43 +0000, Jonathan Perkin <jperkin%joyent.com@localhost> wrote:
Subject: Re: macOS 12.x (Monterey) vs. pkgsrc
>
> It's pretty simple, REQUIRES and PROVIDES must be full paths to the
> libraries. Your "fix" means that the PROVIDES for libavltree and the
> REQUIRES for any packages that depend on it will simply list
> "libavltree.so.1.1" rather than e.g. "/opt/pkg/lib/libavltree.so.1.1",
> and any attempt to install those packages using pkgin or similar will
> result in a "libavltree.so.1.1 is not available on this system" or
> similar error.
I'm beginning to understand much more about macOS ld(1) and dyld(1), and
while my old hack only bypasses the install problem, the current
situation is no better and is not actually going to achieve ANY good
result.
On Darwin/macOS if Mansour's recommendation is more or less followed,
along with using the proper ".dylib" naming convention, then the library
looks like this:
$ otool -L /opt/pkg/lib/libyajl.dylib
/opt/pkg/lib/libyajl.dylib:
@rpath/libyajl.2.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
(I haven't got the right version in there yet)
Then when you link a program with that library, like this:
$ cc -o json_reformat json_reformat.o -L/opt/pkg/lib -Wl,-rpath,/opt/pkg/lib -lyajl
You end up with the binary saying this:
$ otool -L json_reformat
json_reformat:
@rpath/libyajl.2.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
and the secret sauce for finding the library is recorded in the binary's
LC_RPATH:
$ otool -L json_reformat | grep -A2 LC_RPATH <
cmd LC_RPATH
cmdsize 32
path /opt/pkg/lib (offset 12)
(and just to be clear, the resulting binary does run and work)
In any case treating the LC_ID_DYLIB field as a plain pathname
requirement of the library itself is just plain and obviously very
wrong.
In fact the value of LC_ID_DYLIB can be changed in a number of more
complex ways at build time, the best of which Mansour showed, but for
the purposes of pkgsrc finding _OTHER_ requirements for the library
itself, it must simply be ignored by pkgsrc, as more or less as I've
shown.
Using install_name_tool(1) to bash a library's LC_ID_DYLIB field with
the absolute pathname for that library is also arguably wrong (for all
package libraries, not just those produced by BSD Makefiles) even though
it papers over a number of issues for non-BSD-Makefile packages.
In any case my hack was slightly incorrect. The correct "otool -XL"
line to ignore is the one that matches the output of "otool -XD":
--- check-shlibs-macho.awk.~1.9.~ 2022-02-25 13:29:44.287094249 -0800
+++ check-shlibs-macho.awk 2022-03-03 12:54:02.914508668 -0800
@@ -69,6 +69,9 @@
}
function checkshlib(DSO, needed, found) {
+ cmd = "otool -XD " shquote(DSO) " 2>/dev/null"
+ cmd | getline instnm
+ close(cmd)
cmd = "otool -XL " shquote(DSO) " 2>/dev/null"
while ((cmd | getline) > 0) {
if ($0 !~ /^\t/)
@@ -85,6 +88,13 @@
ndirs = split(cross_destdir ":" destdir, check_dirs, ":")
for (lib in needed) {
#
+ # ignore the "install_name" -- libraries don't normally depend
+ # upon themselves, and it will likely contain special tokens
+ # like "@rpath/"
+ #
+ if (lib == instnm)
+ continue;
+ #
# Ensure we don't have any WRKDIR references.
#
if (lib == wrkdir ||
Even with all else in pkgsrc untouched, the above fix is now correct and
complete.
However for effective shared libraries from packages using BSD Makefiles
on Darwin/macos, the above fix _and_ both of Mansour's LDFLAGS are all
necessary, along with an as-yet implemented hack to link the ELF name to
lib${LIB}.dylib (and maybe auto-add that to the PLIST). Adding the
right LDFLAGS for setting the version numbers would be good too.
> Anyway, rather than more talk, I already committed a fixup of the
> library as I suggested earlier, and the package now builds on macOS.
That change is going to be effectively useless and actually quite
short-sighted and counterproductive in the long run.
In actual fact no shared library built by any package using BSD
Makefiles within pkgsrc as-is currently can be used as a shared library
on Darwin/macOS. That's because they're not installed with a file named
like "lib${LIB}.dylib".
The only really robust long-term fix will be to update bootstrap-mk-files
so that they actually "Do All The Right Things" when creating shared
libraries on Darwin/macOS, with one of those necessary things being to
use the "lib${LIB}.dylib" naming convention, another very necessary
thing being to use "-Wl,-rpath,${PREFIX}/lib" when linking against a
shared library.
> Just to be clear I'm definitely in favour of someone working on an
> update for bootstrap-mk-files. The problem is that it's an absolutely
> crazy amount of work to verify that merging over a decade of changes
> from upstream won't break any of the 20+ platforms we support, and as
> yet nobody has stepped up to be responsible for doing this, mostly
> because the risk is monumental and the rewards are microscopic.
That's why I suggest moving over to Simon's mk-files. They're already
portable to Darwin/macOS, and even to several other interesting
platforms, including (mostly) in how they build, install, and use shared
libraries for all those target platforms. I suspect, but have not yet
confirmed, that Simon's mk-files already cover all pkgsrc targets, and
of course adding new targets to them is relatively easy (assuming access
for testing).
In theory Simon's mk-files can be used directly as a drop-in
replacement, though of course they may need some fine tuning for the
pkgsrc environment.
That will get rid of most of the maintenance problem, at least so long
as Simon is willing to take any changes upstream that might be necessary
to tweak his mk-files for pkgsrc. It is also sad that devel/bmake is
way behind Simon's current release, though perhaps switching to his
mk-files would improve this situation for the future.
It is also a much bigger win for packages using BSD Makefiles since they
can trust pkgsrc bmake for more standard cross-platform support instead
of, for example, separately packaging and depending on Simon's mk-files.
If I can get enough of my development environment fixed and built on my
new machine I may be able to try this soon-ish, but I also have to get
on to upgrading non-macOS machines too!
--
Greg A. Woods <gwoods%acm.org@localhost>
Kelowna, BC +1 250 762-7675 RoboHack <woods%robohack.ca@localhost>
Planix, Inc. <woods%planix.com@localhost> Avoncote Farms <woods%avoncote.ca@localhost>
Attachment:
pgpmGUlMJz8GT.pgp
Description: OpenPGP Digital Signature