On 2020-08-15 10:56, Lai, Peter PW wrote:
So in the absence of a PREFER_PKGSRC setting and PREFER_NATIVE set to a list of specific packages, dependencies not in the PREFER_NATIVE list will be linked preferentially from pkgsrc over native if both are installed, otherwise to whichever is available. This by itself seems fairly reasonable to me, with some obvious risks especially on Linux. The problem here is that it doesn't account for pre-installed packages linked against a native dependency before the pkgsrc equivalent dependency was installed.-----Original Message----- From: Jason Bacon [mailto:outpaddling%yahoo.com@localhost] Sent: Friday, August 14, 2020 5:23 PM To: Lai, Peter PW <peter.lai2%prattwhitney.com@localhost>; pkgsrc-users%netbsd.org@localhost Subject: [External] Re: RHEL/Centos 7 builtin vs pkgsrc zlib clash for Makefiles that include devel/zlib/buildlink3.mk On 2020-08-10 12:05, Lai, Peter PW wrote:-----Original Message----- From: Jason Bacon [mailto:outpaddling%yahoo.com@localhost] Sent: Monday, August 10, 2020 10:00 AM To: Lai, Peter PW <peter.lai2%prattwhitney.com@localhost>; pkgsrc-users%netbsd.org@localhost Subject: [External] Re: RHEL/Centos 7 builtin vs pkgsrc zlib clash for Makefiles that include devel/zlib/buildlink3.mk On 2020-08-10 05:50, Lai, Peter PW wrote:Environment: * RHEL 7.7 (Centos 7 for those playing at home I guess) in Azure. * Toolchain: Software collection (SCL) devtoolset-9 * Yum installs native zlib-devel as a dependency for openssl-devel * openssl-devel was installed in order to support PREFER_NATIVE=opensslObservations: In this environment because zlib-devel is installed, bootstrap by defaultpicks up native zlib (/usr/include/zlib.h). When devel/libxml2 is then built, it is built against native zlib (libz.so.1 => /lib64/libz.so.1).Next, some packages that also include "../../devel/zlib/buildlink3.mk" suchnginx is built, it pulls in devel/zlib. This is weird because RHEL 7 includes zlib 1.2.7 which satisfies zlib>=1.2.3 in buildlink3.What happens after is that libxml2 and downstream libxml2 dependencies(e.g. databases/postgresql) break because they see the pkgsrc zlib first:/mnt/datadisk01/pkg/bin/postgres: /mnt/datadisk01/pkg/lib/libz.so.1: no version information available (required by /mnt/datadisk01/pkg/lib/libxml2.so.2) # ldd /mnt/datadisk01/pkg/lib/libxml2.so.2 /mnt/datadisk01/pkg/lib/libxml2.so.2: /mnt/datadisk01/pkg/lib/libz.so.1: no version information available (required by /mnt/datadisk01/pkg/lib/libxml2.so.2) /mnt/datadisk01/pkg/lib/libxml2.so.2: /mnt/datadisk01/pkg/lib/libz.so.1: no version information available (required by /mnt/datadisk01/pkg/lib/libxml2.so.2) I worked around this by adding zlib to PREFER_NATIVE when re-buildingnginx, but is there a better way to resolve this? This looks somewhat related to the continuing "pkg_install fails when archivers/xz is installed" thread because RPATH manipulation could be involved, but I am opening a separate thread in case it is not. This does look like the same issue that we've been discussing regarding xz and it's a great example of the kind of problems that are avoided by using PREFER_PKGSRC=yes (and/or PREFER_NATIVE=no). What's your motive for using native openssl instead of pkgsrc? The only issue I've seen with the latter is the need to manually install security/mozilla-rootcerts* or equivalent to make curl, etc. work withhttps.If there's some other disadvantage to pkgsrc openssl, maybe we should try to correct it.The RHEL version of OpenSSL is backported/patched 1.0.2k with the FIPSmodule compiled (and validated through vendor affirmed basis as per https://urldefense.com/v3/__https://www.openssl.org/docs/fips.html__;!! MvWE!V8OL2UV1NdFLyP770_BxDCbRbIfJxyghAQxVL9hpOXoMsXRu58- 41nFt2k-C-GdOu-Q44Q$ ). So provisional compliance with FIPS 140-2 level 1 can be met for binaries linked to wrapper module when the system & kernel are fips enabled and the binary uses the required calling convention to autoconfigure or manually configure FIPS operation mode: We fork the pkgsrc repo and patch Makefiles for downstream builds' configure knobs where applicable to enable the calling of openssl's fips mode for those apps. I don't think this is worth correcting on the pkgsrc/NetBSD level for folks who need this niche requirement but I sent the message because the xz thread seemed related when dealing with a non-openssl-related base vs pkgsrc library issue... I bootstrapped on CentOS 7 using my modified bootstrap script (attached) with zlib-devel and openssl-devel preinstalled, so pkgsrc should be able to use native versions of both. After installing libxml2, I verified that it's using pkgsrc zlib. Packages that depend on openssl are not picking up the Yum installation, though, because it's one revision too old. Yum's zlib meets the current requirements, though, so as a test I added it to PREFER_NATIVE and checked dependencies for libarchive before and after. This confirmed that my mk.conf configuration is working as expected. So I'm not sure why your installation was using pkgsrc zlib. Did you have anything in PREFER_* besides PREFER_NATIVE=openssl? Linux centosdev.ceas bacon ~ 1072: (pkgsrc): yum list installed|grep zlib zlib.x86_64 1.2.7-18.el7 @anaconda zlib-devel.x86_64 1.2.7-18.el7 @base Linux centosdev.ceas bacon ~ 1073: (pkgsrc): yum list installed | grep openssl openssl.x86_64 1:1.0.2k-19.el7 @anaconda openssl-devel.x86_64 1:1.0.2k-19.el7 @base openssl-libs.x86_64 1:1.0.2k-19.el7 @anaconda Linux centosdev.ceas bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive 1074: (pkgsrc): grep PREFER ~/Pkgsrc/pkg-test/etc/mk.conf PREFER_PKGSRC= yes PREFER_NATIVE= openssl [ Install pkgsrc libxml2 ] Linux centosdev.ceas bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive 1075: (pkgsrc): ldd ~/Pkgsrc/pkg-test/lib/libxml2.so|grep libz libz.so.1 => /home/bacon/Pkgsrc/pkg-test/lib/libz.so.1 (0x00007fb48f8df000) Linux centosdev.ceas bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive 1076: (pkgsrc): bmake show-depends bzip2>=1.0.3:../../archivers/bzip2 xz>=5.0.0:../../archivers/xz zlib>=1.2.3:../../devel/zlib openssl>=1.1.1dnb2:../../security/openssl [ Add zlib to PREFER_NATIVE ] Linux centosdev.ceas bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive 1078: (pkgsrc): grep PREFER ~/Pkgsrc/pkg-test/etc/mk.conf PREFER_PKGSRC= yes PREFER_NATIVE= openssl zlib Linux centosdev.ceas bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive 1079: (pkgsrc): bmake show-depends bzip2>=1.0.3:../../archivers/bzip2 xz>=5.0.0:../../archivers/xz openssl>=1.1.1dnb2:../../security/opensslLibxml2 will pick up native libz fine when built first. The problem was www/nginx which pulled in pkgsrc libz while still linking to native openssl. After installing pkgsrc libz to satisfy nginx, libxml2 broke. Adding zlib to PREFER_NATIVE resolved the issue. I didn't have anything else in PREFER_NATIVE except for openssl
Bookstrapping with --prefer-pkgsrc (which will soon be the default for Linux) should avoid this issue by installing the pkgsrc dep regardless of whether there's a native equivalent available.
I don't see a simple way to address this issue without setting PREFER_PKGSRC during bootstrap. Trying to avoid these issues using RPATH/RUNPATH or equivalent is looking a bit messy, if it's even feasible.
I still see a slight potential for problems if someone explicitly installs a pkgsrc package listed in PREFER_NATIVE, or if a bug causes it to be installed as a dep, but I'm not sure that's worth trying to prevent.