-----Original Message-----
From: Jason Bacon [mailto:outpaddling%yahoo.com@localhost]
Sent: Saturday, August 15, 2020 3:07 PM
To: Lai, Peter PW <peter.lai2%prattwhitney.com@localhost>; pkgsrc-users%netbsd.org@localhost
Subject: [External] Re: RHEL/Centos 7 builtin vs pkgsrc zlib clash for Makefiles
that include devel/zlib/buildlink3.mk
On 2020-08-15 10:56, Lai, Peter PW wrote:
-----Original Message-----
From: Jason Bacon [mailto:outpaddling%yahoo.com@localhost]
Sent: Friday, August 14, 2020 5:23 PM
To: Lai, Peter PW <peter.lai2%prattwhitney.com@localhost>;
pkgsrc-users%netbsd.org@localhost
Subject: [External] Re: RHEL/Centos 7 builtin vs pkgsrc zlib clash
for Makefiles that include devel/zlib/buildlink3.mk
On 2020-08-10 12:05, Lai, Peter PW wrote:
-----Original Message-----
From: Jason Bacon [mailto:outpaddling%yahoo.com@localhost]
Sent: Monday, August 10, 2020 10:00 AM
To: Lai, Peter PW <peter.lai2%prattwhitney.com@localhost>;
pkgsrc-users%netbsd.org@localhost
Subject: [External] Re: RHEL/Centos 7 builtin vs pkgsrc zlib clash
for Makefiles that include devel/zlib/buildlink3.mk
On 2020-08-10 05:50, Lai, Peter PW wrote:
Environment:
* RHEL 7.7 (Centos 7 for those playing at home I guess) in Azure.
* Toolchain: Software collection (SCL) devtoolset-9
* Yum installs native zlib-devel as a dependency for openssl-devel
* openssl-devel was installed in order to support PREFER_NATIVE=
openssl
Observations:
In this environment because zlib-devel is installed, bootstrap by
default
picks up native zlib (/usr/include/zlib.h). When devel/libxml2 is
then built, it is built against native zlib (libz.so.1 => /lib64/libz.so.1).
Next, some packages that also include
"../../devel/zlib/buildlink3.mk" such
nginx is built, it pulls in devel/zlib. This is weird because RHEL
7 includes zlib
1.2.7 which satisfies zlib>=1.2.3 in buildlink3.
What happens after is that libxml2 and downstream libxml2
dependencies
(e.g. databases/postgresql) break because they see the pkgsrc zlib
first:
/mnt/datadisk01/pkg/bin/postgres:
/mnt/datadisk01/pkg/lib/libz.so.1:
no version information available (required by
/mnt/datadisk01/pkg/lib/libxml2.so.2)
# ldd /mnt/datadisk01/pkg/lib/libxml2.so.2
/mnt/datadisk01/pkg/lib/libxml2.so.2:
/mnt/datadisk01/pkg/lib/libz.so.1: no version information
available (required by /mnt/datadisk01/pkg/lib/libxml2.so.2)
/mnt/datadisk01/pkg/lib/libxml2.so.2:
/mnt/datadisk01/pkg/lib/libz.so.1: no version information
available (required by /mnt/datadisk01/pkg/lib/libxml2.so.2)
I worked around this by adding zlib to PREFER_NATIVE when
re-building
nginx, but is there a better way to resolve this? This looks
somewhat related to the continuing "pkg_install fails when
archivers/xz is installed" thread because RPATH manipulation could
be involved, but I am opening a separate thread in case it is not.
This does look like the same issue that we've been discussing
regarding xz and it's a great example of the kind of problems that
are avoided by using PREFER_PKGSRC=yes (and/or
PREFER_NATIVE=no).
What's your motive for using native openssl instead of pkgsrc? The
only issue I've seen with the latter is the need to manually
install
security/mozilla-rootcerts* or equivalent to make curl, etc. work
with
https.
If there's some other disadvantage to pkgsrc openssl, maybe we
should try to correct it.
The RHEL version of OpenSSL is backported/patched 1.0.2k with the
FIPS
module compiled (and validated through vendor affirmed basis as per
https://urldefense.com/v3/__https://www.openssl.org/docs/fips.html__;!!
MvWE!V8OL2UV1NdFLyP770_BxDCbRbIfJxyghAQxVL9hpOXoMsXRu58-
41nFt2k-C-GdOu-Q44Q$ ). So provisional compliance with FIPS 140-2
level 1 can be met for binaries linked to wrapper module when the
system & kernel are fips enabled and the binary uses the required
calling convention to autoconfigure or manually configure FIPS
operation mode: We fork the pkgsrc repo and patch Makefiles for
downstream builds' configure knobs where applicable to enable the
calling of openssl's fips mode for those apps.
I don't think this is worth correcting on the pkgsrc/NetBSD level for
folks who need this niche requirement but I sent the message because
the xz thread seemed related when dealing with a non-openssl-related
base vs pkgsrc library issue...
I bootstrapped on CentOS 7 using my modified bootstrap script
(attached) with zlib-devel and openssl-devel preinstalled, so pkgsrc
should be able to use native versions of both. After installing
libxml2, I verified that it's using pkgsrc zlib. Packages that
depend on openssl are not picking up the Yum installation, though,
because it's one revision too old. Yum's zlib meets the current
requirements, though, so as a test I added it to PREFER_NATIVE and
checked dependencies for libarchive before and after. This confirmed
that my mk.conf configuration is working as expected.
So I'm not sure why your installation was using pkgsrc zlib. Did you
have anything in PREFER_* besides PREFER_NATIVE=openssl?
Linux centosdev.ceas bacon ~ 1072: (pkgsrc): yum list installed|grep
zlib
zlib.x86_64 1.2.7-18.el7 @anaconda
zlib-devel.x86_64 1.2.7-18.el7 @base
Linux centosdev.ceas bacon ~ 1073: (pkgsrc): yum list installed |
grep openssl
openssl.x86_64 1:1.0.2k-19.el7 @anaconda
openssl-devel.x86_64 1:1.0.2k-19.el7 @base
openssl-libs.x86_64 1:1.0.2k-19.el7 @anaconda
Linux centosdev.ceas bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
1074: (pkgsrc): grep PREFER ~/Pkgsrc/pkg-test/etc/mk.conf
PREFER_PKGSRC= yes PREFER_NATIVE= openssl
[ Install pkgsrc libxml2 ]
Linux centosdev.ceas bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
1075: (pkgsrc): ldd ~/Pkgsrc/pkg-test/lib/libxml2.so|grep libz
libz.so.1 => /home/bacon/Pkgsrc/pkg-test/lib/libz.so.1
(0x00007fb48f8df000)
Linux centosdev.ceas bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
1076: (pkgsrc): bmake show-depends
bzip2>=1.0.3:../../archivers/bzip2
xz>=5.0.0:../../archivers/xz
zlib>=1.2.3:../../devel/zlib
openssl>=1.1.1dnb2:../../security/openssl
[ Add zlib to PREFER_NATIVE ]
Linux centosdev.ceas bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
1078: (pkgsrc): grep PREFER ~/Pkgsrc/pkg-test/etc/mk.conf
PREFER_PKGSRC= yes PREFER_NATIVE= openssl zlib
Linux centosdev.ceas bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
1079: (pkgsrc): bmake show-depends
bzip2>=1.0.3:../../archivers/bzip2
xz>=5.0.0:../../archivers/xz
openssl>=1.1.1dnb2:../../security/openssl
Libxml2 will pick up native libz fine when built first. The problem
was www/nginx which pulled in pkgsrc libz while still linking to
native openssl. After installing pkgsrc libz to satisfy nginx, libxml2
broke. Adding zlib to PREFER_NATIVE resolved the issue. I didn't have
anything else in PREFER_NATIVE except for openssl
So in the absence of a PREFER_PKGSRC setting and PREFER_NATIVE set to a
list of specific packages, dependencies not in the PREFER_NATIVE list will be
linked preferentially from pkgsrc over native if both are installed, otherwise
to whichever is available. This by itself seems fairly reasonable to me, with
some obvious risks especially on Linux. The problem here is that it doesn't
account for pre-installed packages linked against a native dependency before
the pkgsrc equivalent dependency was installed.
Bookstrapping with --prefer-pkgsrc (which will soon be the default for
Linux) should avoid this issue by installing the pkgsrc dep regardless of
whether there's a native equivalent available.
I don't see a simple way to address this issue without setting
PREFER_PKGSRC during bootstrap. Trying to avoid these issues using
RPATH/RUNPATH or equivalent is looking a bit messy, if it's even feasible.
I still see a slight potential for problems if someone explicitly installs a pkgsrc
package listed in PREFER_NATIVE, or if a bug causes it to be installed as a dep,
but I'm not sure that's worth trying to prevent.
JB