RE: [External] Re: RHEL/Centos 7 builtin vs pkgsrc zlib clash for Makefiles that include devel/zlib/buildlink3.mk

["Lai, Peter PW" <peter.lai2%prattwhitney.com@localhost>]

> -----Original Message-----
> From: Jason Bacon [mailto:outpaddling%yahoo.com@localhost]
> Sent: Saturday, August 15, 2020 3:07 PM
> To: Lai, Peter PW <peter.lai2%prattwhitney.com@localhost>; pkgsrc-users%netbsd.org@localhost
> Subject: [External] Re: RHEL/Centos 7 builtin vs pkgsrc zlib clash for Makefiles
> that include devel/zlib/buildlink3.mk
> 
> On 2020-08-15 10:56, Lai, Peter PW wrote:
> >> -----Original Message-----
> >> From: Jason Bacon [mailto:outpaddling%yahoo.com@localhost]
> >> Sent: Friday, August 14, 2020 5:23 PM
> >> To: Lai, Peter PW <peter.lai2%prattwhitney.com@localhost>;
> >> pkgsrc-users%netbsd.org@localhost
> >> Subject: [External] Re: RHEL/Centos 7 builtin vs pkgsrc zlib clash
> >> for Makefiles that include devel/zlib/buildlink3.mk
> >>
> >> On 2020-08-10 12:05, Lai, Peter PW wrote:
> >>>> -----Original Message-----
> >>>> From: Jason Bacon [mailto:outpaddling%yahoo.com@localhost]
> >>>> Sent: Monday, August 10, 2020 10:00 AM
> >>>> To: Lai, Peter PW <peter.lai2%prattwhitney.com@localhost>;
> >>>> pkgsrc-users%netbsd.org@localhost
> >>>> Subject: [External] Re: RHEL/Centos 7 builtin vs pkgsrc zlib clash
> >>>> for Makefiles that include devel/zlib/buildlink3.mk
> >>>>
> >>>> On 2020-08-10 05:50, Lai, Peter PW wrote:
> >>>>> Environment:
> >>>>>
> >>>>> * RHEL 7.7 (Centos 7 for those playing at home I guess) in Azure.
> >>>>> * Toolchain: Software collection (SCL) devtoolset-9
> >>>>>
> >>>>> * Yum installs native zlib-devel as a dependency for openssl-devel
> >>>>>
> >>>>> * openssl-devel was installed in order to support PREFER_NATIVE=
> >>>> openssl
> >>>>> Observations:
> >>>>>
> >>>>> In this environment because zlib-devel is installed, bootstrap by
> >>>>> default
> >>>> picks up native zlib (/usr/include/zlib.h). When devel/libxml2 is
> >>>> then built, it is built against native zlib (libz.so.1 => /lib64/libz.so.1).
> >>>>> Next, some packages that also include
> >>>>> "../../devel/zlib/buildlink3.mk" such
> >>>> nginx is built, it pulls in devel/zlib. This is weird because RHEL
> >>>> 7 includes zlib
> >>>> 1.2.7 which satisfies zlib>=1.2.3 in buildlink3.
> >>>>> What happens after is that libxml2 and downstream libxml2
> >>>>> dependencies
> >>>> (e.g. databases/postgresql) break because they see the pkgsrc zlib
> first:
> >>>>> /mnt/datadisk01/pkg/bin/postgres:
> /mnt/datadisk01/pkg/lib/libz.so.1:
> >>>>> no version information available (required by
> >>>>> /mnt/datadisk01/pkg/lib/libxml2.so.2)
> >>>>>
> >>>>> # ldd /mnt/datadisk01/pkg/lib/libxml2.so.2
> >>>>> /mnt/datadisk01/pkg/lib/libxml2.so.2:
> >>>>> /mnt/datadisk01/pkg/lib/libz.so.1: no version information
> >>>>> available (required by /mnt/datadisk01/pkg/lib/libxml2.so.2)
> >>>>> /mnt/datadisk01/pkg/lib/libxml2.so.2:
> >>>>> /mnt/datadisk01/pkg/lib/libz.so.1: no version information
> >>>>> available (required by /mnt/datadisk01/pkg/lib/libxml2.so.2)
> >>>>>
> >>>>> I worked around this by adding zlib to PREFER_NATIVE when
> >>>>> re-building
> >>>> nginx, but is there a better way to resolve this? This looks
> >>>> somewhat related to the continuing "pkg_install fails when
> >>>> archivers/xz is installed" thread because RPATH manipulation could
> >>>> be involved, but I am opening a separate thread in case it is not.
> >>>> This does look like the same issue that we've been discussing
> >>>> regarding xz and it's a great example of the kind of problems that
> >>>> are avoided by using PREFER_PKGSRC=yes (and/or
> PREFER_NATIVE=no).
> >>>>
> >>>> What's your motive for using native openssl instead of pkgsrc?  The
> >>>> only issue I've seen with the latter is the need to manually
> >>>> install
> >>>> security/mozilla-rootcerts* or equivalent to make curl, etc. work
> >>>> with
> >> https.
> >>>> If there's some other disadvantage to pkgsrc openssl, maybe we
> >>>> should try to correct it.
> >>> The RHEL version of OpenSSL is backported/patched 1.0.2k with the
> >>> FIPS
> >> module compiled (and validated through vendor affirmed basis as per
> >>
> https://urldefense.com/v3/__https://www.openssl.org/docs/fips.html__;!!
> >> MvWE!V8OL2UV1NdFLyP770_BxDCbRbIfJxyghAQxVL9hpOXoMsXRu58-
> >> 41nFt2k-C-GdOu-Q44Q$  ). So provisional compliance with FIPS 140-2
> >> level 1 can be met for binaries linked to wrapper module when the
> >> system & kernel are fips enabled and the binary uses the required
> >> calling convention to autoconfigure or manually configure FIPS
> >> operation mode: We fork the pkgsrc repo and patch Makefiles for
> >> downstream builds' configure knobs where applicable to enable the
> calling of openssl's fips mode for those apps.
> >> I don't think this is worth correcting on the pkgsrc/NetBSD level for
> >> folks who need this niche requirement but I sent the message because
> >> the xz thread seemed related when dealing with a non-openssl-related
> >> base vs pkgsrc library issue...
> >> I bootstrapped on CentOS 7 using my modified bootstrap script
> >> (attached) with zlib-devel and openssl-devel preinstalled, so pkgsrc
> >> should be able to use native versions of both.  After installing
> >> libxml2, I verified that it's using pkgsrc zlib.  Packages that
> >> depend on openssl are not picking up the Yum installation, though,
> >> because it's one revision too old.  Yum's zlib meets the current
> >> requirements, though, so as a test I added it to PREFER_NATIVE and
> >> checked dependencies for libarchive before and after.  This confirmed
> that my mk.conf configuration is working as expected.
> >>
> >> So I'm not sure why your installation was using pkgsrc zlib.  Did you
> >> have anything in PREFER_* besides PREFER_NATIVE=openssl?
> >>
> >> Linux centosdev.ceas  bacon ~ 1072: (pkgsrc): yum list installed|grep
> >> zlib
> >> zlib.x86_64 1.2.7-18.el7                     @anaconda
> >> zlib-devel.x86_64 1.2.7-18.el7                     @base
> >>
> >> Linux centosdev.ceas  bacon ~ 1073: (pkgsrc): yum list installed |
> >> grep openssl
> >> openssl.x86_64 1:1.0.2k-19.el7                  @anaconda
> >> openssl-devel.x86_64 1:1.0.2k-19.el7                  @base
> >> openssl-libs.x86_64 1:1.0.2k-19.el7                  @anaconda
> >>
> >> Linux centosdev.ceas  bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
> >> 1074: (pkgsrc): grep PREFER ~/Pkgsrc/pkg-test/etc/mk.conf
> >> PREFER_PKGSRC=        yes PREFER_NATIVE=        openssl
> >>
> >> [ Install pkgsrc libxml2 ]
> >>
> >> Linux centosdev.ceas  bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
> >> 1075: (pkgsrc): ldd ~/Pkgsrc/pkg-test/lib/libxml2.so|grep libz
> >>       libz.so.1 => /home/bacon/Pkgsrc/pkg-test/lib/libz.so.1
> >> (0x00007fb48f8df000)
> >>
> >> Linux centosdev.ceas  bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
> >> 1076: (pkgsrc): bmake show-depends
> >> bzip2>=1.0.3:../../archivers/bzip2
> >> xz>=5.0.0:../../archivers/xz
> >> zlib>=1.2.3:../../devel/zlib
> >> openssl>=1.1.1dnb2:../../security/openssl
> >>
> >> [ Add zlib to PREFER_NATIVE ]
> >>
> >> Linux centosdev.ceas  bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
> >> 1078: (pkgsrc): grep PREFER ~/Pkgsrc/pkg-test/etc/mk.conf
> >> PREFER_PKGSRC=        yes PREFER_NATIVE=        openssl zlib
> >>
> >> Linux centosdev.ceas  bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
> >> 1079: (pkgsrc): bmake show-depends
> >> bzip2>=1.0.3:../../archivers/bzip2
> >> xz>=5.0.0:../../archivers/xz
> >> openssl>=1.1.1dnb2:../../security/openssl
> >>
> > Libxml2 will pick up native libz fine when built first. The problem
> > was www/nginx which pulled in pkgsrc libz while still linking to
> > native openssl. After installing pkgsrc libz to satisfy nginx, libxml2
> > broke. Adding zlib to PREFER_NATIVE resolved the issue. I didn't have
> > anything else in PREFER_NATIVE except for openssl
> So in the absence of a PREFER_PKGSRC setting and PREFER_NATIVE set to a
> list of specific packages, dependencies not in the PREFER_NATIVE list will be
> linked preferentially from pkgsrc over native if both are installed, otherwise
> to whichever is available. This by itself seems fairly reasonable to me, with
> some obvious risks especially on Linux. The problem here is that it doesn't
> account for pre-installed packages linked against a native dependency before
> the pkgsrc equivalent dependency was installed.
> 
> Bookstrapping with --prefer-pkgsrc (which will soon be the default for
> Linux) should avoid this issue by installing the pkgsrc dep regardless of
> whether there's a native equivalent available.
> 
> I don't see a simple way to address this issue without setting
> PREFER_PKGSRC during bootstrap.  Trying to avoid these issues using
> RPATH/RUNPATH or equivalent is looking a bit messy, if it's even feasible.
> 
> I still see a slight potential for problems if someone explicitly installs a pkgsrc
> package listed in PREFER_NATIVE, or if a bug causes it to be installed as a dep,
> but I'm not sure that's worth trying to prevent.
> 
>      JB
> 

My disconnect here is why would www/nginx have preferred pkgsrc libz when not explicitly stated by any makefile? It just includes zlib/buildlink3.mk in the exact way libxml2 does and none of nginx nor libxml2 options.mk prefers a zlib in any other configuration than default (which at the time was use native if found).