Re: pkg_install fails when archivers/xz is installed

[Jason Bacon <outpaddling%yahoo.com@localhost>]

On 2020-07-29 17:26, Greg Troxel wrote:
> Jason Bacon <outpaddling%yahoo.com@localhost> writes:
>
> > The point here is not to add system lib dirs, but to exclude
> > ${PREFIX}/lib.  It's my understanding that RPATH overrides all other
> > search paths, so this would ensure that pkg_install binaries don't get
> > fooled into using the wrong libs by LD_LIBRARY_PATH and such.
> That is not my understanding at all.  I expect LD_LIBRARY_PATH, any
> RPATH values, and system libs, in that order.  I would suggest that you
> experimentally validate this rather than make assumptions.
This was not an assumption.

From the ld man page:

           The linker uses the following search paths to locate required
           shared libraries:

           1.  Any directories specified by -rpath-link options.

           2.  Any directories specified by -rpath options.  The difference
               between -rpath and -rpath-link is that directories 
specified by
               -rpath options are included in the executable and used at
               runtime, whereas the -rpath-link option is only effective at
               link time. Searching -rpath in this way is only supported by
               native linkers and cross linkers which have been configured
               with the --with-sysroot option.

           3.  On an ELF system, for native linkers, if the -rpath and
               -rpath-link options were not used, search the contents 
of the
               environment variable "LD_RUN_PATH".

           4.  On SunOS, if the -rpath option was not used, search any
               directories specified using -L options.

           5.  For a native linker, search the contents of the environment
               variable "LD_LIBRARY_PATH".

etc...

Verification for GNU ld, showing that -rpath overrides use of liblzma in 
$LD_LIBRARY_PATH.

Linux centosdev.ceas  bacon ~ 596: (pkgsrc): cat lzma-test.c
#include <stdio.h>
#include <sysexits.h>
#include <lzma.h>

int     main(int argc,char *argv[])

{
    lzma_allocator  *allocator;

    lzma_index_init(allocator);
    return EX_OK;
}

Linux centosdev.ceas  bacon ~ 597: (pkgsrc): echo $LD_LIBRARY_PATH
/home/bacon/Pkgsrc/pkg/lib

Linux centosdev.ceas  bacon ~ 598: (pkgsrc): cc -o lzma-test lzma-test.c
/tmp/cc2n17BS.o: In function `main':
lzma-test.c:(.text+0x17): undefined reference to `lzma_index_init'
collect2: error: ld returned 1 exit status

Linux centosdev.ceas  bacon ~ 599: (pkgsrc): cc -o lzma-test lzma-test.c 
-llzma

Linux centosdev.ceas  bacon ~ 600: (pkgsrc): ldd lzma-test
    linux-vdso.so.1 =>  (0x00007ffd9a5c6000)
    liblzma.so.5 => /home/bacon/Pkgsrc/pkg/lib/liblzma.so.5 
(0x00007fa404594000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fa4041c6000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa403faa000)
    librt.so.1 => /lib64/librt.so.1 (0x00007fa403da2000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa4047b9000)

Linux centosdev.ceas  bacon ~ 601: (pkgsrc): cc -o lzma-test lzma-test.c 
-Wl,-rpath=/usr/lib64 -llzma

Linux centosdev.ceas  bacon ~ 602: (pkgsrc): ldd lzma-test
    linux-vdso.so.1 =>  (0x00007ffc65bc2000)
    liblzma.so.5 => /usr/lib64/liblzma.so.5 (0x00007fb4a5967000)
    libc.so.6 => /usr/lib64/libc.so.6 (0x00007fb4a5599000)
    libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007fb4a537d000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fb4a5b8d000)

> Sure, but these people are doing things not following best practices, to
> put it politely, and lots of things are going to break.  It's perfectly
> possible to install things with --prefix=$HOME/stuff and not need
> LD_LIBRARY_PATH becuase correct software puts $HOME/stuff/lib in RPATH
> in those binaries.  I have done this (on NetBSD) with no difficulty.
>
> I don't feel responsible for accomodating such practices, or that we
> need to or even should contort to deal with this case.  Are you saying
> that pkgsrc binaries should have some magic mechanism to override the
> normal OS handling and protect people from bad judgement?  This just
> doesn't make sense to me.
It's not a matter of whose fault it is, but whether we want to deal with 
PRs that could have been avoided.  If I can easily prevent end-users 
from shooting themselves in the foot without impinging on their freedom, 
I will always choose to do so.  Not just for their sake, but it saves my 
valuable time solving the same problems over and over or trying to 
convince people not to set LD_LIBRARY_PATH in their .bashrc.  Lots of 
HPC users do things like this, and yes my pkgsrc binaries are immune to 
it as long as they use rpath properly.
> > As I've stated before on this list, I'm leery of having my pkgsrc
> > installations rely on anything outside their control.  On CentOS
> > systems, for example, most Yum packages never get upgraded until you
> > upgrade the major version of the OS.  They only get bug fixes and
> > security patches.  So while pkgsrc is keeping up with the latest
> > releases, "base" liblzma is standing still for the several years the
> > system is in service and eventually it may be too old for pkgsrc to
> > use.
> I realize you don't like this, but I don't think it is at all rational
> to expect to build software on an OS and then allow random OS updates
> and expect that built software to still work.  Software in general is
> built against and ABI and if you change the ABI you have to start over.
> What you want is basically impossible, if any base system libs or
> binaries are useed in the resulting pkgsrc build.
>
> In pkgsrc we have pkg_rolling-replace to rebuild things when
> depdendencies change.  If you update the base OS then you more or less
> need new packages.  It's only more or less because in NetBSD we leave
> the old shlibs.  In $RANDOM_LINUX, I have no idea what the rules are.
This is exactly my point.  The fewer dependencies come from outside 
pkgsrc, the less often OS or other non-pkgsrc updates will break 
pkgsrc.  This is especially true if those dependencies come from 
something like Debian packages in a non-LTS distro, where ABIs are 
changing all the time.  A Debian update to lzma won't affect pkgsrc 
users who use pkgsrc lzma.  That's one reason I always set 
PREFER_PKGSRC=yes.  With this set, pkgsrc binaries use very little from 
base besides libc.
> > On Debian bases systems, it could mean a library getting upgraded
> > behind our backs by apt upgrade, resulting in an ABI/API change that
> > suddenly breaks a pkgsrc binary that's been working fine until now.
> That is why it is only sound to use pkgsrc binaries on Linux which are
> built against a configuration-managed set of base systems libraries.  I
> don't know why you are insisting that it should be possible to work
> against some arbitrary, shifting base.  If you mean there is some FHS
> that specifies a fixed ABI (with fixed shlib versions), and to build for
> that defined ABI, ok, but then you need to back off from "this lib is on
> my system" to "this lib is in the specification."
I don't know how you reached this conclusion, which is basically the 
opposite of what I'm saying.  I steer clear a shifting base as much as 
possible by using dependencies from pkgsrc instead.  On RHEL/CentOS, 
ABIs are preserved as much as possible through an entire major version 
of the OS to maintain compatibility for commercial software, so aging 
libraries is the problem there, rather than a shifting base.  But even 
on bleeding-edge linux distros, changes to the compiler and libc are far 
less frequent than other random libraries, so updates are unlikely to 
cause trouble when pkgsrc builds most other dependencies itself.
> I am talking about freetype2 or something like that.  If you don' have a
> stable base system, you need to configure things to depend on pkgsrc
> whenever that is possible.  If you want to propose changing pkgsrc to
> basically never trust Linux base system versions of things, if it is at
> all avoidable, I think that's an entirely reasonable thing to propose.
I don't think any core pkgsrc change is necessary since this is already 
fairly well-supported by PREFER_PKGSRC/PREFER_NATIVE.  I would only aim 
to educate people about their use.

    JB