Re: pkg_install fails when archivers/xz is installed

[Greg Troxel <gdt%lexort.com@localhost>]

Jason Bacon <outpaddling%yahoo.com@localhost> writes:

> The point here is not to add system lib dirs, but to exclude
> ${PREFIX}/lib.  It's my understanding that RPATH overrides all other
> search paths, so this would ensure that pkg_install binaries don't get
> fooled into using the wrong libs by LD_LIBRARY_PATH and such.

That is not my understanding at all.  I expect LD_LIBRARY_PATH, any
RPATH values, and system libs, in that order.  I would suggest that you
experimentally validate this rather than make assumptions.

> LD_LIBRARY_PATH should be respected in some situations, but I don't
> think this is one of them.  I think allowing users to break pkg_add or
> other critical tools just by setting an env variable would be bad PR
> for pkgsrc.  I can't think of a situation where it would be wise for
> an end-user to force pkgsrc binaries to use an outside library instead
> of the one that's vetted by pkgsrc.

I don't understand your logic.  If someone sets LD_LIBRARY_PATH the
system will follow it (except for Mac and SIP).  So I don't see how you
get to "allow" or "not allow".  We only get to say "if you do that, and
the pkgsrc install breaks, then you get to keep both pieces".  I agree
that most uses of LD_LIBRARY_PATH are a bad idea.

> When I see people using LD_LIBRARY_PATH, it's invariably for some
> cave-man installed software in their home directory and they don't
> expect it to have side-effects on things installed by a package
> manager.

Sure, but these people are doing things not following best practices, to
put it politely, and lots of things are going to break.  It's perfectly
possible to install things with --prefix=$HOME/stuff and not need
LD_LIBRARY_PATH becuase correct software puts $HOME/stuff/lib in RPATH
in those binaries.  I have done this (on NetBSD) with no difficulty.

I don't feel responsible for accomodating such practices, or that we
need to or even should contort to deal with this case.  Are you saying
that pkgsrc binaries should have some magic mechanism to override the
normal OS handling and protect people from bad judgement?  This just
doesn't make sense to me.

>> Option 4 is to figure out why pkgsrc liblzma is ever built on this
>> system, when the base system version is adequate.  Basically we need to
>> have a consistent decision about good enough or not good enough, and
>> this is generally true about bhiltin detection.  We have had cases where
>> base had version X and that was good enough for some things but other
>> things needed X+1 and pulled in pkgsrc.  This went really badly and we
>> resolved it by making builtin detection require a version that was
>> generally good enough, not just for the current program, but anything
>> normal.
>>
>> Why is the library installed in pkgsrc?  What if you pkg_delete that?
>> If something depends on that, why doesn't it use the base version?

> As I've stated before on this list, I'm leery of having my pkgsrc
> installations rely on anything outside their control.  On CentOS
> systems, for example, most Yum packages never get upgraded until you
> upgrade the major version of the OS.  They only get bug fixes and
> security patches.  So while pkgsrc is keeping up with the latest
> releases, "base" liblzma is standing still for the several years the
> system is in service and eventually it may be too old for pkgsrc to
> use.

I realize you don't like this, but I don't think it is at all rational
to expect to build software on an OS and then allow random OS updates
and expect that built software to still work.  Software in general is
built against and ABI and if you change the ABI you have to start over.
What you want is basically impossible, if any base system libs or
binaries are useed in the resulting pkgsrc build.

In pkgsrc we have pkg_rolling-replace to rebuild things when
depdendencies change.  If you update the base OS then you more or less
need new packages.  It's only more or less because in NetBSD we leave
the old shlibs.  In $RANDOM_LINUX, I have no idea what the rules are.

> On Debian bases systems, it could mean a library getting upgraded
> behind our backs by apt upgrade, resulting in an ABI/API change that
> suddenly breaks a pkgsrc binary that's been working fine until now.

That is why it is only sound to use pkgsrc binaries on Linux which are
built against a configuration-managed set of base systems libraries.  I
don't know why you are insisting that it should be possible to work
against some arbitrary, shifting base.  If you mean there is some FHS
that specifies a fixed ABI (with fixed shlib versions), and to build for
that defined ABI, ok, but then you need to back off from "this lib is on
my system" to "this lib is in the specification."

> With all the platforms we support, I think there are just too many
> possible variants of a given library and too many possible problems
> for us to keep up with, so I always set PREFER_PKGSRC to minimize
> unpleasant surprises.

That is wise because you are obviously failing to achieve adequate
soundness to make relying on base system ABI sensible.  I don't mean
that as a slur; relying on something that is not really fixed with no
ABI changes just isn't going to work.  If people expect to run your
binaries on systems which are from the CM point of view uncontrolled,
you are going to have trouble.  Perhaps you can hand-build the min
prereqs for pkgsrc into /usr/pkgbootstrap and then bootstrap with that
in a way that ignores /usr/{bin,lib} exactly, so the resulting /usr/pkg
only depends on libc and the system call interface.

> Setting acceptable versions will help, but versions aren't always the
> only variable.  Sometimes there are build options to enable/disable
> features that could affect us.  I've been hit by this a few times in
> the past, but I don't recall the specific software in question.  Maybe
> boost?

I am talking about freetype2 or something like that.  If you don' have a
stable base system, you need to configure things to depend on pkgsrc
whenever that is possible.  If you want to propose changing pkgsrc to
basically never trust Linux base system versions of things, if it is at
all avoidable, I think that's an entirely reasonable thing to propose.


>> I am assumign that you have not changed the base system during all
>> this.  (If you did, the answer is don't do that!)
>>
> One change: "yum remove xz-devel" and bootstrapped a new pkgsrc tree,
> which is working fine now.

If you boottsrap and build after your change, that's not a change.
That's a new pkgsrc on a different system, which is ok.  What I meant is
that after bootstrap, you can't make ABI changes in the base system.