Re: Packages not honouring CFLAGS

[David Holland <dholland-pkgtech%netbsd.org@localhost>]

On Sun, Aug 09, 2015 at 04:55:25PM +0200, Pierre Pronchery wrote:
 > Some packages are not affected by the environment - including mine
 > (deforaos-*). Those which do are apparently fine already; I think this
 > is how GNU autoconf catches the relevant flags for instance.

Right, the way you add things for autoconf packages is by setting
CFLAGS et al in the environment when running configure; it explicitly
fetches them there and inserts them where needed.

 > As a software author, I do not like when the build process is silently
 > affected by the environment. This has bitten me a lot in the past,
 > particularly when working on Debian packages: they do (did?) not filter
 > the environment in any way while building, with many packages shipping
 > impossible to build without the same (undocumented) environment as the
 > maintainer - or worse, building differently.
 > 
 > This is why the build system of my DeforaOS stack does not accept any
 > default value from the environment, but allows them to be overridden
 > explicitly.

I agree, and we are far from alone on this. I don't really approve of
the autoconf behavior either; I would rather all such things were
explicit arguments and the environment contents were entirely
ignored.

Unfortunately, the way make variables are arranged is such that the
only really useful way to pass extra settings into make is via the
environment.

The upshot of all this is that honoring external CFLAGS (from pkgsrc
or wherever) in general requires explicit patching.

For simple packages this is usually no more complex than
   -CFLAGS=-O2 -Wall
   +CFLAGS+=-O2 -Wall
in a couple places; it is not hard to do and the problem is only that
there's a lot of it.

 > > If you want to make a significant dent in this you need (a) a way to
 > > determine if CFLAGS and LDFLAGS were actually honored for any given
 > > build product (PIE will probably do), (b) a way to make sure the
 > > output of all package builds conforms (this probably requires a custom
 > > check pass to find/test all executables), (c) run an ongoing bulk
 > > build series dedicated to the proposition, and (d) start slogging.
 > 
 > Yes, this is what I am currently doing. It works well for CFLAGS so far.
 > I suspect it will be more difficult for LDFLAGS, but the work on CFLAGS
 > should help determining which packages are affected.

A lot more packages honor LDFLAGS than CFLAGS, because of the need to
honor -R arguments generated by bl3. Some also honor LIBS so that
Solaris builds can inject -lsocket -lnsl. In most cases this required
someone to explicitly patch things... when I do it I try to make sure
that CFLAGS gets honored too, but that isn't (or hasn't been anyway)
general practice.

Anyway, if you are willing to do this it's undoubtedly worth doing;
it's just that it's a lot of work and a bit quixotic... do send the
build results to pkgsrc-bulk (and post a progress summary from time
to time) as at least some people will undoubtedly help out. (I might;
I have very little time for anything any more, but minor patching of
this kind is the sort of thing that can be done in random small slots
so it might happen.)

 > > > But even then, paving the way for proper
 > > > CFLAGS/LDFLAGS support should also relieve the wrappers from dealing
 > > > with situations with flags mixing compilers, platforms or whatnot.
 > > 
 > > That is unlikely. I'm not sure what you think constitutes 'proper'
 > > CFLAGS/LDFLAGS support.
 > 
 > If a package expects to be built by, say, MSVC and uses flags specific
 > for it, then a wrapper for pkgsrc has to understand these flags to
 > remove (or convert) them. Since we support many platform/compiler
 > combinations, each wrapper should then (in theory) understand each of
 > those combinations, for every underlying tool ($CC, $LD, $AR...).

In theory the way it works is that pkgsrc provides a uniform
compilation environment that looks like it has a typical Unix cc/gcc,
and packages are supposed to be configured to issue tool arguments to
match; then if you're actually using Buggix SuperDuperC++ the wrappers
translate the semistandard arguments to the compiler-specific ones
actually understood.

Packages that detect (or expect to be told) that they're on bizarro
platforms and issue compiler flags to match tend to have to be
patched; but most packages don't do this as it breaks building with
gcc on those platforms. And those that do usually have manky build
systems that need patching anyway.

As the world continues to diverge we might eventually need to support
full N-to-N conversion of flags instead of just 1-to-N. That will
require having packages declare what environment the build is
expecting to be talking to, as well as a way of translating pkgsrc-
originating CFLAGS or LDFLAGS on the way into the build system as well
as translating them back on the way out. This would be a big chunk of
work... so far it hasn't been necessary.

-- 
David A. Holland
dholland%netbsd.org@localhost