Re: Bootstrapping pkgsrc with a compiler

[Greg Troxel <gdt%lexort.com@localhost>]

Brook Milligan <brook%nmsu.edu@localhost> writes:

> In my experience, setting USE_PKGSRC_GCC=yes and GCC_REQD=8 leads to
> all packages (except of course prerequisites for the compiler and the
> compiler itself) to be built with gcc8.  If you also set the

Indeed.  But this is not "all packages".  It is "all packages except for
a pretty good-sized chunk".  This is what I think matters, and my
impression is that you think it doesn't.  Let's be careful to be
precise, even if we disagree on what matters.

> always-libgcc option when building the compiler, then everything will
> be linked to that compiler’s library as well.  I have not done this

I see the always-libgcc as a red herring in the compiler choice ABI
stability discussion.  It seems clear that if you build with gcc8 then
you have to link with gcc8's libs (or maybe N >8, but 5 is not ok).  So
I view that libgcc stuff as needing fixing regardless, and for this
discussion will assume it is fixed.

> for a bulk build, but that certainly seems to be the case for a
> selection of 500+ packages that I have tried recently.  From the
> documentation, that is also certainly what those variables are
> intended to accomplish.


> Thus, the answer to your question (1) is 100% built with gcc8 and 0%
> built with gcc5.  Of course, if a package itself sets GCC_REQD=9, then
> that would be built with a different compiler.  We could instead fail
> because of the mismatch.

No - you just said that gcc8 and the things needed to build with gcc8
are built with the base system compiler.   An example is gsed.

As for fail vs elevate, that's another separable discussion.  Until we
have a scheme to basically require a recent version for all, it isn't
reasonable to make the behavior be fail vs elevate.

> I think we are both concerned about consistency and stability.  I am
> not seeing how that is threatened by installing a selected compiler
> during bootstrap and sticking with that single choice throughout the
> build of packages.  I think that is exactly what those options do,

That is not what I said at all.   I am concerned about the compiler and
the packages needed to build them being built with a different
compiler.  If all packages in a given tree were truly built with the
same compiler, that would be fine.

> modulo package-specific requests for a higher version which we could
> handle if we want.

Right, this is separable.


> It is true that using these variables leaves the prerequisites being
> built with a different compiler.  Perhaps that is a critical point.
> If so, then we need a make replace or something equivalent on that set
> so that everything in the main tree is built with a single compiler.

Yes, this is the critical point and I have been trying very hard to be
clear on it.  (I acknowledge that in practice people are getting away
with it.)

> Or perhaps we need to do something more complex as part of the
> bootstrap.  To handle that, perhaps the bootstrap should install the
> compiler and all its prerequisites in one place and then use that
> exclusively to build things installed elsewhere.  Is that what you
> meant by /usr/pkg/COMPILER?

Yes, that's what I meant.

> I think that still requires only a single
> bootstrap to do, one that is not particularly different than what I
> suggested.  If the prerequisites are the issue, then it seems this is
> the best way to guarantee that all packages installed in the main tree
> are compiled with the same compiler.

I think the prerequisites are an issue.  I think that usually we can get
away with it, but with C++ there is no guarantee of ABI stability and
thus technically what you want to do is undefined behavior.  So I don't
want us to construct a scheme that relies on undefined behavior, when I
see no good reason to do so.

what I meant is

  1) a) bootstrap once, to /usr/pkg/COMPILER, mostly (entirely?) as we
        do now
     b) in /usr/pkg/COMPILER, build gcc8

  2) a) bootstrap to /usr/pkg, using the gcc8 above as the compiler
     b) build packages, which will lead to installing gcc-libs


Of course step 1 can be invoked by 2, so it is only "one bootstrap" that
does some things twice from the user POV.   But in fact a lot of things
are done twice.  I think that's ok:

  soundness is precious
  user brain time is valuable
  cpu time is pretty cheap


As someone else said, this is a bit like pbulk

  use pkgsrc to build meta stuff in one tree
  use that tree to build another

> Perhaps I am confused about these variables, either in intent or
> effect.  If so, please correct me.  Otherwise, I think it comes down
> to (i) whether the prerequisites are ok or should be replaced once the
> right compiler is built and (ii) whether we should fail if a package
> requires an even more recent compiler than is selected.

Those are key questions.  I think

  (i) it is not ok to have prereqs built with another compiler installed
  such that they might be used as a dependency.  This is UB, and there
  is no good reason to accept it.  Certainly nobody has put forth a
  compelling argument, and nobody has pointed to ABI stability rules
  (for gnu and C++, and for clang, and for every other compiler pkgsrc
  supports).

  (ii) I tend to say fail by default, but this is separable.  (This
  leads to saying you need to use a single new compiler.)

  (iii) the libgcc issue is separable too

  (iv) things that can be kept separate should be kept separate, as
  complexity estimates N + M is less than (N + M) ^ 1.5, waving hands
  wildly.