Re: Bootstrapping pkgsrc with a compiler

[Brook Milligan <brook%nmsu.edu@localhost>]

> On Jul 3, 2020, at 8:21 AM, Greg Troxel <gdt%lexort.com@localhost> wrote:
> 
> Brook Milligan <brook%nmsu.edu@localhost> writes:
> 
>> However, I encounter systems in which the native compiler is
>> inadequate for general use and will not reliably compile all of
>> pkgsrc.  It is also the case that one might wish to use a different,
>> pkgsrc-supplied, compiler even if the native one is adequate.
>> 
>> In those cases, it is necessary to bootstrap pkgsrc, build a pkgsrc
>> compiler with the native compiler, then reset things like mk.conf to
>> direct pkgsrc to use the new compiler for the remainder.
> 
> That is Jason's approach with auto-pkgsrc-setup.

I think that is right.  However, his approach involves running yet another script.  I feel it would be an improvement to include what is needed in the bootstrap script in a way that can be configured with a file or command options.  Why should we require another layer on top of our bootstrap in order to bootstrap?

>> I am raising the possibility that those steps could be combined into a
>> bootstrap with compiler step, followed by normal use of pkgsrc, which
>> would have been already configured in the first step to use the
>> bootstrapped compiler.  I feel this would make using non-native
>> compilers with pkgsrc, a definite need in some cases, much more
>> feasible.
> 
> I don't really understand what the bootstrap/compiler stage does, as it
> has to build some things with the native compiler.  Are those then "make
> replaced"?

I was not thinking that they would be.  I’m not even sure why they would need to be, as they would not be if you set GCC_REQD to something and built all of pkgsrc with it.  The primary point here is whether that can be set up during the bootstrap phase or must only be done later.

I see three use cases.

(i) The native compiler is fine and should be used for all of pkgsrc.  This seems to be assumed to be the common case, and the bootstrap script of course supports this so nothing special is required.

(ii) The native compiler is fine but one wishes to choose a different pkgsrc-supplied compiler for building packages.  The bootstrap script does not need special support for this.  Prior to a normal bootstrap, the makefile fragment needs to include USE_PKGSRC_GCC=yes and GCC_REQD=8 (or whatever version).  After bootstrap, the next package requiring compiling will depend on the selected compiler so it will be built as a normal dependency.  Everything up through building this compiler will be done by the native compiler and not replaced later.

(iii) The native compiler is _not_ fine in general, but is sufficient to build whatever pkgsrc compiler is required.  The bootstrap script currently does not support this and manual intervention is required.  That manual intervention involves the following: (a) normal bootstrap, (b) build and install some compiler from pkgsrc, (c) modify etc/mk.conf to include USE_PKGSRC_GCC=yes and GCC_REQD=, (d) build other packages, which will now depend on the previously built pkgsrc compiler.  As in all other cases, the compiler in use will have been built by the native compiler and not replaced later.  As my naive minimal patch indicates, however, it is not a big leap to have the bootstrap script support this use case directly and build the compiler during the bootstrap phase.  If it did, then all of these cases can be supported by a single bootstrap process with a single run of the bootstrap script; each would be selected by command options.  If nothing else, this makes this case much less error prone and does not require extra steps.  It may also lead to better consistency and easier compiler selection logic.  However, those directions depend more on the selection of the default bootstrap than on the basic capacity to use a single bootstrap script for all of these use cases.

> I increasingly wonder about the less magic and seemingly simpler strategy of:
> 
> 1) bootstrap pksrc with the native compiler to /usr/pkg/COMPILER
>   build some compiler

I think you are suggesting introducing a new directory under /usr/pkg that would contain whatever compiler is built during the bootstrap, right?

I think you are also thinking that if you later built the same or different compiler it would go into its normal location, for example /usr/pkg/gcc8.

Is that correct?

> 2) bootstrap pkgsrc to /usr/pkg, setting the compiler to be used as
>   /usr/pkg/COMPILER.
> 
> Perhaps step 2 is already doable with just something like
> --compiler=/usr/pkg/COMPILER/gcc7/bin/gcc and it's a small matter to add
> a script to do two bootstraps.

Why do we need two bootstraps?  You are still relying on the native compiler to build whatever compiler is being used to build pkgsrc and you are not replacing the compilers with anything later.  (I am confused why you were thinking of ever using “make replace”, though.)

Is there any substantive difference between /usr/pkg/COMPILER and /usr/pkg/gcc8 (or whatever version)?  I’m not seeing one and so it feels like needless complexity.

> Or, to add
> 
>  --bootstrap-compiler=gcc7
> 
> that will run the sub-bootstrap as a subroutine and then use it.

Please do not confuse the trivial patch I sent with what would be the real implementation of the idea.  Of course, the selection logic would be backed by command line options like you suggest here in order to do the configuration.  I was only trying to illustrate what could be the minimal impact of actually using whatever options were selected.  That is to say, a real patch would include command option logic that would ultimately execute some properly configured version of the small bit of code that I actually patched manually.  I felt that we could discuss the idea of this independently of the command option logic, especially if we can see from the patch what execution path would be influenced.

> One thing that's important is to have an easy way for this to be
> declared as default so that bulk builds use it, if we decide a certain
> approach is the way to go.  This perhaps includes NetBSD where we are
> seeing that the base system compiler on 7 is really too old and the one
> on 8 is already too old.

The obvious way to declare defaults would be the same as for anything in the bootstrap, wouldn’t it?  Just have default values for the appropriate command options.  Again, that is an entirely separate discussion from whether or not the "single bootstrap with selected compiler” approach makes sense as an option.

It seems that we are a bit off track and making this more complex than it really needs to be.  If feel if we can focus on the use cases I identified above (are there others?) we can also focus on how to address them in the simplest way.

Cheers,
Brook