Re: gcc*-libs help needed: rpath needs fixing

[Jonathan Perkin <jperkin%mnx.io@localhost>]

* On 2024-09-10 at 10:18 BST, Thomas Klausner wrote:

> On Sat, Sep 07, 2024 at 06:11:07PM +0100, Jonathan Perkin wrote:
> > Yes, this is what I'd do (though not at bootstrap time), but I think that
> > due to the NetBSD release model this is unlikely to be accepted, given it
> > would mean that you'd no longer be able to ship lots of packages for the
> > most recent release (any package that has GCC_REQD+=11 or newer).
>
> It might need a mindset and pkgsrc bulk machine setup change, but if
> we get the setup to be easy enough...?
>
> What is the current procedure in your setup?

There's a tl;dr at the bottom, but I've tried to describe this fully to 
avoid any misunderstanding.  My setup is more complicated than one for 
NetBSD would be, as I use the same files for all of my builds so there's 
a bunch of variable usage in them, as well as the fact that building on 
SmartOS has some additional complications like there being no compiler 
on the system at all, and various tools needing to come from pkgsrc as 
they are not provided by the system (or are incompatible).

Each build uses what I call a "tools" prefix, that provides anything 
that the bulk build needs to operate.  Most people running bulk builds 
should be familiar with this, and will usually have /usr/pbulk as a 
separate prefix that contains the pbulk tools.  I extend this to also 
contain a bootstrap compiler, as well as any tools that are in the 
bootstrap path for the compiler I've chosen for my main prefix.

For SmartOS this tools prefix is installed under /opt/tools, and the 
main prefix I build for is /opt/local.  For NetBSD this would likely be 
/usr/pbulk and /usr/pkg.  To keep things simple, assume that I'm talking 
about how to build /opt/local with GCC 13 (building /opt/tools itself is 
similar, but it uses a previous /opt/local build to bootstrap itself).

I have three files that are included in the target mk.conf.  They are 
all shared across multiple builds, and so most things use variables that 
are set earlier in each specific mk.conf.

 https://github.com/TritonDataCenter/pkgbuild/blob/master/include/exttools.mk

  - This sets up TOOLS_PLATFORM.<tool> for anything in the compiler
    bootstrap path, to significantly reduce the number of dependencies
    that the compiler package has.  So for example:

      TOOLS_PLATFORM.m4=       /opt/tools/bin/m4
      TOOLS_PLATFORM.makeinfo= /opt/tools/bin/makeinfo
      TOOLS_PLATFORM.perl=     /opt/tools/bin/perl
      ...

    means I no longer have to bootstrap any of these packages in order
    to build the compiler.  On SmartOS this is critical because a lot of
    these packages and their dependencies will need libgcc, and that
    would cause circular dependencies.

 https://github.com/TritonDataCenter/pkgbuild/blob/master/include/pkgsrc-gcc.mk

  - This selects either GCCBASE for packages that are in the bootstrap
    path, or enables USE_PKGSRC_GCC=yes for everything else, along with
    GCC_REQD+=13 to ensure that the same compiler is used for everything.

 https://github.com/TritonDataCenter/pkgbuild/blob/master/include/gcc-libs.mk

  - This one ensures that _USE_GCC_SHLIB is set for all packages (thus
    making them depend on gcc13-libs) unless they are in the skip list
    at the top.  This is mostly a hack for the fact that USE_GCC_RUNTIME
    is missing in many packages, and is needed on illumos due to
    incompatibilities between the GCC and libc unwinders.  It's possible
    you can skip this entirely on NetBSD.

tl;dr:

For NetBSD, here's likely what you'd need to do, assuming GCC 13 is 
chosen as the pkgsrc compiler, due to being suitable for building all of 
current pkgsrc without having the pain of GCC 14 brokenness yet.

 - Build /usr/pbulk and include any tools that are in the lang/gcc13
   bootstrap path.

 - Write a mk.conf that uses those tools when building lang/gcc13 or any
   of its remaining dependencies (see exttools.mk).

 - Add to mk.conf an additional section that sets:

     .if ${PKGPATH:Mgcc/dep} || ${PKGPATH:M...} ...
     GCCBASE=			/usr/pbulk/path/to/gcc
     .else
     GCC_REQD+=			13
     USE_PKGSRC_GCC=		yes
     USE_PKGSRC_GCC_RUNTIME=	yes
     .endif

    with PKGPATH:M for any remaining lang/gcc13 dependency, plus itself
    (see pkgsrc-gcc.mk).

  - Optionally handle gcc-libs.mk.  I don't know if this is required on
    NetBSD.

That way you'd have a binary package set where 99.9% of packages
will be built with the same GCC 13 compiler, while the packages will 
only depend on the smaller gcc13-libs package, saving space and 
downloads.

Happy to answer any further questions.  I know this stuff is more 
complicated than it needs to be, but a lot of that can't be helped, and 
once you have it set up it's very little maintenance.  I use the same 
files above for all of my releases, and all they need to do is select 
which compiler version they target and where it's located.

--
Jonathan Perkin   -   mnx.io   -   pkgsrc.smartos.org
Open Source Complete Cloud   www.tritondatacenter.com