tech-pkg archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: editors/emacs29 and gcc14-libjit default option breakage
Am Tue, 29 Apr 2025 00:36:46 +0200
schrieb Thomas Klausner <wiz%gatalith.at@localhost>:
> On Tue, Apr 29, 2025 at 12:21:07AM +0200, Dr. Thomas Orgis wrote:
> > Reading a bit on the gcc wiki about this jit thing, I figured out that
> > apart from the question which version of libgccjit would fit my
> > toolchain, I need such to successfully run the conftest:
> >
> > LIBRARY_PATH=/sw/compiler/gcc-13.2.0/lib:/sw/compiler/gcc-13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0:$LIBRARY_PATH ./conftest
>
> To me this looks like a problem in your toolchain. If these files are
> needed for libgccjit, aren't they needed for other stuff as well? If
> so, shouldn't these paths be in the system-wide ld.so.conf file?
The libgccjit build runs just fine and links in all libraries it needs.
The usual „problem” my toolchain has that I did not modify its spec
files to hardcode rpath flags to its own directories, to locate
libstdcd++, for example, as that would break all LD_RUN_PATH use.
Putting things into ld.so.conf is out of question since I am dealing
with multiple toolchains a user can choose per shell session via
Environment Modules. It's all in PATH-like environment variables.
The toolchain has no trouble locating its own libraries. The binary
using libgccjit is special in that it needs directy run-time access to
internal gcc libraries. Quoting
https://gcc.gnu.org/onlinedocs/jit/internals/index.html#working-on-the-jit-library
LD_LIBRARY_PATH
libgccjit.so is dynamically linked into client code, so if
running against a locally-built library, LD_LIBRARY_PATH needs
to be set up appropriately. The library can be found within the
“gcc” subdirectory of the build tree:
file libgccjit.so*
libgccjit.so: symbolic link to `libgccjit.so.0'
libgccjit.so.0: symbolic link to `libgccjit.so.0.0.1'
libgccjit.so.0.0.1: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, not stripped
PATH
The library uses a driver executable for converting from .s
assembler files to .so shared libraries. Specifically, it looks for
a name expanded from
${target_noncanonical}-gcc-${gcc_BASEVER}${exeext} such as
x86_64-unknown-linux-gnu-gcc-5.0.0.
Hence PATH needs to include a directory where the library can locate this executable.
The executable is normally installed to the installation bindir
(e.g. /usr/bin), but a copy is also created within the “gcc”
subdirectory of the build tree for running the testsuite, and for
ease of development.
LIBRARY_PATH
The driver executable invokes the linker, and the latter needs to
locate support libraries needed by the generated code, or you will
see errors like:
ld: cannot find crtbeginS.o: No such file or directory
ld: cannot find -lgcc
ld: cannot find -lgcc_s
Hence if running directly from a locally-built copy (without
installing), LIBRARY_PATH needs to contain the “gcc” subdirectory
of the build tree.
I do not have trouble with LD_LIBRARY_PATH since we use RPATH. I do not
have trouble with PATH, as
/sw/compiler/gcc-13.2.0/bin/x86_64-pc-linux-gnu-gcc-13.2.0 is found
just fine and in PATH due to the loaded module. But LIBRARY_PATH. In my
module, I actually add /sw/compiler/gcc-13.2.0/lib and
/sw/compiler/gcc-13.2.0/lib64 (did not manage/bother to get rid of the
distinction there) to LIBRARY_PATH and LD_RUN_PATH. But not
/sw/compiler/gcc-13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0/. Trying to
add that to the conftest call in the emacs29 configure-env is really funny:
$ LIBRARY_PATH="/sw/compiler/gcc-13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0:$LIBRARY_PATH" ./conftest
ld: cannot find -lc: No such file or directory
libgccjit.so: error: error invoking gcc driver
This works outside the pkgsrc/cwrapper environment. Inside
configure-env … this … is … just … broken? I mean, even
$ LIBRARY_PATH="/sw/compiler/gcc-13.2.0/lib/gcc/x86_64-pc-linux-gnu/13.2.0:$LIBRARY_PATH:/lib/x86_64-linux-gnu/" ./conftest
ld: cannot find -lc: No such file or directory
libgccjit.so: error: error invoking gcc driver
All the while
-rw-r--r-- 1 root root 283 Mar 6 23:46 /lib/x86_64-linux-gnu/libc.so
clearly exists. So this is very much not OK on Debian 12 with pkgsrc
and my toolchain. I repeated the test I describe below for Ubuntu 22.04
also here, and was able to both build and run the conftest with
/bin/gcc from debian and my /sw/compiler/gcc-13.2.0/bin/gcc, specifying
either of the private gcc directories in LIBRARY_PATH.
Within the pkgsrc buildlink/wrapper environment, there are additional
hurdles. I did not figure out yet why it even refuses to find libc.
> I don't know enough about how they work on Linux though.
>
> > I need LIBRARY_PATH set to locate all internal gcc libraries,
> > including crtbeginS.o. I don't need that for any other ordinary build.
> > This jit thing is special sauce. The toolchain knows where these files
> > are (though RPATH is a different matter, hence PASSHRU_RPATHDIRS is
> > needed).
>
> Or perhaps add them in the gccjit package build, if it's really only
> needed there.
This is a misunderstanding: gccjit builds and installs just fine. It is
the _use_ of libgccjit that fails when building emacs29. And with use I
mean compile, link and _run_, which is another compile/link … JIT.
>
> > I see two issues. Wait, no: three issues.
> >
> > 1. Isn't gccjit something too esoteric to suggest as default for a text
> > editor (at least emacs is in that category, not OS or IDE;-)?
>
> It works fine for me on NetBSD, and is a noticable speedup. You can
> disable it on Linux, but of course I'd prefer if it was fixed there
> instead.
For now we migh have to at least default the option to off until someone
figured out how this works properly on Linux systems.
> It doesn't seem to matter - on my NetBSD system, the base system has
> gcc12, and the emacs package uses libgcc14-jit without problems.
I do not see a guarantee for that, anywhere. If libgccjit is to be a
thing for my installs, I will build and install it together with gcc
and binutils. And then pkgsrc hopefully supports builtin libgccjit.
(Reminds me of committing changes about openmp.mk to use builtin openmp
lib where it exists. Pulling in clang's openmp library is not what I
normally want.).
> > 3. Does this work for anyone, actually? Using system gcc? Using pkgsrc
> > gcc? How?
>
> On NetBSD, yes, from pkgsrc as-is without any changes.
OK. Cannot argue with that;-)
> I guess the libgcc*jit packages need changes for Linux (and perhaps
> other operating systems) and I had hoped that people more versed in
> their operating systems would take care of that.
> Thomas
Just looking at the wiki page again and seeing the forced CFLAGS+=-fPIC
in the package …
The configure-time option --enable-host-shared is needed when
building the jit in order to get position-independent code.
This will slow down the regular compiler by a few percent.
Hence when packaging gcc with libgccjit, please configure and
build twice:
once without --enable-host-shared for most languages, and
once with --enable-host-shared for the jit
Did you intentionally decide against --enable-host-shared?
---- Ubuntu 22.04 below, Debian 12 with my toolchain above ----
Also, I don't find a statement about mixing libgccjit and base gcc
versions. There seems to be the assumption that you build those
together from the same source tree. Checking Ubuntu, I see
that they package
- libgccjit0: the library itself, version 12.3.0
- libgccjit-$version-dev: development packages, depending on gcc-$version
each and libgccjit0 ≥ $version
So they make use of the ABI backwards compatibility for the runtime
library, but separate development environments for gcc versions
matching their libgccjit. I just assume that they build
libgccjit0-$version with gcc-$version … building libgccjit-14 with
gcc-11 seems to be at least something less tested, if at all.
Trying to test this with pkgsrc on Ubuntu, using pkgsrc's libgccjit, I
see that I need to add --disable-multilib to the libgccjit build,
otherwise it wants to build 32 bit code, too, and misses respective
libraries. I don't have those (all) installed, but the base gcc
supports multilib. I don't do that in my custom toolchains and hence I
did not face that specific error on my main system.
The emacs29 build also fails on Ubuntu 22.04 with stock gcc 11. I can
reproduce the conftest like this, without any LIBRARY_PATH, LD_RUN_PATH
or LD_LIBRARY_PATH, base gcc and libgccjit from pkgsrc in
/data/pkg/gcc14/.
$ cat conftest.c
#include <libgccjit.h>
#include <stdlib.h>
#include <stdio.h>
int
main (int argc, char **argv)
{
gcc_jit_context *ctxt;
gcc_jit_result *result;
ctxt = gcc_jit_context_acquire ();
if (!ctxt)
exit (1);
gcc_jit_type *int_type =
gcc_jit_context_get_type (ctxt, GCC_JIT_TYPE_INT);
gcc_jit_function *func =
gcc_jit_context_new_function (ctxt, NULL,
GCC_JIT_FUNCTION_EXPORTED,
int_type, "foo", 0, NULL, 0);
gcc_jit_block *block = gcc_jit_function_new_block (func, "foo");
gcc_jit_block_end_with_return (
block,
NULL,
gcc_jit_context_new_rvalue_from_int (ctxt, int_type, 1));
result = gcc_jit_context_compile (ctxt);
if (!result)
exit (1);
typedef int (*fn_type) (void);
fn_type foo =
(fn_type)gcc_jit_result_get_code (result, "foo");
if (!foo)
exit (1);
if (foo () != 1)
exit (1);
gcc_jit_context_release (ctxt);
gcc_jit_result_release (result);
return 0;
}
$ gcc -o conftest -I/data/pkg/include -I/data/pkg/gcc14/include \
-L/data/pkg/gcc14/lib -Wl,-R/data/pkg/gcc14/lib conftest.c -lgccjit
$ ldd conftest
linux-vdso.so.1 (0x00007ffdf7860000)
libgccjit.so.0 => /data/pkg/gcc14/lib/libgccjit.so.0 (0x00007f804fa6d000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f804f815000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f804f7f9000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f804f5cd000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f804f4e6000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f804f4c4000)
/lib64/ld-linux-x86-64.so.2 (0x00007f805220a000)
$ LANG=C ./conftest
ld: cannot find crtbeginS.o: No such file or directory
ld: cannot find -lgcc: No such file or directory
ld: cannot find -lgcc_s: No such file or directory
libgccjit.so: error: error invoking gcc driver
$ LANG=C LIBRARY_PATH=/data/pkg/gcc14/lib ./conftest
ld: cannot find crtbeginS.o: No such file or directory
ld: cannot find -lgcc: No such file or directory
ld: cannot find -lgcc_s: No such file or directory
libgccjit.so: error: error invoking gcc driver
I need to refer not to the libgccjit install directory, but to the
internal directories of the system gcc, which are _not_ in ld.so.conf
(and do not belong there).
$ LANG=C LIBRARY_PATH=/lib/gcc/x86_64-linux-gnu/11 ./conftest
$ echo $?
0
This directory usually is not for public consumption. It's internal to
gcc. But a program using libgccjit does need to know it, apparently. So
on NetBSD, this works as-is? Is this gcc-specific directory in
ld.so.conf or equivalent? That would cause havoc if using a different
gcc release, I presume. For the conftest, I can point LIBRARY_PATH to
any of the multiple gcc lib dirs I have …
$ for d in /lib/gcc/x86_64-linux-gnu/*; do if test -e "$d/crtbeginS.o"; then echo $d:; LANG=C LIBRARY_PATH=$d ./conftest && echo yes; fi; done
/lib/gcc/x86_64-linux-gnu/11:
yes
/lib/gcc/x86_64-linux-gnu/12:
yes
/lib/gcc/x86_64-linux-gnu/7:
yes
/lib/gcc/x86_64-linux-gnu/7.5.0:
yes
/lib/gcc/x86_64-linux-gnu/9:
yes
That superficially seems to work, but I feel uneasy about answering the
question which gcc is actually used as backend for libgccjit now. I
_guess_ a libgccjit built along with a full gcc install would not need
LIBRARY_PATH and use its install prefix, and the gcc objects there.
Right now I have no idea how this works for you on NetBSD. Can you tell
me where your crtbeginS.o is and how it is located?
Also on Ubuntu, it is consistent that the conftest never works in the
pkgsrc build environment. It seems the private gcc library path is
blocked out by the wrappers. This is also the case for Ubuntu with the
system compiler, where I did not mess with the toolchain. I think it is
not surprising that buildlink and wrappers may interfere with the
operation of libgccjit. I find it amazing that it seems to work on
NetBSD.
Anyhow, I'll disable this option and will remember to try to build
libgccjit in my future gcc toolchains. It may be useful to other
projects, if anyone is not just using llvm for that purpose.
Can we agree to disable this by default for emacs until someone fixed
it for Linux? Or only enable it on tested platforms? I still think that
the mixing of versions should be reconsidered, but this is not my
problem, then, yet.
Alrighty then
Thomas
--
Dr. Thomas Orgis
HPC @ Universität Hamburg
Home |
Main Index |
Thread Index |
Old Index