Re: rust - volunteers sought...

[Havard Eidnes <he%NetBSD.org@localhost>]

>> It's unfortunate that the rust compiler has so many problems.
>
> Yes, it is.

Not to excuse it, but ... rust is large and depends on a working
LLVM, which again requires a working C++ compiler.  The rust project
is also quite quick to adopt new LLVM versions, ref.:

rust      embedded
version   LLVM

1.72.1    16.0.5
1.73.0    17.0.2
1.74.0    17.0.4
1.74.1    17.0.4
1.75.0    17.0.6
1.76.0    17.0.6
1.77.0    17.0.6
1.78.0    18.1.2

Luckily, it still accepts an external LLVM >= version 16.x, instead
of insisting on building with the bundled LLVM.

Using the bundled LLVM (which the cross-compilation of rust more or
less dictates) hasn't been entirely smooth:

1) 1.75.0 on NetBSD/sparc64 10.0 appears to mis-compile the embedded
   LLVM, using the in-tree gcc 10.5.0.  Building natively with the
   in-tree gcc results in SIGILL failure, ref.
   https://github.com/rust-lang/rust/issues/117231
   For that version of rust this was worked around by insisting
   on GCC >= 12 when building with the embedded LLVM on sparc64 on
   10.x.

   Meanwhile, building rust on NetBSD/sparc64 9.2 has consistently
   succeeded for all the above rust versions.

2) We now appear to have an "opposite problem" on sparc64 10.0:
   building with the embedded LLVM now causes a SIGILL failure even
   when building with pkgsrc gcc 12, while building with an external
   LLVM 16.0.6nb1 succeeds, and the result can build and run code.

3) 1.73.0 failed on powerpc, due to a bug in LLVM, fixed in 1.74.0.
   Ref. https://github.com/rust-lang/rust/issues/116845.  We carried
   the patch for this for 1.73.0 in pkgsrc.

4) Meanwhile, the 32-bit arm ports, unless they SEGV for other
   reasons, continue to bump into the "virtual address space is too
   small to self-host rust", particularly since I think they have a
   limit to the data segment just below the 2GB mark.

Now, sparc64 and 32-bit powerpc are possibly not sufficiently
"mainstream CPUs" that they are well-represented in rust's
continuous integration testing(?) (and LLVMs?).

The arm 32-bit error was "rustc SEGVs far into the stage2 rust
build", while if I now re-try the 1.77.2 build, it now appears to
exhaust virtual memory:

LLVM ERROR: out of memory
Allocation failed

rustc exited with signal: 6 (SIGABRT) (core dumped)

and this happens on re-try as well, during build of the
rustc-std-workspace-core crate.

While 64-bit arm *may* just be our resource limits for stack size
which is "too small", though if I double the hard stack limit, it
still bombs with

thread 'rustc' has overflowed its stack
fatal runtime error: stack overflow

Did not run successfully: signal: 6 (SIGABRT) (core dumped)

so I'm not sure the above error message is fully reliable...

During the discussion of the 32-bit arm bug, reported upstream in
https://github.com/rust-lang/rust/issues/123551 it appears that
"nobody else is so crazy as to try to build the arm7 rust compiler
natively", because "cross compilation is a thing".  This makes me
question whether the rust testing on 32-bit arm is sufficient to
uncover the SEGV problem I've been seeing.  The failure is far from
"immediate".

So... perhaps a major thing which plays into this is which version
of gcc is used to cross-compile LLVM.  My current cross-compilation
setup has:

$ for c in /u/*/tools/bin/*--netbsd-gcc; do
$c --version
done 2>&1 | grep NetBSD
aarch64--netbsd-gcc (NetBSD nb2 20190226) 7.4.0
aarch64_be--netbsd-gcc (NetBSD nb1 20210411) 10.3.0
powerpc--netbsd-gcc (NetBSD nb3 20190319) 7.4.0
x86_64--netbsd-gcc (NetBSD nb3 20190319) 7.4.0
mips64el--netbsd-gcc (NetBSD nb4 20200810) 7.5.0
mipsel--netbsd-gcc (NetBSD nb3 20231008) 10.5.0
riscv64--netbsd-gcc (NetBSD nb1 20220722) 10.4.0
sparc64--netbsd-gcc (NetBSD nb4 20200810) 7.5.0
$ 

OK, OK, these are perhaps maintained via the "build once, use
forever" maintenance mode, but brings up the question whether
there's a mapping between (target-CPU, LLVM-version) to the
minimally required gcc version to build a working LLVM.

The reason it is as above is that many platforms are cross-built to
target the lowest OS version where rust is supported by pkgsrc,
which mostly means 9.0.  The cross-compile is also always done with
the embedded LLVM so as to not have a binary dependency on an
external LLVM for the bootstrap rust compiler or the rust-bin
package.

The question is if this version of gcc is now (for some target
platforms?)  getting to be "too old" to build more recent versions
of LLVM?  However, bumping the gcc version used for cross-
compilation is not going to be straight-forward if we still want
backward compatibility with 9.x, since I think newer g++ depends on
a newer C++ shared library which isn't present on the "plain 9.0"
target system.


Regards,

- Håvard