Cross-buliding rust

[Havard Eidnes <he%NetBSD.org@localhost>]

Hi,

on and off, "because it's there", I've been working on getting rust
running on NetBSD/macppc.

The normal route to build rust is to start from a "bootstrap kit"
which is a binary (possibly an earlier version) of rust, as larger
parts of the compiler is written in rust.

However, to build rust for a "new" OS/arch combination (such as any of
the NetBSD powerpc ports) requires doing a cross-build.  Internally
rust has a copy of llvm embedded, and llvm is already able to generate
powerpc code.  Additionally, the NetBSD/powerpc support was already
added to rust, I beleive primarily via this pull request:

  https://github.com/rust-lang/rust/pull/48281

So, after quite a few trial and error attempts (more about the
failures I hit and how they were worked around below), these are the
steps to repeat the cross-build:

1) You need a full build of tools and user-land for the target
   platform.  This is most easily produced by our build.sh script,
   building a full release.  The setup I used assumes that you'll keep
   the tools/ and dest/ directories under a common root.

   Uncomment these parts of lang/rust/Makefile and modify to suit
   where this common root is:

   #CROSS_ROOT=            /u/macppc
   #MAKE_ENV+=             CROSS_ROOT=${CROSS_ROOT}

2) Tell the wrapper script used to run the cross-compiler which GNU
   target it's supposed to build for.  This is part of the "tools"
   binary names:

   #GNU_CROSS_TARGET=      powerpc--netbsd
   #MAKE_ENV+=             GNU_CROSS_TARGET=${GNU_CROSS_TARGET}

3) You need to tell rust the "triplet" for your target, and to use a
   wrapper script to run the cross-compiler.  You need to instruct
   rust to build a compiler to be run on the target ("host") as well
   as include code generation for the intended run-time target
   ("target").  Uncomment:

   #TARGET=                powerpc-unknown-netbsd
   #CONFIGURE_ARGS+=       --host=${TARGET}
   #CONFIGURE_ARGS+=       --target=${TARGET}
   #CONFIGURE_ARGS+=       --set=target.${TARGET}.cc=${.CURDIR}/files/gcc-wrap
   #CONFIGURE_ARGS+=       --set=target.${TARGET}.cxx=${.CURDIR}/files/c++-wrap
   #CONFIGURE_ARGS+=       --set=target.${TARGET}.linker=${.CURDIR}/files/gcc-wrap
   #CONFIGURE_ARGS+=       --set=target.${TARGET}.ar=${CROSS_ROOT}/tools/bin/${GNU_CROSS_TARGET}-ar

4) You need to instruct rust where openssl can be found, uncomment:

   #MAKE_ENV+=             OPENSSL_DIR=/usr

5) The rust build wants to use the target binaries from the
   http-parser package.  You need to ensure that package's contents
   (for the target) is installed in ${CROSS_ROOT}/usr/pkg.

6) The rust build wants to use libssh2.  Not only that, but it
   actually wants to use a bug-fixed non-formally-released version of
   libssh2.  It has a copy of the source embedded within itself, and
   it builds it, but there is an undiagnosed error in the build
   causing libssh2.a to be built but not found during the subsequent
   use of it, as it's not installed in the internal directory where it
   is apparently supposed to be found during linking.

   I beleive I've only seen this when building natively on
   NetBSD/macppc so you may not need to consder this when
   cross-building.  My native build machine has a copy of libssh2.a
   sitting in its /usr/lib (from the rust build, not from the
   package).

7) If you have llvm installed already, perhaps an earlier version
   (e.g. llvm 5.*) you need to deinstall it, otherwise the wrong
   include files will be used.

With that you should be set to cross-build a rust bootstrap by simply
doing "make" in the lang/rust package.  The result will be found in
${WRKSRC}/build/dist/, the files needed are (in my case)

  rust-std-1.29.0-powerpc-unknown-netbsd.tar.gz
and
  rustc-1.29.0-powerpc-unknown-netbsd.tar.gz


Below follows a summary of the various build errors I experienced and
which gave riste to the various points above:

A) "dead lock detected".  This is an error message from our ld.elf_so
and which might occur if you run the cross build on a multiprocessor,
as the rust compiler will by default try to use all you cores (i.e.
runs threaded) and apparently *sometimes* does an operation our run-
time linker disapproves of.  The root cause has yet to be diagnosed.

Workaround: restart the build.


B) If you have an earlier llvm package (for the moment e.g. 5.0.0)
installed, you might hit

cargo:warning=../rustllvm/PassWrapper.cpp: In function 'void LLVMRustPrintTargetCPUs(LLVMTargetMachineRef)':
cargo:warning=../rustllvm/PassWrapper.cpp:314:57: error: 'const class llvm::MCSubtargetInfo' has no member named 'getCPUTable'
cargo:warning=   const ArrayRef<SubtargetFeatureKV> CPUTable = MCInfo->getCPUTable();
cargo:warning=                                                         ^
cargo:warning=../rustllvm/PassWrapper.cpp: In function 'void LLVMRustPrintTargetFeatures(LLVMTargetMachineRef)':
cargo:warning=../rustllvm/PassWrapper.cpp:331:58: error: 'const class llvm::MCSubtargetInfo' has no member named 'getFeatureTable'
cargo:warning=   const ArrayRef<SubtargetFeatureKV> FeatTable = MCInfo->getFeatureTable();
cargo:warning=                                                          ^
exit code: 1

Fix: deinstall the llvm package, restart the build.


C) If you didn't point OPENSSL_DIR correctly, you might hit

   Compiling openssl-sys v0.9.28
error: failed to run custom build command for `openssl-sys v0.9.28`
process didn't exit successfully: `/usr/pkgsrc/lang/rust/work/rustc-1.28.0-src/build/x86_64-unknown-netbsd/stage2-tools/release/build/openssl-sys-720f6e16fbe661b9/build-script-build` (exit code: 101)
--- stdout
cargo:rerun-if-env-changed=POWERPC_UNKNOWN_NETBSD_OPENSSL_LIB_DIR
cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
cargo:rerun-if-env-changed=POWERPC_UNKNOWN_NETBSD_OPENSSL_INCLUDE_DIR
cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
cargo:rerun-if-env-changed=POWERPC_UNKNOWN_NETBSD_OPENSSL_DIR
cargo:rerun-if-env-changed=OPENSSL_DIR
run pkg_config fail: "Cross compilation detected. Use PKG_CONFIG_ALLOW_CROSS=1 to override"

Fix: Point OPENSSL_DIR=/usr and use the sysroot of your target, as is
done by the gcc-wrap script.


D) If you don't use the gcc-wrap script, and don't redirect the
includes to use the target's include files, you might hit:

/var/tmp//cctsAmmO.s: Assembler messages:
/var/tmp//cctsAmmO.s:123: Error: unrecognized opcode: `rorw'
/var/tmp//cctsAmmO.s:1511: Error: unrecognized opcode: `rorw'
/var/tmp//cctsAmmO.s:1561: Error: unrecognized opcode: `rorw'
/var/tmp//cctsAmmO.s:1705: Error: unrecognized opcode: `rorw'
/var/tmp//cctsAmmO.s:1767: Error: unrecognized opcode: `rorw'

This is caused by the compiler picking up native include files with
inline assembly of the build host.  Not what you want.

Fix: use the gcc-wrap script, pointing includes into your target's
destdir.


E) Failure to install the http-parser package in the target destdir
might lead to this build failure:

CMakeFiles/Makefile2:87: recipe for target 'src/CMakeFiles/git2internal.dir/all' failed
Makefile:129: recipe for target 'all' failed

--- stderr
fatal: not a git repository (or any of the parent directories): .git
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_CXX_COMPILER
    CMAKE_CXX_FLAGS


/usr/pkgsrc/lang/rust/work/rustc-1.28.0-src/src/vendor/libgit2-sys/libgit2/src/netops.c:15:25: fatal error: http_parser.h: No such file or directory
compilation terminated.

Fix: install the http-parser package (binary for the target, of
course) in the target's destdir's usr/pkg/.


F) Failure to point library searching into the target's destdir might
result in attempts at linking the build host's libraries into the
target binaries, which won't work, obviously:

  = note: /u/macppc/tools/lib/gcc/powerpc--netbsd/5.5.0/../../../../powerpc--netbsd/bin/ld: cannot find -lhttp_parser
          /u/macppc/tools/lib/gcc/powerpc--netbsd/5.5.0/../../../../powerpc--netbsd/bin/ld: skipping incompatible /usr/lib/libssl.so when searching for -lssl
          /u/macppc/tools/lib/gcc/powerpc--netbsd/5.5.0/../../../../powerpc--netbsd/bin/ld: skipping incompatible /usr/lib/libssl.a when searching for -lssl
          /u/macppc/tools/lib/gcc/powerpc--netbsd/5.5.0/../../../../powerpc--netbsd/bin/ld: skipping incompatible /usr/lib/libcrypto.so when searching for -lcrypto

etc.

Fix: Use the gcc-wrap script also for linking; it will instruct the
linker to search the target's destdir for libraries.


G) Failure to find the http_parser library:

  = note: /u/macppc/tools/lib/gcc/powerpc--netbsd/5.5.0/../../../../powerpc--netbsd/bin/ld: cannot find -lhttp_parser
          collect2: error: ld returned 1 exit status

Fix: as above for E), install the http-parser package for the target
in in the target destdir's usr/pkg/.


The above are all errors experienced while cross building, I also hit
a few snags using the initial bootstrap kit on the target:


H) Initially, librustc_codegen_llvm-llvm.so had an undefined symbol
"backtrace":

error: couldn't load codegen backend "/usr/pkgsrc/lang/rust/work/rust-bootstrap/lib/rustlib/powerpc-unknown-netbsd/codegen-backends/librustc_codegen_llvm-llvm.so": "/usr/pkgsrc/lang/rust/work/rust-bootstrap/lib/rustlib/powerpc-unknown-netbsd/codegen-backends/librustc_codegen_llvm-llvm.so: Undefined PLT symbol \"backtrace\" (symnum = 23441)"

Fix: patch-src_libstd_build.rs now causes -lexecinfo to be included
when linking rust -- thanks to Ryo for the patch.  (I've not figured
out why this wasn't a problem for NetBSD/amd64.)


I) The build didn't know what "macppc" was:

unknown cpu type: macppc

Fix: patch-src_bootstrap_bootstrap.py now has code to use `uname -p`
on NetBSD, since `uname -p` is reasonably reliable on NetBSD, at least
in comparison to others...


J) You might get a build error compiling the furst rust file, similar
to this:

   Compiling serde v1.0.70
Expected no forward declarations!
!537 = <temporary!> !{}
scope points into the type hierarchy
!538 = !DILocation(line: 1394, scope: !534)
LLVM ERROR: Broken function found, compilation aborted!
error: Could not compile `serde`.

I got a hint on rustc's IRC channel that this might be related the
generation of debug information.

Fix: This is the various --disable-debug* options which will be
supplied on NetBSD/powerpc and also resulted in the -Cdebuginfo=0
setting of RUSTFLAGS in patch-src_bootstrap_bootstrap.py.  Admittedly
the latter could perhaps be made conditional on NetBSD/powerpc, but
isn't at the moment.


K) You might experience error related to the build of libssh2-sys:

error: could not find native static library `ssh2`, perhaps an -L flag is missing?

error: aborting due to previous error

error: Could not compile `libssh2-sys`.

This is the same error as mentioned above in 6).

Fix: I did (somewhat cheesily, I have yet to find a proper fix):

find work/rustc* -name libssh2.a
su
cp <where-it-was-found> /usr/lib


L) The build of XZ Utils 5.2.3 might fail with

checking for ld used by /usr/pkgsrc/lang/rust/files/gcc-wrap... no

Fix: If, for diagnostic purposes, you need to echo the compiler
invocation command, echo it to stderr, not stdout, so that "gcc-wrap
-print-prog-name=ld" can work as intended and not include spurious
output.  (The gcc-wrap has this info in a comment prior to the
commented-out correct echo command.)


M) You may hit the error from B) also when running on the target.

Fix: deinstall the llvm package.


Best regards,

- Håvard