Re: rust on 32-bit arm

[Havard Eidnes <he%NetBSD.org@localhost>]

So...

the saga continues, and it's not good news for the combination of
32-bit arm and a desire to self-host rust.

On NetBSD/evbarm 9.0 w/32-bit earmv7hf, the last version of rust
which in my testing could be built natively was 1.76.0, and that
has earned us pkgsrc/lang/rust176 and a redirect there from
pkgsrc/lang/rust/rust.mk.

In the mean time, NetBSD/evbarm 10.0 w/32-bit earmv7hf has been
able to build rust natively, just barely, for a while, for at
least versions 1.80.0 and 1.81.0 (my test summary shows
un-diagnosed errors for 1.77.1 and 1.78.0, but that *may* have
been due to use of thread-local storage and bugs in NetBSD
related to that).

The normal flow of bits for rust is typically:

1) for a given version, I cross-build rust for all the targets we
   support (except for amd64 which is built upstream).  This
   produces what we refer to "bootstrap kits", which consist of
   binaries kits of rustc and of rust-std.  These kits are BTW
   also used by the corresponding rust-bin package. 

2) If you want to build rust natively, you must have the previous
   version's "bootstrap kit" to do that.  For NetBSD/amd64 this
   relies on "upstream binaries", while what I build in #1 from
   the previous version are used for our other rust-supporting
   platforms.

   In a pinch, if there has been trouble with cross-building the
   previous version for a given target but the current version
   builds OK, it is possible to use the "same version" binary
   bootstrap kit to build the rust compiler natively.

The rust bits for NetBSD/evbarm 32-bit earmv7hf are cross-built
against a "target root" populated with 9.0_STABLE binaries and
the corresponding cross-compiler ("tools" in our parlance).  The
strange thing is that those same resulting rust binaries behave
quite different on NetBSD/evbarm 9.0 and 10.0; the 10.0 version
has been able to natively build rust, while the 9.0 system has
not.

The bad news is that my prediction that the bloat of rust has now with
rust version 1.82.0 exceeded the ability of 32-bit arm systems to
self-host rust (i.e. being able to build the rust compiler natively).
I expect that the 10.0 systems will be able to use the new rust-bin
versions to build other rust packages successfully, while that looks
to be problematic for the 9.0 32-bit arm systems, hence rust176.

More details:

The initial failing of trying to self-host rust 1.82.0 was

   memory allocation of 278532 bytes failed

   Did not run successfully: signal: 6 (SIGABRT) (core dumped)

OK, from earlier attempts, I know that the default data segment
size limit on 32-bit arm is from

src/sys/arch/arm/include/arm32/vmparam.h:

vmparam.h:#ifndef       MAXDSIZ
vmparam.h:#define       MAXDSIZ         (1856*1024*1024)        /* max data size */

and as far as I have understood, this is to allow some slop for
ASLR and space for the stack within 2GB which is a hard limit for
user-space per-process virtual memory(?)

And that limit was bumped by yours truly on 8 Oct 2020 in revision
1.56, bumped from 1536M, exactly triggered by this ("building rust
natively") problem (I see the commit message says 1836, but the code
is correct, and 1856 is 2048-128(MAXTSIZ)-64(MAXSSIZ) as indicated in
the commit message.

I did a successive automated "top -b -I" in a shell loop with "sleep
10" while I did my last build attempt, and got

load averages:  1.08,  0.73,  0.35;               up 1+08:43:54        16:49:01
42 processes: 1 runnable, 40 sleeping, 1 on CPU
CPU states:  0.0% user,  0.0% nice, 25.0% system,  0.0% interrupt, 74.9% idle
Memory: 1816M Act, 795M Inact, 20K Wired, 110M Exec, 1252M File, 47M Free
Swap: 2048M Total, 14M Used, 2034M Free / Pools: 327M Used / Network: 

  PID USERNAME PRI NICE   SIZE   RES STATE       TIME   WCPU    CPU COMMAND
 5102 he        78    0  1580M 1225M RUN/0       5:58 92.53% 92.53% rustc
    0 root     124    0     0K  972K syncer/3   26:13 12.30% 12.30% [system]
 7510 he        43    0  6136K 1508K CPU/0       0:00  0.00%  0.00% top

load averages:  1.07,  0.74,  0.35;               up 1+08:44:08        16:49:15
42 processes: 40 sleeping, 2 on CPU
CPU states:  0.2% user,  0.0% nice, 26.3% system,  0.0% interrupt, 73.4% idle
Memory: 1787M Act, 855M Inact, 20K Wired, 110M Exec, 1282M File, 17M Free
Swap: 2048M Total, 14M Used, 2034M Free / Pools: 327M Used / Network: 

  PID USERNAME PRI NICE   SIZE   RES STATE       TIME   WCPU    CPU COMMAND
 5102 he        78    0  1580M 1225M uvnfp1/0    6:08 89.21% 89.21% rustc
    0 root     126    0     0K  836K CPU/1      26:18 15.62% 15.62% [system]
 7852 he        43    0  6136K 1512K CPU/0       0:00  0.00%  0.00% top

I'm pretty sure that rustc is dumping core at this point, it did
"uvnfp1/0" the next time around, and then vanished in the next
display.  And the corresponding error message was

memory allocation of 167772164 bytes failed
rustc exited with signal: 6 (SIGABRT) (core dumped)

That's 160MB, and should according to my math have brought the SIZE up
to 1740MB, but god knows how many other successful largish memory
allocations rust did in that intervening 10 seconds while "top" was
not looking.

I'm going to update my rust issue with the information that
self-hosting on 32-bit arm is now basically hopeless, but last time I
complained about this, the feedback was basically "cross-compiling is
a thing", and precious few if any in the rust camp seemed to care.

So it looks increasingly possible that even for NetBSD/evbarm 10.0 on
32-bit arm, we will have to start recommending the use of rust-bin
and/or we'll have to create a pkgsrc/lang/rust181 package which by the
looks of it will be the last one which can be self-built on 32-bit
arm.  The pkgsrc/lang/rust176 package would still be needed for
NetBSD/evbarm 9.0 on 32-bit arm, and that version (and also 1.81.0
after a while) becomes increasingly old and that's troublesome for the
rust environment where reliance on brand new language features is all
too common.

Best regards,

- Håvard