Re: [mrustc] progress on binary-less rust bootstrap (on NetBSD-9 amd64)

[Dave B <spam%netbsd.dberg.net@localhost>]

On Mon, Oct 11, 2021 at 09:25:01AM -0400, Greg Troxel wrote:
> 
> Dave B <spam%netbsd.dberg.net@localhost> writes:
> 
> > For source purists: it appears to be possible now to boostrap rustc
> > 1.39.0 on NetBSD-9 amd64 completely from source code--without
...
> That's really interesting.
> 
> I can see muliple threads of effort and progress on any of them is
> probably useful:
> 
>   package mrustc (wip first, usually) including your changes to it

This is done now.  I'm hoping people will try it out and step by
step we can get it working well.

=== commit message: ===
  1st release to wip of DLB's pkgsrc attempt of Mutabah's Rust
  Compiler.  [Currently for rust 1.39, only]
  
  Builds are not seamless yet (sometimes requiring repeated
  invocations, e.g., of "make update", to get past sporadic
  "guard page"--and other possibly memory related--compile-time
  errors); and in addition, the packages need to be better tuned for
  pkgsrc correctness by a knowledgeable pkgsrc developer.
  
  For example, a big win for troubleshooting might be to separate
  out the relatively reliable llvm build stage into its own sub-
  package if that's possible; so as to avoid always doing a very
  expensive rebuild of that phase when it isn't actually necessary.
  
  Also, there's probably NetBSD bias in the patches & packages; and
  it may be possible to improve this to benefit other platforms.
  
  Moreover, these were constructed and tested under a private copy
  of pkgsrc, in .../lang , so they may immediately fail and need to
  be re-configured for .../wip
  
  That said, with adequate RAM [~12G], swap [~10G], storage [~35G]
  and determination, these can successfully complete all the way
  through the packaging & installation (on a NetBSD 9/amd64 system)
  of rustc & cargo binaries--built from rustc sources, but using the
  compiled-from-C++-source mrustc toolchain only (i.e., no rustc
  binaries downloaded during the bootstrap).
  
  Invocation of the resulting exe's with --help or --version args
  currently works (and the exe's SHOULD also pass ~99% of the rustc
  test suite... but I haven't re-run the suite lately).
=======================

>   get the mrustc changes upstreamed (I think it's really important in
>   pkgsrc to get fixes upstreamed, so that upstreams know non-Linux use
>   happens, and so that we don't need to keep merging patches.

Good point, although I haven't done so yet, in part because I'm
hoping the patches will get some review attention--and we'll
therefore be able to correct them first if/as needed.  Of course,
now (and this only STRENGTHENS your point!), I'm MONTHS behind
upstream...  I locked on to a snapshot back in Sept.  I had to do
that because this undertaking was already difficult, and a moving
target would have been too much.

But even once these mrust packages are stable & approvable with
the snapshot we have, I'm guessing it still might be valuable to
shift to a much newer snapshot and see how that goes, rather than
submit patches against too old an upstream codebase, yes?

>   Add some sort of scheme to build a rustc from mrustc.

This is now done too (albeit, only preliminarily):

- the mrust-mrustc package builds mrustc from its (c++) source,
- some other mrust-* pkgs build other needed pieces (from srcs),
- and the mrust-rustc package builds rustc (from rust-lang.org
  source code), using the mrustc toolchain, only -- so, no binary
  downloads involved
- and finally, the mrust-cargo package builds cargo, similarly
  (since rustc & cargo are both needed when building the jump to
  the next rustc version)

but a full bootstrap would do yet one or more additional stages...
(e.g., rebuilding rustc with the "mrustc-built rustc", and/or
running the rustc testsuite, and/or building a "1.40" rustc with
one of these newly built "1.39" rustc's).  These are the steps
I haven't done (at least not in pkgsrc form).

>   Add some scheme to be able to bootstrap forward.  It could be a lot of
>   steps.  Right now we only have one rust version, but it might be that
>   there should be some way to have older ones as well, or something that
>   builds them without needing packages.  This is proabbly the only part
>   that is likely to have real disagreements.

I think someone in the mrustc community may have written a script
to chain forward all the way from an mrustc-built, 1.39 rustc to
a 1.54 rustc (or maybe it's 1.55, .56 or .57 at this point);
however, I don't know anything else about this yet.  Also, I
believe the mrustc author has made some notable progress on
direct-to-1.54 support in mrustc itself.

> So my advice is to get mrustc into wip, even if the package doesn't even
> sort of build, any progress or documentation of what you are doing is
> helpful.  Once in wip, other people tend to just make improvements, as
> the process of mailing partial package tarballs and unpacking them is
> avoided.

That will be great, because it's likely there are things in the
packages that could really use the attention of more experienced/
expert netbsd and/or pkgsrc developers or users.

Speaking of which, thank you very much for the advice and feedback
you've already given on this.

Best,
-Dave