Current-Users archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: Rust, pkgsrc

On 11/07/18 09:27, Chavdar Ivanov wrote:

On 11/06/18 18:50, Havard Eidnes wrote:
Yes. I had a similar problem.  The build would fill up the
/tmp/ directory and die from exhausted resources.  I had /tmp/
created with tmpfs and had a constraint of 64M.  The answer for
me was to create /tmp in /etc/fstab with tmpfs and no size
constraint.  Then Rust would build, but it still took a long
Yes, rust is an absolute Pig resource-wise.  Not only does it
carry a copy of llvm inside itself (it most probably has to), it
also carries a nubmer of other packages inside.  Plus, the build
builds most parts at least twice if I've been able to observe

When building on NetBSD/amd64 8.0, I noticed that the work/
directory after a "make" consumes in the order of 10G disk space,
possibly more when cross-building (my current work/ is 13G, which
is an unfinished cross-build because I hit build issues...).

It is conceivable that the storage could be reduced somewhat (but
probably not by much?) by tweaking src/bootstrap/ to
say -Cdebuginfo=0 instead of 2 for RUSTFLAGS.

So that you have an idea what to expect:

My amd64 build host has 8G real memory, and a 2G tmpfs /tmp, and
... it didn't run out of space anywhere :) On this particular
host (i7 3rd-gen, 4 real cores, 8 w/HT, pkgsrc and system on SSD)
the build of 1.29.2 completed in a little over 2 hours wall-clock
time, csh's "time" report at the end of the build was

40468.007u 1958.277s 2:04:42.18 567.0%  0+0k 10302+100556io 129329pf+0w
On my 16GB HP Envy, 4c/8t, I got overnight:
Build completed successfully in 4:22:05
make  14672.75s user 36893488148207.93s system -2147483648% cpu 4:43:37.28 total
with /tmp just a standard directory, tmpfs as usual on /var/shm. This is more than twice as slow compared to the last few rust builds, which also took just over two hours.

I wonder if something was changed in the build process, previous builds were highly parallelized, with all the threads being seen busy during various stages, whereas this time I didn't see anything like this.

All my builds were off unmodified pkgsrc lang/rust directory with nothing in /etc/mk.conf that I can see to affect this.

I suspect the ridiculous times for system and cpu are from zsh 'time'.

Follow up: 'make test' killed the machine, there was no coredump. It had fully loaded the system, though - with all 8 threads up to 100%.

Previous versions usually completed the test, albeit with errors. This is on
# uname -a
NetBSD tarkus 8.99.25 NetBSD 8.99.25 (GENERIC) #0: Sat Nov  3 03:33:22 GMT 2018 sysbuild@ymir:/home/sysbuild/sysbuild/amd64/obj/home/sysbuild/src/sys/arch/amd64/compile/GENERIC amd64

Second attempt with the same build quickly errored, perhaps with leftovers from the previous. I'll probably wait for the next rust release before trying again.

So, yes, the build makes fairly good use of the multiple cores;
notice the 567.0%, which, if I've understood correctly, indicates
approx. 5.7* parallelism on average.

The last version I managed to build on one of my NetBSD/macppc
8.0 machines (a single-core 1.5GHz G4 Mac Mini, 1GB memory) was
1.29.2, the build took nearly 29 hours wallclock time.  This one
doesn't have a tmpfs, and has a single file system with ~40G
free, so it also didn't run into any barriers on the /tmp front


- Håvard

Home | Main Index | Thread Index | Old Index