NetBSD-Users archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: Building NetBSD 11 Release from Source Fails on Linux/MacOS
Dear all,
Thank you very much for your input and for considering my issue.
I am replying first now, since I needed some time to experiment, collect data, think, and to pull my hair.
This is a really weird issue...
Your feedback suggests that there are users who build NetBSD 11 from sources on Linux hosts.
Since RVP mentios to test in QEMU, and to eliminate factors from my host environment, I decided to systematically test when building a release works and when it fails.
In virtual machines and on multiple hosts directly.
Essentially, I tested building the release on all computers that I can easily get hold of :)
To experiment, I use the following method:
## a) Building in Virtual Machines
To build NetBSD 11 from source on virtual machines, I create a shallow clone of the sources on the host.
(`git clone --depth 1 https://github.com/NetBSD/src.git -b netbsd-11 netbsd-11-src`)
Then, with Vagrant and VirtualBox, I create five virtual machines (each with eight cores), where the VM name indicates the OS type and version.
These are: ubuntu22.04, ubuntu24.04, debian-13, popos-22.04, and freebsd14.
Each VM mounts the source tree from the host to make sure all builds are made from the same sources.
Then, I build two versions of the tool chain.
Once in the VMs default environment and once in a minimal environment in which only a minimal PATH is set and in which LC_ALL is set to C, see 1.) and 2.) respectively.
1. `./build.sh -U -u -j8 -m amd64 -a x86_64 -O $HOME/obj tools`
2. `env -i HOME="$HOME" PATH=/usr/bin:/bin LC_ALL=C ./build.sh -U -u -j8 -m amd64 -a x86_64 -O $HOME/obj-small-env tools`
Each tool chain is built into a respective OBJ dir, which are only visible in the VM.
The split of building the tool chain (and later the release) in these two environments, is to rule out that there is an issue that arises from how environments are configured.
The reason for building the tools in a separate step, is to have separate log files.
Once the two tool chains are built, I build the release, again in the two environments as above:
1. `./build.sh -U -u -j8 -m amd64 -a x86_64 -O $HOME/obj release`
2. `env -i HOME="$HOME" PATH=/usr/bin:/bin LC_ALL=C ./build.sh -U -u -j8 -m amd64 -a x86_64 -O $HOME/obj-small-env`
## b) Building on "Bare Metal" (directly on a host OS)
On "bare metal", I do the same as for the virtual machines above.
I shallowly clone the sources, build the tools in the default environment and once in the minimal environment (`env -i HOME="$HOME" PATH=/usr/bin:/bin LC_ALL=C`), and subsequently, I build the release in the default and in the minimal environment.
I do that for ubuntu22.04 and popos-24.04 (both on a Thinkpad x240), popos-22.04 (on a Thinkpad P14s Gen.3, AMD Ryzen5), MacOS Sequoia (on a MacBook Pro M4, once in the default ZSH environment and once in a Bash env.), MacOS Montery (on an Intel i7 MacBook Pro 2015, only in the default Zsh env.).
## Results
The two tables below list the results of my experiment.
"tools" and "release" are the argument of build.sh respectively.
YES means the respective build step was successful and NO means the build step failed.
| OS | VM VirtualBox | VM VirtualBox Min. Env. |
|---------------|-----------------------|-------------------------|
| ubuntu22.04 | tools YES release YES | tools YES release YES |
| ubuntu24.04 | tools YES release YES | tools YES release YES |
| debian-13 | tools YES release YES | tools YES release YES |
| popos-22.04 | tools YES release YES | tools YES release YES |
| freebsd14 | tools YES release YES | tools YES release YES |
| OS | Bare Metal | Bare Metal Min. Env. |
|--------------------|-----------------------|-----------------------|
| ubuntu22.04 | tools YES release NO | tools YES release NO |
| popos-22.04 | tools YES release NO | tools YES release NO |
| popos-24.04 | tools YES release NO | tools YES release NO |
| MacOS Sequoia ZSH | tools YES release NO | tools YES release NO |
| MacOS Sequoia BASH | tools YES release NO | tools YES release NO |
| MacOS Montery | tools YES release YES | tools YES release YES |
That is, in the virtual machines, I can build a NetBSD 11 release from source for all the configurations.
It looks as if it does not matter if LC_ALL=C or a default value as Ramiro reported (or C.UTF-8 or en_US.UTF-8 as for me in most cases).
The release just builds as expected.
The results for building directly on a host are less successful in my case.
On all the three versions of Linux that I tried on the two Thinkpads, building the tools succeeds but then building the release fails when building libgmp, specifically when nbm4 preprocesses asm files as described in my previous emails.
On MacOS Sequoia, the default shell is ZSH, so I try to build in that and also in BASH (after chsh -s /bin/bash).
In both cases, I can only build the tool chain but not the release.
Then, last night, when I was thinking that there is perhaps an issue with building on a physical host instead of a virtual machine, I learned that this is not the case either.
On MacOS Montery, I can build a release, also directly from the default ZSH.
The results of my experiments make me believe that the precise values of LANG, LC_ALL, and LC_CTYPE or other environment variables are not relevant for the issue I am observing.
In the VMs, builds succeed no matter if they are set to C, C.UTF-8 or en_US.UTF-8.
These results leave me scratching my head.
## How to reproduce my issue?
Since I would like to be able to share an environment in which the issue I am having is reproducible and especially since it is working for RVP in Pop_OS 22.04 in QEMU, I thought I create a VM image that I can share with you.
But it would take me some time to create either a QEMU disk image or a VirtualBox disk image.
Additionally, I would not know directly how I can share such a big file with you.
Therefore, I am describing a manual way to reproduce my issue:
- Get a current Pop_OS release (https://system76.com/download-pop/):
```
$ cd ~/Downloads
$ wget https://iso.pop-os.org/24.04/amd64/generic/28/pop-os_24.04_amd64_generic_28.iso
$ echo "8d78e500fc95da765a91cddcf2ab7ba71ce4e7997e4f008456d4d872ee8f8221 pop-os_24.04_amd64_generic_28.iso" | sha256sum --check
```
- Write it to a USB key (for me it is in /dev/sda)
`$ sudo dd if=./pop-os_24.04_amd64_generic_28.iso of=/dev/sda bs=4M status=progress`
- Boot into the live environment. There, in the terminal, I do the following:
- Get the NetBSD 11 sources (on the live environment on the USB key, there is not enough space for the sources, so they have to be stored on a mounted disk):
```
cd /path/to/disk
git clone --depth 1 https://github.com/NetBSD/src.git -b netbsd-11 netbsd-11-src
```
- Note, I conducted all of my experiments on the head:
```
$ cd /path/to/disk/netbsd-11-src
$ git log
commit 4198f5370c187384b1de46b9b0fcd04462a420e0 (grafted, HEAD -> netbsd-11, origin/netbsd-11)
Author: martin <martin%NetBSD.org@localhost>
Date: Mon Nov 2 16:34:32 2026 +0000
Welcome to 11.1!
```
- Now, build the tool chain:
`./build.sh -U -u -P -j$(lscpu | grep "^CPU(s):" | awk '{print $2}') -m amd64 -a x86_64 -O $HOME/obj tools`
- Since the release build always fails when nbm4 is preprocessing files while building libgmp, one can directly make only this dependency with the respective make wrapper:
`$HOME/obj/tooldir.Linux-*-generic-x86_64/bin/nbmake-amd64 -C netbsd-11-src/external/lgpl3/gmp/lib/libgmp
- That step fails with the following output:
```
/path/to/disk/obj/tooldir.Linux-7.0.11-76070011-generic-x86_64/bin/nbm4 -DCONFIG_TOP_SRCDIR=/path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp/../../dist -I/path/to/disk/netbsd-11-src/ external/lgpl3/gmp/lib/libgmp -I/path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp/dummy -I/path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp/../../dist/ -I/path/to/disk/ netbsd-11-src/external/lgpl3/gmp/lib/libgmp/../../dist/mpn -I/path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp -DPIC -DOPERATION_sub_err1_n /path/to/disk/netbsd-11-src/external/lgpl3/ gmp/lib/libgmp/../../dist/mpn/x86_64/aors_err1_n.asm > sub_err1_n.s
nbm4: unexpected end of input, unclosed parenthesis:
/path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp/../../dist/mpn/x86_64/aors_err1_n.asm at line 225
*** Failed target: sub_err1_n.s
*** Failed command: /path/to/disk/obj/tooldir.Linux-7.0.11-76070011-generic-x86_64/bin/nbm4 -DCONFIG_TOP_SRCDIR=/path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp/../../dist -I/path/to/ disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp -I/path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp/dummy -I/path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp/../../dist/ -I/path/ to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp/../../dist/mpn -I/path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp -DPIC -DOPERATION_sub_err1_n /path/to/disk/netbsd-11-src/external/ lgpl3/gmp/lib/libgmp/../../dist/mpn/x86_64/aors_err1_n.asm > sub_err1_n.s
*** Error code 1
Stop.
nbmake: stopped making "all" in /path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp
```
- Running the failing command, i.e., nbm4, `/path/to/disk/obj/tooldir.Linux-7.0.11-76070011-generic-x86_64/bin/nbm4 -DCONFIG_TOP_SRCDIR=/path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp/../../dist -I/path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp -I/path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp/dummy -I/path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp/../../dist/ -I/path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp/../../dist/mpn -I/path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp -DPIC -DOPERATION_sub_err1_n /path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp/../../dist/mpn/x86_64/aors_err1_n.asm`
shows the following:
```
nbm4: unexpected end of input, unclosed parenthesis:
/path/to/disk/netbsd-11-src/external/lgpl3/gmp/lib/libgmp/../../dist/mpn/x86_64/aors_err1_n.asm at line 225
```
The environment in that Pop_OS 24.04 is as follows:
```
- `uname -a`:
Linux pop-os 7.0.11-76070011-generic #202606011647~1784215097~24.04~4974880 SMP PREEMPT_DYNAMIC Thu J x86_64 x86_64 x86_64 GNU/Linux
- `lsb_release -a`:
Distributor ID: Pop
Description: Pop!_OS 24.04 LTS
Release: 24.04
Codename: noble
- `ls -la $(which sh)`:
lrwxrwxrwx 1 root root 4 Mar 31 2024 /usr/bin/sh -> dash
- `ls -la $(which $SHELL):`
-rwxr-xr-x 1 root root 1446024 Mar 31 2024 /bin/bash
- C compiler version:
cc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- Linker version:
GNU ld (GNU Binutils for Ubuntu) 2.42
Copyright (C) 2024 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
- Std dynamic linker/loader version:
ld.so (Ubuntu GLIBC 2.39-0ubuntu8.8) stable release version 2.39.
Copyright (C) 2024 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
- Stdlib version:
ldd (Ubuntu GLIBC 2.39-0ubuntu8.8) 2.39
Copyright (C) 2024 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
LANG=C.UTF-8
LANGUAGE=
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_PAPER="C.UTF-8"
LC_NAME="C.UTF-8"
LC_ADDRESS="C.UTF-8"
LC_TELEPHONE="C.UTF-8"
LC_MEASUREMENT="C.UTF-8"
LC_IDENTIFICATION="C.UTF-8"
LC_ALL=
Excerpt of environment (`env`):
SHELL=/bin/bash
LANG=C.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
Experiment conducted on Git commit (https://github.com/NetBSD/src.git):
4198f5370c187384b1de46b9b0fcd04462a420e0
```
## What now?
I believe, that it is not my environment settings that make the release build fail.
Since I build from the same sources, I would believe it is also not the nbm4 binary that has some issues.
By the way, in my experiment, I also double checked for all cases that the config.m4 files in the dist and lib directories are the same.
I checked with diff and sha256sum. The files are identical in all cases (also in those where the build fails). So this is likely neither the issue.
Now, my current hypothesis is that nbm4 somehow calls out to something from the host that is not part of the tool chain that causes the issue (but what could that be?).
I would have to validate that, by comparing and testing the reproducible build of nbm4 that fails on a host in one of the VMs where a previous build succeeds.
I will see when I will find time for that experiment.
The reason why I am spending so much time and effort on this issue is that I am preparing a new course for next semester, in which I would like that students build their own OS (NetBSD or FreeBSD) from sources as a part of their project work.
From experience I know that many students either have some MacBooks or, since they took another course with me, have a version of Pop_OS installed.
That is, likely they would see a similar error as I do.
I would like that they do not stumble over this in the beginning of the course directly.
Consequently, I appreciate all your help and feedback!
## PS
If required, I can share the collected information about environment configurations, versions of compilers, libraries, etc. with you.
But it got quite a bit of data, and I am hoping that my example to reproduce my issue above in the Pop_OS live environment is perhaps sufficient.
@RVP: In all Linuxes in my experiment the stdlib is GLIBC, no musl.
For FreeBSD it is libc (or how it is called in the BSD world :) )
For MacOS, I believe it is libc too. But would have to check again in the logged data.
I would guess none of the students comes with an environment
with musl either.
On Thu, 2026-11-05 at 11:23 +0100, Ramiro Aceves wrote:
> [You don't often get email from ea1abz%gmail.com@localhost. Learn why
> this is important at
> https://aka.ms/LearnAboutSenderIdentification ;]
>
> O Wed, Nov 04, 2026 at 03:04:33PM -0500, Greg Troxel wrote:
> > What locale are you using? Try again with LC_ALL=C or
> > however it is you
> > turn off locale stuff.
> >
> > I'm not saying that you should have to do that. Just
> > that it's a good
> > thing to check.
>
>
> Hello Greg,
>
> This was my locale:
>
> ramiro@debian-nuc8i7:~/NetBSD$ locale
> LANG=es_ES.UTF-8
> LANGUAGE=
> LC_CTYPE="es_ES.UTF-8"
> LC_NUMERIC="es_ES.UTF-8"
> LC_TIME="es_ES.UTF-8"
> LC_COLLATE="es_ES.UTF-8"
> LC_MONETARY="es_ES.UTF-8"
> LC_MESSAGES="es_ES.UTF-8"
> LC_PAPER="es_ES.UTF-8"
> LC_NAME="es_ES.UTF-8"
> LC_ADDRESS="es_ES.UTF-8"
> LC_TELEPHONE="es_ES.UTF-8"
> LC_MEASUREMENT="es_ES.UTF-8"
> LC_IDENTIFICATION="es_ES.UTF-8"
> LC_ALL=
>
>
> Now the locale is:
>
> ramiro@debian-nuc8i7:~/NetBSD$ locale
> LANG=C
> LANGUAGE=
> LC_CTYPE="C"
> LC_NUMERIC="C"
> LC_TIME="C"
> LC_COLLATE="C"
> LC_MONETARY="C"
> LC_MESSAGES="C"
> LC_PAPER="C"
> LC_NAME="C"
> LC_ADDRESS="C"
> LC_TELEPHONE="C"
> LC_MEASUREMENT="C"
> LC_IDENTIFICATION="C"
> LC_ALL=C
>
>
> I downloaded a fresh source copy and built again:
>
>
>
> ===> Tools built to /home/ramiro/NetBSD/netbsd-11-
> src/./obj/tooldir.Linux-6.12.113+deb13-amd64-x86_64
> ===> build.sh ended: Thu Nov 5 09:41:33 CET 2026
> ===> Summary of results:
> build.sh command: ./build.sh -U -u -j8 -m amd64
> -a x86_64 -O ./obj tools
> build.sh started: Thu Nov 5 09:33:46 CET 2026
> NetBSD version: 11.1
> MACHINE: amd64
> MACHINE_ARCH: x86_64
> Build platform: Linux 6.12.113+deb13-amd64
> x86_64
> HOST_SH: /usr/bin/sh
> No $TOOLDIR/bin/nbmake, needs building.
> Bootstrapping nbmake
> share/mk MAKECONF: /etc/mk.conf
> MAKECONF file: /etc/mk.conf (File not found)
> TOOLDIR path: /home/ramiro/NetBSD/netbsd-
> 11-src/./obj/tooldir.Linux-6.12.113+deb13-amd64-x86_64
> DESTDIR path: /home/ramiro/NetBSD/netbsd-
> 11-src/./obj/destdir.amd64
> RELEASEDIR path: /home/ramiro/NetBSD/netbsd-
> 11-src/./obj/releasedir
> Created /home/ramiro/NetBSD/netbsd-11-
> src/./obj/tooldir.Linux-6.12.113+deb13-amd64-
> x86_64/bin/nbmake
> Updated makewrapper: /home/ramiro/NetBSD/netbsd-
> 11-src/./obj/tooldir.Linux-6.12.113+deb13-amd64-
> x86_64/bin/nbmake-amd64
> Tools built to /home/ramiro/NetBSD/netbsd-11-
> src/./obj/tooldir.Linux-6.12.113+deb13-amd64-x86_64
> build.sh ended: Thu Nov 5 09:41:33 CET 2026
> ===> .
>
> real 7m47.821s
> user 36m5.311s
> sys 2m28.899s
>
> make release started at: Thu Nov 5 09:44:25 CET 2026
> make release finished at: Thu Nov 5 11:12:20 CET 2026
> ===> Successful make release
> ===> build.sh ended: Thu Nov 5 11:12:20 CET 2026
> ===> Summary of results:
> build.sh command: ./build.sh -U -u -j8 -m amd64
> -a x86_64 -O ./obj release
> build.sh started: Thu Nov 5 09:44:25 CET 2026
> NetBSD version: 11.1
> MACHINE: amd64
> MACHINE_ARCH: x86_64
> Build platform: Linux 6.12.113+deb13-amd64
> x86_64
> HOST_SH: /usr/bin/sh
> share/mk MAKECONF: /etc/mk.conf
> MAKECONF file: /etc/mk.conf (File not found)
> TOOLDIR path: /home/ramiro/NetBSD/netbsd-
> 11-src/./obj/tooldir.Linux-6.12.113+deb13-amd64-x86_64
> DESTDIR path: /home/ramiro/NetBSD/netbsd-
> 11-src/./obj/destdir.amd64
> RELEASEDIR path: /home/ramiro/NetBSD/netbsd-
> 11-src/./obj/releasedir
> Updated makewrapper: /home/ramiro/NetBSD/netbsd-
> 11-src/./obj/tooldir.Linux-6.12.113+deb13-amd64-
> x86_64/bin/nbmake-amd64
> Successful make release
> build.sh ended: Thu Nov 5 11:12:20 CET 2026
> ===> .
>
> real 87m54.932s
> user 569m23.716s
> sys 44m38.003s
>
>
>
> It ended just fine.
>
> Regards.
> Ramiro.
>
Home |
Main Index |
Thread Index |
Old Index