Re: Failing to build a bootable custom kernel

[Jared Barnak <jared.j.barnak%gmail.com@localhost>]

> You might want to watch a recent talk by riastradh@ at EuroBSDCon that
> gives an overview of build.sh and stuff: https://youtu.be/gm-F3GnM8yM
> and skim throgh BUILDING. While build.sh wraps things up nicely, the
> NetBSD build system is still just makefiles, and you can still use
> good old make inside some specific directory to build just the stuff
> you need. On a fast modern box you can build everything in a
> reasonable time, but it's still a bit of an overkill for just the
> kernel :).

Thanks! I actually watched it and that is what originally inspired me. And I
have read `src/BUILDING`. There's just some parts that confuse me.

For the script itself, I settled with:

```
#!/usr/bin/sh

export DESTDIR=/home/jared/Projects/netbsd/obj/destdir
export RELEASEDIR=/home/jared/Projects/netbsd/obj/releasedir

cd src/
./build.sh -u -O ../obj -U -m amd64 -N1 -j24 -V MKCROSSGDB=yes tools
./build.sh -u -O ../obj -U -m amd64 -N1 -j24 kernel.gdb=DEBUG_KERNEL
```

for now. It works (I think), but I don't have any gunzipped images. I have a
raw kernel under

`obj/sys/arch/amd64/compile/DEBUG_KERNEL/netbsd-DEBUG_KERNEL.debug`

And this seems to align with what is says under the make target

```
kernel.gdb=kconf
              Build a new kernel with debug information.  Similar to the
              above kernel=kconf operation, but creates a netbsd.gdb file
              alongside of the kernel netbsd, which contains a full
              symbol table and can be used for debugging (for example
              with a cross-gdb built by MKCROSSGDB).
```

I do see that there is a `releasekernel=kconf` target which does create
a gzip(1)ed copy of the kernel. It creates a
`obj/releasedir/amd64/binary/kernel/netbsd-DEBUG_KERNEL.gz`.

In "How to get started hacking NetBSD", I was confused
on where <machine>.img.gz file come from and was confused by the
following command:

```
# Create a disk image.
mkdir ~/vm
cd ~/obj/releasedir/evbarm-aarch64/binary/gzimg
gunzip -c <arm64.img.gz >~/vm/disk.img
# Make it a sparse 10 GB image.
cd ~/vm
dd if=/dev/zero of=disk.img oseek=10g bs=1 count=1
# Make a symlink to the kernel.
KERNDIR=~/obj/sys/arch/evbarm/compile/GENERIC64
ln -sfn $KERNDIR/netbsd.img .
```

But booting into QEMU is proving challenging, the slide provided in the
video specifies the following

But the `-c` flag writes to stdout, and I don't understand why that's
needed. Can't I use the uncompressed kernel created with
`kernel.gdb=DEBUG_KERNEL`?

To boot into Qemu, I have

```
#!/urs/bin/sh

/usr/bin/qemu-system-x86_64 \
    -kernel = ./obj/releasedir/amd64/binary/kernel/netbsd-DEBUG_KERNEL \
    -append = "root=dk1" \
    -M virt \
    -m 1g \
    -drive if=none,file=disk.img,id=disk,format=raw \ # Created with
`dd if=/dev/zero of=disk.img seek=10 bs=1 count=1`
    -device virtio-blk-device,drive=disk \
    -device virtio-rng-device \
    -nic user,model=e1000 \
    -nographic \
    -machine 'ubuntu'
```

And this yields the following error"

```
WARNING: Image format was not specified for
'./obj/releasedir/amd64/binary/kernel/netbsd-DEBUG_KERNEL.gz' and
probing guessed raw.
         Automatically detecting the format is dangerous for raw
images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
qemu-system-x86_64: root=dk1: drive with bus=0, unit=0 (index=0) exists
```

Any suggestions?


On Mon, Apr 8, 2024 at 8:05 PM Valery Ushakov <v.ushakov%gmail.com@localhost> wrote:
>
> [gmail seems to require from senders antispam setup, so I can't reply
> directly.  it's better to use the mailing list]
>
> -------- Forwarded Message --------
> Subject:        Re: Failing to build a bootable custom kernel
> Date:   Tue, 9 Apr 2024 02:54:33 +0300
> From:   Valery Ushakov <uwe%stderr.spb.ru@localhost>
> To:     Jared Barnak <jared.j.barnak%gmail.com@localhost>
>
>
>
> On Mon, Apr 08, 2024 at 19:16:35 -0400, Jared Barnak wrote:
>
> > > Install programs, libraries, and documentation into DESTDIR.
> > > Few files will be installed to DESTDIR/dev, DESTDIR/etc,
> > > DESTDIR/root or DESTDIR/var in order to prevent user supplied
> > > configuration data from being overwritten.
> >
> > So to clarify, userspace programs (the distribution) will be installed
> > in `DESTDIR` and `RELEASEDIR` will have the kernel image. Is this
> > right?
>
> Long long time ago NetBSD used to get built and installed directly
> into the live running system. That was changed around 1.6 time when
> we've got build.sh and crossbuild infrastructure. When DESTDIR was
> the root of your system, you didn't want stuff installed in you live
> /etc :)
>
> Unless you are going out of you way, you are no longer building like
> that. Instead the system gets built into DESTDIR, and in unpriveleged
> mode the ownership and permission info is recorded in a metalog, not
> as actual ownership/permissions.
>
> My guess is that the above paragraph used to talk about your / but was
> changed to talk about DESTDIR. It might use some further edits...
>
> Anyway, "distribution" will get you complete userland built into
> DESTDIR. "sets" will pack up that DESTDIR into base.tgz &co (using
> metalog info to get the files the right ownership/permissions inside
> the tars) and "release" does all that and also builds necessary
> kernels and installers.
>
>
> > > If you only need a custom kernel, why do you ask build.sh to install
> > everything?
> >
> > I have two use cases:
> >
> > 1.) Run netbsd on a laptop and do live kernel debugging over a serial port
> >
> > 2.) Run the netbsd kernel with debug symbols in QEMU to create network
> > drivers
>
> If you only need a custom kernel you can either use kernel or
> kernel.gdb arguments to build.sh. Though I never use them and do it
> the old way (build.sh kernel=... just automates them).
>
> In the old days you'd just use config(1) and make(1) to build a new
> kernel. E.g.
>
> vax$ cd /usr/src/sys/arch/vax/conf
> vax$ config GENERIC
> vax$ cd ../compile/GENERIC
> vax$ make dependall
>
> You can still do that today when building a native kernel from the
> same release branch, the only difference is that you need to tell the
> makefile to use the native tools:
>
> vax$ make USETOOLS=no dependall
>
> But across releases you need to have cross-tools (build.sh ... tools)
> and then you can do exactly the same song and dance but with
> cross-tools:
>
> amd64$ PATH=$TOOLDIR/bin:$PATH # populated by build.sh tools
> amd64$ cd /usr/src/sys/arch/vax/conf
> amd64$ nbconfig GENERIC
> amd64$ cd ../compile/GENERIC
> amd64$ nbmake-vax dependall
>
> build.sh kernel= basically ensures that you have the tools and then
> does the above config/make thing. Sice I have tooldir with a bunch of
> cross tools for various machines (all in once place), I don't need to
> waste time checking that tools are up to date.
>
>
> > Anyway, I was able to make it work this script:
> > ```
> > #!/usr/bin/sh
> >
> > export DESTDIR=/home/jared/Projects/netbsd/obj/destdir
> > export RELEASEDIR=/home/jared/Projects/netbsd/obj/releasedir
> >
> > cd src/
> > ./build.sh -u -O ../obj -U -m amd64 -N3 -j24 release kernel=DEBUG_KERNEL
> > ./build.sh -u -O ../obj -U -m amd64 -N3 -j24 install-image
> > ```
> >
> > And, I'm left with these two directories:
> >
> > ```
> > netbsd/obj/releasedir/amd64/binary/kernel
> > ├── MD5
> > ├── netbsd-GENERIC.gz
> > ├── netbsd-GENERIC_KASLR.gz
> > ├── netbsd-GENERIC.symbols.gz
> > ├── netbsd-INSTALL.gz
> > ├── netbsd-INSTALL.symbols.gz
> > ├── netbsd-INSTALL_XEN3_DOMU.gz
> > ├── netbsd-XEN3_DOM0.gz
> > ├── netbsd-XEN3_DOMU.gz
> > └── SHA512
> > ```
> >
> > and
> >
> > ```
> > netbsd/obj/releasedir/images
> > ├── MD5
> > ├── NetBSD-10.99.10-amd64-bios-install.img.gz
> > ├── NetBSD-10.99.10-amd64-install.img.gz
> > └── SHA512
> > ```
> >
> > And for further clarification: `netbsd-GENERIC.gz` could be loaded to
> > QEMU and I'd run something like `file netbsd-GENERIC.symbols` to load
> > the kernel symbols. And I could create a bootable USB drive with
> > `NetBSD-10.99.10-amd64-install.img.gz`, right?
>
> You might want to watch a recent talk by riastradh@ at EuroBSDCon that
> gives an overview of build.sh and stuff: https://youtu.be/gm-F3GnM8yM
> and skim throgh BUILDING. While build.sh wraps things up nicely, the
> NetBSD build system is still just makefiles, and you can still use
> good old make inside some specific directory to build just the stuff
> you need. On a fast modern box you can build everything in a
> reasonable time, but it's still a bit of an overkill for just the
> kernel :).
>
> Yes, the above script more or less replicates what releng does to make
> a release and to publish daily builds (http://releng.netbsd.org/), but
> as I said, for what you want to do, building just the kernel is way
> faster and easier.
>
>
>
> > On Mon, Apr 8, 2024 at 8:26 AM Valery Ushakov <uwe%stderr.spb.ru@localhost> wrote:
> > >
> > > On Sun, Apr 07, 2024 at 19:28:24 -0400, Jared Barnak wrote:
> > >
> > > > export DESTDIR=/home/jared/Projects/netbsd/obj/destdir
> > > [...]
> > > > ./build.sh -u -O ../obj -U - -E m amd64 -N3 -j24
> > > > install=/home/jared/Projects/netbsd/obj/destdir
> > > > ```
> > > >
> > > > which then yields the following error
> > > >
> > > > ```
> > > > [ same error repeated ]
> > > > pax: Unable to copy
> > > > /home/jared/Projects/netbsd/obj/destdir/./usr/share/tmac/mm/se_ms.cov,
> > > > file would overwrite itself
> > >
> > > You are asking build.sh to install files from destdir into destdir
> > > which pax duly reports.
> > >
> > >
> > > > I'm trying to build a custom kernel so that I am able to debug it
> > over real
> > > > and virtualized systems (i.e., use kdb on real hardware and gdb in
> > qemu).
> > >
> > > If you only need a custom kernel, why do you ask build.sh to install
> > > everything?
> > >
> > > ISTR build.sh has an option to build just the kernel. kernel.gdb=CONF
> > > according to the usage. (I usually just do nbconfig plus
> > > nbmake-$machine by hand, b/c that's what build.sh will eventually do
> > > anyway and I personally find it easier and clearer to cut out the
> > > extra scripting layer).
> > >
> > >
> > > -uwe
>
> -uwe