Filesystem problems and init.root

[Brendon Costa <bcosta%avdat.com.au@localhost>]

Hi all,

I have been attempting to build a NetBSD system that runs from a FAT32
formatted USB stick. I had a method of doing this that was working with
NetBSD 3.1 but am upgrading to 4.0 and there have been all sorts of
problems in doing so, i am unsure if the problems are in mistakes i have
made or in NetBSD 4.0. Following describes the problem(s) and how i
produce them.

The problem i am seeing is that most commands fail with an: "I/O error"
or "Input/output error". Sometimes i get an: "Exec format" error when
trying to run chroot, and with an older method of achieving all this
(that worked with NetBSD 3.1), commands such as mount, have failed in
the past with segfaults and sometimes with EDEADLK error return code
(11). Though i think the last were possibly caused because i did not
have a complete libexec directory at one point in the script where i was
null-mounting certain directories over others.

Basically, what i want to achieve is to boot a NetBSD system from a
FAT32 USB stick (bootable USB) and have the system look as thought its
root filesystem is from a filesystem stored in a compressed disk image:
full.imz. I dont really care how i go about it, the method i am using
has come from weeks of playing with different configurations. If someone
knows a "proper" way of achieving this that i could not find, i would
love to hear of it.

I used to have a working method of achieving this with NetBSD 3.1. I
have since tried upgrading to 4.0 and the old method does not work with
NetBSD 4.0. I also since encountered: sysctl init.root that i heard is
available in NetBSD 4.0 so i have changed the method i am using to work
with the init.root sysctl value instead of my original method which
involved a lot of null/union mounts over a minimal root filesystem. I
will describe the new method i am using and a little bit on the old
method near the end of the email. Sorry for the length, i have tried to
provide as much information as possible in case i am doing something
incorrect and the problem is not with NetBSD 4.0.


Basically i have a FAT32 (I require that it be FAT32) USB stick with the
following files in its root:
* full.imz
A compressed vnd image which contains extracted base.tgz, etc.tgz,
man.tzg and comp.tgz (and a few other small mods that are un-important
for this discussion at the moment). The contents of this is what i want
to be/look-like the "root" filesystem of the booted USB.

* netbsd.img
A SMALL disk image that has the kernel with a built in initial ramdisk
(containing just the binaries necessary to get my "real" root filesystem
from full.imz setup). I found this necessary to get things started.

* Other files for syslinux, it loads netbsd.img into ram as a ramdisk (I
could not get the multiboot to work. Yes i did have the kernel option
enabled. It would hang or reset just when trying to start to run the
kernel). So using the ramdisk method from syslinux, it will load the
image into ram, the hook interrupt 15 (from memory) and make it look
like a proper disk. This is then used to boot the small "disk" image
that contains the kernel.




So as i understand it, the basic method for doing this is:
1) Boot the kernel somehow (with a minimal embedded ram-disk containing
some tools necessary to setup a proper root filesystem: netbsd.img)

2) Using a modified rc script in the embedded ramdisk, setup a new
"root" filesystem to use. This rc script does the following:
    * Mount the USB stick to: /mnt/flash
    * vnconfig the image found at: /mnt/flash/full.imz
    * Mount the vnd0a device to: /mnt/root
    * Create a mfs: /mnt/mfs
    * I populate the mfs with the same directory structure + permissions
as /mnt/root has. This is done because by default the union mount i do
later will always create new directories in the mfs with ownership and
privileges for root. Not the same as the underlying file-system. So to
avoid directories like /home/mprt ending up being owned by the root
user, i pre-create it (and all others) with correct permissions in the
memory filesystem.
    * Union mount the mfs over /mnt/root (So we can temporarily write to it)
    * Setup /mnt/root as the new root filesystem using a chroot by:
sysctl init.root=/mnt/root
    * Exit the rc script, which returns to init and should then get init
to run a second rc script (because the value of init.root has changed)
from the new root filesystem (According to what i read about init.root)

So following is the procedure i use to create all this in some detail:
* Build the NetBSD 4.0 kernel

I am using NetBSD 4.0-RELEASE source CD. I am building for the i586
platform (Have only "options I586_CPU" enabled).

I have a slightly custom kernel, though have been careful to ensure that
the compressed vnd options are available. If you need me to post the
kernel config file let me know. In addition to support the ramdisk i use
the following:
    options MEMORY_DISK_HOOKS
    options MEMORY_DISK_IS_ROOT
    options MEMORY_DISK_SERVER=1
    options MEMORY_DISK_ROOT_SIZE=20480
    options MEMORY_RBFLAGS=0

* Create a tarball for the "full" filesystem (Includes all of base, etc,
comp and man.tgz plus a few minor modifications).
Also adds a fstab file and a few other necessary things

* Create a minimal.tgz that contains a few necessary binaries used by my
special rc script, including: init, mount mount_msdos, mount_union,
vnconfig ...
Also adds a fstab file and a few other necessary things
This also contains a special rc script that looks like the following
(Removed some error handling and echo statements for clarity):

VND=vnd3
FLASH=/mnt/flash
FULL=full.imz
DEV=/dev/sd0e

export HOME=/
export PATH=/sbin:/bin:/usr/sbin:/usr/bin
umask 022

# Mount USB and then the full.imz file through vnd device.
mount -r $DEV $FLASH
FULL=`ls $FLASH | grep -i ^$FULL$`
vnconfig -czv ${VND} $FLASH/$FULL
mount -r -t ffs /dev/${VND}a /mnt/root

# Create a new mfs that we will union over the readonly /mnt/root
mount -t mfs -o -s=15m /dev/null /mnt/mfs
cwd=`pwd`
cd /mnt/root
find . -type d | pax -r -w -dv -pe /mnt/mfs/
result=$?
cd $cwd
mount -t union /mnt/mfs /mnt/root

# Exit the rc script to return to init, but tell init to chroot into
/mnt/root and run that rc script.
sysctl -w init.root=/mnt/root





* Create the full.imz file  (Commands described are not exact but close
enough)
dd if=/dev/zero of=full.img count=524288
vnconfig -vc vnd0 full.img
echo "$full_disklabel" | disklabel -R -r vnd0 /dev/stdin
newfs /dev/vnd0a
mount /dev/vnd0a mnt
tar -xzpf full.tgz -C mnt/

then i clean up...
then i compress the image:
vndcompress full.img full.imz


* Create the netbsd.img file (Commands described are not exact but close
enough)
This contains the kernel with an initial ramdisk, and then puts it into
a small "disk image" that is booted as a ramdisk by syslinux. I
initially tried just booting the kernel image using the syslinux
mboot.c32, but the booting "froze" or reset my PC just at the start of
booting the kernel, i.e. after lines looking like:
12345+3456456....:


# Create initial root fs and embed it into the kernel
makefs -s 10485760 -t ffs minimal.img minimal
mdsetimage -vs netbsd-mprt_rd minimal.img

# Create small disk image that will contain the kernel and a boot
dd if=/dev/zero of=netbsd.img count=30720
vnconfig -vc vnd0 netbsd.img 512/32/64/15
fdisk -vvaifu -b 15/64/32 -0s 169/32/30688 vnd0
echo "$small_disklabel" | disklabel -R -r vnd0 /dev/stdin
newfs /dev/rvnd0a
mount /dev/vnd0a mnt
cp netbsd-rd.gz mnt/netbsd.gz
installboot -v /dev/rvnd0a bootxx_ffsv1


* Finally i copy these files to my USB stick, and then configure
syslinux to boot the small disk image: "netbsd.img"

------------------------------
So when my rc script runs following is what i see:
* It successfully mounts the USB MSDOS filesystem
* It successfully finds and configures the vnd device
* It successfully mounts the vnd device fine at /mnt/root
* It successfully creates the mfs and mounts it under /mnt/mfs

-- From here is where i start seeing problems --
* Sometimes the find or pax command(s) fail, sometimes they complete
successfully. Following are the types of failures i see:
    - With the current setup, i only seem to see "I/O Error" failures
    - I have in the past when using the old method seen segfaults in
applications like find, pax and mount, and the apps would sometimes
return error code 11 (EDEADLK). With my old method the main difference
was that I didn't use the sysctl init.root=/mnt/root to chroot init into
a different root directory, but manually null mounted and union mounted
sub-directories of /mnt/root over the equivalent sub-directories of / in
my rc script. This method used to work well in NetBSD 3.1 but fails
miserably in 4.0

* If the find/pax command completes successfully, then the sysctl to set
init.root always seems to work successfully, however when it returns to
the init process. The machine hangs for about 30sec and then i a asked
to enter the location of a shell script with no error given. Doing so, i
can run all sorts of commands manually, but i expected that the init
process would have run the rc script found in /mnt/root/etc/rc and
started booting the system. Following is a list of a few things i have
tested manually under the shell at this point:

* chroot /mnt/root /bin/sh
This always fails with: "Input/output error", in the past it used to
fail with: "Exec format" error.

* Sometimes the filesystem under /mnt/flash (The USB stick filesystem)
seems to just "disappear". I have not seen this since i started mounting
that filesystem readonly, however occasionally this used to happen when
it was writable and i was not modifying that filesystem in any way. I
found when this occurred, mount reported that everything was still
mounted fine, and the vnd filesyste under /mnt/root also could not be
read. I then unmounted everything, and tried re-mounting and there were
device errors. Then i removed the USB stick and plugged it back in, and
i could then re-mount everything fine.

Basically my question(s) are:
* Am i doing something wrong?
* Are these issues likely to be caused by problems in the kernel or
userland of NetBSD 4.0?
* Is there a better way of achieving all this?
* Where can i go from here?

I understand this is not the most concise or minimal test case showing
the problem. It bothers me that it seems so complicated to boot a NetBSD
system from a compressed image on a FAT32 USB stick, but i cant find a
way to do things more simply. If there is any more information i can
provide that will help diagnose the problem(s) more please let me know.


Thanks,
Brendon.