Consider the system with these filesystems mounted:

/dev/wd0e on / type ffs (read-only, local)
mfs:10 on /dev type mfs (synchronous, local)
mfs:43 on /mfs type mfs (synchronous, noatime, local)
/etc on /permanent/etc type null (local)
/mfs/etc on /etc type null (local)

Let us remount the root read-write:

# mount -u -w /

Let us create two files in /permanent/etc/, which is a null mount of
/etc on /dev/wd0e:

# echo garbage can > /permanent/etc/garbage
# echo garbage can > /permanent/etc/garbagecan

Now, let us remount the root read-only, and reboot:

# mount -u -r /
# reboot

When the system comes back up, I list the files I had created
in /permanent/etc/:

$ ls -l /permanent/etc/garbage*
-rw-r--r--  1 root  wheel  12 May 17 10:22 /permanent/etc/garbage
-rw-r--r--  1 root  wheel   0 May 17 10:23 /permanent/etc/garbagecan

Surprise!  /permanent/etc/garbagecan was truncated to zero length.

Let me start over from scratch.  I will repeat the steps, adding one step:
before remounting the root read-only, I will sync filesystems:

# sync ; sync ; sync
# mount -u -r /
# reboot

When the system comes back up, I list the files.  This time, the last
file I wrote has not been truncated!

$ ls -l /permanent/etc/garbage*
-rw-r--r--  1 root  wheel  12 May 17 10:22 /permanent/etc/garbage
-rw-r--r--  1 root  wheel  12 May 17 10:23 /permanent/etc/garbagecan

I am surprised that 'mount -u -r /' does not flush filesystem caches to
disk; I am *very* surprised that it nevertheless leaves the system in
a state where the caches will not be flushed to disk when I reboot!

I believe this is an instance of PR 30525, "remounting ffs read-only
(mount -ur) does not sync metadata".  Is it as simple as calling
ffs_sync(,MNT_WAIT,,) from ffs_mount whenever there is a rw->ro
transition?

The null mount *does* seem to be involved with the problem.  If I do
not write garbagecan through the null mount (/permanent/etc/garbagecan
-> /etc/garbagecan), then /etc/garbagecan will not be truncated after
I reboot.

Dave

-- 
David Young             OJC Technologies
dyoung@ojctech.com      Urbana, IL * (217) 278-3933