The following three recipies will reproduce this bug:

Recipe 1, layer mount, courtesy dyoung@
  newfs /dev/rwd1a
  mount_ffs /dev/wd1a /mnt
  mkdir /mnt/foo
  mkdir /mnt/bar
  mount_null /mnt/foo /mnt/bar
  echo test > /mnt/bar/test1
  echo test > /mnt/bar/test2
  mount_ffs -o update,ro /dev/wd1a /mnt

Recipe 2: reference to unlinked file, courtesy atatat@
  newfs /dev/rwd1a
  mount_ffs /dev/wd1a /mnt
  touch /mnt/file
  tail -f /mnt/file &
  rm /mnt/file
  mount_ffs -o update,ro /dev/wd1a /mnt

Recipe 3: async mount, as per this pr.
  newfs /dev/rwd1a
  mount_ffs -o async /dev/wd1a /mnt
  echo test > /mnt/test
  mount_ffs -o update,ro /dev/wd1a /mnt

Analysis:

At the time of remount, only files open for writing are synced.
Unfortunately, each of the above cases have different reasons why
there may be unflushed outstanding changes on files which are no
longer open for writing.

In case 1, layer mounts delay vnode inactivation until the vnode is
reclaimed, allowing flushes to remain unsynced.

In case 2, the unreferenced file needs to be handled similarly
to a file open for writing.

In case 3, async mounts delay inode updates indefinitely.

Putting a simple ffs_sync call in the remount has a couple
of minor problems, which need to be handled:
  - the readonly flag is already set on the mount point by
    that time, so the update call will be ignored.
  - referenced but unlinked files need to be inactivated
    rather than simply flushed.
  - ffs_sync skips over vnodes if it cannot get the vnode lock
    on first try, it probably shouldn't when MNT_WAIT

These issues should be managable, I'll work on a fix.

Darrin