Fixes for ffs/logging issues for 5.0, for review

[Andrew Doran <ad%netbsd.org@localhost>]

Hi,

Here's a patch that fixes a number of issues with ffs. Please review. If no
major issues are identified I'm planning to check it in after 24h because
the problems it addresses must be fixed for 5.0.

        http://www.netbsd.org/~ad/ffs-fixes3.diff

The most interesting change is for performance issues with wapbl on
"multiuser" systems. On an 8 core xeon with the patch, running "build.sh
-j16 release" produces these numbers:

async    8096.7u 1827.4s 26:31.19 623.6% 0+0k 6721+10531io 44619pf+0w
logging  8102.3u 1823.6s 26:43.10 619.1% 0+0k 6774+17864io 44374pf+0w
softdep  8114.3u 1827.1s 27:03.75 612.2% 0+0k 6852+14453io 44423pf+0w
vanilla  8121.5u 1850.0s 28:01.66 592.9% 0+0k 3311+567055io 44806pf+0w

Previously it would have taken hours with -o log. In fact it took so long
I never let it finish. Summary of changes:

- A fix for an issue that can lead to "ffs_valloc: dup" due to dirty cg
  buffers being invalidated. Problem discovered and patch by dholland@.

- If the syncer fails to lazily sync a vnode due to lock contention,
  retry 1 second later instead of 30 seconds later.

- Flush inode atime updates every ~10 seconds (this makes most sense with
  logging). Presently they don't hit the disk for read-only files or devices
  until the file system is unmounted (or sync() manually called). I am sure
  a lot of people have seen FS unmount taking many seconds, this is why. It
  would be better to trickle the updates out but that would require more
  extensive changes. (It may make sense to enable the change as-is only for
  logging file systems.)

- Fix issues with file system corruption, busy looping and other nasty
  problems when logging and non-logging file systems are intermixed,
  with one being the root file system.

- For logging, do not flush metadata on an inode-at-a-time basis if the sync
  has been requested by ioflush. Previously, we could try hundreds of log
  sync operations a second due to inode update activity, causing the syncer
  to fall behind and metadata updates to be serialized across the entire
  file system. Instead, burst out metadata and log flushes at a minimum
  interval of every 10 seconds on an active file system (happens more often
  if the log becomes full). Note this does change the operation of fsync()
  etc.

- With the flush issue fixed, re-enable concurrent metadata updates in
  vfs_wapbl.c.

Andrew