Re: WAPBL/cache flush and mfi(4)

[Thor Lancelot Simon <tls%panix.com@localhost>]

On Fri, Aug 24, 2012 at 06:05:54PM +0200, Manuel Bouyer wrote:
> 
> First, I was a bit dissapointed by performances compared to an older
> system with a perc/5 controller. Then I noticed that while running
> bonnie++ on /data2, some processes would hang in tstile.
> I tracked it down to this scenario:
> - WAPBL wants to issue a cache flush for one of the fs. It's doing this
>   with the filesystem's journal lock held.
> - the cache flush hits mfi0. This is translated to a command which
>   flushes the controller+disk caches. But this command flushes the
>   whole controller's cache, not only data associated with sd0.
>   With 1Gb of dirty data to write back it takes 2 to 3 seconds.

If the controller has a nonvolatile cache, it should ignore the
flush.  If you have the awkward cache where the controller cache
is NV but the disks have volatile caches which are switched on,
then the disks' caches must be flushed but the controller's
should not.

If the controller's firmware can't do this you probably need
to advise the user to turn off the local caches on the disks,
and skip the controller-level flush in the driver if the
controller indicates its cache is NV.

I have always thought we should use finer grained barriers
(in terms of SCSI or modern ATA, writes with ordered tags,
which we wait for, to force all preceding simple-tagged
requests out first) instead of full cache flushes, but
it is noteworthy that Linux started with that approach and
hurriedly backed away from it.  If we intend to go there we
must understand as best we can why they changed their minds
and why we should still use the way they rejected.

Thor