NetBSD-Users archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: state or future of LFS?

On Apr 13, 2009, at 12:21 PM, Miles Nordin wrote:
"cs" == Chuck Swiger <> writes:
   cs> Yeah, well, relevant facts don't care about opinions: whether
   cs> you acknowledge them or wish to ignore them is up to you.

and kool-aid apple zealot ranting doesn't care about
decisively-demonstrated experience

I'm actually a tequila-drinking BSD fan who likes FreeBSD for general server purposes, NetBSD for embedded/flash-driven appliance roles, and OSX for a desktop. As for "ranting", I'll leave that for those better suited to it-- just as I'll abandon this thread as a waste of my time if you continue the ad hominem attacks.

   cs> That's absolutely right-- only, I mentioned the special API
   cs> (that's the F_FULLFSYNC bit you quoted above).

yes.  you mentioned it.  It's existence is the problem.  wtf does
fsync() itself do, then?  serve as a time-wasting decoy!  Enshrouded
in befuddling documentation to hide their benchmark-inflating lie.

Next time, when you don't know what something does, UTSL & RTFM:

Fsync() causes all modified data and attributes of fildes to be moved to a permanent storage device. This normally results in all in- core modi- fied copies of buffers for the associated file to be written to a disk.

Note that while fsync() will flush all data from the host to the drive (i.e. the "permanent storage device"), the drive itself may not physi- cally write the data to the platters for quite some time and it may be
     written in an out-of-order sequence.

Specifically, if the drive loses power or the OS crashes, the application may find that only some or none of their data was written. The disk drive may also re-order the data so that later writes may be present,
     while earlier writes are not.

This is not a theoretical edge case. This scenario is easily reproduced
     with real world workloads and drive power failures.

For applications that require tighter guarantees about the integrity of their data, Mac OS X provides the F_FULLFSYNC fcntl. The F_FULLFSYNC fcntl asks the drive to flush all buffered data to permanent storage. Applications, such as databases, that require a strict ordering of writes should use F_FULLFSYNC to ensure that their data is written in the order
     they expect.  Please see fcntl(2) for more detail."

Does that answer your question clearly enough? If you still find this documentation "befuddling", I suppose I might see whether I can file a radar with the appropriate parties to clarify, but IMO the above documentation is more complete than, say:

   cs> As far as I can tell (ie, from looking at the code), by
   cs> default OSX does synchronous updates to data and async updates
   cs> to filesystem metadata because it trusts the journaling
   cs> mechanism to keep the metadata consistent

nobody does synchronous updates to data.  You are saying it has no
write cache at all.  Of course there is a write cache.

My point is that MySQL calls fsync() on Mac OS X, and Mac OS X does
not synchronously update the data before returning.  period.  I do not
care about metadata, nor does MySQL.  Are you saying I'm wrong and
MySQL is wrong?

Evidently: yes.

I've seen plenty of these debates about what fsync() does on various platforms before, as well as the corresponding debate on the interaction with the write-cache of the drives, and the main difference between OSX and other platforms isn't whether fsync() returns before the data is written out to the drives, but whether the platform lets fsync() return without confirming whether the disk itself has flushed the data to the platters or not.

If write-caching is enabled and the disks are not mounted using -o sync or equivalent, *all* of the platforms I am familiar with let fsync() cheat in the sense that they do not wait until the disk cache itself has been flushed before returning.

What's interesting about OSX isn't that it does the same thing here that other platforms do, but that it has the optional capability to ensure transaction ordering and writes completing through the write- cache on a per-descriptor or system-wide basis using fsync(), without needing to disable write-caching entirely for other processes.

   cs> But let's focus on just what other Unices do with fsync():

yes, this is pretty interesting.

But the point is, the MySQL guys observed corruption problems related
to broken fsync() on Mac OS X, only, not on other unixes.  I do not
doubt that other unixes have broken fsync() commands, especially
w.r.t. sending flush commands to drives with write caches.  Then there
are supposedly lying drives, broken iSCSI stacks, VM implementations
that discard ATA sync commands, all sorts of gremlins.  and
furthermore the fsync API is limited.  And all this is worth talking

Fine. However, I've seen no evidence from you or from the MySQL link provided earlier which suggests that OSX is doing anything different with fsync() than the other platforms: without specifics, I would be willing to conclude that the data corruption being reported was either due to a bug with MySQL expecting Linux kernel semantics or with the drive re-ordering cached writes and had something go wrong (ie, OS bug or power yanked).

I suppose it's also possible that OSX itself has changed significantly since the MySQL issue you reported, but my recollection is that these fsync() changes and a related integration of lockf() vs. flock() handling were all made way back around 2000-2002 timeframe (ie, MacOS X 10.0 / 10.1). If the issue is still present in 10.3 or later, it should be obvious and easy to reproduce with current OS versions, no...?

However none of that overshadows the FACT that all these broken
limited unixes still permit a mostly-functional MySQL implementation.
To achieve feature parity on Mac OS X, MySQL had to use this
proprietary made-up-bullshit Apple-only API.  so do not tell me
fsync() on Mac OS X is not broken w.r.t. other Unixes.  it is.  You
have to do ``simon-sez, fsync'' or else it does nothing.

You're either mistaken or are deliberately wrong. I've provided pointers to the exact place in the XNU source code where fsync(2) is implemented, and noted that it waits until the data blocks are written out by buf_flushdirtyblks() because MNT_WAIT is being set. Feel free to read the sources yourself.

The effect of the "proprietary made-up-bullshit Apple-only API" is that it provides tighter semantics for fsync() that what POSIX requires (not that that is hard), with the intention of enforcing write ordering and guaranteeing that the data really is written all the way to disk platters and not just to the write cache of the drive-- this is nice for databases which ought to require ACID semantics.

And this is a clear win when you compare some benchmark package that doesn't know
about simon-sez on Apple yet---you'll never catch the lie because
people don't do cord-yank tests of their filebench runs.  It's a nasty
trick Apple pulled!

You're welcome to believe that if you like, but I can't see which facts are being used to support such a conclusion. Frankly, going by the MySQL benchmarks I've seen, the performance improvements by having gettimeofday() be handled by a fast commpage mechanism rather than via a system call makes so much more of a difference to MySQL that the performance difference with fsync() is down in the noise.

(To my mind, that says more about MySQL's code and the focus of Linux upon system-call micro-benchmarks than it proves about anything else, but YMMV....)

And all the stuff about metadata is a little bit silly because the
only metadata that's relevant to fsync is the mtime.  This
create/delete/rename stuff is mostly unrelated to fsync, other than a
few corner cases like creates and size-increases need to be pushed
(and, according to SQLite3, they generally are).  [ ... ]

You're ignoring edge cases which are vital to consider: for example, when you write enough data that the file needs to change representation (ie, from fitting in the direct inode list to indirect in FFS, or indirect to double-indirect, etc or similar equivalent with HFS+ B-trees), the file metadata has to also change and an indirect pointer block allocated for the write of the data contents to be visible.

In the case of FFS, these changes happen whenever you grow a file past 80KB in size (assuming 10 direct links per inode with 8K block size), which happens plenty often.


Home | Main Index | Thread Index | Old Index