>>>>> "wk" == Wouter Klouwen <dublet%acm.org@localhost> writes: >>>>> "cc" == Christopher Chen <muffaleta%gmail.com@localhost> writes: wk> The only problems occur when the drive fills up, usually wk> around 80%. These problems are crashes, but I've not seen any wk> data loss. I ran it like 0.5 - 1 decade ago and found that it seemed to need some tools around it to be useful as more than /usr/obj scratchspace such as: * a plain defragmenter, or the Holy Grail: access-pattern-observing data-reorganizing cleaner * a log roll-forward agent. at some point it was truncating the log at boot instead of rolling it, meaning that the filesystem was still guaranteed to be consistent after a cord-yank, but some data which was committed to the disk and could have been rescued according to the design, was discarded instead because of missing implementation. this may affect fsync() reliability, not sure. * was NFS working? * an lfsck that can deal with all situations that plausibly arise in practice, and do it without running out of memory I think some of these were added after I stopped using it. The key advantage in practice was blazingly-fast performance for temporally-local workloads like builds or write-only workloads like untarring things. people often used such workloads for informal testing, so LFS would beat the pants off FFS. wk> IMO, there is no alternative for LFS on NetBSD. I've been using ZFS on Solaris, and it's available performantly on FreeBSD. In my experience and from the zfs-discuss list, it's not as safe/stable as UFS/FFS/ext3. The Sun people have riddled it with assertions and panics such that it'll often find something unexpected and simply refuse to let you touch your data. I think they might actually be on the right track with this approach, but it sounds like they haven't arrived yet. There are a bunch of obstructionist appologists over there making a whole littany of excuses for the lost pools casting FUD (some of it reasonable) on every part of the system except ZFS, but the bottom line is, the guys with hundreds of filesystems and formal datacenter SAN setups say they are losing ZFS pools a lot more often than they lost UFS/vxfs filesystems. ZFS is still missing the defragmenter and the lfsck, from the LFS todo list. However, it does consistently beat UFS for performance. and it has all those other fancy features they blogflog, but to me the most interesting feature came up in list discussion recently. ZFS, I think, shares LFS's property that it always recovers to some state the filesystem passed through in the minutes leading up to a crash. POSIX guarantees no such thing, nor is any such thing implied by a working version of POSIX's ``optional'' (ignored on Mac OS X) fsync() call. The ZFS/LFS guarantee makes so much intuitive sense to application developers that they often assume they have it, but they don't, and they've never been promised it, and they've never had it on Unix until now, until this latest batch of filesystems. This guarantee makes unnecessary most of the bizarre tricks sqlite3 does to survive cord-yanking: http://sqlite.org/atomiccommit.html If we can come up with a word for it, achieve it, and start bragging about it, it may become expected, and writing programs might be a lot nicer. You should care about this because BDB does *NOT* do all those sqlite3 tricks. Also, casual things, like if I yank the cord during 'make' do I need to 'make clean' or can I just restart it? The answer for FFS/UFS is that you probably need to 'make clean', while for ZFS or LFS (even without the roll-forward agent) supposedly you can safely restart the build. and of course things like Postfix queues and maildir's and NFSv3 correctness are even more worry-free than before (on ZFS where fsync() works, but not necessarily on LFS), if you were worried about them before which you probably weren't. The downside is that so far filesystems that can keep this promise, like ZFS and LFS (and probably WaFS?), are the filesystems that never overwrite blocks in-place, so they are prone to fragmentation and do particularly badly with database workloads. In fact I expect they'd do badly with any nested-filesystem situation, a filesystem inside a file, like vnconfig or a virtual machine disk image or iSCSI backing store. This is becoming a really common use-case, and AIUI UFS is still beating ZFS on this one. another cool thing about ZFS they don't discuss much is the way the POSIX filesystem layer is separated from the ``transactional object store'' back-end. There are several non-filesystem things which use the back-end directly, such as: * uncached zvol's (iSCSI backing stores), * pNFS (unfinished), * Lustre (unfinished). I'm dreaming someone will one day invent a non-POSIX storage API which databases can use to solve the performance problem while remaining copy-on-write and snapshottable. zvol's are already working on FreeBSD at least, in fact over there they are constantly misunderstanding how you're supposed to use ZFS and putting UFS's inside zvol's because they think ZFS is like geom. wk> WAPBL may do what you want, it may, but does not share this ZFS/LFS consistency characteristic I just described. The filesystems that do metadata logging (ext3, xfs, hfs+journal) don't. With LFS's log-STRUCTURED, all the data goes into the log, and with ZFS all data is copy-on-write while most journaled filesystems put only metadata in the log and overwrite data in the traditional way. Metadata-only logging also doesn't share all the performance characteristics (neither the good ones nor the bad ones :). cc> this happened before I think when we went to cc> UBC (I think so, anyway! I don't think it was UVM). IIRC for a while it was working in general, but mmap() was not working. I think that was fixed but don't rember for sure. cc> Konrad picked it up and got it more or less working by 2, but cc> yeah, I imagine it's not a priority for him. was anyone besides Konrad ever working on it? seems kind of not sustainable. Anyway to me it almost looks as though we're poised to skip right over the 1 - 5 TB filesystems that FFS+softdep can't handle, towards 100TB filesystems created and consumed by large clusters. Catching up with where Linux was a decade ago is hardly interesting. Some things to consider as you scale filesystems to extremely large numbers of spindles, is that not only fsck time matters: * there will be an O(n) ``scrub'' operation to read every block. This is non-optional: RAID's need this operation to catch 1 bad disk before it turns into 2 bad disks. You cannot just write your data and expect it to quietly stay there, never reading it, and expect to be magically interrupted with a hardware error-report if it ever disappears. NetApp scrubs every week. Make sure your scrub is able to run to completion, ~weekly, without killing performance. If scrub kills performance it'll have to run very very fast. and you cannot reset scrub and start it from 0 upon operations that happen frequently, for example ZFS used to upon taking a snapshot until they fixed it. * when disks go bad, can you resilver them? For example, suppose you come up with something having 100 disks, taking 1 week to resilver, and only 1 disk can resilver at a time. SATA annual failure rate is ~2%/yr and probably much higher in the first month. this isn't workable. With wide arrays you need to not only tolerate multiple failures but be able to run multiple concurrent resilvers. and still, remember, you must be able to run scrub to completion---you cannot interrupt scrub and restart it from 0 upon resilver (as ZFS currently does) if you have so many disks in the pool that you're always resilvering something. * backup and restore. suppose you back up to another pool of the same size, and you have really good incremental backps, ``replication'' like NetApp has. great, but is the pool so big that it's DEPENDENT on the incremental feature to backup in a human-scale amount of time, that it'll take a month to restore it completely from backup? In that case you will have to run two pools, and upon failure be prepared to invert their roles and replicate in the other direction. with such a long restore window maybe 2 pools isn't enough. In ahy case you can't just do a plain old restore like in the old days with such large pools. ultimately I think soemthing like ZFS will not scale because it bottlenecks all data through a single kernel image. It will have to be something using SCSI multiple-initiator features like OCFS, or else a big filesystem above many small filesystems like Lustre, pNFS, GlusterFS. A better design might be to duplicate the ZFS split between object-store and POSIX fs, but crack it across a network layer, and build something Lustre-like to begin with that operates in a degraded localhost mode when you want a plain filesystem. the other thing that's needed is a native NAND flash filesystem with really good cord-yank characteristics (Linux JFFS2 does no write buffering at all). but, JFFS2 isn't working well with the current designs where FLASH size >> RAM size.
Description: PGP signature