-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi

I have written this project proposal for this year's summer of code. You can
read it here

http://wasabi.fiit.stuba.sk/~haad/netbsd/soc_pro_ext3.html


Here is short version: <<EOF
General

The Ext2 file system is the de-facto standard, Unix-like file system used on
Linux installations. Ext2 does not have journaling capabilities, so Ext3 was
built on top of it to add them without breaking compatibility with Ext2. Ext3 is
now a stable journaled file system used on lots of Linux installations.

NetBSD currently fully supports the Ext2 file system at the kernel level.
Unfortunately there is no support for the new features included in Ext3,
although Ext3 file systems can be mounted provided that their journal is clean.
It would be very nice if NetBSD had Ext3 file system support because the system
could immediately gain a journaled file system as well as compatibility with Linux.

NetBSD as operating system really need good, stable journal file system, today
disks and raids become more and more bigger with size about 1TB or more. FFS was
not designed for disks size like this. We have problems with file system sizes
over 2TB (nor FFS or FFS2 is suitable for this size) good ext3/ext4 support will
give away these problems.

EXT3 file system features:

    *

      Journaling
    *

      Over 16TB file system size

   1.

      Journaling

      In a nutshell, the journal in ext3fs meaning is a regular file which
stores whole metadata (and optionally data) blocks that have been modified,
prior to writing them into the filesystem. This means it is possible to add a
journal to an existing ext2 file system without the need for data conversion.

      When changes to the filesystem (e.g. a file is renamed) they are stored in
a transaction in the journal and can either be complete or incomplete at the
time of a crash. If a transaction is complete at the time of a crash (or in the
normal case where the system does not crash), then any blocks in that
transaction are guaranteed to represent a valid filesystem state, and are copied
into the filesystem. If a transaction is incomplete at the time of the crash,
then there is no guarantee of consistency for the blocks in that transaction so
they are discarded
   2.

      Availability

      By contrast, ext3 does not require a file system check, even after an
unclean system shutdown, except for certain rare hardware failure cases (e.g.
hard drive failures). This is because the data is written to disk in such a way
that the file system is always consistent. The time to recover an ext3 file
system after an unclean system shutdown does not depend on the size of the file
system or the number of files; rather, it depends on the size of the "journal"
used to maintain consistency.
   3.

      Data Integrity

      Using the ext3 file system can provide stronger guarantees about data
integrity in case of an unclean system shutdown. You choose the type and level
of protection that your data receives. You can choose to keep the file system
consistent, but allow for damage to data on the file system in the case of
unclean system shutdown; this can give a modest speed up under some but not all
circumstances. Alternatively, you can choose to ensure that the data is
consistent with the state of the file system; this means that you will never see
garbage data in recently-written files after a crash.

Linux use journal block device to manage journals for their filesystems like
ext3... . I think that NetBSD need something similar to Linux's JBD(Journal
block device).

Journal block device

Linux use for journaling JBD Journal Block Device. JBD provides atomicity in
operations. It was design to add journaling capabilities on a block device. The
ext3 filesystem code will inform the JBD of modifications it is performing
(called a transaction). he journal supports the transactions start and stop, and
in case of crash, the journal can replayed the transactions to put the partition
back in a consistent state fast.

Good journal API can be used in our non journaled filesystems e.g ffs. Main goal
of my Soc project should be design and implementation of good journal API and
then implement ext3fs support.

JBD API is used to open,load,commit and administer journal transactions on
device. In Linux JBD is defined in fs/jbd/ and include/linux/jbd.h.

JBD use these objects in their API: handle,transaction,journal.

   1.

      Handle is single atomic update on filesystem. Handle is a group of
writes/updates on disk that should be performed atomically.
   2.

      Handles can be stored in groups called transactions. Only transactions are
flushed to journal. Transaction is atomicity in nature because consists only
from atomic handles. When transaction is being committed it can have these states:
         1.

            Running: the transaction currently is live and can accept new
handles. In a system only one transaction can be in the running state.
         2.

            Locked: the transaction does not accept any new handles but existing
handles are not complete. Once all the existing handles are completed, the
transaction goes to the next state.
         3.

            Flush: all the handles in a transaction are complete. The
transaction is writing itself to the journal.
         4.

            Commit: the entire transaction log has been written to the journal.
The transaction is writing a commit block indicating that the transaction log in
the journal is complete.
         5.

            Finished: the transaction is written completely to the journal. It
has to remain there until the blocks are updated to the actual locations on the
disk.


Extending our ext2fs support

Our ext2fs implementation is located src/sys/ufs/ext2fs/. I will use this path
when explicitly define another path. For linux paths I implicitly mean
/usr/src/linux/fs/ext3/ path.
Ext3fs SuperBlock

I have to extend our super block structure defined in ext2fs.h to support ext3fs
journal options used. In our superblock structure there is padding included
which can be used for adding new features.
Also struct m_ext2fs need to have a least new journal mounted flag. If we want
EXT3 ACL support structures for struct ext3_acl_header,struct ext3_acl_entry are
needed.
Journal

A journal is a log that internally manages updates for a single block device.
The updates first are stored in the journal and then are reflected to their real
locations on the disk. The area belonging to the journal is managed like a
circular-linked list. That is, the journal reuses its area when the journal is full.

User land part

I have to write usable BSD license mke2fs program, and e2fsck if we want to use
ext3 file system without additional packages from pkgsrc. Here I will also write
new or extend our mount_e2fs to support journaling.
Documentation

Write good documentation about development process so other developers can use,
include it to NetBSD internals book.

EOF

I'm working on this proposal now ,so it's work in progress now , but I want to
discuss this project here.

Regards
- ---------------------------------------------------------------
Adam Hamsik
ICQ 249727910
jabber haad@jabber.org
- ---------------------------------------------------------------
There are 10 kinds of people in the world. Those who understand
binary numbers, and those who don't.
				
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (NetBSD)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFF+yYL9Wt2FT7y228RAj5RAKCCr3Uc8Gk8o00lsAyfSUraUj2BfwCfcRO2
Hn5KKd6GKRRZ2CMPbyE0N8g=
=T1j9
-----END PGP SIGNATURE-----