Re: Implementation Proposal for Subfile GSoC

[Gerhard Sittig <Gerhard.Sittig%gmx.net@localhost>]

Since I'm mostly ignorant to that subject, I'm just going to reference
the "neighbour projects" and then shut up again.

On Sun, May 25, 2008 at 12:13 +0200, Hubert Feyrer wrote:
> 
> On Sun, 25 May 2008, Adam Burkepile wrote:
> >OS Level - System Calls
> 
> Is the API you suggest compatible with any other operating system?

The "other" systems which have this kind of feature seem to be
- MacOS and its "resource forks"
- Windows (NT style) and "alternative data streams"
- Linux and its support for NTFS ADS (don't know about MacOS resource
  forks handling on Linux)

I cannot tell what MacOS did in previous versions.  OS X seems to
"somehow encode" resource forks into native means of UNIX and DOS file
systems, which may be worth another look.

Windows (ADS in NTFS) can attach several streams to any directory entry,
_not_ just regular files (that is, directories, too -- don't know which
other entries NTFS supports).  There is no official API to learn which
streams are attached to a file system directory entry, the available
methods seem to be undocumented.  But when one knows that there is a
stream and what it's name is, it's possible to address it (don't know
how exactly the API looks, only know that you can access streams you
know about but you cannot -- officially -- learn whether there are
streams).  Since I haven't been too much involved in Windows development
lately, take all of this with a grain of salt.

Linux seems to depend on the file system driver, which either can reside
in the kernel or in user space (via fuse).  Accessing streams is done
very much like accessing traditional ("regular") file system entries,
but there has to be a convention of mapping the base file's and the
stream's name into the file name space (for NTFS it's to concatenate
these separated by a colon, very much like some Windows tools do on the
user visible side).  Getting a list of streams associated with a base
file system entry is done by means of the xattr API (extended attributes
of a file system).  The FUSE based NTFS-3g project does support ADS on
NTFS, and recently fixed an ambiguity in how the associated streams got
listed.  The kernel space NTFS driver is said to not support ADS.


So in summary I think it's like this (maybe on all of the above
systems?).  The traditional notion of the file system is unchanged,
accessing files or traversing directories will be the same.  Optionally
one can associate streams/resources/subfiles with traditional file
system entries, but they won't show up in the traditional listing.  When
a separate API allows to enumerate the streams, one can either access
them by a separate API (Windows) or by file name conventions (Linux).
User space tools _may_ hide these details from the user or might
introduce a convention which the kernel does not provide yet which could
be identical over a set of file system types.  That MacOS X "leaks"
those resource forks into regular files on UNIX or DOS media is an edge
case.

User space tools won't transparently "inherit" stream/resource/subfile
support from the kernel, but should see the file system as they saw it
before (very much like ACL and xattr ignorant tools).  Applications need
to explicitly be taught about the attached data and how to handle them.
A library provided with the system may help, but the kernel probably
should not "help too much" here by extending opendir(2), I guess.


BTW was this the first message on subfiles I'm aware of.  It contained a
lot more details.

  From: Gordon Waidhofer <gww%traakan.com@localhost>
  To: Jan Schaumann <jschauma%netmeister.org@localhost>
  Cc: tech-kern%netbsd.org@localhost, netbsd-advocacy%netbsd.org@localhost
  Subject: RE: Google Summer of Code 2006
  Date: Sat, 15 Apr 2006 13:27:47 -0700
  Message-ID: <OKEHKFHNMFINIBDNPMFGKEFCDJAA.gww%traakan.com@localhost>

The http://www.netbsd.org/contrib/projects.html#addsubfiles URL seems to
have become invalid (it was mentioned in some other message in 2007 or
so).


virtually yours
Gerhard Sittig
-- 
     If you don't understand or are scared by any of the above
             ask your parents or an adult to help you.