That was a really good post, Jason.....

> There are different kinds of file  
> systems out there, and if you want to be "fully-featured" on all of  
> them, sometimes you have to do wacky stuff, like use special files to  
> store certain kinds of information that the file system was not  
> originally designed to store.

I believe I followed that. There is some debate. For example,
does it make sense to store an ACL on a file if the access
control mechanisms are not enforced? Opinions vary. I've never
much worried about it, save for the marshalling questions.

You suggested mapping extattr_get/set to subfile operations. That
might be OK. Don't know. The question was: how does a file system
support both? I'm inferring that you're thinking of VOPs for both
user-land extattrs and subfiles.

I'm saying BSD-UFS-type extended attributes should purely be
an implementation issue under the file system and there should
be no VOPs to get/set extended attributes.

So, let's say there are VOPs for both subfiles and BSD-type EAs.
How does the NFSv4 server know which one to use?

> > To be fair, I'm quite pleased today. That there is the notion
> > of "we should have both" is a kind of concession. It practically
> > admits that the BSD/Linux model of "named thingy" is inadequate.
> > If it was adequate, nobody would suggest both.
> 
> I don't think it's a concession.  I was never arguing that subfiles  
> weren't a good idea.  I was, however, arguing that extattrs were not  
> a BAD idea.

As an internal implementation detail, I probably agree.

> I think each of them has their place, and I still  
> fundamentally think that subfiles are not necessarily the best place  
> to store things that are "extended attributes" of a file from the  
> perspective of the file system implementation.
> 
> > NetBSD has no significant investment or deployment of what
> > is (effectively) admittedly inadequate. So why do it?
> 
> A good reason to do it is compatibility with existing deployed UFS  
> file systems.

I read that to be TrustedBSD and FreeBSD.

Solaris is a UFS file system
with considerable deployment and it has no such API. A shadow inode
is used to contain FSDs. Two FSDs are defined (that I know of), both
are related to POSIX-style ACLs. It is quite extensible and a clever
leveraging of existing file system mechanisms (including dump and fsck).
It makes the Solaris UFS extensible. But there are no APIs to directly
manipulate an FSD.

> Another good reason is, as I've already said, extattrs  
> clearly define something as a "file attribute" as opposed to "just  
> another place to store file data".

File attributes should be subject of rigorous definition,
including: NFS XDR definitions (marshalling and endianess
issues), canonical text (for tar/pax 'x' records), API structures
(operands to getxattr()), and perhaps other formal definitions
such as XML. They should not be thought of as name/value blobs.

I'm unclear which way you view "attributes". Formally or as
blobs.

> Consider the distinct  
> capabilities that might be expressed in an ACL:
> 
> - "read data"
> - "read attributes"
> - "write data"
> - "write attributes"
> - "read security information"
> - "write security information"
> 
> Now, tell me -- what is a subfile?  Is it data?  Is it attribute  
> information?  What about storing security information?  Do you now  
> have magic subfiles?

That is a terrific question. It was the subject of quite
an exchange on the NFSv4 reflector about three months ago.
No conclusion was reached.

The NT extended attributes are an accident of history. The roots
are in the OS/2 HPFS file system. They, too, had these smallish
extended attributes manipulated by get/set interfaces. The idea
was to make HPFS extensible with "plug ins" (they called it something
else) and those added features had a way to store their values.
They couldn't resist exposing EAs to userland applications either.
It ultimately was found to be a bad idea for application data
because it had inadequate capacity. The developers of SMB and NTFS
supported OS/2 EAs as legacy. Nobody I've ever talked to has seen them
used. Rather, they gave way to subfiles. OS/2 HPFS was 15 years ago.
History does repeat itself :)

The upshot is that the "read attributes" and "write attributes" supported
by NTFS and NFSv4 ACEs are not well understood.

Further, NTFS subfiles have proven a vulnerability. Specifically, the
lack of access control. I expect (I've heard nothing, this is just my
guess) Microsoft to address the vulnerability with a tweak to the NTFS
ACLs and enforcement mechanisms.

For now, I believe Solaris is using the access control on the primary
data fork (ACLs or UNIX 9-bit) to apply to the subfiles. NFSv4 and the
Solaris APIs leave the door open when there are better, more broadly
adopted answers.

> 
> I think subfiles have their place.  Consider a movie file that has 2  
> data forks -- one for video, one for audio.

That's like an example I use. I say one video and an audio
for each language: English, Spanish, Mandarin, etc, etc.
Another is a database file with indexes as subfiles.
They help an audience understand subfiles as something
largish rather than smallish.
 
> > Both is not necessary. Just have a way to add bona-fide
> > attributes (like ACLs) internal to the underlying file
> > systems. Use getxattr() API (ala Linux) to manipulate the
> > new bona-fide attributes. No need to expose the storage
> > mechanism for attributes to the API.
> 
> I don't think that the current APIs expose the storage mechanism at  
> all.

OK. This is important. I'd like to understand what you
are saying.

Are you saying the FreeBSD extattr_get/set() et al is not
going to be a NetBSD API? Further, is there going to be
no VOPs for these? Is it really a purely internal UFS mechanism?
If that's what you are saying, hurray! But I don't want to
put words in your mouth.

> In fact, consider HFS+ in 10.4, which even overlaps the two in  
> certain circumstances... HFS+ in 10.4 supports extended attributes  
> natively.  It also supports subfiles (the resource and data forks)  
> natively.  There are a couple of special extattrs  
> ("com.apple.ResourceFork" and "com.apple.FinderInfo") that can be  
> used to get/set the data in the file's resource fork and Finder info  
> (the former which can also be opened as a named fork, i.e.  
> "file/..namedfork/rsrc", which in principle could also be opened with  
> openat()).

I've not included HFS+ in my research but I'd like to. Just
haven't had time since I did my paper last year. I think it's
a good idea to keep an eye on Solaris and the Apple folks. I
know both groups are very attentive to NFSv4 and new file
features.

I ran across somebody who said that NTFS only supports subfiles
because they wanted to make migration of Mac files easier (read
snatch the customer). Once the mechanism was in place, though,
engineers found other fun things to do with it like attach
image thumbnails. I'm told Microsoft has been restrained in
using subfile because of MS-DOS legacy volumes, and further
that the sun is setting on that concern. Expect a lot of
subfile use over the next few years.

> 
> It puzzles me that you remain focused on NFSv4 in this discussion.

NFSv4 is a relatively easy way to raise a broader concern I
call "interchange". This includes over-the-wire, like NFSv4,
and over-the-fence, like tar(1). Should tar(1) use 'x' records
or 'E' records (see Solaris tar(1)). When to use which? Shouldn't
there be an attempt at convergence by all operating systems
on the answers to those sorts of questions? Shouldn't I be able
to generate a tar(1)-ball on Solaris and have it load properly
on NetBSD? And vice-versa? Shouldn't I have a consitent result
wether NFS is involved or not in the generate/load/both?

Anything looks worthwhile and -- let's just say it -- fun
out of context. Interchange issues are a wonderful way to
test and vet strawman designs. My sense is that that hasn't
happened with BSD/Linux style EAs because there aren't crisp
answers to how they are linked to NFSv4, how well they meet
expectations set by Solaris and NetApp, how tarballs are made
and moved between systems, and because the course
of events so much resembles the history of OS/2 HPFS.

> We also need to be concerned about the actual disk-backed file  
> systems that might be served out by NFSv4.  Like it or not, those  
> file systems might not support all of NFSv4's groovy features (like  
> subfiles, for example).  There are a lot of lumps to go around... how  
> many and what kind of APIs we have to provide access to certain kinds  
> of file information is only one aspect.

Yes. Of course. Nobody could disagree with that. Those are the
sorts of questions that should be raised and answered before
unfortunate APIs are implemented that lead to severe interoperability
goinks.

> 
> -- thorpej

Thanks for the response, Jason.

Regards,
	-gww