Re: sql api for ufs

[George Georgalis <george%galis.org@localhost>]

On Wed 09 Jun 2010 at 03:53:58 PM +0100, Ian Clark wrote:
>On 7 June 2010 20:08, George Georgalis <george%galis.org@localhost> wrote:
>> On Mon 07 Jun 2010 at 05:07:43 PM +0200, Kees Nuyt wrote:
>>>
>>>That could be implemented as an appication with a set of
>>>virtual tables, backed by the readdir() and stat() system
>>>calls.
>>>I haven't heard of any implementation, although fossil
>>>http://www.fossil-scm.org/index.html/doc/tip/www/index.wiki
>>>has a few routines which might be interesting.
>>
>> Interesting approach. I've never heard of virtual tables.
>> Unfortunately, I don't know anything about the backend of
>> generating indices or optimizing db queries either. But I
>> see what you are getting at, maybe I can work it out.
>>
>[snip]
>
>Apologies if I'm misunderstanding you here, but are you interested in
>a unified fs/database type affair, similar to what Microsoft attempted
>(and failed at) with WinFS?

Cannot answer, I haven't heard of WinFS? But you are the second to
ask that question.  What is the purpose of your question?


>The Be Filesystem and this interview with it's developers may be of interest:-
>
>http://www.theregister.co.uk/2002/03/29/windows_on_a_database_sliced/
>http://www.letterp.com/~dbg/practical-file-system-design.pdf


Wow - interesting article and awesome paper! Obviously I've not time
to read the whole book but I scanned it. Well enough to get an
idea of the detailed scope.

The *only* reason I'm looking to put a db on top of the filesystem
is to optimize reads. I'm not interested in developing any special
db tricks or getting in any transaction race. The only thing fancy
I'm interested in is some additional columns (tags, notes, md5,
path, mime) which aren't stored in the regular fs struct; exactly how they
are stored would be an implementation detail, but I imagine as a
regular file in the filesystem, a la sqlite db.

The only tricky thing I can see would be an interface
with the fs journal/log to accommodate rapid optimization
index update. ie only rebuild the parts of the index which are out
of date because of writes (and *sigh* atimes but maybe save that
for later).

Even the file and backup management stuff I imagined operating
through the fs calls, it is in their invocation that I was
referring to the atomic process. 

The last thing I want to do is reinvent filesystem
manipulation. I don't even want to do anything fancy with the
db implementation. The goal here is to do rapid queries of data
that may not be online, or off in some http protocol that doesn't
accommodate good bandwidth or seek ability. 

Correct me if I'm wrong but utilizing modern db optimizing index
of mostly already written data (not md5 etc) in the filesystem
for quick queries of mostly archive (user home) data is NOT the
shortcoming of BeOS fs/db or WinFS? -because that's really what
I'm going for---an interface to the data sprawl.

-George