Re: GSoC 2011 project proposal [Add kqueue support to GIO]

[Dmitry Matveev <me%dmitrymatveev.co.uk@localhost>]

Hi Julio,

Thank your for your feedback! And sorry for a delayed answer.

> As mentioned above, this does not only help the desktop; it helps
> servers as well.  You may want to do a bit more research in this area
> and explain your findings in your proposal, because it will affect
> your design decisions.

Sure, file system notifications are important on servers as well. As
you have already mentioned, it is used by Mediatomb; it is also used
by some administrative tools like lsyncd, iwatch and probably Samba.

This aspect imposes some constraints on the kqueue backend and the
emulation layer - it should be more fast and more reliable.

> >   1. A set of patches for GIO;
>
> Try to elaborate a bit on how you plan to integrate these upstream

It would be perfect if the original GIO developers will accept the
final patches to GIO's upstream :) I think we should contact them
before start. In the alternative case it is quite possible to have
NetBSD's own GIO branch for pkgsrc and to keep it in sync with the
original repository.

> >   2. [probably] Userspace inotify-over-kqueue library sources;
>
> Elaborate on how you think this library should be distributed.  I.e.
> as a separate software package, or bundled right into NetBSD.  Each
> approach has its advantages/disadvantages, and you should explain 
> what
> these are in the proposal.

Distributing inotify emulation library as a separate package would be
flexible - it makes this library optional and users may choose, to
install it or not to install. But (for me) this approach has a general
disadvantage: it is a library, it is a dependency package and thus
all the packages that depend on it should be updated to include
this dependency. I think it would be simplier and straightforward to
include the emulation into the base system - I stand for simplicity :)

> As for the compatibility library: take a look at ATF
> (http://www.netbsd.org/~jmmv/atf/).  It's what we are using in NetBSD
> for our tests and this library should integrate seamlessly.

Great, I will study it.

> Don't postpone tests until a late stage.  Develop tests as you go.
> When you are writing code, you can see more easily what the tricky
> code paths are.  If you postpone the development of tests until a 
> very
> later stage, you will still be able to write simple tests, but you 
> may
> not remember what the tricky cases are.  Also, having tests along the
> way ensure that your code is easily testable and that you don't
> introduce regressions early on.

Surely, I have planned to use unit tests at the implementation
stage :). At the late stage I had more generic tests in mind - testing
the introduced functionality in its consumer applications, such as
Nautilus.

> Also, in general, don't postpone reviews until late. It is nice if
> you keep your mentor involved while you develop your code. Reviewing
> a ton of code on the deadline is painful; reviewing the code
> incrementally is much easier, and you may get some interesting
> comments along the way!

Agree.

> Would be nice to provide a detailed description of what kqueue lacks
> during the early stages of the project.  It might be the case that
> it's impossible to extend kqueue to implement some important
> functionality... in which case we could be looking at a different
> project altogether (e.g. implement inotify in the kernel)!

I have found that kqueue does not have analogues to the following
inotify events:

1. IN_ACCESS        - File was read from
2. IN_OPEN          - File was opened
3. IN_CLOSE_WRITE   - File was closed (and was open for writing)
4. IN_CLOSE_NOWRITE - File was closed (and was not open for writing)
5. IN_MOVED_FROM    - File was moved away from a watched directory
6. IN_MOVED_TO      - File was moved into a watched directory

There is also the IN_UNMOUNT event. According to manual, kqueue has
the NOTE_REVOKE event to signal that the underlying fileystem was
unmounted, but I could not reach this behaviour. The fact that
kqueue requires an open file descriptor per entry complicates the
monitoring on removable devices.

Inotify also provides an additional information for several
notifications, i.e. for MOVED_FROM/MOVED_TO events it also supplies
a name of the file moved. Technically for kqueue this information
can be passed from kernel via kevent.data (if to consider this field
as a pointer).

Determining the possibility of extending kqueue with this additional
functionality requires a deeper analysis, I have planned it on the
community bonding period.

Julio, may I already submit the updated proposal on the official GSoC
portal?

Best regards,
Dmitry