Re: GSoC 2011 project proposal [Add kqueue support to GIO]

[Julio Merino <jmmv%NetBSD.org@localhost>]

On Mon, Mar 28, 2011 at 11:04 PM, Dmitry Matveev 
<me%dmitrymatveev.co.uk@localhost> wrote:
> Hello everybody!

Hello Dmitry,

Thanks for writing up this proposal.  It seems pretty complete and
work on this project would deliver a much more usable desktop (and
server!) NetBSD system.

> I am a 5th year student of Nizhny Novgorod State Technical University,
> Russia.
>
> Please excuse me for CCing tech-kern@ list in this message, I have
> asked some questions regarding the project in tech-pkg@ but there was
> no answer.

Sorry for the delay; I really meant to ask your original email but I
have been away from the computer until now :-)

As a general comment, cross-posting to different NetBSD mailing lists
is not generally well seen.  It is better to repost your message
several times, once per list, to prevent followups from spreading to
multiple lists.

> Here is my proposal for "Add kqueue support to GIO" GSoC project. Any
> comments are welcome.

Comments inline.

> What is the goal of the project?
> --------------------------------
>
> The goal of the project is to add kqueue support to GIO. GIO, the Glib
> IO layer, currently uses inotify interface to monitor changes in the
> filesystem, so it does work on Linux and does not work on NetBSD. The
> importance of the file monitoring grows with the evolution of modern
> desktop environments. Semantic desktops, local file indexers and even
> usual file managers, such as Nautilus, use it to provide an updated
> information to user. GIO is used in many applications, and providing
> a full-featured file monitoring to it will significantly increase
> the end-user convenience as well as application compatibility.

As mentioned above, this does not only help the desktop; it helps
servers as well.  You may want to do a bit more research in this area
and explain your findings in your proposal, because it will affect
your design decisions.

As an example, several server packages have file-notification
functionality.  One that I'm using is Mediatomb: the lack of proper
file notifications makes it pretty much unusable because polling is a
very slow alternative when the trees to monitor are huge.  I'm sure I
have also seen imap servers offering file notification functionality,
but I can't remember which these are off the top of my head.  (For
these packages, an inotify-emulation layer may be more suitable if
they do not use glib, instead of patching their code to add direct
kqueue support.)

> If the time will permit (I assume it will), it would be also possible
> to implement an userspace library that will emulate the inotify
> interface on the top of kqueue. It will help applications that do not
> use GIO but use inotify directly.

As you mention, time should permit to implement both approaches; it'd
be nice to plan both of them in your proposal.  If things turn out to
be more complex than they seem now during the development of the
project, that's fine, we can readjust the goals to focus on one
implementation only.  But it's better to plan for a longer term ;-)

> What will be the deliverables of the project?
> ---------------------------------------------
>
> In the form in which I see the project, the final deliverables are
> expected to be:
>   1. A set of patches for GIO;

Try to elaborate a bit on how you plan to integrate these upstream.

>   2. [probably] Userspace inotify-over-kqueue library sources;

Elaborate on how you think this library should be distributed.  I.e.
as a separate software package, or bundled right into NetBSD.  Each
approach has its advantages/disadvantages, and you should explain what
these are in the proposal.

>   3. Documentation files describing general design decisions;
>   4. A set of sample programs to ensure that is the introduced
>      functionality does work (tests).

Cool; tests are a very nice part of this.  For GIO, the tests should
be implemented using whatever framework the glib guys are using in
their codebase.  Such tests need to be sent upstream so that the
developers can periodically ensure that your new code works, and the
tests need to be readily runnable from NetBSD (make test in pkgsrc).

As for the compatibility library: take a look at ATF
(http://www.netbsd.org/~jmmv/atf/).  It's what we are using in NetBSD
for our tests and this library should integrate seamlessly.

> Milestones and schedule
> -----------------------
>
> 1. Community bonding period
>   * Getting familiar with GIO file monitoring implementation
>   * Researching the way to isolate it from inotify
>   * Probably studying kqueue/inotify implementations on the kernel
>     side
>   * Discussing the ways of implementation with mentor.
>
> 2. May 23 - June 23: the core work
>   * Isolating GIO from inotify with a compile-time abstration level
>   * Implementing the kqueue backend
>
> 3. June 24 - July 10: wrap up for this milestone
>   * Reviews
>   * Bug fixes
>   * Tests

Don't postpone tests until a late stage.  Develop tests as you go.
When you are writing code, you can see more easily what the tricky
code paths are.  If you postpone the development of tests until a very
later stage, you will still be able to write simple tests, but you may
not remember what the tricky cases are.  Also, having tests along the
way ensure that your code is easily testable and that you don't
introduce regressions early on.

> 4. July 11 - August 1: the rest of the work
>   * Implementing the user-space inotify interface over kqueue
>
> 5. August 2 - August 7: offline
>   * Probably I will miss this days
>
> 6. August 8 - August 22: final stage
>   * Reviews
>   * Bug fixes
>   * Tests
>   * Delivery

Same thing re. tests here.  Also, in general, don't postpone reviews
until late.  It is nice if you keep your mentor involved while you
develop your code.  Reviewing a ton of code on the deadline is
painful; reviewing the code incrementally is much easier, and you may
get some interesting comments along the way!

> Some implementation notes
> -------------------------
>
> Obviously my project will use kqueue(). The abstraction layer for GIO
> will be implemented as a compile-time Strategy pattern.
>
> There is a tricky moment with inotify-over-kqueue emulation. The
> problem is that inotify provides some types of notifications that
> kqueue does not - i.e. IN_MOVED_FROM/IN_MOVED_TO with the additional
> information and so on. For it I am going to use the standard file
> system routines such as readdir(), stat() and so on.

Would be nice to provide a detailed description of what kqueue lacks
during the early stages of the project.  It might be the case that
it's impossible to extend kqueue to implement some important
functionality... in which case we could be looking at a different
project altogether (e.g. implement inotify in the kernel)!

> However the frequent usage of these system calls will result in a
> negative impact on the performance, so I consider the possibility to
> extend kqueue to support additional events. This moment (as well as
> others :)) is open for discussion.

Extending kqueue would be nice/preferred, if it is possible.

Thanks for your proposal and keep up the good work!

-- 
Julio Merino / @jmmv