pkgsrc-Changes-HG archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
[pkgsrc/trunk]: pkgsrc/sysutils/fam Add kqueue support. This way, famd can b...
details: https://anonhg.NetBSD.org/pkgsrc/rev/f44f90522e6f
branches: trunk
changeset: 482009:f44f90522e6f
user: jmmv <jmmv%pkgsrc.org@localhost>
date: Sun Oct 17 19:20:53 2004 +0000
description:
Add kqueue support. This way, famd can be notified of changes to files and
directories in "real time", without having to periodically poll(2) for them
after several seconds. This improves the responsiveness of applications
using famd (specially GNOME) when changes to files occur from the "outside"
(they are notified from changes immediately), or even from the application
itself (for example, have you ever tried to rename several files from
Nautilus? just a PITA).
To enable kqueue, you have to pass the 'kqueue' option to the package.
I'm not enabling it by default because it needs testing (but it should be
enabled in a future). Furthermore, I'd like to send these patches to the
FAM developers for comments too. And a review from somebody knowing kqueue
could be good! (this is the first time I use the kqueue interface, so I'm
not sure if everything is right).
A description on how this works can be found in the files/IMonKQueue.c++
file. Note that, due to FAM's design, the easiest way to do this change
is to "emulate" imon functionality.
While here, add an rc.d script for famd, in case the user prefers to run
it at system startup instead of from inetd (I created it while developing
the kqueue functionality, so it's a good moment to add it). Adjust the
MESSAGE accordingly.
Bump PKGREVISION to 4.
diffstat:
sysutils/fam/MESSAGE | 19 +-
sysutils/fam/MESSAGE.kqueue | 10 +
sysutils/fam/Makefile | 32 ++-
sysutils/fam/distinfo | 4 +-
sysutils/fam/files/IMonKQueue.c++ | 407 ++++++++++++++++++++++++++++++++++++++
sysutils/fam/files/famd.sh | 17 +
sysutils/fam/files/imon-compat.h | 53 ++++
sysutils/fam/patches/patch-ag | 10 +-
8 files changed, 536 insertions(+), 16 deletions(-)
diffs (truncated from 631 to 300 lines):
diff -r 1758fb299f83 -r f44f90522e6f sysutils/fam/MESSAGE
--- a/sysutils/fam/MESSAGE Sun Oct 17 16:37:02 2004 +0000
+++ b/sysutils/fam/MESSAGE Sun Oct 17 19:20:53 2004 +0000
@@ -1,17 +1,20 @@
===========================================================================
-$NetBSD: MESSAGE,v 1.3 2004/03/28 22:00:04 minskim Exp $
+$NetBSD: MESSAGE,v 1.4 2004/10/17 19:20:53 jmmv Exp $
-FAM uses RPC and is usually started by the inetd(8) superserver. You need
-to enable it manually, by issuing the following steps:
-
-1) Add FAM to the system's portmapper by appending the following line to
- the /etc/rpc file:
+Because FAM uses RPC, you have to add the following line to the system's
+portmapper file (/etc/rpc):
sgi_fam 391002 fam # File Alteration Monitor
-2) Add FAM to the system's superserver by appending the following line to
- the /etc/inetd.conf file:
+After that, restart the rpcbind(8) server.
+
+Furthermore, you have to enable the FAM daemon. There are two ways to do
+this: through inetd(8) or as a standalone server. If you prefer the former
+way, add the following line to /etc/inetd.conf:
sgi_fam/1-2 stream rpc/tcp wait root ${PREFIX}/sbin/famd famd
+and reload inetd(8)'s daemon. However, if you prefer the later way, use
+the provided famd rc.d(8) script.
+
===========================================================================
diff -r 1758fb299f83 -r f44f90522e6f sysutils/fam/MESSAGE.kqueue
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/sysutils/fam/MESSAGE.kqueue Sun Oct 17 19:20:53 2004 +0000
@@ -0,0 +1,10 @@
+===========================================================================
+$NetBSD: MESSAGE.kqueue,v 1.1 2004/10/17 19:20:53 jmmv Exp $
+
+FAM has been built with kqueue support. Consider raising the amount of
+open files the kernel can handle (i.e., change kern.maxfiles using
+sysctl(8)). FAM will consume a maximum of a half of that limit; this can
+be quickly exhausted if you monitor large directories (a quite common thing
+if you navigate through your filesystem using Nautilus).
+
+===========================================================================
diff -r 1758fb299f83 -r f44f90522e6f sysutils/fam/Makefile
--- a/sysutils/fam/Makefile Sun Oct 17 16:37:02 2004 +0000
+++ b/sysutils/fam/Makefile Sun Oct 17 19:20:53 2004 +0000
@@ -1,8 +1,8 @@
-# $NetBSD: Makefile,v 1.17 2004/10/16 14:40:23 jmmv Exp $
+# $NetBSD: Makefile,v 1.18 2004/10/17 19:20:53 jmmv Exp $
#
DISTNAME= fam-2.7.0
-PKGREVISION= 3
+PKGREVISION= 4
CATEGORIES= sysutils devel
MASTER_SITES= ftp://oss.sgi.com/projects/fam/download/stable/ \
ftp://ftp.tuwien.ac.at/opsys/linux/gentoo/distfiles/ \
@@ -26,10 +26,36 @@
EGDIR= ${PREFIX}/share/examples/fam
CONF_FILES= ${EGDIR}/fam.conf ${PKG_SYSCONFDIR}/fam.conf
+RCD_SCRIPTS= famd
+
SUBST_CLASSES+= paths
SUBST_MESSAGE.paths= "Fixing hardcoded paths."
-SUBST_STAGE.paths= post-patch
+SUBST_STAGE.paths= pre-configure
SUBST_FILES.paths= man/famd.conf.5 man/famd.8
SUBST_SED.paths= -e 's,/usr/local/etc/,${PKG_SYSCONFDIR}/,g'
+PKG_OPTIONS_VAR= PKG_OPTIONS.fam
+PKG_SUPPORTED_OPTIONS= kqueue
+
+.include "../../mk/bsd.options.mk"
+
+.if !empty(PKG_OPTIONS:Mkqueue) && ${OPSYS} == "NetBSD"
+CPPFLAGS+= -DHAVE_KQUEUE
+LIBS+= -lpthread
+
+SUBST_CLASSES+= kqueue
+SUBST_MESSAGE.kqueue= "Enabling kqueue monitoring."
+SUBST_STAGE.kqueue= pre-configure
+SUBST_FILES.kqueue= configure
+SUBST_SED.kqueue= -e 's,IMonNone,IMonKQueue,g'
+
+MESSAGE_SRC= ${.CURDIR}/MESSAGE ${.CURDIR}/MESSAGE.kqueue
+
+.include "../../mk/pthread.buildlink3.mk"
+.endif
+
+post-extract:
+ ${CP} ${FILESDIR}/IMonKQueue.c++ ${WRKSRC}/src
+ ${CP} ${FILESDIR}/imon-compat.h ${WRKSRC}/src
+
.include "../../mk/bsd.pkg.mk"
diff -r 1758fb299f83 -r f44f90522e6f sysutils/fam/distinfo
--- a/sysutils/fam/distinfo Sun Oct 17 16:37:02 2004 +0000
+++ b/sysutils/fam/distinfo Sun Oct 17 19:20:53 2004 +0000
@@ -1,4 +1,4 @@
-$NetBSD: distinfo,v 1.14 2004/10/16 14:40:23 jmmv Exp $
+$NetBSD: distinfo,v 1.15 2004/10/17 19:20:53 jmmv Exp $
SHA1 (fam-2.7.0.tar.gz) = 6c2316f02acf89a41c42ffc3d7fd9cf5eada83a8
Size (fam-2.7.0.tar.gz) = 301974 bytes
@@ -8,7 +8,7 @@
SHA1 (patch-ad) = b8e621acd36811a76a84af82e6f2b5962973e344
SHA1 (patch-ae) = 225a0bd5195be3d3d75edf021b27bed19d84dc15
SHA1 (patch-af) = 57946b3837479b641bb002620ae41008f49af995
-SHA1 (patch-ag) = fa5889ad6d93af72d7efe83784caf61b2ac39d6a
+SHA1 (patch-ag) = 978fa2a582c5f9d2c8660c0a8d933211e97ef500
SHA1 (patch-ah) = d7763198df76d1f0783342a8961b59879e8e1241
SHA1 (patch-ai) = b80aafbb3849fc8c828b6829d8975b910e4d0fd5
SHA1 (patch-aj) = 39391961fd7929d6a5fb49ecb492585cb821afaa
diff -r 1758fb299f83 -r f44f90522e6f sysutils/fam/files/IMonKQueue.c++
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/sysutils/fam/files/IMonKQueue.c++ Sun Oct 17 19:20:53 2004 +0000
@@ -0,0 +1,407 @@
+// $NetBSD: IMonKQueue.c++,v 1.1 2004/10/17 19:20:53 jmmv Exp $
+//
+// Copyright (c) 2004 Julio M. Merino Vidal.
+//
+// This program is free software; you can redistribute it and/or modify it
+// under the terms of version 2 of the GNU General Public License as
+// published by the Free Software Foundation.
+//
+// This program is distributed in the hope that it would be useful, but
+// WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Further, any
+// license provided herein, whether implied or otherwise, is limited to
+// this program in accordance with the express provisions of the GNU
+// General Public License. Patent licenses, if any, provided herein do not
+// apply to combinations of this program with other product or programs, or
+// any other product whatsoever. This program is distributed without any
+// warranty that the program is delivered free of the rightful claim of any
+// third person by way of infringement or the like. See the GNU General
+// Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, write the Free Software Foundation, Inc., 59
+// Temple Place - Suite 330, Boston MA 02111-1307, USA.
+
+// ------------------------------------------------------------------------
+
+// imon emulation through kqueue
+// -----------------------------
+//
+// The code in this file provides an imon-like interface to FAM using kqueue,
+// the kernel event notification mechanism found in NetBSD and OpenBSD.
+//
+// The idea is the following: a thread, kqueue_monitor, simulates the kernel
+// part of imon. This thread can receive commands (ioctl(2)s) and produces
+// notifications when there is something to notify. The thread is constantly
+// running in the background, calling kevent(2) to see if there are new
+// events in the monitored files since the last call.
+//
+// Communication with kqueue_monitor is accomplished by using two pipes.
+// pipe_out is used by the monitor to provide notifications; i.e., it is the
+// same as the read end of the regular /dev/imon device, and produces
+// compatible messages. On the other hand we have pipe_in, which is used
+// to give commands to the monitor (express and revoke); we can't emulate
+// ioctl(2)s from user space, so we have to go this route.
+//
+// Why we use pipe_in to provide commands to the thread, instead of some
+// mutexes? If we used mutexes, we'd have to give kevent(2) a timeout, to
+// let it "reload" the list of changes to be monitored in case it was
+// externally modified. By using a pipe, we can tell kqueue(2) to monitor
+// it for us, and let kevent(2) immediately return when there is a command
+// to process.
+//
+// However, there is a little problem when using kqueue instead of imon or
+// polling. kqueue(2) works by monitoring open file descriptors, instead
+// of inodes on the disk. Therefore we must keep all files being monitored
+// open, and the number of open files can quickly raise in some environments.
+// This is why the code unlimits the number of open files in imon_open and
+// sets a reasonable maximum based on kern.maxfiles (to avoid overflowing
+// it quickly). If we overflow this limit, the poller will enter the game
+// (because we will return an error).
+//
+// Known problem: if we receive *lots* of events quickly, we may end up
+// locked in pipewr. I haven't located where the problem is, but I
+// suspect of the read code in read_handler in IMon.c++. To reproduce,
+// run the test program provided by fam against a local directory, say
+// /tmp/foo, and run the following:
+// cd /tmp/foo; for f in $(jot 1000); do touch $(jot 100); rm *; done
+//
+// Having said all this, let's go to the code...
+
+// ------------------------------------------------------------------------
+
+#include "IMon.h"
+#include "Log.h"
+
+#include "config.h"
+#include "imon-compat.h"
+
+#include <sys/event.h>
+#include <sys/param.h>
+#include <sys/resource.h>
+#include <sys/sysctl.h>
+#include <sys/time.h>
+
+#include <assert.h>
+#include <fcntl.h>
+#include <pthread.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <map>
+
+// ------------------------------------------------------------------------
+
+// devino is a structure that holds a device/inode pair. It is used as an
+// indentifier of files managed by imon.
+struct devino {
+ dev_t di_dev;
+ ino_t di_ino;
+
+ bool operator<(const struct devino& di) const
+ { return (di_dev < di.di_dev) || (di_ino < di.di_ino); }
+};
+
+// imon_cmd simulates commands thrown to imon as ioctl(2)s (but remember
+// we use a pipe).
+struct imon_cmd {
+#define IMON_CMD_EXPRESS 0
+#define IMON_CMD_REVOKE 1
+ int ic_type;
+
+ // imon identifies files through a device/inode pair.
+ struct devino ic_di;
+
+ // A pipe that will be used to receive the result of the command
+ // (asynchronously).
+ int ic_stat[2];
+
+ // If this is an 'express' command, we need the descriptor to monitor.
+ int ic_fd;
+};
+
+// ------------------------------------------------------------------------
+
+static int max_changes;
+static int last_change;
+static int kqueue_fd;
+static int pipe_in[2], pipe_out[2];
+static pthread_t kevent_thread;
+static struct kevent *changes;
+
+typedef std::map<struct devino, int> DEVINOFD_MAP;
+static DEVINOFD_MAP devino_to_fd;
+typedef std::map<int, struct devino> FDDEVINO_MAP;
+static FDDEVINO_MAP fd_to_devino;
+
+// ------------------------------------------------------------------------
+
+static void *kqueue_monitor(void *data);
+static void process_command(void);
+
+// ------------------------------------------------------------------------
+
+int
+IMon::imon_open(void)
+{
+ // Get the kernel event queue. We only need one during all the life
+ // of famd.
+ kqueue_fd = kqueue();
+ if (kqueue_fd == -1)
+ return -1;
+
+ // Create "emulation" pipes.
+ if (pipe(pipe_in) == -1) {
+ close(kqueue_fd);
+ return -1;
+ }
+ if (pipe(pipe_out) == -1) {
+ close(kqueue_fd);
+ close(pipe_in[0]); close(pipe_in[1]);
+ return -1;
+ }
+
+ // Get the maximum number of files we can open and use it to set a
+ // limit of the files we can monitor.
+ size_t len = sizeof(max_changes);
+ if (sysctlbyname("kern.maxfiles", &max_changes, &len, NULL, 0) == -1)
+ max_changes = 128;
+ else
+ max_changes /= 2;
+
+ // Unlimit maximum number of open files. We don't go to RLIM_INFINITY
+ // to avoid possible open descriptor leaks produce a system DoS. 75%
+ // of the system limit seems a good number (we request more than the
+ // number calculated previously to leave room for temporary pipes).
+ // We need to be root to do this.
+ uid_t olduid = geteuid();
+ seteuid(0);
+ struct rlimit rlp;
+ rlp.rlim_cur = rlp.rlim_max = max_changes * 3 / 2;
Home |
Main Index |
Thread Index |
Old Index