Subject: RE: "knowing" when a file is available in a directory
To: None <port-i386@netbsd.org>
From: Steven Grunza <grunza@ulticom.com>
List: port-i386
Date: 10/11/1999 09:39:23
Be very careful of using the file system's open() and renaming commands
to indicate when a file is ready to be processed.  I worked on a project
where this was done.  Everything worked great until full load was
applied.  We quickly found out that opening and renaming several
thousand files every minute takes several minutes (even with four 200MHx
R4600 processors and four separate SCSI busses with stripped drives).  I
ended up creating a separate thread to create and rename the files.
This thread provided simple "give me a file" and "rename this file to
this name" functions to the main thread.  Crude but it worked; however,
I did learn to avoid the temptation of using file system function calls
for "real-time" file status info.  Since you seem to working with a
system that can handle more than one page per second, everything might
look and work great until you try to hit that 1+ pps performance mark.

Since you may change OS's in the future, your best bet would seem to be
to control the process creating the files since any hooks relied on in
one OS may not be available in another.  Of course, this would be a good
way to force the use of NetBSD vs. Solaris since custom driver code is
much easier to create when you've got the source....

Steven G.	steven_grunza@ieee.org

> -----Original Message-----
> From: port-i386-owner@netbsd.org [mailto:port-i386-owner@netbsd.org]On
> Behalf Of abs
> Sent: Saturday, October 09, 1999 6:46 PM
> To: Eric McWhorter
> Cc: Bill Sommerfeld; port-i386@netbsd.org
> Subject: Re: "knowing" when a file is available in a directory
>
>
> On Wed, 6 Oct 1999, Eric McWhorter wrote:
> > I read the first page of a document on portalfs and it mentioned a
> > daemon.  If I read that right, then I think the overhead
> would be too
> > high and I'd probably get better performance by getting the
> receiving
> > process to somehow cooperate.  Good suggestion, though.  I will
> > back-burner some portalfs experiments.
>
> 	If you can hook into the recieving process then just get it to
> 	put the file into one dir, then rename into into another when
> 	complete  (or rename within the samedir to a given suffix)
>
> 	Low tech it may be, but simple is good :)
>