Re: pwrite() and threads

[Sad Clouds <cryintothebluesky%googlemail.com@localhost>]

On Wed, 26 May 2010 14:27:47 +0000
Wouter Klouwen <dublet%acm.org@localhost> wrote:

> Depends how you want the rest of the system behave when it's locking
> your entire thread because it's writing. I/O can be very slow and if
> the OS is already writing a bunch of data, you may get some serious
> delays.

If I/O is saturated then there is not much you can do. You can only go
as fast as the slowest component in your system. No matter how big your
buffers are, or how many threads you have, if the producer overrunning
the consumer with data, sooner or later you'll hit the bottleneck. If
you're logging data and the buffers are full, you have to flush them,
event if you have to wait for disk I/O. The other option is discarding
data.

The logging package I'm writing is designed to solve the following
issues:

- MT safe. When initialising the package, you specify max number of
concurrent threads. Each thread has its own log buffer, which can be
flushed manually, or automatically when it gets full. Threads register
with the log package and get a thread ID, which they use for logging
data.

- Multiple logs with multiple destinations. Each log is identified by
a log ID and each log can be sent to different destinations, e.g.
syslog, /var/log/all.log and so on.

- Support different "flavours" of syslog, i.e. MT safe syslog_r/syslog,
and regular syslog that needs locking.

This is mainly to provide easy to use logging package for network
servers with different logs and large numbers of threads, e.g. Sun
Niagara processor.