This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig33C459C25BD4D38D2FD9D21E
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Bill Studenmund wrote:
>
> No, he's not.
>
First let me apologize.. I overreacted a bit, but I'm dealing with these
symptoms for a few days already and have become quite pissed of at..
no-one/nothing in particular. :) it's frustrating to see a powerful
server (dual 2GHz Xeon, 3G RAM, SCSI RAID) dealing with itself instead
of applications, and with no apparent reason.

The scenario is the following:
- the app reads a large file (few GB), does some data processing and
writes out results in another large file
- the data processing itself can be memory hungry, and there are a lot
of lookups into the BerkeleyDB database
- BerkeleyDB mmaps() a shared 512MB file into memory (its cache) and
uses it as its working store

I have to admit that things were working just fine before I've made BDB
to use the mmap()-ed region as its cache instead of the malloc/sbrk heap.

But as I have mentioned in one of my previous posts, I can have only 1GB
of sbrk() memory and then BDB competes with my memory-hungry
application, which is not good... in some cases I would have then to
switch to disk-based BDB hash instead of in-memory hash, slowing the
processing...

 >
> the pages. So you fill up all of RAM, then have to wait for the ioflush
> system to turn on and flush data out.
>
But why is it so SLOW? I mean, I'd prefer everything grinding to a halt
for like 10 secs with 100% CPU and disk utilization if it's possible.
But ioflush is at 10% CPU and cca. 1 MB/sec disk and it can take more
than a minute to finish. At least that's what top and iostat report.

My data processing program is anyhow blocked until ioflush finishes and
can't do any useful work. So if it has to block everything - can I make
it finish quickly? :)

>
> The problem is if all or most all the dirtying is happening in one file.
 >
and this is indeed the case.

 >
> We do nothing until we try to clean that one file, then we clean
> everything at once. And we drag the system to its knees in the mean time.
>
ufff.. thanks for the explanation.

is there anything I could do on the application level to fix this? It's
a bit 'dirty' to adapt the application to the inefficiencies in the OS,
but I'm willing to do it if it's not a major effort (i.e. less than
installing another OS).

btw, I'd hate installing another OS. I have a really neat setup already
and switching to another OS would be an unproductive loss of time of at
least one day on my part.

Is this maybe fixed in the NetBSD-3 branch?

Best regards,
   Zeljko.

--------------enig33C459C25BD4D38D2FD9D21E
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (FreeBSD)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFC5U4yUIHQih3H6ZQRAq/5AJ9SZvkcRM4qW0okDlAO3QTgbnBZWwCgwkl0
1CHo4VxybwC7miKqCZhCoDo=
=UgcL
-----END PGP SIGNATURE-----

--------------enig33C459C25BD4D38D2FD9D21E--