Mike Cheponis writes:
[...]
>
>Could you please be more specific than "unmitigated pain"? 
>
>Also, let's consider the "typical" piece of h/w that I think will be common
>when I finish writing this code: An Alpha or Merced at >= 500 MHz
>with at least 1/4 GB of DRAM and around 50 GB of disk.
>
>-mac
>
Hi Mike,

I don't mean to be rude here, but I think you should consider
getting the "Sorting and Searching" volume from Knuth's series if
you want to run programs that use tens of gigabytes of temporary
storage.  (Look at the sections about out-of-core algorithms.)

We've got some 1 GB FreeBSD machines here (SMP), and we often want
to run programs that are bigger than available "core." Usually,
these programs are spice-level circuit simulators simulating large
digital circuits (I think we've done as many as 400k transistors
using the BSIM2 models), and they want to use something like 1.5
GB of memory.  Note that they only want to make three sequential
passes or so through the memory, so it's not like they are going
to send the machine to thrash-hell.  Still, performance is really
quite bad.  When these jobs get going, all kinds of terribly annoying
(but not very surprising things) happen.. NFS gets slow, xterms
are almost unusable, and the disks, of course, go absolutely crazy
for about half an hour.  (The machine we mainly beat up like this
is a dual PPro200/512k with 1 GB of RAM, 5 GB of swap, and a 16 GB
filesystem ccd'd over four disks.  All the disks are fast/wide 7200
rpm.) Using swap space to make up for the fact that you're a lazy
programmer is simply not advisable (at least if you easily get
bored of watching the disk activity light flicker).  

Almost without exception, even a dumb out-of-core algorithm (like
a little malloc() wrapper someone mentioned, or that you could
probably whip up in a couple of hours) will perform better than
swap space which has to "guess" what you're going to do with the
memory next, whereas you probably have information about it that
you just can't tell the VM system.  (If you wondered, the guy who
wrote the circuit simulator has this excuse:  he says that it's
not his fault that the program uses swap because the correct solution
is just to buy more RAM.  Too bad that we couldn't find any 128
meg FPM modules...)

   Regards,
     Mika Nystrom <mika@cs.caltech.edu>
     Asynchronous Systems Architecture Project
     Department of Computer Science
     California Institute of Technology