> This sounds a lot like a distributed shared memory system, a topic
> near and dear to my heart.  If you haven't already, you may 
> want to visit some
> of the pages listed on 
> http://www.pdcl.eng.wayne.edu/~sumit/research/dsm.html
> (which I chose at random from a web search for the DSM I'm 
> most familiar with, Munin).


Exactly. 

The real problem in addition to just swapping a process to another machine
is what to do with its shared memory.  I had thought about making it really
simple and create the process there with all of it's pages marked not 
present and the VM system would cause them to be loaded on demand.  Since
Shared memory is mapped in VM space, processes on different nodes could
demand pages from one machine to another.  I think this fits in your model
too, except for the fork() and exec() and process balancing part.  The
problem that the NUMA folks have had to solve is how to schedule processes
so that contention is minimized on the low latency interconnect.  The other
problem that is unique to this approach is what if a machine dies?  True
the OS's on the other machines are still running, but processes on surviving
machines might be borrowing address space that is no longer present.  My
initial thought is to cause all those programs to core-dump.  There is no
issue with kernel pages in my scheme, because these are pages that would
never leave the local swap domain.


Andy