Perhaps a good way to analyze what's going wrong is to add some kind
of logging to the VM system; allocate (say) a 1MB buffer in kernel
memory (wired, presumably) and log each decision by the VM system.
Then, when you have driven the system into the pathological state,
freeze the log and dump it out.  (Recognizing that automatically might
be hard.)

Some important questions: is there a convenient way for a kernel task
to automatically flush this log to a disk file?  (Hmm, I guess the
accounting log should provide an adequate model.)  How many decisions
does the VM system make in a second, and how many decisions would 1MB
of buffer hold?  (I picked 1MB since it's a reasonable fraction of my
16MB machine; someone with a 64MB machine (especially since it's the
big-memory people complaining the most) might more easily afford a
larger fraction of memory.)