>... SuperCache

I do believe I finally found the problem.  (There may actually
be multiple problems, but this is the one that would be causing
trouble in NetBSD, just as it was in BSD/OS.)

When I wrote the original sun4c code, I had the the pmap `zero
page' and `copy page' functions use non-cached writes to the
destination pages since the cache is write-through anyway and I
did not want to displace anything from the cache (not that I know
even now whether the sun4c does a cache allocate on write -- there
may be no need to map the target page uncached).

The sun4m code preserves these same semantics, but on the SuperSPARC,
this does not work as desired.  The cache is physically indexed
and physically tagged, and cache data is writeback at the lowest
cache level (L2 if MXCC/SuperCache, else L1).  The CPU uses
cache-coherent operations (32 byte MBus transactions) during a
cacheable load or store to keep things consistent.  A non-cacheable
load or store will use a smaller (e.g., 8 byte) MBus transaction;
these are not cache-coherent and simply go directly to main memory.

If the Ecache contains data from page P, and you map page P to
virtual V but map it non-cached and write somewhere in V, the write
goes directly to main memory; the Ecache does not snoop this
transaction.  Once you unmap V and map some new address cacheably,
the contents of the Ecache will be used instead; any dirty lines
will be written back to main memory.  Since the Ecache works via
physical addresses, none of the virtual-address coherence code
takes care of this; thus, once you begin paging, the contents of
`reused' pages get corrupted and you get random core dumps, etc.

The problem would occur on non-MXCC machines, but the L1 Dcache is
only 16K (vs 1 MB for the L2 Ecache) and hence the likelihood of
a `bad' cache op is quite small.

This took me *weeks* to find... you have to put together about a
dozen different little bits of information (about the MBus, the
Ecache, the cache coherence protocols, etc.), but then it all makes
sense.

(There are still some other bugs remaining once you fix that, but
those are pretty minor.  The big ugliness is that you need to flush
any Ecache contents for the pages that map the iopte's in the iommu,
but typically there is nothing there to flush, so the missing code
here does not hurt you.  There are also some issues with getting
reliable M and R bits, as described in section 8.5.3 of the big
manual Aaron has.)

Chris