>>bypasses both the D$ and the E$ ...

[kch@kchowksey@hss.hns.com (is this address for real?) writes:]
>Right. I realised that. Of course I was wrong.

I am not sure what this is referring to, but it does remind me of
a little "surprise" we got when we were using ASI_PHYS_USE_EC in
the BSD/OS Ultrasparc port.

The manual describes ASI_PHYS_USE_EC as:
	Physical address, external cacheable only (and refers one
to 6.10, which implies that this is the same as CP=1, CV=0).  The
obvious interpretation is that such an access does not look in the
D-cache.  (After all, how could it -- the D-cache is virtually
indexed, while the address in an ASI_PHYS_USE_EC access is a physical
address.  If there is no virtual address, how can you look up the
virtual index?)

This interpretation is wrong.

A CV=0 access still *looks* in the D-cache.  If the physical address
is found in the D-cache (by tag, presumably -- the D-cache is two
pages wide; presumably the "virtual index" is really a physical
index into both halves, and the one remaining VA bit merely selects
which of the two halves will get used on a cache fill op), the data
are taken directly from the cache.  This means that you can get
"stale" values if you do not flush any given VA-that-used-to-
map-to-that-PA from the D-cache before using ASI_PHYS_USE_EC.

(We built a kind of inverted page table scheme for user pages, and
use physical addresses to chain entries and move TTEs into the
"top" level, where the fast TSB miss handler can find them with a
single lookup.  Kernel text+data are mapped via locked 4MB TTEs,
and kernel pageable space uses a flat TTE table, which limits kernel
VM but simplifies the implementation a lot.  We had a lot of trouble
with the physical address chains getting scragged, until we figured
out this bizarre interpretation of "bypassing the cache".)

Chris