Cleaning out some old mail...

>>  HyperSPARC's MMU does not peek a modification of PTE.

I believe this is more generally true (i.e., it applies to other
Mbus-based sparcs as well).  There are all kinds of interesting
potential races between on-chip store buffers and MMU updates of
ref/mod bits.  (Note, losing or gaining a bogus ref bit is not
fatal, and gaining a bogus mod bit is not fatal, but losing a mod
bit could be fatal.)  The code I use is this:

/*
 * Update a valid PTE using the SWAP instruction, per SuperSPARC/MXCC
 * manual, in such a way as to avoid losing R/M bits even if the CPU does
 * a table walk while we are looking at the PTE.  This seems like overkill
 * on a uniprocessor, but following the manual is rarely a bad idea...
 *
 * Return the previous PTE.
 *
 * Note that this is not (yet) multiprocessor-ready.
 */
static int
updatepte(va, apte, bis, bic)
	vm_offset_t va;
	int *apte, bis, bic;
{
	int oldval, swapval;
	volatile int *vpte = (volatile int *)apte;

	/* MP: lock out other cpus here */

#ifdef notdef
	__asm__ __volatile__("stbar");	/* ??? */
#endif

	/*
	 * Replace PTE with zero until the zero stays -- any change
	 * should occur only if the MMU updates a ref or mod bit.  By
	 * using a SWAP instruction, the write will be atomic even in
	 * the presence of MP-caches and store buffers.
	 */
	oldval = 0;
	do {
		swapval = 0;
		swap(swapval, vpte);
		tlbflushpg4m(va);
		oldval |= swapval;
	} while (*vpte != 0);

	/*
	 * Now that the PTE is 0, write a new value atomically.  Use
	 * SWAP again to avoid having to drain the store buffer.
	 */
	swapval = (oldval & ~bic) | bis;
	swap(swapval, vpte);

	/* MP: unlock */

	return (oldval);
}

Chris