port-mips/55139: Misunderstanding on R5000 L2 cache Page_Invalidate(S) op size

[Izumi Tsutsui <tsutsui%ceres.dti.ne.jp@localhost>]

>Number:         55139
>Category:       port-mips
>Synopsis:       Misunderstanding on R5000 L2 cache Page_Invalidate(S) op size
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    port-mips-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Apr 03 18:05:00 +0000 2020
>Originator:     Izumi Tsutsui
>Release:        NetBSD 9.0
>Organization:
>Environment:
System: NetBSD 9.0
Architecture: mips
Machine: R5000 mips
>Description:
In src/sys/arch/mips/mips/cache_r5k.c, there is inconsistent size
definitions for block size of cache flush ops:

---
#define	mips_r5k_round_page(x)	round_line(x, PAGE_SIZE)
#define	mips_r5k_trunc_page(x)	trunc_line(x, PAGE_SIZE)

void
r5k_sdcache_wbinv_range(register_t va, vsize_t size)
{
	uint32_t ostatus, taglo;
	register_t eva = mips_r5k_round_page(va + size);

	va = mips_r5k_trunc_page(va);

	ostatus = mips_cp0_status_read();
	mips_cp0_status_write(ostatus & ~MIPS_SR_INT_IE);

	__asm volatile("mfc0 %0, $28" : "=r"(taglo));
	__asm volatile("mtc0 $0, $28");

	for (; va < eva; va += (128 * 32)) {
		cache_op_r4k_line(va, CACHEOP_R4K_HIT_WB_INV|CACHE_R4K_SD);
	}

	mips_cp0_status_write(ostatus);
	__asm volatile("mtc0 %0, $28; nop" :: "r"(taglo));
}
---

I.e. the start address and end address are truncated/rounded up
per PAGE_SIZE:
>> #define	mips_r5k_round_page(x)	round_line(x, PAGE_SIZE)
>> #define	mips_r5k_trunc_page(x)	trunc_line(x, PAGE_SIZE)

but "(128 * 32)" (i.e. 4096) is used in the flush address iteration:
>>	for (; va < eva; va += (128 * 32)) {
>>		cache_op_r4k_line(va, CACHEOP_R4K_HIT_WB_INV|CACHE_R4K_SD);
>>	}

The former part was changed in rev 1.17:
 http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/mips/mips/cache_r5k.c#rev1.17
>> - don't clear KX when disabling interrupts
>> - sign extend addresses as needed
>> - use PAGE_SIZE instead of blindly assuming 4KB
>> now n32 kernels work again on my R5k SGIs
 
Currently MIPS3 kernels seem to use 8192 as PAGE_SIZE by default,
and if the latter (128 * 32) is changed to PAGE_SIZE, most DMA ops
(ahc(4) disk xfers) fail on O2 (DMA ops on O2 are not cache coherent).

According to "User's Manual VR5000 TM, VR10000 TM 64-BIT MICROPROCESSOR
INSTRUCTION" (Document No. U12754EJ) by NEC, the "Page Invalidate (S)"
(R5000 only) op is described as:
---
1.4.19 Page Invalidate (S) (R5000 only)
 The processor will generate a page invalidate by doing a burst of
 128 line invalidates to the secondary cache at the page specified
 by the effective address generated by the CACHE instruction, which
 must be page aligned. Interrupts are deferred during page invalidates.
---

R5000 L2 cacheline size is 32, so the the original implementation
"(128 * 32)" shall be a correct for R5000 Page Invalidate (S) op,
and PAGE_SIZE is not correct. Extra truncation/roundup aginst flush
regions would be harmless except small performance penalty, though.

>How-To-Repeat:
Code inspection.

>Fix:
Revert PAGE_SIZE changes to (128 * 32), or
define and use proper macro for a size of R5K_Page_Invalidate_S op
as Linux does?
 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/mips/mm/sc-r5k.c?h=v5.6.1#n22
>> #define SC_LINE 32
>> #define SC_PAGE (128*SC_LINE)