Sign-extension bug in syscall return path on big-endian N64

[Thor Lancelot Simon <tls%panix.com@localhost>]

I've been chasing a strange problem where, on big-endian N64, ssh bails
out claiming it's setuid because getuid() != geteuid.

This likely doesn't impact many users because, as it turns out, a _very_
high UID is required in order to exhibit the bug.  I obtained one through
another error: accidentally rebuilding the pwd database with the wrong
endianness. ;-)

That aside, after a pile of analysis -- which I heavily relied on Claude
for, but which I confirmed myself -- the issue seems to be an N64 ABI
violation in the syscall return path.

Claude's explanation (which appears correct from my manual examination,
though I *have not* fully dug through the MIPS N64 ABI spec:

	The MIPS N64 ABI requires that all 32-bit integer values
	(including unsigned) are sign-extended to 64 bits in registers.
	GCC enforces this via `PROMOTE_MODE` in `gcc/config/mips/mips.h`
	which sets `UNSIGNEDP=0` for `SImode`.  However, the kernel's
	syscall return path does not sign-extend 32-bit return values.

	Syscall functions store `uint32_t` results into `register_t`
	(`int64_t`) via C assignment, which zero-extends (e.g. `0x80000001`
	becomes `0x00000000_80000001` instead of the ABI-required
	`0xffffffff_80000001`).  The syscall return path in syscall.c
	passes these values directly to userspace in the v0/v1 registers
	without correction.

	When GCC spills a 32-bit return value to the stack with `sd`
	(64-bit store) and reloads it with `lw` (32-bit load), `lw`
	correctly sign-extends per the ABI.  If bit 31 is set, the
	sign-extended reload (0xffffffff_80000001) differs from the
	zero-extended original (0x00000000_80000001), causing comparisons
	to produce incorrect results.

	Any comparison of two 32-bit syscall return values with bit 31
	set can fail.  The most visible manifestation is OpenSSH refusing
	to run with "ssh setuid not supported" because ssh.c checks
	`if (getuid() != geteuid()`, and this comparison fails on big
	endian MIPS N64 when uid >= 0x80000000.

This can also trigger on little-endian N64, of course; the explanation is
specific to big-endian N64 because I triggered the bug by accidentally
running a big-endian system with a little-endian password database!  The
specific manifestation with getuid() != geteuid() is unlikely in general
because normally UID values do not have bit 31 set, but the problem is
not specific to these two system calls.

Triggering the bug is annoying because it requires making a syscall
with a 32-bit return value in a program context in which there is
enough register pressure for GCC to spill and reload the return value
from the stack.  I have a small reproducer but it is Claude-written
code so it's not suitable for contribution to NetBSD.  However, the
fix seems relatively simple (note this is my code, not Claude's!).
Is this OK to commit?

Index: syscall.c
===================================================================
RCS file: /cvsroot/src/sys/arch/mips/mips/syscall.c,v
retrieving revision 1.51
diff -u -r1.51 syscall.c
--- syscall.c	5 Oct 2023 19:41:04 -0000	1.51
+++ syscall.c	10 Apr 2026 19:08:35 -0000
@@ -316,17 +316,41 @@
 
 	switch (error) {
 	case 0:
+		if (SYCALL_RET_64_P(callp)) {
 #if !defined(__mips_o32)
-		if (abi == _MIPS_BSD_API_O32 && SYCALL_RET_64_P(callp)) {
-			/*
-			 * If this is from O32 and it's a 64bit quantity,
-			 * split it into 2 32bit values in adjacent registers.
-			 */
-			mips_reg_t tmp = reg->r_regs[_R_V0];
-			reg->r_regs[_R_V0 + _QUAD_LOWWORD] = (int32_t) tmp;
-			reg->r_regs[_R_V0 + _QUAD_HIGHWORD] = tmp >> 32;
-		}
+			if (abi == _MIPS_BSD_API_O32) {
+				/*
+				 * If this is from O32 and it's a 64bit
+				 * quantity, split it into 2 32bit values
+				 * in adjacent registers.
+			 	 */
+				mips_reg_t tmp = reg->r_regs[_R_V0];
+				reg->r_regs[_R_V0 + _QUAD_LOWWORD] =
+				    (int32_t) tmp;
+				reg->r_regs[_R_V0 + _QUAD_HIGHWORD] =
+				    tmp >> 32;
+			}
 #endif
+			if (abi != _MIPS_BSD_API_O32) {
+				/*
+				 * Sign-extend 32-bit return values
+				 * to 64 bits, as required by the N64
+				 * ABI.  Without this, zero-extended
+				 * 32-bit values from the kernel can
+				 * flow through to userspace where if
+				 * they are spilled to/reloaded from
+				 * the stack under register pressure,
+				 * they may be sign-extended by the
+				 * reload, causing bizarre results like
+				 * getuid() != geteuid() if the UID has
+				 * bit 31 set.
+				 */
+				reg->r_regs[_R_V0] =
+				    (int32_t)(int64_t)reg->r_regs[_R_V0];
+				reg->r_regs[_R_V1] =
+				    (int32_t)(int64_t)reg->r_regs[_R_V1];
+			}
+		}
 #ifdef MIPS_SYSCALL_DEBUG
 		if (p->p_emul->e_syscallnames)
 			printf("syscall %s:", p->p_emul->e_syscallnames[code]);