lib/58781: fesetround and fpsetround are out of sync on some architectures

[campbell+netbsd%mumble.net@localhost]

>Number:         58781
>Category:       lib
>Synopsis:       fesetround and fpsetround are out of sync on some architectures
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Oct 28 19:15:00 +0000 2024
>Originator:     Taylor R Campbell
>Release:        current, 10, 9, ...
>Organization:
The NetBSD Roundation
>Environment:
>Description:
On sparc64, fesetround sets %fsr (floating-point status register), but that only affects float and double arithmetic (IEEE 754 binary32 and binary64), which is computed in hardware by CPU instructions.  Long double arithmetic is computed in software by libc softfloat, which doesn't respect %fsr and instead has its own global variable _softfloat_float_rounding_mode -- which in turn should really be a thread-local variable.

This is probably not the only mistake of this class.  We need to audit the following architectures which have separate fesetround and fpsetround implementations:

1. [ ] aarch64 (libc fpsetround, libm fesetround)
2. [ ] alpha (libc fpsetround, fenv.h fesetround)
3. [ ] arm (libc fpsetround, softfloat fpsetround, libm fesetround)
4. [ ] hppa (libc fpsetround, libm fesetround)
5. [ ] i386 (libc fpsetround, libm fesetround; remember to sync x87 and SSE state)
6. [ ] ia64 (libc fpsetround, fenv.h fesetround)
7. [ ] m68k (libc fpsetround, softfloat fpsetround, fpfake.c fpsetround, fenv.h fesetround)
8. [ ] mips (libc fpsetround, softfloat fpsetround, fenv.h fesetround)
9. [ ] or1k (libc fpsetround, softfloat fpsetround, missing fesetround?)
10. [ ] powerpc (libc fpsetround, softfloat(?) fpsetround, fenv.h fesetround)
11. [ ] powerpc64 (libc fpsetround, softfloat(?) fpsetround, powerpc/fenv.h fesetround)
12. [ ] riscv (libc fpsetround, softfloat/softfloat128 fpsetround, libm fesetround)
13. [ ] sparc (libc fpsetround, softfloat128 fpsetround, libm fesetround)
14. [ ] sparc64 (libc fpsetround, softfloat128 fpsetround, libm fesetround)
15. [ ] x86_64 (libc fpsetround, libm fesetround; remember to sync x87 and SSE state)

Also need to make sure that fegetround, fpgetround, and FLT_ROUNDS all agree on which variable to query.

Looks like sh3 only has fesetround in fenv.h, no fpsetround at all -- which is maybe a separate bug that should be fixed (although fpsetround is a legacy BSD API that has been superseded by C99 fenv.h).
>How-To-Repeat:
On a sparc64 system:

cd /usr/tests/lib/libm && atf-run t_fe_round | atf-report

Notice lots of long double rounding operations in non-default rounding modes fail, like nearbyintl and rintl.
>Fix:
Yes, please!