port-arm/58789: aarch64 long double arithmetic is haunted

[campbell+netbsd%mumble.net@localhost]

>Number:         58789
>Category:       port-arm
>Synopsis:       aarch64 long double arithmetic is haunted
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    port-arm-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Oct 29 18:25:01 +0000 2024
>Originator:     Taylor R Campbell
>Release:        current
>Organization:
The Haunted NetBSD Rintlation
>Environment:
>Description:
Since the resolution to PR lib/57021 (libc __trunctfdf2 conflicts with libgcc on aarch64), we have, on aarch64:

- compiler-rt softfloat128 long double arithmetic linked into libc.so
- libgcc softfloat128 long double arithmetic defined in libgcc_s.so
- executables linked against both by default

The long double arithmetic symbols in question are symbols like __addtf3 and __subtf3.

The two implementations, from compiler-rt and libgcc, are not quite compatible.  For example, compiler-rt softfloat128 long double arithmetic has no concept of floating-point rounding mode, while gcc softfloat128 long double arithmetic respects the rounding mode set in the FPCR (floating-point control register).

I suspect this is why long double functions such as rintl exhibit haunted behaviour like the example below.
>How-To-Repeat:
$ cat rintl1.c
#include <fenv.h>
#include <math.h>
#include <unistd.h>

int
main(void)
{
	volatile long double x = 0x2.00000000000008p+52L;
	volatile long double y, z;
	volatile int c;

	y = rintl(x);
	fesetround(FE_UPWARD);
	z = rintl(x);
	c = x ? 0 : 1;
	write(STDOUT_FILENO, __UNVOLATILE(&y), sizeof(y));
	write(STDOUT_FILENO, __UNVOLATILE(&z), sizeof(z));
	return c;
}
$ cat rintl2.c
#include <fenv.h>
#include <math.h>
#include <unistd.h>

int
main(void)
{
	volatile long double x = 0x2.00000000000008p+52L;
	volatile long double y, z;
	volatile int c;

	y = rintl(x);
	fesetround(FE_UPWARD);
	z = rintl(x);
	c = 1 ? 0 : 1;
	write(STDOUT_FILENO, __UNVOLATILE(&y), sizeof(y));
	write(STDOUT_FILENO, __UNVOLATILE(&z), sizeof(z));
	return c;
}
$ diff -u rintl1.c rintl2.c
--- rintl1.c	2024-10-29 12:48:29.000000000 +0000
+++ rintl2.c	2024-10-29 12:48:09.000000000 +0000
@@ -12,7 +12,7 @@
 	y = rintl(x);
 	fesetround(FE_UPWARD);
 	z = rintl(x);
-	c = x ? 0 : 1;
+	c = 1 ? 0 : 1;
 	write(STDOUT_FILENO, __UNVOLATILE(&y), sizeof(y));
 	write(STDOUT_FILENO, __UNVOLATILE(&z), sizeof(z));
 	return c;
$ make rintl1 rintl2 LDLIBS=-lm DBG=-g\ -O2\ -Wall\ -Werror
cc -g -O2 -Wall -Werror    -o rintl1 rintl1.c -lm
cc -g -O2 -Wall -Werror    -o rintl2 rintl2.c -lm
$ ./rintl1 | hexdump -C
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 34 40  |..............4@|
00000010  00 00 00 00 00 00 00 08  00 00 00 00 00 00 34 40  |..............4@|
00000020
$ ./rintl2 | hexdump -C
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 34 40  |..............4@|
*
00000020

>Fix:
Yes, please!