lib/60015: x87 makes testing signalling NaN difficult

["campbell+netbsd%mumble.net@localhost via gnats" <gnats-admin%NetBSD.org@localhost>]

>Number:         60015
>Category:       lib
>Synopsis:       x87 makes testing signalling NaN difficult
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Feb 17 23:25:00 +0000 2026
>Originator:     Taylor R Campbell
>Release:        current
>Organization:
The NanBSD/i387 Roundation, Inc.
>Environment:
>Description:

	The C23 fpclassify() function and its cousins isnan(),
	isinf(), isnormal(), &c., are required, in implementations
	supporting Annex F (IEC 60559 floating-point arithmetic, more
	popularly known as IEEE 754), _not_ to raise floating-point
	exceptions _even if_ the inputs are signalling NaN:

	> 6. The C classification macros fpclassify, iscanonical,
	> isfinite, isinf, isnan, isnormal, issignaling, issubnormal,
	> iszero, and signbit provide the IEC 60559 operations
	> indicated in the table above provided their arguments are in
	> the format of their semantic type. Then these macros raise no
	> floating-point exceptions, even if an argument is a signaling
	> NaN.  --C23, Annex F, Sec. F.3

	(This language appears to be new in C23; I can't find it in C99
	or C11, though I haven't tracked down the specific change that
	added it in C23.)

	It is tempting to try something like this, as in t_fpclassify.c
	rev. 1.21 for double (simplified for the double-only case):

bool
issignalling(double f)
{
	union { double f; uint64_t i; } u = { .f = f };

	return isnan(f) && (u.i & (DBL_SNANBIT|DBL_QNANBIT)) == DBL_SNANBIT;
}

	Unfortunately, actually using this can prove difficult.  The
	following assertion should be guaranteed not to fire:

		feclearexcept(FE_ALL_EXCEPT);
		x = issignalling(nan);
		assert(fetestexcept(FE_ALL_EXCEPT) == 0);

	But on x87, mere floating-point load instruction (FLDL) to load
	a double (IEEE 754 binary64) into the floating-point register
	stack converts it to long double format (x87 80-bit extended
	precision) which maps signalling NaN to quite NaN and, as a
	side effect, raises FE_INVALID.

	So issignalling might wrongly return false _and_ calling it
	might raise a floating-point exception.

>How-To-Repeat:

	Expected output with the following program:

		issignalling=1 except=0x0

	Actual output on i386:

		issignalling=0 except=0x1

#include <fenv.h>
#include <float.h>
#include <math.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>

#if defined __hppa__ || defined __mips__	/* just, why? */
#  define	QNANBIT	0
#  define	SNANBIT	1
#else
#  define	QNANBIT	1
#  define	SNANBIT	0
#endif

#define	DBL_QNANBIT	(QNANBIT*((uint64_t)1 << (DBL_MANT_DIG - 2)))
#define	DBL_SNANBIT	(SNANBIT*((uint64_t)1 << (DBL_MANT_DIG - 2)))

bool
issignalling(double f)
{
	union { double f; uint64_t i; } u = { .f = f };

	return isnan(f) && (u.i & (DBL_SNANBIT|DBL_QNANBIT)) == DBL_SNANBIT;
}

volatile union {
	double d;
	uint64_t u64;
} u = { .u64 = 0x7ff0000000000001 };

int
main(void)
{
	bool result;
	int except;

	feclearexcept(FE_ALL_EXCEPT);
	result = issignalling(u.d);
	except = fetestexcept(FE_ALL_EXCEPT);

	printf("issignalling=%d except=0x%x\n", result, except);
	fflush(stdout);
	return ferror(stdout);
}

	Relevant excerpt of generated code -- note that the value of
	u.d is loaded with FLDL only to store it on the stack again
	with FSTPL for passing to __isnand, which both raises an
	exception (FE_INVALID) and changes the input to __isnand so
	that it is a quiet NaN:

  16:	dd 05 00 00 00 00    	fldl   0x0
			18: R_386_32	u
  1c:	dd 5c 24 18          	fstpl  0x18(%esp)
  20:	f2 0f 10 44 24 18    	movsd  0x18(%esp),%xmm0
  26:	f2 0f 11 04 24       	movsd  %xmm0,(%esp)
  2b:	e8 fc ff ff ff       	call   2c <main+0x2c>
			2c: R_386_PC32	__isnand
  30:	85 c0                	test   %eax,%eax
  32:	74 50                	je     84 <main+0x84>
  34:	8b 54 24 1c          	mov    0x1c(%esp),%edx
  38:	89 d0                	mov    %edx,%eax
  3a:	c1 e8 13             	shr    $0x13,%eax
  3d:	83 f0 01             	xor    $0x1,%eax
  40:	89 c3                	mov    %eax,%ebx
  42:	83 e3 01             	and    $0x1,%ebx

>Fix:

	Find a flux capacitor and a place to drive 88mph during a
	lightning storm so we can have a word with engineers at Intel
	in 1985.