lib/57250: dtoa mishandles infinite doubles on 32bit big endian machines

[martin%NetBSD.org@localhost]

>Number:         57250
>Category:       lib
>Synopsis:       dtoa mishandles infinite doubles on 32bit big endian machines
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Feb 28 15:05:00 +0000 2023
>Originator:     Martin Husemann
>Release:        NetBSD 10.99.2
>Organization:
The NetBSD Foundation, Inc.
>Environment:
System: NetBSD big-apple.aprisoft.de 10.99.2 NetBSD 10.99.2 (POWERMAC_G5.MP) #95: Tue Feb 14 11:59:21 CET 2023 martin%seven-days-to-the-wolves.aprisoft.de@localhost:/work/src/sys/arch/macppc/compile/POWERMAC_G5.MP macppc
Architecture: powerpc
Machine: macppc
>Description:

While looking at a lint test failure on some machines Roland came up with
this test code:

---8<---
#include <err.h>
#include <errno.h>
#include <math.h>
#include <stdio.h>
#include <stdlib.h>
                                       
static void
lex_floating_constant(const char *cp)
{ 
        errno = 0;
        char *eptr;
        long double ld = strtold(cp, &eptr);
        if (errno != 0)
                warn("errno");

        ld = (double)ld;
        if (isfinite(ld) == 0) 
                warnx("infinite");
        printf("%Lf\n", ld);
}

int
main(int argc, char **argv)
{
        lex_floating_constant(argv[1]);
}
--->8---

compiling and invoking it with 1e309 shows strange output on 32bit big
endian machines:

test: errno: Result too large or too small
test: infinite
Infinity0000000000000000000000000000000000000000000000000000000[...]

while the expected output is:

[/tmp] martin@thirdstage > ./test 1e309
test: infinite
inf
[/tmp] martin@thirdstage > 

The failure happens at least on sparc, powerpc and evbearmv7hf-eb.
It works on 64bit big endian (sparc64) and all little endian machines I
tested.

>How-To-Repeat:
s/a

>Fix:
n/a