lib/50698: hypotf() of small numbers yields infinity

[gson%gson.org@localhost (Andreas Gustafsson)]

>Number:         50698
>Category:       lib
>Synopsis:       hypotf() of small numbers yields infinity
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Jan 23 22:45:00 +0000 2016
>Originator:     Andreas Gustafsson
>Release:        NetBSD 7.0
>Organization:

>Environment:
System: NetBSD
Architecture: x86_64
Machine: amd64
>Description:

The hypotf() function in libm incorrectly returns infinity (!) when
its arguments are small numbers such as 1e-18.

I have tested this on amd64 (7.0, real hardware), i386 (-current,
qemu), and sparc (-current, qemu), and they all behave this way.

>How-To-Repeat:

$ cat test-hypotf.c
#include <math.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv) {
    float a = atof(argv[1]);
    float b = atof(argv[2]);
    float c = hypotf(a, b);
    printf("%.15g\n", c);
    return 0;
}
$ cc test-hypotf.c -lm -o test-hypotf
$ ./test-hypotf 1e-18 1e-18
inf

On a non-NetBSD-machine, the test program gives the correct result:

./a.out 1e-18 1e-18
1.41421363268178e-18

>Fix: