lib/56937: printf(3) long double %a formatting is broken

[kre%munnari.OZ.AU@localhost]

>Number:         56937
>Category:       lib
>Synopsis:       printf(3) long double %a formatting is broken
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Jul 22 13:15:00 +0000 2022
>Originator:     kre%munnari.OZ.AU@localhost
>Release:        NetBSD 9.99.99
>Organization:
>Environment:
System: NetBSD jacaranda.noi.kre.to 9.99.99 NetBSD 9.99.99 (JACARANDA:1.1-20220718) #0: Tue Jul 19 03:35:07 +07 2022 kre%jacaranda.noi.kre.to@localhost:/usr/obj/testing/kernels/amd64/JACARANDA amd64
Architecture: x86_64
Machine: amd64
>Description:
	When printf(3) is asked to print a long double (flost) in %a format
	(eg: printf("%La", (long double)1.0); ) It includes a bit set in the
	result that should not be there.

>How-To-Repeat:
	You can write code easily to test this (I did), but a simple way
	is to install (of not already done) any version (well, recentish)
	of bash ans use its builtin printf(1) - bash uses long double for
	floats in its arithmetic

	jacaranda$ printf %a\\n 1
	0x8.8p-3
	jacaranda$ /usr/bin/printf %a\\n 1
	0x1p+0

	The first of those is from bash, and is clearly nonsense, no
	matter how you express it, the floating point representation
	of 1 has (at most) 1 bit set (in its mantissa), not two.

	The second is NetBSD's printf, which uses regular double for
	its floats, and is correct.

	Note that the difference between 0x8p-3 and 0x1p+0 is
	irrelevant - either is acceptable (those values represent the
	same thing).

	Similarly, from bash:

	jacaranda$ printf '%a ' 1.5 1.25 1.125 1.0625 ; printf \\n
	0x8.cp-3 0x8.ap-3 0x8.9p-3 0x8.88p-3 

	Those values should all have 2 bits set, not the 3 shown.
	And they do when it is just a double being printed:

	jacaranda$ /usr/bin/printf '%a ' 1.5 1.25 1.125 1.0625 ; printf \\n
	0x1.8p+0 0x1.4p+0 0x1.2p+0 0x1.1p+0 

	You can repeat all of this using a simple C program, to demonstrate
	that the problem is not internal to bash somewhere (which is
	what I first assumed).

	The internal format difference between double and long double
	should not be altering the representation (in this form) of any
	of these values, the extra precision or range is not relevant.

	All the other conversions (%e %f %g) seem fine, as does parsing
	floats in any of the formats they're allowed - including converting
	0xX.XXXp[+-]N into float/double/long double.

>Fix:
	I spent some time hunting in src/lib/libc/gdtoa/hdtoa.c
	which is where I think the problem occurs, but ...
	(It doesn't help that I don't know either IEEE or x87
	floating point representations almost at all).