tech-userlevel archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: long double losing mantissa bits



On Tue 19 Jan 2021 at 22:16:32 +0000, Taylor R Campbell wrote:
> It looks like you have an off-by-one error in your exponent handling.
> If you want to scale the fractional part into an integer, you need to
> multiply by 2^57, not by 2^56.

Thank you! Your analysis made me realise that I was miscounting the
bits, and assumed one too many. So several of the test values were a bit
too long and got rounded which confused me.

So I did change the multiplication to 2^57.  But the 2^56 was correct
for the final format, so I just used it to get the lsb for rounding
later, and then shifted it a bit right again.

Final result is at
https://gitlab.com/Rhialto/macro11/-/blob/master/parse.c#L386.

-Olaf.
-- 
Olaf 'Rhialto' Seibert -- rhialto at falu dot nl
___  Anyone who is capable of getting themselves made President should on
\X/  no account be allowed to do the job.       --Douglas Adams, "THGTTG"

Attachment: signature.asc
Description: PGP signature



Home | Main Index | Thread Index | Old Index