tech-userlevel archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: long double losing mantissa bits



> Date: Tue, 19 Jan 2021 22:16:32 +0000
> From: Taylor R Campbell <riastradh%NetBSD.org@localhost>
> 
> This is not an integer, so conversion to uint64_t rounds it to
> nearest, with ties to even, so you get ufrac = 0x80000000000000 =
> 36028797018963968 as shown in the result.

Oops -- one more correction: conversion to uint64_t rounds toward
zero, not to nearest.  The result happens to coincide in this case.

> It looks like you have an off-by-one error in your exponent handling.
> If you want to scale the fractional part into an integer, you need to
> multiply by 2^57, not by 2^56.

Another possibility is that you expected round-to-odd or something
instead of toward-zero or to-nearest -- or perhaps the code works as
intended but the test cases are buggy.


Home | Main Index | Thread Index | Old Index