__hldtoa broken for ld128

[enh <enh%google.com@localhost>]

this was found by fuzzing the LLVM __cxa_demangle on an ld128 Android
system using hwasan, but it turns out no to simply be a buffer
overflow --- the results are just wrong. (which shows how much anyone
uses ld128 in conjunction with %a!)

this was the minimized test case:

free(__cxa_demangle("1\006ILeeeEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE", 0, 0, 0));

which, going down a level to snprintf was working out to something
like this gtest:

  char buf[BUFSIZ];
  union {
    uint64_t a[2];
    long double v;
  } u;
  u.a[0] = UINT64_C(0xcececececececece);
  u.a[1] = UINT64_C(0xcececececececece);
  EXPECT_EQ(41, snprintf(buf, sizeof(buf), "<%La>", u.v));
  EXPECT_STREQ("<-0x1.cecececececececececececececep+3791>", buf);


here's a fix, the diff being relative to the copy of the OpenBSD
source that's used in Android (but the NetBSD source looks the same,
with minor cosmetic differences, and this should fix both):


--- a/libc/upstream-openbsd/lib/libc/gdtoa/hdtoa.c
+++ b/libc/upstream-openbsd/lib/libc/gdtoa/hdtoa.c
@@ -278,18 +278,18 @@ __hldtoa(long double e, const char *xdigs, int
ndigits, int *decpt, int *sign,
                p->ext_fracl >>= 4;
        }
 #ifdef EXT_FRACHMBITS
-       for (; s > s0; s--) {
+       for (; s > s0 + sigfigs - ((EXT_FRACLBITS + EXT_FRACHMBITS) /
4) - 1; s--) {
                *s = p->ext_frachm & 0xf;
                p->ext_frachm >>= 4;
        }
 #endif
 #ifdef EXT_FRACLMBITS
-       for (; s > s0; s--) {
+       for (; s > s0 + sigfigs - ((EXT_FRACLBITS + EXT_FRACHMBITS +
EXT_FRACLMBITS) / 4) - 1; s--) {
                *s = p->ext_fraclm & 0xf;
                p->ext_fraclm >>= 4;
        }
 #endif
-       for (; s > s0; s--) {
+       for (; s > s0 + sigfigs - ((EXT_FRACLBITS + EXT_FRACHMBITS +
EXT_FRACLMBITS + EXT_FRACHBITS) / 4) - 1; s--) {
                *s = p->ext_frach & 0xf;
                p->ext_frach >>= 4;
        }
@@ -300,7 +300,7 @@ __hldtoa(long double e, const char *xdigs, int
ndigits, int *decpt, int *sign,
         * (partial) nibble, which is dealt with by the next
         * statement.  We also tack on the implicit normalization bit.
         */
-       *s = p->ext_frach | (1U << ((LDBL_MANT_DIG - 1) % 4));
+       *s = (p->ext_frach | (1U << ((LDBL_MANT_DIG - 1) % 4))) & 0xf;

        /* If ndigits < 0, we are expected to auto-size the precision. */
        if (ndigits < 0) {


if you want to watch it going wrong live in a way that makes the bug
quite plain, i recommend adding `fprintf(stderr, "%02x // %x\n", *s,
p->ext_<whatever>);` between each `*s =` line and the `>>= 4` line
that follows. that was where i understood what was going wrong.

thanks!