Re: bin/60496: Locale-dependent UTF-8 issue over SSH session

[Robert Elz <kre%munnari.OZ.AU@localhost>]

There is progress on this issue, it hasn't been forgotten!

Given the test program:

#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <locale.h>

int
main(int argc __unused, char **argv __unused)
{
        char buf[129];

        setlocale(LC_ALL, "");
        memset(buf, 0, sizeof buf);

        sprintf(buf, "%'d\n", 123456789);
        write(1, buf, strlen(buf));

        sprintf(buf, "%'40.4f\n", 123456789012.76543);
        write(1, buf, strlen(buf));

        return 0;
}

[Just using the sprintf/write combination as it was easier for debugging
 - the changes are all buried inside the __vfprintf_unlocked_l or
 __vfwprintf_unlocked_l function (as appropriate) and the subordinate
 routines it calls, which are used for all *printf() variations, so this
 should be the same for any kind of printf().]

When built to make (the imaginatively named) binary) "t" I now get:

LC_ALL=nb_NO.UTF-8 ./t
123 456 789
                 123 456 789 012,7654

LC_ALL=nb_NO.UTF-8 ./t | hexdump
0000000 3231 c233 34a0 3635 a0c2 3837 0a39 2020
0000010 2020 2020 2020 2020 2020 2020 2020 3120
0000020 3332 a0c2 3534 c236 37a0 3938 a0c2 3130
0000030 2c32 3637 3435 000a

That is, the grouping characters are properly C2A0
aka \u00A0 (the unbreakable space) which I believe
is what is intended (it is what is in our locale data
anyway, if incorrect that can be a different change).


In my fake test locale, I get:

LC_ALL=x1_X1.UTF-8 ./t | hexdump
0000000 3231 d933 34ac 3635 d937 38ac 0a39 2020
0000010 2020 2020 2020 2020 2020 3120 d932 33ac
0000020 acd9 3534 d936 37ac 3938 d930 31ac e232
0000030 8e97 3637 3435 000a

There the grouping characters are D9AC (\u066c)
and the decimal point is E2978E (\u25ce) - not
because they're particularly likely values to
ever exist, but just because they're not impossible
either:

LC_ALL=x1_X1.UTF-8 ./t
123٬4567٬89
             12٬3٬456٬7890٬12◎7654

(and yes, this fake locale also uses a rather odd grouping
specification, not just every N digits).

I am still working on improving the ATF tests before this
gets committed, but I believe the libc parts are all OK
(wrt this issue anyway) now.   The decimal point change is
identical to (obtained from) the FreeBSD changes to fix
the same problem, the grouping changes are done differently,
FreeBSD's implementation has diverged from ours in other
ways, and it was (for just this) easier to simply reimplement
the changes in ways that fit our current sources.

kre