Weird 8859-1 charset

[Robert Elz <kre%munnari.OZ.AU@localhost>]

Hello all ... Some of you might know I am working on fixing the
printf(3) use of non-ascii decimal point and the grouping characters
(see PR bin/60496 which really should be lib/...)

The fixes for the bugs are relatively straightforward, and made
even easier by FreeBSD having already made them many years ago.

However, obviously (I hope) I'm not going to commit any fixes
without testing them, and while we have locales which have non-ascii
grouping characters, we have none, or none I could find, which have
a non-ascii decimal point.

For local testing I could easily just change some locale without
committing that change, run the test using that locale (one I'd
never normally use locally, like perhaps the no_NO locale from the
PR) and run enough tests to be confident the fixes are working.

But Taylor would have my grouping characters for decimal points
if I fail to commit ATF tests to test the changes, so we can make
sure this problem never happens again, and that can't be done without
installing a locale with a non-ascii decimal point (certainly at
least one - more than one would be better, as each gets just one
decimal point and grouping character, and testing just one variation
of them wouldn't be much of a test).

So I plan on creating a few new locales, using the (reserved) x*
names for country codes.   Given who I am, I'll probably have them
exclusively using English, but natuarlly anyone would be free to
add some variations using other languages, to improve the test
coverage (the decimal point and grouping chars need to work with
any character encoding that might be used with the locale, and as
these will all be fakes, we can make that to include absolutely
everything, if someone can do the work to define them).

Unfortunately I know nothing at all about locales, so I have been
trying to learn.   Our doc (unless I missed something) on this isn't
great, I haven't (so far) even managed to work out how locales get made
(other than message catalogs and the charset files, which aren't the parts
I need to change).   But I shouldn't really need to know that, just copy
existing locales for the files & contents, add the new one(s) to
Makefile.locale (a list of all the locale names), and then just allow the
build system to do its thing.

With that in mind I have been looking at the existing locale definitions
so I can work out what I need to create, in order to build a (fake)
locale definition which is correct (as a locale) even if containing
nonsense as values for a locale (I mean, no'one is likely to really use
a CJK ideograph as a decimal point, are they?   But that's a good way
to test long UTF-8 encodings and non-ascii based charsets.)

And now we finally get to the point of this message:

While looking, I looked in .../src/share/locale/ctype/charset and for
the purposes of this e-mail anyway, the Latin-1 file in particular.

In that we see (inter alia):

	ALPHA		0x40 - 0x56 0x58 - 0x76 0x78 - 0x7f
	LOWER		0x60 - 0x76 0x78 - 0x7f
	PUNCT		0x21 - 0x3f 0x57 0x77
	UPPER		0x40 - 0x56 0x58 - 0x5f

What's with the Ws ???   (0x57 and 0x77)

As best I can tell, 0x57 is 'W' and 0x77 is 'w' in Latin-1
(aka 8859-1) despite the original (ancient) Latin not having a 'W'
character.   If those code points truly are punctuation, used for
what?   The W's are also issing from the TOUPPER and TOLOWER mappings.

Probably now someone will tell me it is all irrelevant, as
nothing uses this file (the ctype/en_US.ISO_8859-1.src file
has no similar oddities).   This possibility is increased given
there is no Latin-9 file, which as best I can work out would
be the name for an equivalent for ISO 8859-15 (the other possibility,
it seems would be Latin-0, which we don't have either).

Any idea what is going on here, someone must have a clue?

If those files (or at least, that one file) isn't used, why hasn't
it just been deleted?

And while I am here, two related questions, more just to satisfy
my curiosity.

First, the Latin-1 file also contains:

	CHARSET		",A"

which a comment suggests is used after an ESC.   Used for what?
Anything that matters?   Who or what assigned the ",A" ?
(Other files in that dir have similar definitions, with different
values.)

And second, I see that along with ctype/en_US.ISO_8859.1.src
there are sibling files en_US.ISO_8859-2.src and en_US.ISO_8859-4.src
but there is also a file en_US.DIS_8859-15.src

What's with the "DIS" ?   Surely no-one believes tha 8859-15 is still
just a draft standard, do they?   Shouldn't that one be changed to
en_US.ISO_8859-15.src ?

kre