tech-userlevel archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

localedef(1) (Was: [OT]Who's in charge of Localization (L10N) in NetBSD?)


> I've also started work on a localedef(1) tool. Again, not enough time
> to complete.

i have started to implement localedef(1) too :-)
but now suspended, because of the specification's unclearness
(and also i have no time to solve these problems).

POSIX locale's localedef(1) specification apparently disregards some
stateful encoding.
charmap can't treat locking-shift escape sequence like ISO/IEC
2022(*1) correctly :-<


OTOH, we have already have these encodings(like ja_JP.ISO2022-JP,
ja_JP.CTEXT and so on).
to keep backward compatibility, we have to support locking-shift
stateful encoding by charmap.

ISO/IEC TR14652(*2) extensions mentioned about how to treat ISO/IEC
2022 escape sequence,
by using <escseq2022> and <include> tags in charmap.
# why not hz-gb2312? n-byte hangle? viqr? and any other stateful encoding?


but it is not enough, if i don't misunderstand spec :-(
following problems still exist:

1. <include><escseq2022> tag can't treat ISO/IEC 2022's single-shift correctly.

using single-shift is not special case, see:
    + ISO-2022-CN-EXT(RFC1922) (*3)
    + eucJP with supplymentaly ideograph(jisx0212) and half-width
     (UI-OSF Application Platform Profile for Japanese Environment
Version 1.1) (*4)


localedef(1) parser can't determin whether
LS2 or SS2, LS3 or SS3, LS2R or SS2R, LS3R or SS3R:

designation -> tag
LS0(SO)         <include> "g0";"g0",...;
LS1(SI)         <include> "g1";"g0",...;
LS2             <include> "g2";"g0",...;
LS3             <include> "g3";"g0",...;
LS1R            <include> "g1";"g1",...;
LS2R            <include> "g2";"g1",...;
LS3R            <include> "g3";"g1",...;

SS2             <include> "g2";"g0",...; /* XXX same as LS2 */
SS3             <include> "g3";"g0",...; /*             LS3 */
SS2R            <include> "g2";"g1",...; /*             LS2R */
SS3R            <include> "g3";"g1",...; /*             LS3R */

because escape sequense is hardcorded in *included* files's
<escseq2022> tag, like:
<escseq2022>    "g0";"g0";"/x28/x48"
<escseq2022>    "g1";"g0";"/x29/x48"

this escape sequence(includes final character) means locking-shift,
not single-shift(0x8E/0x8F).

or we have to treat single-shift as multibyte sequence like Shift_JIS?
<include> "g2";"g0","JISX0208-1990"; # LS2
# XXX: SS2
<half-width-kana-a> /x8E/xAE

2. spec disregards `empty charset designation by default' case, such as:
    + ISO-2022-KR(RFC1557) (*5)


RFC1557 said:

ESC $ ) C       Appears once in the beginning of a line
                before any appearance of SO characters.

so that G1 should be empty in initial state, but we can't control.

or should i avoid this case by using EMPTY designation?
<include> "g0";"g0";"ISO646-US"
<include> "g1";"g0";"EMPTY" /* XXX kludge */
<include> "g1";"g0";"KSC5601"

3. <escseqq2022> doesn't include ESC(0x1b),
hardcording of 0x1b is not Codeset Independent!

4. repertoiremap can't treat many to many mapping

example, JISX0213-2004 <-> UCS-4 conversion need many to one mapping.
(and some arabic too!) but repertoiremap seems only one to one mapping.
using Unicode Named Sequence? how to map them to wchar_t?

and more, more, simple mklocale(1) is lol.

early stage of implementation(=garbage) is here.

very truly yours.
Takehiko NOZAKI<>

Home | Main Index | Thread Index | Old Index