Internet Draft                                               M. Duerst
<draft-duerst-i18n-norm-04.txt>                    W3C/Keio University
Expires in six months                                         M. Davis
                                                                    IBM
                                                         September 2000


              Character Normalization in IETF Protocols


Status of this Memo

This document is an Internet-Draft and is in full conformance
with all provisions of Section 10 of RFC2026.

Internet-Drafts are working documents of the Internet Engineering Task
Force (IETF), its areas, and its working groups.  Note that other
groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time.  It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."

The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt

The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.

This document is not a product of any working group, but may be
discussed on the mailing lists <www-international@w3.org> or
<discuss@apps.ietf.org>.


Abstract

The Universal Character Set (UCS) [ISO10646, Unicode] covers a very
wide repertoire of characters. The IETF, in [RFC 2277], requires that
future IETF protocols support UTF-8 [RFC 2279], an ASCII-compatible
encoding of UCS. The wide range of characters included in the UCS has
lead to some cases of duplicate encodings. This document proposes
that in IETF protocols, the class of duplicates called canonical
equivalents be dealt with by using Early Uniform Normalization
according to Unicode Normalization Form C, Canonical Composition (NFC)
[UTR15]. This document describes both Early Uniform Normalization
and Normalization Form C.


Table of contents

0. Change log
1. Introduction
    1.1 Motivation
    1.2 Notational Conventions
2. Early Uniform Normalization
3. Canonical Composition (Normalization Form C)
    3.1 Decomposition
    3.2 Reordering
    3.3 Recomposition
    3.4 Implementation Notes
4. Stability and Versioning
5. Cases not Dealt with by Canonical Equivalence
6. Security Considerations
Acknowledgements
References
Copyright
Author's Addresses


0. Change Log

Changes from -03 to -04

- Changed intro to make clear this is mainly about canonical
   equivalences
- Made UTR#15, V18.0, the normative description of NFC
- Added subsection on interaction with text processing (3.4.11)
- Added various examples
- Various small wording changes
- Added reference to test file
- Added a note re. terminology (Normalization vs. Canonicalization)

Changes from -02 to -03

- Fixed a bad typo in the title.
- Made a lot of wording corrections and presentation improvements,
   most of them suggested by Paul Hoffman.


1. Introduction

1.1 Motivation

The Universal Character Set (UCS) [ISO10646, Unicode] covers a very
wide repertoire of characters. The IETF, in [RFC 2277], requires that
future IETF protocols support UTF-8 [RFC 2279], an ASCII-compatible
encoding of UCS. The need for round-trip convertion to pre-existing
character encodings has led to some cases of duplicate encodings.
This has lead to uncertainity for protocol specifiers and
implementers, because it was not clear which part of the Internet
infrastructure should take responsibility for these duplicates,
and how.

Besides straight-out duplicates, there are also many cases of
characters that are in one way or another similar. The equivalence
between duplicates is called canonical equivalence. Many of the
equivalences between similar characters are called compatibility
equivalences. This document concentrates on canonical equivalence.
The various cases of similar characters are listed in Section 5.

There are mainly two kinds of canonical equivalences, singleton
equivalences and precomposed/decomposed equivalences. Both of these
can be illustrated using the character A with a ring above. This
character can be encoded in three ways:

1) U+00C5 LATIN CAPITAL LETTER A WITH RING ABOVE
2) U+0041 LATIN CAPITAL LETTER A followed by U+030A COMBINING RING
    ABOVE
3) U+212B ANGSTROM SIGN

The equivalence between 1) and 3) is a singleton equivalence. The
equivalence between 1) and 2) is a precomposed/decomposed equivalence,
where 1) is the precomposed representation, and 2) is the decomposed
representation.

In all three cases, it is supposed to look the same for the reader.
Also, applications may use one or another representation, or even
more than one, but they are not allowed to assume that other
applications will preserve the difference between them.

The inclusion of these various representation alternatives was a
result of the requirement for round trip conversion with a wide range
of legacy encodings as well as of the merger between Unicode and
ISO 10646.

The Unicode Standard from early on has defined Canonical Equivalence
to make clear which sequences of codepoints cases should be treated
as pure encoding duplicates and which sequences of codepoints should
be treated as genuinely different (if maybe in some cases closely
related) data. The Unicode Standard also from early on defined
decomposed normalization, what is now called Normalization Form D
(case 2) in the example above). This is very well suited for some
kinds of internal processing, but decomposition does not correspond
to how data gets converted from legacy encodings and transmitted on
the Internet. In that case, precomposed data (i.e. case 1) in the
example above) is prevalent.

Note: This specification uses the term 'codepoint', and not
       'character', to make clear that it speaks about what the
       standards encode, and not what the end users think about,
       which is not always the same.

Encouraged by many factors such as a requirements analysis of the W3C
[Charreq], the Unicode Technical Committee defined Normalization
Form C, Canonical Composition (see [UTR15]). Normalization Form C
in general produces the same representation as straightforward
transcoding from legacy encodings (See Section 3.4 for the known
exception). The careful and detailled definition of Normalization
Form C is mainly needed to unambigously define edge cases (base
letters with two or more combining characters). Most of these edge
cases will turn up extremely rarely in actual data.

The W3C is adapting Normalization Form C in the form of Early Uniform
Normalization, which means that it assumes that in general, data will
be already in Normalization Form C [Charmod].

This document recommends that in IETF protocols, Canonical Equivalents
be dealt with by using Early Uniform Normalization according to
Unicode Normalization Form C, Canonical Composition [UTR15]. This
document describes both Early Uniform Normalization (in Section 2)
and Normalization Form C (in Section 3). Section 4 contains an
analysis of (mostly theoretical) potential risks for the stability
of Normalization Form C. For reference, Section 5 discusses various
cases of equivalences not dealt with by Normalization Form C.

Note: The terms 'normalization' (such as in 'Normalization Form C')
    and 'canonicalization' (such as in XML Canonicalization) can
    mean virtually the same thing. In the context of the topics
    described in this document, only 'normalization' is used because
    'canonical' is used to distinguish between canonical equivalents
    and compatibility equivalents.

1.2 Notational Conventions

For UCS codepoints, the notation U+HHHH is used, where HHHH is the
hexadecimal representation of the codepoint. This may be followed by
the official name of the character in all caps.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
specification are to be interpreted as described in [RFC2119].


2. Early Uniform Normalization

This section tries to give some guidance on how Normalization Form C
(NFC), described later in Section 3, should be used by Internet
protocols. Each Internet protocol has to define by itself how to use
NFC, and has to take into account its particular needs. However, the
advice in this section is intended to help writers of specifications
not very familliar with text normalization issues, and to try to make
sure that the various protocols use solutions that interface easily
with each other.

This section uses various well-known Internet protocols as examples.
However, such examples do not imply that the protocol elements
mentioned actually accept non-ASCII characters. Depending on the
protocol element mentioned, that may or may not be the case, and may
change in the future. Also, the examples are not intended to actually
define how a specific protocol deals with text normalization issues.
This is the responsibility of the specification for each specific
protocol.

The basic principle for how to use Normalization Form C is Early
Uniform Normalization. This means that ideally, only text in
Normalization Form C appears on the wire on the Internet. This can be
seen as applying 'be conservative in what you send' to the problem
of text normalization. And (again ideally) it should not be needed
that each implemenation of an Internet protocol separately implements
normalization. Text should just be provided normalized from the
underlying infrastructure, e.g. the operating system or the keyboard
driver.

Early normalization is of particular importance for those parts of
Internet protocols that are used as identifiers. Examples would
be URIs, domain names, email addresses, identifier names in PKIX
certificates, identifiers in ACAP, file names in FTP, folder names in
IMAP, newsgroup names in NNTP, and so on. This is due to the following
reasons:

- In order for the protocol to work, it has to be very well defined
   when two protocol element values match and when not.
- Implementations, in particular on the server side, do not in any
   way have to deal with e.g. display of multilingual text, but on
   the other hand have to handle a lot of protocol-specific issues.
   Such implementations therefore should not be bothered with text
   normalization.

For free text, e.g. the content of mail messages or news postings,
Early Uniform Normalization is somewhat less important, but definitely
improves interoperability.

For protocol elements used as identifiers, this document recommends
Internet protocols to specify the following:

- Comparison SHOULD be carried out purely binary (after it has been
   made sure, where necessary, that the texts to be compared are in
   the same character encoding).
- Any kind of text, and in particular identifier-like protocol
   elements, SHOULD be sent normalized to Normalization Form C.
- In case comparison fails due to a difference in text normalization,
   the originator of the non-normalized text is responsible for the
   failure.
- In case implementors are aware of the fact, or suspect, that their
   underlying infrastructure produces non-normalized text, they SHOULD
   take care to do the necessary tests, and if necessary the actual
   normalization, by themselves.
- In the case of creation of identifiers, and in particular if this
   creation is comparatively infrequent (e.g. newsgroup names, domain
   names), and happens in a rather centralized manner, explicit checks
   for normalization SHOULD be required by the protocol specification.



3. Canonical Composition (Normalization Form C)

This section describes Canonical Composition (Normalization Form C,
NFC). The normative specification of Canonical Composition is found in
[UTR15]. The description is done in a procedural way, but any other
procedure that leads to identical results can be used. The result is
supposed to be exactly identical to that described by [UTR15]. If any
differences should be found, [UTR15] must be followed. For each step,
various notes are provided to help understand the description and give
implementation hints.

Given a sequence of UCS codepoints, its Canonical Composition can
be computed with the following three steps:

1. Decomposition   (Section 3.1)
2. Reordering      (Section 3.2)
3. Recomposition   (Section 3.3)

Additional implementation notes are given in Section 3.4.


3.1 Decomposition

For each UCS codepoint in the input sequence, check whether this
codepoint has a canonical decomposition according to the newest
version of the Unicode Character Database (field 5 in [UniData]).
If such a decomposition is found, replace the codepoint in the
input sequence by the codepoint(s) in the decomposition, and
recursivly check for and apply decomposition on the first replaced
codepoint.

Note: Fields in [UniData] are delimited by ';'. Field 5 in [UniData]
    is the 6th field when counting with an index origin of 1. Fields
    starting with a tag delimited by '<' and '>' indicate compatibility
    decompositions; these compatibility decompositions MUST NOT be used
    for Normalization Form C.

Note: For Korean Hangul, the decompositions are not contained in
    [UniData], but have to be generated algorithmically according to
    the description in [Unicode], Section 3.11.

Note: Some decompositions replace a single codepoint by another
    single codepoint.

Note: It is not necessary to check replaced codepoints other than the
    first one due to the properties of the data in the Unicode
    Character Database.

Note: It is possible to 'precompile' the decompositions to avoid
    having to apply them recursively.


3.2 Reordering

For each adjacent pair of UCS codepoints after decomposition, check
the combining classes of the UCS codepoints according to the newest
version of the Unicode Character Database (Field 3 in [UniData]).
If the combining class of the first codepoint is higher than the
combining class of the second codepoint, and at the same time the
combining class of the second codepoint is not zero, then exchange
the two codepoints. Repeat this process until no two codepoints can
be exchanged anymore.

Note: A combining class greater than zero indicates that a codepoint
    is a combining mark that participates in reordering. A combining
    class of zero indicates that a codepoint is not a combining mark,
    or that it is a combining mark that is not affected by reordering.
    There are no combining classes below zero.

Note: Besides a few script-specific combining classes, combining
    classes mainly distinguish whether a combining mark is attached
    to the base letter or just placed near the base letter, and on
    which side of the base letter (e.g. bottom, above right,...) the
    combining mark is attached/placed. Reordering assures that
    combining marks placed on different sides of the same character
    are placed in a canonical order (because any order would visually
    look the same), while combining marks placed on the same side of
    a character are not reordered (because reordering them would change
    the combination they represent).

Note: After completing this step, the sequence of UCS codepoints
    is in Canonical Decomposition (Normalization Form D).


3.3 Recomposition

This section describes recomposition in a top-down manner, first
describing recomposition processing in general (Section 3.3.1), then
describing which pairs of codepoints can be canonically combined
(Section 3.3.2) and then describing the combination exclusions.

3.3.1 Recomposition Processing

Process the sequence of UCS codepoints resulting from Reordering
from start to end. This process requires a state variable called
'initial'. At the beginning of the process, the value of 'initial'
is empty.

For each codepoint in the sequence resulting from Reordering,
do the following:
- If the following three conditions all apply
   - 'initial' has a value
   - the codepoint immediately preceeding the current codepoint
     is this 'initial' or has a combining class not equal to the
     combining class of the current codepoint
   - the 'initial' can be canonically recombined (see Section 3.3.1)
     with with the current codepoint
   then replace the 'initial' with the canonical recombination and
   remove the current codepoint.
- Otherwise, if the current codepoint has combining class zero,
   store its value in 'initial'.

Note: At the beginning of recomposition, there is no 'initial'.
    An 'initial' is remembered as soon as the first codepoint
    with a combining class of zero is found. Not every codepoint
    with a combining class of zero becomes an 'initial'; the
    exceptions are those that are the second codepoint in
    a recomposition. The 'initial' as used in this description
    is slightly different from the 'starter' as defined in [UTR15],
    but this does not affect the result.

Note: Checking the previous codepoint to have a combining class
    smaller than the combining class of the current codepoint
    (except if the previous codepoint is the 'initial' and therefore
    has a combining class of zero) assures that the conditions used
    for reordering are maintained in the recombination step.

Note: Other algorithms for recomposition have been considered, but
    this algorithm has been choosen because it provides a very good
    balance between computational and implementation complexity
    and 'power' of recombination. As an example, assume a text contains
    a U+0041 LATIN CAPITAL LETTER A with a U+030A COMBINING RING ABOVE
    and a U+031F COMBINING PLUS SIGN BELOW. Because the canonical
    reordering puts the COMBINING PLUS SIGN BELOW before the COMBINING
    RING ABOVE, a more straightforward algorithm would not be able
    to recombine this to U+00C5 LATIN CAPITAL LETTER A WITH RING ABOVE
    followed by U+031F COMBINING PLUS SIGN BELOW.

3.3.2 Pairs of Codepoints that can be Canonically Recombined

A pair of codepoints can be canonically recombined to a third
codepoint if this third codepoint has a canonical decomposition into
the sequence of two codepoints (see [UniData], field 5) and this
canonical decomposition is not excluded from recombination. For Korean
Hangul, the redecompositions are not contained in [UniData], but have
to be generated algorithmically according to the description in
[Unicode], Section 3.11.

3.3.3 Combination Exclusions

The exclusions from recombination are defined as follows:

1) Singletons: Codepoints that have a canonical decomposition into
    a single other codepoint (example: U+212B ANGSTROM SIGN).
2) Non-starter: A codepoint with a decomposition starting with
    a codepoint of a combining class other than zero (example:
    U+0F75 TIBETAN VOWEL SIGN UU).
3) Post-Unicode3.0: A codepoint with a decomposition introduced
    after Unicode 3.0 (no applicable example).
4) Script-specific: Precomposed codepoints that are not the
    generally preferred form for their script (example: U+0959
    DEVANAGARI LETTER KHHA).

The list of codepoints for 1) and 2) can be produced directly from
the Unicode Character Database [UniData]. The list of codepoints for
3) can be produced from a comparison between the 3.0.0 version and
the latest version of [UniData], but this may be difficult. The list
of codepoints for 4) cannot be computed. For 3) and 4), the lists
provided in [CompExcl] MUST be used. [CompExcl] also provides lists
for 1) and 2) for cross-checking. The list for 3) is currently empty
because there are at the moment no post-Unicode3.0 codepoints with
decompositions.

Note: Exclusion of singletons is necessary because in a pair of
    canonically equivalent codepoints, the canonical decomposition
    points from the 'less desirable' codepoint to the preferred
    codepoint. In this case, both canonical decomposition and
    canonical composition have the same preference.

Note: For discussion of the exclusion of Post-Unicode3.0
    codepoints from recombination, please see Section 4
    on versioning issues.


3.4 Implementation Notes

This section contains various notes on potential implementation
issues, improvements, and shortcuts. Further notes on implementation
may be found in [UTR15] or in newer versions of that document.

3.4.1 Avoiding Decomposition, and Checking for Normalization Form C

It is not always necessary to decompose and recompose. In particular,
any sequence that does not contain any of the following is already in
Normalization Form C:

- Codepoints that are excluded from recomposition (see Section 3.3.3)
- Codepoints that appear in second position in a canonical
   recomposition
- Hangul Jamo codepoints (U+1100-U+11F9)
- Unassigned codepoints

If a contiguous part of a sequence satisfies the above criterion all
but the last of the codepoints are already in Normalization Form C.

The above criteria can also be used to easily check that some data
is already in Normalization Form C. However, this check will reject
some cases that actually are normalized.

3.4.2 Unassigned Codepoints

Unassigned codepoints (codepoints that are not assigned in the
current version of Unicode) are listed above to avoid claiming that
something is in Normalization Form C when it may indeed not be, but
they usually will be treated differently from others. The following
behaviours may be possible, depending on the context of normalization:

- Stop the normalization process with a fatal error. (This should be
   done only in very exceptional circumstances. It would mean that the
   implementation will die with data that conforms to a future version
   of Unicode.)
- Produce some warning that such codepoints have been seen, for
   further checking.
- Just copy the unassigned codepoint from the input to the output,
   running the risk of not normalizing completely.
- Checking that the program-internal data is up to date via the
   Internet.
- Distinguish behaviour depending on which range of codepoints
   the unassigned codepoint has been found

3.4.3 Surrogates

When implementing normalization for sequences of UCS codepoints
represented as UTF-16 code units, care has to be taken that pairs of
surrogate code units that represent a single UCS codepoint are treated
appropriately.

3.4.4 Korean Hangul

There are no interactions between normalization of Korean Hangul and
the other normalizations. These two parts of normalization can
therefore be carried out separately, with different implementation
improvements.

3.4.5 Piecewise Application

The various steps such as decomposition, reordering, and
recomposition, can be applied to appropriately choosen parts of a
codepoint sequence. As an example, when normalizing a large file,
normalization can be done on each line separately because line
endings and normalization do not interact.

3.4.6 Integrating Decomposition and Recomposition

It is possible to avoid full decomposition by noting that
decomposition of a codepoint that is not in the exclusion list can be
avoided if it is not followed by a codepoint that can appear in second
position in a canonical recomposition. This condition can be
strengthened by noting that decomposition is not necessary if the
combining class of the following codepoint is higher than the highest
combining class obtained from decomposing the character in question.
In other cases, a decomposition followed immediately by a
recomposition can be precalculated. Further details are left to the
reader.

3.4.7 Decomposition

Recursive application of decomposition can be avoided by a
preprocessing step that calculates a full canonical decomposition
for each character with a canonical decomposition.

3.4.8 Reordering

The reordering step basically is a sorting problem. Because the
number of consecutive combining marks (i.e. consecutive codepoints
with combining class greater than zero) is usually extremely small,
a very simple sorting algorithm can be used, e.g. a straightforward
bubble sort.

Because reordering will occur extremely locally, the following
variant of bubble sort will lead to a fast and simple implementation:

- Start checking the first pair (e.g. the first two codepoints).
- If there is an exchange, and we are not at the start of the
   sequence, move back by one codepoint and check again.
- Otherwise (i.e. if there is no exchange, or we are at the start of
   the sequence) and we are not at the end of the sequence, move
   forward by one codepoint and check again.
- If we are at the end of the sequence, and there has been no exchange
   for the last pair, then we are done.

3.4.9 Conversion from Legacy Encodings

Normalization Form C is designed so that in almost all cases,
one-to-one conversion from legacy encodings (e.g. iso-8859-1,...)
to UCS will produce a result that is already in Normalization Form C.

The one exception to this known at the moment is code page 1252
(charset=windows-1258, for Vietnamese, [windows-1258]). This character
encoding uses a kind of 'half-precomposed' encoding, whereas
Normalization Form C uses full precomposition for the characters
needed for Vietnamese. As an example, U+1EAD LATIN SMALL LETTER A
WITH CIRCUMFLEX AND DOT BELOW is encoded as U+00E2 LATIN SMALL LETTER
A WITH CIRCUMFLEX followed by U+0323 COMBINING DOT BELOW in
code page 1252, but U+1EAD is the normalized form.

3.4.10 Uses of UCS in Non-Normalized Form

One known case where the UCS is used in a way that is not in
Normalization Form C is a group of users using the UCS for Yiddish.
The few combinations of Hebrew base letters and diacritics used to
write Yiddish are available precomposed in UCS (example: U+FB2F
HEBREW LETTER ALEF WITH QAMATS). On the other hand, the many
combinations used in writing the Hebrew language are only available
by using combining characters.

In order to lead to an uniform model of encoding Hebrew, the
precomposed Hebrew codepoints were excluded from recombination. This
means that Yiddish using precomposed codepoints is not in
Normalization Form C.

3.4.11 Interaction with Text Processing

There are many operations on text strings that can create
non-normalized output even if the input was normalized. Examples are
concatenation (if the second string starts with one of the characters
discussed is Section 3.4.1) or case changes (as an example, 1E98
LATIN SMALL LETTER W WITH RING ABOVE does not have a precomposed
capital equivalent).

3.4.12 Implementations and Test Suites

Implementation examples can be found at [Charlint] (Perl), [ICU]
(C/C++) and [Normalizer] (Java).

A huge file with test cases for normalization is avaliable as part of
Unicode 3.0.1 [NormTest].


4. Stability and Versioning

Defining a normalization form for Internet-wide use requires that
this normalization form stays as stable as possible. Stability for
Normalization Form C is mainly achieved by introducing a cutoff
version. For precomposed characters encoded up to and including this
version, in principle the precomposed version is the normal form, but
precompomposed codepoints introduced after the cutoff version are
decomposed in Normalization Form C.

As the cutoff version, version 3.0 of Unicode and the second edition
of ISO/IEC 10646-1 have been choosen. These are aligned codepoint-by-
codepoint. They are both widely and integrally available, i.e. they
do not reqire the application of updates ammendments.

The rest of this section discusses potential threats to the stability
of Normalization Form C, the probability of such threats, and how to
avoid them. [UniPolicy] documents policies adopted by the Unicode
Consortium to limit the impact of changes on existing implementations.

The analysis below shows that the probability of the various threats
is extremely low. The analysis is provided here to document the
awareness of these treats and the measures that have to be taken to
avoid them. This section is only of marginal importance to an
implementer of Normalization Form C or to an author of an Internet
protocol specification.


4.1 New Precomposed Codepoints

The introduction of new (post-Unicode 3.0) precomposed codepoints is
not a threat to the stability of Normalization Form C. Such codepoints
would just provide an alternate way of encoding characters that can
already be encoded without them, by using a decomposed form. The
normalization algorithm already provides for the exclusion of such
characters from recomposition.

While Normalization Form C itself is not affected, such new codepoints
would affect implementations of Normalization Form C, because such
implementations have to be updated to correctly decompose the new
codepoints.

Note: While the new codepoint may be correctly normalized only by
updated implementations, once normalized neither older nor updated
implementations will change anything anymore.

Because the new codepoints do not actually encode any new characters
that could not be encoded before, because the new codepoints would not
actually be used due to Early Uniform Normalization, and because of
the above implementation problems, encoding new precomposed characters
is superfluous and should be very clearly avoided.


4.2 New Combining Marks

It is in theory possible that a new combining mark would be encoded
that is intended to represent decomposable pieces of already existing
encoded characters. In case this indeed would happen, problems for
Normalization Form C can be avoided by making sure the precomposed
character that now has a decomposition is not included in the list
of recoposition exclusions. While this helps for Normalization Form
C, adding a canonical decomposition would affect other normalization
forms, and it is therefore highly unlikely that such a canonical
decomposition will ever be added in the first place.

In case new combining marks are encoded for new scripts, or in case a
combining mark is introduced that does not appear in any precomposed
character yet, then the appropriate normalization for these characters
can easily be defined by providing the appropriate data. However,
hopefully no new encoding ambiguities are introduced for new scripts.


4.3 Changed Codepoints

A major threat to the stability of Normalization Form C would come
from changes to ISO/IEC 10646/Unicode itself, i.e. by moving around
characters or redefining codepoint or by ISO/IEC 10646 and Unicode
evolving differently in the future. These threats are not specific to
Normalization Form C, but relevant for the use of the UCS in general,
and are mentioned here for completeness.

Because of the very wide and increasing use of the UCS thoughout the
world, the amount of resistance to any changes of defined codepoints
or to any divergence between ISO/IEC 10646 and Unicode is extremely
strong. Awareness about the need for stability in this point, as well
as others, is particularly high due to the experiences with some
changes in the early history of these standards, in particular with
the reencoding of some Korean Hangul characters in ISO/IEC 10646
amendment 5 (and the corresponding change in Unicode). For the IETF
in particular, the wording in [RFC 2279] and [RFC 2781] stresses the
importance of stability in this respect.


5. Cases not dealt with by Canonical Equivalence

This section gives a list of cases that are not dealt with by
Canonical Equivalence and Normalization Form C. This is done to help
the reader understand Normalization Form C and its limits. The list in
this section contains many cases of widely varying nature. In many
cases, a viewer, if familiar with the script in question, will be able
to distinguish the various variants.

Internet protocols can deal in various ways with the cases below. One
way is to limit the characters e.g. allowed in an identifier so that
all but one of the variants are disallowed. Another way is to assume
that the user can make the distinction him/herself. Another is to
understand that some characters or combinations of characters that
would lead to confusion are very difficult to actually enter on any
keyboard; it may therefore not really be worth to exclude them
explicitly.

    - Various ligatures (Latin, Arabic, e.g. U+FB01 LATIN SMALL LIGATURE FI
      vs. U+0066 LATIN SMALL LETTER F followed by U+0069 LATIN SMALL LETTER I)

    - Croatian digraphs (e.g. U+01C8 LATIN CAPITAL LETTER L WITH SMALL LETTER J
      vs. U+004C LATIN CAPITAL LETTER L followed by U+006A LATIN SMALL 
LETTER J)

    - Full-width Latin compatibility variants (e.g. U+FF21 FULLWIDTH LATIN
      CAPITAL LETTER A vs. U+0041 LATIN CAPITAL LETTER A)

    - Half-width Kana and Hangul compatibility variants (e.g. U+FF76 HALFWIDTH
      KATAKANA LETTER KA vs. U+30AB KATAKANA LETTER KA)

    - Vertical compatibility variants (U+FE35 PRESENTATION FORM FOR VERTICAL
      LEFT PARENTHESIS vs. U+0028 LEFT PARENTHESIS)

    - Superscript/subscript variants (numbers and IPA, e.g. U+00B2 SUPERSCRIPT
      TWO)

    - Small form compatibility variants (e.g. U+FE6A SMALL PERCENT SIGN)

    - Enclosed/encircled alphanumerics, Kana, Hangul,... (e.g. U+2460 CIRCLED
      DIGIT ONE)

    - Letterlike symbols, Roman numerals,... (e.g. U+210E PLANCK CONSTANT vs.
      U+0068 LATIN SMALL LETTER H)

    - Squared Katakana and Latin abbreviations (units,..., e.g. U+334C SQUARE
      MEGATON)

    - Hangul jamo representation alternatives for historical Hangul

    - Presence or absence of joiner/non-joiner and other control
      characters

    - Upper case/lower case distinction

    - Distinction between Katakana and Hiragana

    - Similar letters from different scripts
      (e.g. "A" in Latin, Greek, and Cyrillic)

    - CJK ideograph variants (glyph variants introduced due to the
      source separation rule, simplifications)

    - Various punctuation variants (apostrophes, middle dots,
      spaces,...)

    - Ignorable whitespace, hyphens,...

    - Ignorable accents,...

Many of the cases above are identified as compatibility equivalences
in the Unicode database. [UTR15] defines Normalization Forms KC and
KD to normalize compatibility equivalences. It may look attractive
to just use Normalization Form KC instead of Normalization Form C for
Internet protocols. However, while Canonical Equivalence, which forms
the base of Normalization Form C, deals with a very small number of
very well defined cases of complete equivalence (from an user point
of view), Compatibility Equivalence comprises a very wide range of
cases that usually have to be examined one at a time. If the domain
of acceptable characters is suitably limited, such as for program
identifiers, then NFKC may be a suitable normalization form.



6. Security Considerations

Security problems can result from:
- Improper implementations of normalization. For example, in
   certificate chaining, if the program validating a certificate chain
   mis-implements normalization rules, an attacker might be able to
   spoof an identity by picking a name that the validator thinks is
   equivalent to another name.
- The fact that normalization maps several input sequences to the same
   output sequence. If a digital signature calculation includes
   normalization, this can make it slightly easier to find a fake
   document that has the same digest as a real one.
- The use of normalization only in part of the applications. In
   particular, if software used for security purposes, e.g. to create
   and check digital signatures, normalizes data, but the applications
   actually using the data do not normalize, it can be very easy to
   create a fake document that can claim to be the real one but
   produces different behaviour.
- Different behavior in programs that do not respect canonical
   equivalence.

Security-related applications therefore MAY check for normalized
input, but MUST NOT actually apply normalization unless is can be
guaranteed that all related applications also apply normalization.



Acknowledgements


The earliest version of this Internet Draft, which dealt with quite
similar issues, was entitled "Normalization of Internationalized
Identifiers" and was submitted in July 1997 by the first author while
he was at the University of Zurich. It benefited from ideas, advice,
criticism and help from: Mark Davis, Larry Masenter, Michael Kung,
Edward Cherlin, Alain LaBonte, Francois Yergeau, and others.

For the current version, the authors were encouraged in particular by
Patrick Faltstrom and Paul Hoffman. The discussion of potential
stability threats is based on contributions by John Cowan and Kenneth
Whistler. Some security threats were pointed out by Masahiro
Sekiguchi. Further contributions are due to Dan Oscarson.



References

    [Charlint]     Martin Duerst. Charlint - A Character Normalization
                   Tool. <http://www.w3.org/International/charlint>.

    [Charreq]      Martin J. Duerst, Ed. Requirements for String
                   Identity Matching and String Indexing. World Wide
                   Web Consortium Working Draft.
                   <http://www.w3.org/TR/WD-charreq>.

    [Charmod]      Martin J. Duerst and Francois Yergeau, Eds.
                   Character Model for the World Wide Web. World Wide
                   Web Consortium Working Draft.
                   <http://www.w3.org/TR/charmod>.

    [CompExcl]     The Unicode Consortium. Composition Exclusions.
       <ftp://ftp.unicode.org/Public/UNIDATA/CompositionExclusions.txt>

    [ICU]          International Components for Unicode.
                   <http://oss.software.ibm.com/icu/>.

    [ISO10646]     ISO/IEC 10646-1:2000. International standard --
                   Information technology -- Universal multiple-octet
                   coded character Set (UCS) -- Part 1: Architecture
                   and basic multilingual plane, and its Amendments.

    [Normalizer]   The Unicode Consortium. Normalization Demo.
          <http://www.unicode.org/unicode/reports/tr15/Normalizer.html>

    [NormTest]     Mark Davis. Unicode Normalization Test Suite.
          <http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt>

    [RFC2119]      Scott Bradner. Key words for use in RFCs to
                   Indicate Requirement Levels, March 1997.
                   <http://www.ietf.org/rfc/rfc2119.txt>

    [RFC 2277]     Harald Alvestrand, IETF Policy on Character Sets and
                   Languages, January 1998.
                   <http://www.ietf.org/rfc/rfc2781.txt>

    [RFC 2279]     Francois Yergeau. UTF-8, a transformation format of
                   ISO 10646. <http://www.ietf.org/rfc/rfc2781.txt>

    [RFC 2781]     Paul Hoffman and Francois Yergeau. UTF-16, an
                   encoding of ISO 10646.
                   <http://www.ietf.org/rfc/rfc2781.txt>

    [Unicode]      The Unicode Consortium. The Unicode Standard,
                   Version 3.0. Reading, MA, Addison-Wesley Developers
                   Press, 2000. ISBN 0-201-61633-5.

    [UniData]      The Unicode Consortium. UnicodeData File.
                 <ftp://ftp.unicode.org/Public/UNIDATA/UnicodeData.txt>
                   For explanation on the content of this file, please
                   see
                <ftp://ftp.unicode.org/Public/UNIDATA/UnicodeData.html>

    [UniPolicy]    The Unicode Consortium. Unicode Consortium Policies.
                   <http://www.unicode.org/unicode/standard/policies>

    [UTR15]        Mark Davis and Martin Duerst. Unicode Normalization
                   Forms. Unicode Technical Report #15, Version 18.0.
            <http://www.unicode.org/unicode/reports/tr15/tr15-18.html>,
                   also on the CD of [Unicode].

    [windows-1258] Microsoft Windows Codepage: 1258 (Viet Nam).
            http://www.microsoft.com/globaldev/reference/sbcs/1258.htm>


Copyright

Copyright (C) The Internet Society, 2000. All Rights Reserved.

This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph
are included on all such copies and derivative works.  However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other
than English.

The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE."



Author's Addresses

         Martin J. Duerst
         W3C/Keio University
         5322 Endo, Fujisawa
         252-8520 Japan
         mailto:duerst@w3.org
         http://www.w3.org/People/D%C3%BCrst/
         Tel/Fax: +81 466 49 1170

         Note: Please write "Duerst" with u-umlaut wherever
               possible, e.g. as "D&252;rst" in HTML and XML.

         Mark E. Davis
         IBM Center for Java Technology
         10275 North De Anza Bouleward
         Cupertino 95014 CA
         U.S.A.
         mailto:mark.davis@us.ibm.com
         http://www.macchiato.com
         Tel: +1 (408) 777-5850
         Fax: +1 (408) 777-5891


