Subject: Comments on draft-hoffman-utf16-04

From: Juliusz Chroboczek <[email-masked]>
Date: 13 Aug 1999 19:56:28 +0100

Sir,

I would like to object to the approach proposed in
draft-hoffman-utf16-04.

Hoffmans's draft proposes three different `character-set' labels for
UTF-16 text -- `UTF-16', `UTF-16BE', and `UTF-16LE'.  One of those --
`UTF-16' -- actually represents two different formats, distinguished
by an initial Byte Order Mark (BOM).

All of this is done in order to allow applications to send data in
their native endianness, and thus gain a small measure of performance.
However, byte swapping is a trivial operation, which can be done in
one machine instruction on most CISC architectures (either exchange or
rotate), and in at most five instructions on RISC architectures (move,
shift, shift, or, and).  Thus, the gain in performance is negligible.

The introduction of the initial Byte Order Mark (BOM) creates numerous
oportunities for corrupting text.  While Hoffman seems to adequately
cover most of those, the mistakes involved are dangerously easy to
make, even in the presence of a precise standard.  Typical mistakes
include:

* omitting to take the BOM into account on read;
* omitting to discard the BOM after decoding;
* omitting to write a BOM;
* omitting to discard the BOM when concatenating data (files with a
  BOM cannot just be concatenated, the BOM of the second file needs to
  be deleted);
* taking data without a BOM as being in local order (which might be
  little-endian) rather than in Network Order.

I would like to point out that most of those have been recorded as
being made by existing software.

While Hoffman's draft encourages applications to use `UTF-16LE' and
`UTF16-BE' (which do not require the usage of a BOM) in preference to
`UTF-16', this recommendation is not likely to be followed, as
implementors are likely to just send data without trying to interface
with the MIME tagging mechanisms of their applications.  Furhermore,
even if this recommendation were followed, we would still be left with
two different formats for UTF-16 data.

The alternative, of course, is to define a single format for UTF-16
which mandates the data to be in Network Order and not to contain an
explicit BOM.  The performance cost of this approach would be
negligible, while greatly simplifying the interaction of applications.

In the light of the above, I believe that there is no justification
for introducing 4 distinct formats for UTF-16 data as Hoffman does.

Sincerely,

                                        Juliusz Chroboczek