Subject: RE: Last Call: draft-klensin-net-utf8 (Unicode Format for Network Interchange) to Proposed Standard

From: "Karlsson, Kent" <[email-masked]>
Date: Mon, 14 Jan 2008 11:13:32 +0100

John C Klensin wrote:

--Frank Ellermann wrote:
> >...
> > Hopefully somebody can confirm that IND is correct, or not.
> > For HT and FF I hope the final version will somehow express
> > that both are not really bad, and as far as they're bad FF is
> > worse than HT. 

See http://www.itscj.ipsj.or.jp/ISO-IR/077.pdf, which, somewhat
to my surprise, says that IND is an LF clone. However, IND has
long been deprecated, and never got any noticable use, and is even
REMOVED from ECMA 48. So I think it is safe to ignore IND. Indeed,
I would prefer it not be mentioned in the document we're discussing.

(I would like to say the same about NEL, but NEL is alive
and the native line separator/terminator in EBCDIC bases systems,
and may escape as NEL rather than be converted to something else.)

> I'm open to consensus about changes for either HT or FF, but the
> theory of "bad" that was used to construct the spec was:
> 
> (i) If a "spacing" control has the effect of setting the
> position of the next character, it is "bad" unless that position
> is unambiguous.   In addition, things are "bad" unless they are
> necessary in running text (as distinct from faking things that
> are better handled in markup, followed by either device-specific
> output or standard page representations, neither of which are
> normal text).

There is also another issue. If HT is converted to (presumably)
a sequence of SP, you will mess up bidi text. (See one of the
other mails I send at about the same time as this one.)

> It is unambiguous for SP.  It is unambiguous for CRLF.
> Independent of the "what is a line-end" problem, it is somewhat
> ambiguous for CR or LF alone and for IND.  It is ambiguous for

Even though IND was, for some strange reason, defined as an LF
clone, it has long been deprecated, and AFAIK never saw any
popular use. I think it is best left forgotten and left in silence.
Note also that it is not only deprecated, but even REMOVED from
ISO/IEC 6429 (ECMA 48).

> HT.  It would be ambiguous for FF except that FF is assigned
> fairly clear semantics in NVT -- "FF" is not a line ending

Of course it is line ending. So is "raw" LF. That the new line
(under some circumstances) may be strangely indented is irrelevant.

> (CRLF FF is needed)

That is a combination I haven't heard of before and I DON'T
think it should be regarded as one NLF. There are TWO NLFs there,
CRLF and then FF.

> and as Bob Braden noted, there is a fairly clear
> rule that FF is to be interpreted as "top of next page" if one

Sure. But the line before it is also ended (no matter where the
top of next page line begins).

> knows what a page is and as "blank line" otherwise.  But that
> rule is sufficiently often ignored to call for considerable
> caution about FF, and the text now contains a cautionary note
> for that reason.

I agree that there should be caution, but not in the shape and
form it has in the draft we are discussing.

> There is an interesting demonstration of the law of unintended
> consequences here.  If we could tell that a string was
> unambiguously UTF-8 (or whatever) by looking at it, even if it
> contains nothing but ASCII characters, then there would be no
> reason to try to make net-utf8 a proper superset of NVT.  If we

I don't see why you really need to carry on the (unworkable in
a more general setting than ASCII, in particular it is unworkable
for the UCS) idea of using carriage return and BS for strange
overstriking. Even for ASCII, the ONLY aspect of that that worked
moderately well was using <BS, _> (or similar) for underlining.
But note also that underlining can be achieved also in the UCS
(without using kludges line <BS, _> for that) without the use
of a higher level protocol by instead using U+0332, COMBINING LOW
LINE. Though using a higher level protocol for getting underlining
is preferable (consider searching),  COMBINING LOW LINE would
still be much preferable over <BS, _> (or similar).

> could do that, we could also do away with the entire "next line"
> debate by prohibiting even CRLF and requiring the use of LS

LS would be a bad idea. See my other email (sent at approx. the
same time as this one). You would get (to you) unexpected effects
from bidi processing.

		/Kent Karlsson

> (U+2028).  In retrospect, there might have been considerable
> advantages to forcing the ASCII- UTF-8 distinction by requiring
> that UTF-8 strings all start with a BOM, but it is far too late
> for that (and probably not, on balance, a good idea despite its
> advantages).  So I don't see how to get there from here -- we
> are stuck, for historical reasons, with CRLF on the wire as what
> The Unicode Standard calls NLF (incidentally, Unicode 5.0,
> Section 5.8, provides significant insight into the complexity of
> this problem and probably should have been referenced.  It would
> be even more helpful had Table 5-2 included identifying CRLF as
> a standard Internet "wire" form of NLF, not just binding that
> form to Windows.
_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "Karlsson, Kent" <[email-masked]>
Date: Mon, 14 Jan 2008 11:13:32 +0100

John C Klensin wrote:

		/Kent Karlsson

---

From: "Karlsson, Kent" <[email-masked]>
Date: Mon, 14 Jan 2008 11:08:37 +0100

Frank Ellermann wrote:

> John C Klensin wrote:
> 
> > It is ambiguous for HT.
> 
> Yes, but we typically don't care about this in protocols as
> long as it behaves like one or more spaces.  I think that's

Well, they don't exactly behave like a sequence of spaces.
(See below.)

> the idea of "WSP = SP / HTAB ; white space" in RFC 4234bis,
> waiting for its STD number.
> 
> We talked about the 4234bis issue of "trailing white space",
> which could cause havoc when it is silently removed, and a
> "really empty line" is not the same as an "apparently empty
> line" (i.e. CRLF CRLF vs. CRLF 1*WSP CRLF).
> 
> A similar robustness principle would support to accept old
> "HTAB-compression" or "HTAB-beautification" (e.g. as first

Doing what you call 'old "HTAB-compression"' is a bad idea,
for several reasons (that I don't detail here, but for one:
see below).

> character in a folded line).  In other words WSP, not only
> SP.  It is clear that the outcome is ambiguous, but in some
> protocols I care about (headers in MIME, mail, and news)
> *WSP or 1*WSP are acceptable.   Admittedly it is a pain when
> signatures need white space canonicalization.  But replacing
> *WSP by *SP would only simplify this step, not get rid of it.
> 
>  [About CRLF]
> > Unicode 5.0, Section 5.8, provides significant insight into
> > the complexity of this problem and probably should have
> > been referenced.  It would be even more helpful had Table
> > 5-2 included identifying CRLF as a standard Internet "wire"
> > form of NLF, not just binding that form to Windows.
> 
> Indeed, this chapter offers significantly *broken* insight
> for our purposes.  What they found was a horrible mess, then

Hmm, a minor mess, but not that horrible.

> they introduced wannabe-unambiguous LS + PS, and what they
> arrived at was messier than before.  Claiming that CRLF is 

They were introduced in Unicode 1.1, long before the text for
section 5.8 was drafted (originally as UTS 10).

One important point that you have missed is that LS and PS,
and the difference between THEM, are essential to the bidi
algorithm. What is or may be done with other NLFs is basically
a hack (most NLF are treated as if they were PS).

Note also that two LSes in sequence don't make a PS...

Assume for the moment that we were using only LS (as John
has suggested as a possible ideal). This would imply that
the bidi algorithm would consider the ENTIRE document,
however many thousands of pages it may span, as a SINGLE
paragraph. Thus, if there is any bidi processing, none of
the text can be displayed until the entire text has been
read in and bidi levelled as a whole, etc. That may have
some display effects (bidi controls, LRE, RLE, LRO, RLO,
span at most to the end of a paragraph, so if paragraph
ends are replaced by LS, the bidi controls may span more
text than they did originally). The hack is to regard all
NLFs except LS, VT, and FF as PS. Since CRLF (say) may occur
inside of what is actually a paragraph, this has some display
effects (limiting bidi controls range more than they were
originally), but at least bidi processing can be done piece
by piece of the text.

Bidi control codes are not talked about in the document
we are discussing...

> "windows" is odd for DOS + OS/2 users, it is also at odds
> with numerous Internet standards - precisely the reason why
> we need your draft.  
> 
> The chapter talks about line and paragraph separators without
> mentioning relevant ASCII controls such as RS.  On the other

RS (and GS and FS) are regarded the same as PS for bidi
processing, even though they are not mentioned in section 5.8.
But I would agree that using RS, GS, FS, or (even worse) IND
would be aberrant.

US is regarded as similar to a HT for bidi processing (so should
HTJ, but isn't by default). Note that HT is NOT treated the same
as a sequence of spaces for bidi processing. HT always has the
paragraph bidi level, which is not necessarily the case for spaces.
This DOES affect display, in that HT always "moves" according to
the paragraph level, while spaces may (and often do) move opposite
to the paragraph level. So **DON'T** imply that HT should be 
replaced by spaces; such a replacement WILL have ill display
effects.

However, I do think that these four characters should not be
used.

> hand it mentions MS Word interna which are nobody's business
> outside of MS Word.

I guess you refer to VT (LINE TABULATION). The real reason it
is mentioned is that the C and C++ standards give a special
escape for it (\v, which according to C implies a return to
beginning of line). If it were not for that, I would agree
that VT is not very interesting (though it does provide for
a hack to distinguish line separation from paragraph separation
by ignoring the "tabulation" aspect of VT, also for pure 8-bit
character encodings).

	/Kent Karlsson

> It is interesting, but IMO unusable for net-utf8.