Subject: RE: Last Call: draft-klensin-net-utf8 (Unicode Format for Network Interchange) to Proposed Standard

From: "Kent Karlsson" <[email-masked]>
Date: Mon, 7 Jan 2008 22:30:51 +0100

Comment on draft-klensin-net-utf8-07.txt:

--------------------------

"Network Virtual Terminal (NVT)" occurs first in Appendix A.
The explanation of the abbreviation should (also) be given at
the first occurence of "NVT" in the document.

--------------------------

Section 2, point 2, "Line-endings..."

       "discussion.  The newer control characters IND (U+0084) and NEL
       ("Next Line", U+0085) might have been used to disambiguate the"

I have a hard time figuring out what IND was supposed to be used for,
but I don't think it was for line endings. Chain printer "font" change is
the closest I get... (http://www.freepatentsonline.com/3699884.html).

NEL is used in EBCDIC originally (IIUC), and still used in EBCDIC...

The description "might have been used to disambiguate" is more
appropriate for U+2028 and U+2029.

--------------------------

       "it, lines end in CRLF and only in CRLF.  Anything that does not
       end in CRLF is either not a line or is severely malformed."

The sentence starting with "Anything" seems  severely malformed...
You don't really meant to say "Anything", I hope. "Using other line
ending or line separation conventions" perhaps. And "severely
malformed", I hope you did not mean that either. "is lacking in
conversion to 'net-utf8'/'net-Unicode'" perhaps.

To be "rescrictive in what one emits and permissive/liberal in
what one receives" might be applicable here.

Upon reciept, the following SHOULD be seen as at least line ending
(or line separating), and in some cases more than that: 

LF, CR+LF, VT, CR+VT, FF, CR+FF, CR (not followed by NUL...),
NEL, CR+NEL, LS, PS
where
LF	U+000A
VT	U+000B
FF	U+000C
CR	U+000D
NEL	U+0085
LS	U+2028
PS	U+2029

even FS, GS, RS
where
FS	U+001C
GS	U+001D
RS	U+001E
should be seen as line separating (Unicode specifies these as havingFrom [email-masked] Wed Jan 09 10:09:16 2008
From: "Kent Karlsson" <[email-masked]>
To: <[email-masked]>
Date: Mon, 7 Jan 2008 22:30:51 +0100
Cc: [email-masked]
Subject: RE: Last Call: draft-klensin-net-utf8 (Unicode Format for Network
	Interchange) to Proposed Standard

Comment on draft-klensin-net-utf8-07.txt:

--------------------------

"Network Virtual Terminal (NVT)" occurs first in Appendix A.
The explanation of the abbreviation should (also) be given at
the first occurence of "NVT" in the document.

--------------------------

Section 2, point 2, "Line-endings..."

       "discussion.  The newer control characters IND (U+0084) and NEL
       ("Next Line", U+0085) might have been used to disambiguate the"

I have a hard time figuring out what IND was supposed to be used for,
but I don't think it was for line endings. Chain printer "font" change is
the closest I get... (http://www.freepatentsonline.com/3699884.html).

NEL is used in EBCDIC originally (IIUC), and still used in EBCDIC...

The description "might have been used to disambiguate" is more
appropriate for U+2028 and U+2029.

--------------------------

       "it, lines end in CRLF and only in CRLF.  Anything that does not
       end in CRLF is either not a line or is severely malformed."

The sentence starting with "Anything" seems  severely malformed...
You don't really meant to say "Anything", I hope. "Using other line
ending or line separation conventions" perhaps. And "severely
malformed", I hope you did not mean that either. "is lacking in
conversion to 'net-utf8'/'net-Unicode'" perhaps.

To be "rescrictive in what one emits and permissive/liberal in
what one receives" might be applicable here.

Upon reciept, the following SHOULD be seen as at least line ending
(or line separating), and in some cases more than that: 

LF, CR+LF, VT, CR+VT, FF, CR+FF, CR (not followed by NUL...),
NEL, CR+NEL, LS, PS
where
LF	U+000A
VT	U+000B
FF	U+000C
CR	U+000D
NEL	U+0085
LS	U+2028
PS	U+2029

even FS, GS, RS
where
FS	U+001C
GS	U+001D
RS	U+001E
should be seen as line separating (Unicode specifies these as having bidi
property B, which effectively means they are paragraph separating).

Apart from CR+LF, these SHOULD NOT be emitted for net-utf8, unless
that is overriden by the protocol specification (like allowing FF, or CR+FF).
When faced with any of these in input **to be emitted as net-utf8**, each
of these SHOULD be converted to a CR+LF (unless that is overridden
by the protocol in question).

--------------------------

Section 2, point 3:

You have made an exception for FF (because they occur in RFCs?).
I think FF SHOULD be avoided, just like VT, NEL, and more (see above).
Even when it is allowed, it, and CR+FF, should be seen as line separating.

You have also (by implication) dismissed HT, U+0009. The reason for this in
unclear. Especially since HT is so common in plain texts (often with some
default tab setting). Mapping HT to SPs is often a bad idea. I don't think a
default tab setting should be specified, but the effect of somewhat (not wildly)
different defaults for that is not much worse than using variable width fonts.

SP, U+0020, is nowadays not seen as a control character, not even in
your own text... (same paragraph).

--------------------------

   "However, because they were optional in NVT applications
   and this specification is an NVT superset, they cannot be prohibited
   entirely." 

Why not? Why must this be a strict NVT superset? I think it would be rather
important to rule these strange beasts out from net-utf8. These were really
ASCII (ISO 646) features, but have been ruled out much before Unicode.

--------------------------

   "The most important of these rules is that CR MUST NOT
   appear unless it is immediately followed by LF (indicating end of
   line) or NUL."

I don't see how that follows (read: that does not follow).

--------------------------

     "[ISO10646]
              International Organization for Standardization,
              "Information Technology - Universal Multiple- Octet Coded
              Character Set (UCS) - Part 1: Architecture and Basic
              Multilingual Plane"", ISO/IEC 10646-1:2000, October 2000."

That seems a bit old... Better with the current revision:

ISO/IEC 10646:2003   Information technology -- Universal Multiple-Octet Coded
Character Set (UCS)

with the amendments (which I don't think you should reference explicitly):
ISO/IEC 10646:2003/Amd 1:2005  Glagolitic, Coptic, Georgian and other characters
ISO/IEC 10646:2003/Amd 2:2006  N'Ko, Phags-pa, Phoenician and other characters
(and more amendments in the works).

--------------------------

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 10 Jan 2008 09:17:16 +0100

On Mon, Jan 07, 2008 at 10:30:51PM +0100,
 Kent Karlsson <[email-masked]> wrote 

> LF, CR+LF, VT, CR+VT, FF, CR+FF, CR (not followed by NUL...),

The whole point of the Internet-Draft on Net-UTF8 is to limit the size
of the zoo of line endings. Accepting "everything in Unicode which
looks like a line ending" seems strange to me. Do you know any
Internet *protocol* which accepts several line endings? (Some Internet
*applications* do so, in the name of the robustness principle, but for
a protocol, I think it is a really bad idea.)

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 10 Jan 2008 11:33:11 +0100

On Mon, Jan 07, 2008 at 03:33:52PM -0500,
 The IESG <[email-masked]> wrote 
 a message of 24 lines which said:

> - 'Unicode Format for Network Interchange '
> <draft-klensin-net-utf8-07.txt> as a Proposed Standard

This is a long-awaited standard, addressing one of the few missing
pieces in the internationalisation of the Internet protocols. 

As mentioned in the appendix C, some protocols would benefit from this
new standard.

I've read and studied the -07 draft and my opinion is that it is good
for publication as a PS.

Some choices are - for me - mostly arbitrary (such as using CR+LR for
line endings instead of LS) but the important thing is that there is
*one* choice.

The delicate balance between what we specify (using MUST or SHOULD)
and the recognition that people will do strange things and that
implementations should be prepared, is, IMHO, right.

I appreciate also the historical materials in appendices A and B and I
wish more RFCs have such background information.

---

From: "Kent Karlsson" <[email-masked]>
Date: Thu, 10 Jan 2008 09:59:43 +0100

> a protocol, I think it is a really bad idea.)

Please reread my comment. I wrote:
| Apart from CR+LF, these SHOULD NOT be emitted for net-utf8, unless
| that is overriden by the protocol specification (like allowing FF, or CR+FF).
| When faced with any of these in input **to be emitted as net-utf8**, each
| of these SHOULD be converted to a CR+LF (unless that is overridden
| by the protocol in question).

I.e. this is about conversion/normalisation of input that is *TO BE*
sent as Net-UTF-8.

As for the recieving side the same considerations as for the (SHOULD)
requirement (point numbered 4 on page 4) for NFC in Net-UTF-8 applies.
The reciever cannot be sure that NFC has been applied. Nor can it be
sure that conversion of all line endings to CR+LF (there-by loosing
information about their differences) has been applied.

	/kent k

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 10 Jan 2008 15:21:09 +0100

> pieces in the internationalisation of the Internet protocols. 

+1

> I appreciate also the historical materials in appendices A and B
> and I wish more RFCs have such background information.

+2

Hopefully somebody can confirm that IND is correct, or not.  For
HT and FF I hope the final version will somehow express that both
are not really bad, and as far as they're bad FF is worse than HT. 

My impression from reading the draft was exactly the opposite, FF
not too bad, HT really bad, that's odd for protocols allowing WSP.

 Frank

---

From: John C Klensin <[email-masked]>
Date: Thu, 10 Jan 2008 11:58:27 -0500

Kent,

I will try to address the comments that are essentially
editorial after the Last Call closes, but you have raised a few
points that have been discussed over and over again (not just in
the net-utf8 context) and that I think are worth my responding
to now (in addition to comments already made by others).  FWIW,
I'm working on several things right now that are of much higher
priority to me than net-utf8, so, especially since I think that
authors should generally be quiet during Last Call and let other
responses accumulate, are likely to cause comments from me to
come very slowly.

> what one receives" might be applicable here.

What we have consistently found out about the robustness
principle is that it is extremely useful if used as a tool for
interpreting protocol requirements. Otherwise it is useful only
in moderation.  When we have explicit requirements that
receivers accept garbage, we have repeatedly seen the
combination of those requirements with the robustness principle
used by senders to say "we can send any sort of garbage and you
are required to clean it up".   That does not promote either
interoperability or a smoothly-running network.

The question of why net-utf8 expect receivers to clean up
normalization but does not expect them to tolerate aberrant
line-endings is a reasonable one, for which see below.  But I
think your invocation of the robustness principle is
inappropriate.

> Upon receipt, the following SHOULD be seen as at least line
> ending (or line separating), and in some cases more than that: 
> 
> should be seen as line separating (Unicode specifies these as
> having bidi property B, which effectively means they are
> paragraph separating).

There are two theories about how to construct a protocol.  One
puts all responsibility on the sender; the other puts all
responsibility on the receiver.  A different manifestation of
this is the difference between protocols with very few options
which are therefore supported in the same one by everyone and
protocols with many options that, in practice, are selectively
supported and for which finding an interoperable client-server
pair can be difficult. The "many options" and "receiver
responsibility" approaches tend, if nothing else, tend to create
N**2 problems, where every receiver has to understand every
possible sender format and option and combination of them,
rather than dealing with a single form to which everything is
converted on sending and, if necessary, de-converted on receipt.

While one often needs to seek a balance in practice, I and many
others believe that the Internet has been better served by the
first option in each of these pairings, resulting in a minimum
of variations on the wire.  People such as Marshall Rose and
Mike Padlipsky have been much more elegant on this subject than
I can be, but the Internet's history of "just one form on the
wire" for text types at the applications layer goes back to at
least RFC 20 (which is one reason it was cited in the current
draft).

There is also a security issue associated with this.  When there
is a single standard form, we know how to construct digital
signatures over the text.   When there are lots of things that
are to be "treated as" the same, and suggestions that various
systems along the line might appropriately convert one form into
another, methods for computing digital signatures need their own
canonicalization rules and rules about exactly when they are
applied.  That can be done, but, as I have suggested in several
other places in this note, why go out of our way to make our
lives more complex?

> Apart from CR+LF, these SHOULD NOT be emitted for net-utf8,
> unless that is overriden by the protocol specification (like
> allowing FF, or CR+FF). When faced with any of these in input
> **to be emitted as net-utf8**, each of these SHOULD be
> converted to a CR+LF (unless that is overridden by the
> protocol in question).

While I may not have succeeded (and, if I didn't, specific
suggestions would be welcome), the net-utf8 draft was intended
to be very specific that it didn't apply to protocols that
didn't reference it, nor was its use mandatory for new
protocols.  That means that a protocol doesn't need to
"override" it; it should just not reference it.  Yes, I think it
makes a new protocol harder to write if it doesn't reference
net-utf8 than if it does.  It may also generate "do you really
need to do this" pushback against such protocols and against
protocols that try "all of this _except_" references to this
document.  That is, IMO, as it should be.  But, if other forms
can be justified for particular applications, then they should
be.   

It just makes no sense, at least to me, to include a lot of text
in a spec whose purpose is to tell applications that don't use
the spec what to do.

We made an exception for FF --while cautioning against its use--
because it is permitted in NVT and fairly widely in text streams
and because some reasonable interpretation of its semantics are
moderately well-understood.   On the other hand, it comes with
some cautions and, if there were consensus to remove the
exception, I wouldn't personally hesitate to do that.

Part of the issue here is separate from the "don't mess with the
other control characters" principle and the issue of
line-endings.  When ASCII came along, people were fairly
optimistic about using plain text streams to specify formatting
and page layout, partially because our expectations for those
things were very low (one available font with no variations,
always fixed-width and fixed-size).  We figured out a rather
long time ago that we needed markup  and printer-specific
controls to handle these things well (I believe the first
instance in code may have been Jerry Saltzer's original RUNOFF
in the first half of the 60s, but it has been a long time in any
event).  To the extent to which the output of those programs
requires "control characters", rather than, e.g., the "terminal
control" functions specified in ISO/IEC 6549 (more or less ANSI
X3.64 or ECMA48).   Today, we tend to use XML or other forms of
markup on the input side and are normally pretty clear that the
print-form output isn't a normal text stream of the type at
which net-utf8 is targeted.

> I think FF SHOULD be avoided, just like VT, NEL, and

I think the cautions about use of FF are just about that strong,
but it does have significant current use (albeit not an Internet
text-stream line separator).   One could put HT with it and
explain that it should be used only when its interpretation (as
a fixed number of spaces or a jump to a well-establish column)
is known, but, because those are rarely known, I/we made another
choice.

> should be seen as line separating.

That has never been permitted in the protocols that reference,
even implicitly, NVT.  Why make things more complicated now by
(i) introducing more flexibility for its own sake and putting
more burden on receivers and (ii) giving bodies of text that use
FF a different interpretation under this specification than they
have under NVT.  Deliberately introducing incompatibilities, it
seems to me, requires much more justification than  added
flexibility.

> You have also (by implication) dismissed HT, U+0009. The
> reason for this in unclear. Especially since HT is so common
> in plain texts (often with some default tab setting). Mapping
> HT to SPs is often a bad idea. I don't think a default tab
> setting should be specified, but the effect of somewhat (not
> wildly) different defaults for that is not much worse than
> using variable width fonts.

But you have just summarized the reasons for avoiding HT.  We
don't have any standard that would give it unambiguous
semantics.  There is no way to incorporate tab column settings
(in characters or millimeters) in a text stream, so one can't
even disambiguate with an in-band option.  That makes HT
appropriate in marked-up text (which might or might not have
better ways to specify what is wanted) or when options are being
transmitted out of band, but not in running text streams.   If
there is consensus that this needs to be addressed more
explicitly in the document, we can try to do so.

--On Wednesday, 09 January, 2008 23:30 +0100 Kent Karlsson
<[email-masked]> wrote:

> B.t.w. many programs (long ago) had a bug that deleted the
> last line if if was not ended with a LF.

Not that long ago.  I discovered, at a recent IETF meeting, a
printer and printer driver that would drop the entire last page
of a document, or drop the document entirely, if it didn't end
in CR LF.  I think the technical term for that is "nasty bug",
not something that requires protocol changes.

> As an additional
> comment, I think that the Net-UTF-8 document should state
> that the last line need not be ended by CR+LF (or any other
> line end/separator), though it should be. This is just as a
> matter of normalising the line ends for Net-UTF8, not for
> UTF-8 in general.

So now End of Document (however that is expressed) is also a
line-ending?  Unfortunately, as we have discovered many times
with email, an implied line-ending gets one into lots of trouble
about just when it is implied and, in particular, whether
digital signatures should be computed with the normal
line-ending sequence inserted as implied or over the document as
sent.   Again, these problems are much more easily dealt with by
specifying explicitly what is to be put on the wire, making the
sending system convert things to that format as needed, and
treating bugs as bugs rather than justification for making the
standard forms more complex.

--On Thursday, 10 January, 2008 09:59 +0100 Kent Karlsson
<[email-masked]> wrote:

> As for the receiving side the same considerations as for the
> (SHOULD) requirement (point numbered 4 on page 4) for NFC in
> Net-UTF-8 applies. The reciever cannot be sure that NFC has
> been applied. Nor can it be sure that conversion of all line
> endings to CR+LF (there-by loosing information about their
> differences) has been applied.

This is, at least to me, a more interesting problem.  On the one
hand, there are no constraints due to backward compatibility
with NVT.  On the other, there are at least two real constraints:

(i) There is not a single normalization form.  Four are
standardized and others, for more or less specific purposes, are
floating around (e.g., without getting tied up in terminology
niceties about what is a normalization form and what is
something else, nameprep uses, to a first order approximation,
NFKC+lowercasing).  There has never been a clear recommendation
as to which one should used globally (The Unicode Standard
discusses considerations and tradeoffs... quite appropriately,
IMO).  In order to avoid chaos, some systems and packages force
particular normalizations on whatever passes through them (by
contrast, I'm not aware of anything that goes out of its way to
convert CRLF into NEL.  From a Unicode standpoint, it would make
more sense to convert CRLF to U+2028 (which, strangely to me,
doesn't appear on your list above) but, again, AFAK, no one does
that as a matter of routine either).   The net result of this is
that, if we have a string that starts out in some normalization
form (even NFC) that is then passed across the network, it may
then end up in the hands of the receiving subsystem in, e.g.,
NFD.  So it is important, pragmatically and whether we like it
or not, that the receiver check or apply normalization
regardless of what requirements me make on the sender.  The
digital signature issues are similar -- if one wants two bodies
of text to have the same signature value if they are considered
equivalent requires normalization to get even near equivalency.
Put differently, treating a body of text that is  unnormalized
on receipt as a bug to be rejected just doesn't make practical
sense, while treating text strewn with assorted characters that
might be line-ends (or, in some cases, might be something else)
doesn't.

thanks for the comments, thoughts, and careful reading.

    john

---

From: John C Klensin <[email-masked]>
Date: Thu, 10 Jan 2008 12:19:12 -0500

--On Thursday, 10 January, 2008 15:21 +0100 Frank Ellermann
<[email-masked]> wrote:

>...
> Hopefully somebody can confirm that IND is correct, or not.
> For HT and FF I hope the final version will somehow express
> that both are not really bad, and as far as they're bad FF is
> worse than HT. 

I'm open to consensus about changes for either HT or FF, but the
theory of "bad" that was used to construct the spec was:

(i) If a "spacing" control has the effect of setting the
position of the next character, it is "bad" unless that position
is unambiguous.   In addition, things are "bad" unless they are
necessary in running text (as distinct from faking things that
are better handled in markup, followed by either device-specific
output or standard page representations, neither of which are
normal text).

It is unambiguous for SP.  It is unambiguous for CRLF.
Independent of the "what is a line-end" problem, it is somewhat
ambiguous for CR or LF alone and for IND.  It is ambiguous for
HT.  It would be ambiguous for FF except that FF is assigned
fairly clear semantics in NVT -- "FF" is not a line ending (CRLF
FF is needed) and as Bob Braden noted, there is a fairly clear
rule that FF is to be interpreted as "top of next page" if one
knows what a page is and as "blank line" otherwise.  But that
rule is sufficiently often ignored to call for considerable
caution about FF, and the text now contains a cautionary note
for that reason.

There is an interesting demonstration of the law of unintended
consequences here.  If we could tell that a string was
unambiguously UTF-8 (or whatever) by looking at it, even if it
contains nothing but ASCII characters, then there would be no
reason to try to make net-utf8 a proper superset of NVT.  If we
could do that, we could also do away with the entire "next line"
debate by prohibiting even CRLF and requiring the use of LS
(U+2028).  In retrospect, there might have been considerable
advantages to forcing the ASCII- UTF-8 distinction by requiring
that UTF-8 strings all start with a BOM, but it is far too late
for that (and probably not, on balance, a good idea despite its
advantages).  So I don't see how to get there from here -- we
are stuck, for historical reasons, with CRLF on the wire as what
The Unicode Standard calls NLF (incidentally, Unicode 5.0,
Section 5.8, provides significant insight into the complexity of
this problem and probably should have been referenced.  It would
be even more helpful had Table 5-2 included identifying CRLF as
a standard Internet "wire" form of NLF, not just binding that
form to Windows.

> My impression from reading the draft was exactly the opposite,
> FF not too bad, HT really bad, that's odd for protocols
> allowing WSP.

See above.

    john

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 11 Jan 2008 07:52:50 +0100

John C Klensin wrote:

> It is ambiguous for HT.

Yes, but we typically don't care about this in protocols as
long as it behaves like one or more spaces.  I think that's
the idea of "WSP = SP / HTAB ; white space" in RFC 4234bis,
waiting for its STD number.

We talked about the 4234bis issue of "trailing white space",
which could cause havoc when it is silently removed, and a
"really empty line" is not the same as an "apparently empty
line" (i.e. CRLF CRLF vs. CRLF 1*WSP CRLF).

A similar robustness principle would support to accept old
"HTAB-compression" or "HTAB-beautification" (e.g. as first
character in a folded line).  In other words WSP, not only
SP.  It is clear that the outcome is ambiguous, but in some
protocols I care about (headers in MIME, mail, and news)
*WSP or 1*WSP are acceptable.   Admittedly it is a pain when
signatures need white space canonicalization.  But replacing
*WSP by *SP would only simplify this step, not get rid of it.

 [About CRLF]
> Unicode 5.0, Section 5.8, provides significant insight into
> the complexity of this problem and probably should have
> been referenced.  It would be even more helpful had Table
> 5-2 included identifying CRLF as a standard Internet "wire"
> form of NLF, not just binding that form to Windows.

Indeed, this chapter offers significantly *broken* insight
for our purposes.  What they found was a horrible mess, then
they introduced wannabe-unambiguous LS + PS, and what they
arrived at was messier than before.  Claiming that CRLF is 
"windows" is odd for DOS + OS/2 users, it is also at odds
with numerous Internet standards - precisely the reason why
we need your draft.  

The chapter talks about line and paragraph separators without
mentioning relevant ASCII controls such as RS.  On the other
hand it mentions MS Word interna which are nobody's business
outside of MS Word.  It is interesting, but IMO unusable for
net-utf8.

 Frank

---

From: "Kent Karlsson" <[email-masked]>
Date: Mon, 14 Jan 2008 11:11:27 +0100

> points that have been discussed over and over again (not just in

I raised a number of non-editorial issues that you did not
address below...

> responses accumulate, are likely to cause comments from me to

I think this document is at least one draft, maybe two or three
drafts, away from being of sufficient clarity and of sufficient
quality to become a standards document. In addition you state that
you don't have time right now to deal with this. I would therefore
suggest that the document be withdrawn from last call, to allow
time for clearing up the document.

> line-endings is a reasonable one, for which see below.  But I

I would not refer to most other line separators (that is how they
are best seen) as in any way "aberrant". Except for RS, GS, FS and
IND, they are not aberrant, and they are in no way malformed,
irregular, or any such. I agree that the situation is not ideal.
But I do think it is perfectly manageable without undue effort.

I do, however, regard using pure CR or BS to achieve accenting
or underlining to be highly aberrant, malformed, irregular and
unmanageable. (But that aberrant zoo you want to keep...)

> think your invocation of the robustness principle is

I'm not sure why...

> other places in this note, why go out of our way to make our

You already have NFC as a SHOULD not a SHALL. Which makes your
argument here entirely moot.

> protocols.

Agreed. But that is not related to any of my comments.

>  That means that a protocol doesn't need to

I think it is perfectly reasonable for a protocol to define a
"profile" of Net-UTF-8; e.g. saying that "[use Net-UTF-8] except that
FF and CR+FF are allowed, and that FF is converted to CR+FF [while
normally those would have been converted to CR+LF]." Note: that
was only an example.

> in a spec whose purpose is to tell applications that don't use

Of course not. I did not say so either.

> exception, I wouldn't personally hesitate to do that.

See my original comment, and above, for how I think this should be
resolved.

> FF a different interpretation under this specification than they

FF is always line separating (though it has been rarely used,
fortunately): if you change page, the line preceding the change of
page is of course ended (though the paragraph need not end).
Even when interpreted as an "empty line", it is line separating
(as it should be).

> But you have just summarized the reasons for avoiding HT.  We

No I have not. You tried to give some arguments, but I'm far from
persuaded. (Nor, it seems, is Frank Ellerman.) In practice the "problems"
are not worse that those of using variable width fonts. And that is
just about an absolute necessity when going beyond Latin/Greek/Cyrillic,
and also commonly used for Latin/Greek/Cyrillic, however detrimental
it is for "ASCII art". Unless the default tab setting is really
wacko, which it usually isn't.

> (in characters or millimeters) in a text stream, so one can't

Yes there is:

0088;<control>;Cc;0;BN;;;;;N;CHARACTER TABULATION SET;;;;

**NOT** that I suggest using that! Definitely not! I'm just pointing
out that there **is** an already defined control code for setting
tab stops, that however has clear disadvantages, and is outdated.

> explicitly in the document, we can try to do so.

I (and apparently at least also Frank Ellerman) think that HT
should be allowed in Net-UTF-8. The default settings for tab stops
for plain text seems to work well.

You silently seems to suggest that original HT should be replaced
by one or more spaces. But how many spaces in each instance? I think
it would be better to keep the HT (which I agree is not an ideal
character, but it is very common) as is. Note that I do NOT suggest
to ever replace spaces with HT. Doing that would be a really bad
idea (but still seen sometimes, with ill effects; like for the
subject line I "got" for this message...).

> not something that requires protocol changes.

As the current Net-UTF-8 draft is written, that printer behaviour
seems entirely within what is permissible (for printing, say,
Net-UTF-8 plain text documents).

What do expect to happen if other line separation than CR+LF
is/are used? Rejection of text, error messages, ignoring/deleting
them, treating them as spaces, or what?

> line-ending?  

Unless the text piece is used as a fragment (to be inserted/appended
to something else), end-of-document without explicit line-end should
end the (last) line, rather than be an error.

(And if there is a page paradigm, end-of-document (for a "complete"
document, i.e. not used as a fragment) also implies end-of-last-page,
even if there is no FF at the end.)

> Unfortunately, as we have discovered many times

Conversion to EBCDIC usually converts line endings (like CRLF) to NEL.
It is not absolutely certain that NEL is converted to another
line ending/separation upon conversion to a non-EBCDIC encoding.

Unicode normalisation conversion is also, in principle and as yet,
much less likely than line ending conversion.

> that as a matter of routine either).   The net result of this is

U+2028 does occur in my list above. IIRC it is used as "native"
line separator in at least one system (SymbianOS). Some programs
in other systems can also save files using LS as line separator.

> thanks for the comments, thoughts, and careful reading.

As I mentioned, I think the document we are discussing needs a few
more drafts before it may be in good enough shape to be reissued
as "Last Call".

	/Kent Karlsson

---

From: "Kent Karlsson" <[email-masked]>
Date: Mon, 14 Jan 2008 11:11:27 +0100

I'm not sure why...

Yes there is:

	/Kent Karlsson

---

From: John C Klensin <[email-masked]>
Date: Tue, 15 Jan 2008 02:31:17 -0500

--On Monday, 14 January, 2008 11:11 +0100 Kent Karlsson
<[email-masked]> wrote:

> quality to become a standards document. In addition you state
> that you don't have time right now to deal with this. I would
> therefore suggest that the document be withdrawn from last
> call, to allow time for clearing up the document.

Kent,

I think we have a difference of opinion here that has been
fairly well summarized in your exchanges with Frank.   Let me
try to identify the difference in a different way by examining
three possible cases:

	(1) We have a collection of protocols in the IETF that
	use text in lines and data transmission models that are
	both very simple and very dependent on a clear and
	precise definition of "line".  That definition has
	traditionally involved a CRLF sequence and that alone.
	
	(2) We have other protocols (and are likely to have more
	in the future) that use other definitions of text,
	usually, but not always, involving some form of markup
	or highly structured data.  HTML and XML are just two of
	those.
	
	(3) At least in principle, we could have something
	intermediate -- no markup in the XML sense, but
	significant use of contemporary Unicode characters and
	directives that become more or less specific about
	breaks and non-breaks, "soft" and "hard" line-endings,
	paragraph endings, and, as I had explained to me
	recently, the distinction between spaces and tabs in
	Unicode's "bidi" algorithm.

Now the net-utf8 work is targeted exclusively at the first case.
Even more specifically, it has been clear since the request to
get this written up and standardized came out of an Applications
Area meeting a few years ago that it would need be designed so
that all of the sensible and plausible uses of "NVT" would be
valid for it, without changes.   Based on the discussions on the
Apps Area discussion list over the last 18 months or so and the
comments on the IETF list since the Last Call started, I believe
that, modulo some tuning and clarifications, it is largely done.

>From reading your last three notes, it appears to me that you
would prefer the third protocol definition instead.   While I
find it interesting that the IETF has apparently not needed
anything like that so far (at least not that I've noticed), I
think creating such a definition would be a perfectly reasonable
activity.   But it doesn't seem to me that it is reasonable for
you to say that the net-utf8 document should be withdrawn from
Last Call and run through multiple additional drafts because it
doesn't solve the (rather different) problem you would like to
solve in the way in which you would like to solve it.

Like Ned Freed and probably you and others, I would like to
think that, if we were doing our character stream and line
definitions today, with all of Unicode and its various
formatting and layout characters in front of us, we would make
some decisions differently from the decisions made in the first
half of the 1970s (when I assume some readers of this note were
very, very young).  If we need that third protocol, then that
may be the place to make different decisions.  But, because of
those old decisions and a huge installed base the reflects them,
there are constraints on what can be done with net-utf8.  The
current specification reflects those constraints (or at least I
think it does).   Most of the suggestions you are making do not.

Assuming that the third protocol were written, we could then
look forward to many interesting arguments as to whether it, or
net-utf8, or something else entirely, should be specified for
use in a particular protocol or situation.   But that argument,
it seems to me, would be worthwhile.  Trying to insist that
net-utf8 be turned into something else entirely is not, IMO.

    john

---

From: <[email-masked]>
Date: Tue, 15 Jan 2008 11:12:49 -0000

> Even more specifically, it has been clear since the request 
> to get this written up and standardized came out of an 
> Applications Area meeting a few years ago that it would need 
> be designed so that all of the sensible and plausible uses of 
> "NVT" would be
> valid for it, without changes.

In that case, this is *NOT* truly net-utf8 as one would understand
it in normal English. Instead it is the Unicode version of NVT.

It seems to me that there is room for a standard Net-UTF8
for future new protocols, that sticks closely to the Unicode
standards and tries to be transparent to arbitrary UTF-8
streams. This newer UTF8 standard would take its line-ending
cues from the Unicode regular expression rules, i.e. a 
command line ending could potentially be detected using
a Unicode compatible RE engine. Unicode Whitespace can be used
to delimit words and arguments in command lines. One might even
leverage the existence of PS to mark the end of the command lines
paragraph, for instance an SMTP like protocol would not need a DATA
keyword because a PS could be used to mark the beginning of
data.

I think it is a good idea to have an update to NVT that allows 
for Unicode, but I don't think that the need for reverse
compatibility kludges which make it incompatible with the
Unicode standard require this document to not masquerade as the
definitive format for Unicode on the wire.

If the nomenclature is changed a bit to make it clear that
this is an update of NVT, then I think many of the arguments
against it fall away. Of course this means that there will
be another document at some point with a different form of
UTF8 on the wire, but it doesn't hurt to give future protocol
designers a clear choice.

Is it better to stick with the installed base and make incremental
improvements, or should we break with the past and start afresh
using the hard-earned knowledge of building that installed base?
This draft should not have to wait for that decision to be made
rather it should go to RFC status sooner, rather than later, so
that attention can be focussed on the second option, and what 
comes after that.

--Michael Dillon

---

From: "Frank Ellermann" <[email-masked]>
Date: Wed, 16 Jan 2008 05:28:20 +0100

Michael Dillon wrote:

> In that case, this is *NOT* truly net-utf8 as one would 
> understand it in normal English. Instead it is the Unicode
> version of NVT.

It's the UTF-8 version of net-ascii.  Admittedly I had no
idea what "net-ascii" is until I read John references to
RFCs with numbers below 821, excl. lost in space RFC 97 ;-)

<http://purl.net/xyzzy/-a9/netascii%20OR%20%22net-ascii%22>

 Frank

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 21 Jan 2008 10:22:46 +0100

On Tue, Jan 15, 2008 at 11:12:49AM -0000,
 [email-masked] <[email-masked]> wrote 
 a message of 59 lines which said:

> it in normal English.

[Personnally, I like the term and it seems to me, in my broken
english, quite clear.]

> Instead it is the Unicode version of NVT.

Do you mean that a simple change of the name of the protocol would be
sufficient to address your concerns?

If so, based on John Klensin's excellent explanations in
http://www1.ietf.org/mail-archive/web/ietf/current/msg49905.html, I
suggest:

1) Really plain text, as in NVT, with almost only CRLF sequences as
formatting: Unicode-NVT (currently in draft-klensin-net-utf8-07.txt)

2) Markup text as in XML: out of scope.

3) Plain text but using the full power of Unicode formatting: Net-UTF8
(currently not in any I-D, not widely in use and I'm skeptical about
it, despite the fact that it is technologically tempting)

Will it be better for you? (Not for me but may be the rough consensus
is against me.)