Subject: Gen-ART and OPS-Dir review of draft-ietf-json-text-sequence-09

From: "Black, David" <[email-masked]>
Date: Fri, 5 Dec 2014 14:51:04 +0000

This is a combined Gen-ART and OPS-Dir review.  Boilerplate for both follows ...

I am the assigned Gen-ART reviewer for this draft. For background on
Gen-ART, please see the FAQ at:

<http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>.

Please resolve these comments along with any other Last Call comments
you may receive.

I have reviewed this document as part of the Operational directorate's ongoing
effort to review all IETF documents being processed by the IESG.  These comments
were written primarily for the benefit of the operational area directors.
Document editors and WG chairs should treat these comments just like any other
last call comments.

Document: draft-ietf-json-text-sequence-09
Reviewer: David Black
Review Date: Dec 5, 2014
IETF LC End Date: Dec 5, 2014
IESG Telechat date: Dec 18, 2014

Summary: This draft is on the right track, but has open issues
 		described in the review.

This draft specifies a format that packs multiple JSON texts into a
single string.  The ASCII RS (0x1E) character is used to separate texts,
and a linefeed is appended to each text to ensure that a complete text
always ends with a whitespace character.

All of the open issues are minor - the most important ones center on
treatment of incomplete JSON texts - that appears to be an afterthought
in this draft and needs more attention.  I also found a couple of
minor issues in the Security and IANA Considerations sections, both of
which are almost nits.

Major issues: None.

Minor issues:

[A] Section 2.1:

   If parsing of such an octet string as a JSON text fails, the parser
   SHOULD nonetheless continue parsing the remainder of the sequence.

That's not quite right - there are two levels of parsing, JSON
sequence parsing and JSON text parsing of each text in the sequence,
both of which might be implemented in a single-pass parser.  For such an
implementation, the above sentence could be (mis-)read to imply that the
JSON text parse should resume from the point at which it failed, which
would be silly (although I've seen heroic PL/1 parsers do exactly that).
Instead, the parse needs to skip ahead to the next RS, ignoring the rest
of the JSON text that failed to parse.  I suggest:

   If parsing of such an octet string as a JSON text fails, and the
   octet string is followed by an RS octet, the parser
   SHOULD nonetheless skip ahead to that RS octet and continue parsing
   the remainder of the sequence from there.

That also covers the case where there is nothing more to parse after the
JSON text that caused the parse failure.

[B] Section 2.3:

Is incremental parsing of a JSON text within a sequence allowed, or
is the parser required to not produce any results until the parse of
the entire text is successful?  I'd expect that incremental parsing
is ok (so results may be produced from a text that ultimately fails
to parse), and I think that's worth stating.

[C] Section 2.4:

   Parsers MUST check that any JSON texts that are a top-level number
   include JSON whitespace ("ws" ABNF rule from [RFC7159]) after the
   number, otherwise the JSON-text may have been truncated.

That reference to the "ws" rule doesn't get the job done because that
rule allows a match to no characters - it's of the form ws = *( ... )
where ... is the list of whitespace characters.  What's needed here is
a rule of the form vws = 1*( ...) to force there to be at least one
whitespace character, but see the next issue for a better way to deal
with this topic by pulling the appended LF into the sequence parse
instead of the text parse.

[D] I wonder whether the possibility of incomplete texts ought to be
encoded into the parsing rules to directly catch JSON texts that must
be incomplete because the last character is not LF, e.g.:

     JSON-sequence = *(1*RS (possible-JSON / truncated-JSON / empty-JSON))
     RS = %x1E; "record separator" (RS), see RFC20
     possible-JSON = 1*(not-RS) LF ; attempt to parse as UTF-8-encoded
                               ; JSON text (see RFC7159)
     truncated-JSON = *(not-RS) not-LFRS); truncated, don't attempt
					; to parse as JSON text
     empty-JSON = LF ; only the LF appended by the encoder, nothing to parse

     not-RS = %x00-1D / %x1F-FF; any octet other than RS
     not-LFRS = %x00-09/ %x1B-1D / %x1F-FF; any octet other than RS or LF

Note that this won't detect all incomplete JSON texts, because LF is allowed
within a JSON text (and this should be stated).

[E] Section 3 - Security Considerations

Incomplete and malformed JSON texts can be used to attack JSON parsers -
that should be pointed out, as I don't see that in RFC 7159's security
considerations and incomplete texts are a relevant consideration for
this draft.

[F] Section 4 - IANA Considerations

   Security considerations: See <this document, once published>,
   Section 3.

   Interoperability considerations: Described herein.

   Published specification: <this document, once published>.

   Applications that use this media type: <by publication time
   <https://stedolan.github.io/jq> is likely to support this format>.

Replace all three instances of the angle bracketed text.  The first two
instances should be RFC references (e.g., RFC XXXX) w/a note to the RFC
Editor to insert the number of the RFC when published.  The third instance
should be resolved now, or could have an RFC Editor note added indicating
that the author will resolve that during Authors 48 hours.

Nits/editorial comments:

idnits didn't like the reference to RFC 20 for ASCII:

  ** Downref: Normative reference to an Unknown state RFC: RFC   20

RFC 5234 (ABNF) uses this, which looks like a better reference:

   [US-ASCII]  American National Standards Institute, "Coded Character
               Set -- 7-bit American Standard Code for Information
               Interchange", ANSI X3.4, 1986.

--- Selected RFC 5706 Appendix A Q&A for OPS-Dir review ---

Most of these questions are n/a because this draft describes a format
that will be used in other protocols to which RFC 5706's concerns would apply.

A.1.4   Have the Requirements on other protocols and functional
       components been discussed?

The specification of the interaction of the JSON sequence parser with the
JSON text parser is not as clear as it should be for incomplete or malformed
JSON texts.  See Minor Issues [A]-[E] above.

A.1.8   Are there fault or threshold conditions that should be reported?

Yes, incomplete JSON texts - this is covered in sections 2.3 and 2.4. 

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: Barry Leiba <[email-masked]>
Date: Fri, 5 Dec 2014 17:38:54 -0500

>                Interchange", ANSI X3.4, 1986.

Except that (1) many IETF documents do use RFC 20 and (2) the RFC 20
reference is not for ASCII: it's for RS, the Record Separator
character, which is explained in RFC 20, Section 5.2.

Barry

---

From: "Black, David" <[email-masked]>
Date: Fri, 5 Dec 2014 22:54:04 +0000

Ok, it is a nit, as is much of what idnits produces ;-).
I have no problem ignoring idnits on this one ...

Thanks,
--David

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Barry Leiba
> Sent: Friday, December 05, 2014 5:39 PM
> To: Black, David
> Cc: [email-masked]; General Area Review Team ([email-masked]); ops-
> [email-masked]; [email-masked]; [email-masked]
> Subject: Re: Gen-ART and OPS-Dir review of draft-ietf-json-text-sequence-09
>

---

From: John C Klensin <[email-masked]>
Date: Fri, 05 Dec 2014 22:51:48 -0500

Barry (and IESG generally),

This has come up multiple times and will undoubtedly keep coming
up, especially since RFC 20 is a stable reference to one
particular version of ASCII and actually includes the code
tables while X3.4-1968 (the version to which it refers) is
largely unobtainable today (only the current version is) and
ANSI X3.4, aka ANSI/INCITS 4, is not a stable reference without
a date.

Most, perhaps all, versions of ANSI X3.4 (and ANSI/INCITS 4)
also specify the repertoire and coding, i.e., the CCS, but not
what we would call the encoding form today (in the case of RFC
20, the familiar "seven bits in and eight bit byte with high
order bit always zero").  So, for most IETF purposes, RFC 20
really should be the normative reference for ASCII (or, if one
prefers, "US-ASCII").

RFC 20 has status "Unknown" only because it comes from a time
that predates both the IETF and our use of the term "standard"
(with or without qualifications) to describe Internet technical
specifications.

So, rather than go through a discussion about downrefs and the
like every time RFC 20 is referenced from a Standards-Track
specification, I suggest that the IESG reclassify it to Internet
Standard and waste as little more time doing so as possible.  

The implementation report is that, whether they explicitly
reference RFC 20 or not, substantially every application-layer
protocol we have depends on the ASCII CCS and encoding form
specified in that RFC.  In addition, RFC 5234 and its
predecessors are heavily dependent on ASCII so that
substantially any specification that depends on ABNF is also an
ASCII implementation.

Thanks,
    john

--On Friday, December 05, 2014 17:38 -0500 Barry Leiba
<[email-masked]> wrote:

>>    [US-ASCII]  American National Standards Institute, "Coded
>>    Character Set -- 7-bit American Standard Code for
>>                Information Interchange", ANSI X3.4, 1986.
> 
> Except that (1) many IETF documents do use RFC 20 and (2) the
> RFC 20 reference is not for ASCII: it's for RS, the Record
> Separator character, which is explained in RFC 20, Section 5.2.
> 
> Barry
>

---

From: "Stewart Bryant (stbryant)" <[email-masked]>
Date: Sat, 6 Dec 2014 15:02:44 +0000

If it is just for IETF purposes it could be added to the downref list without being reclassified.

Stewart

> On 6 Dec 2014, at 03:52, John C Klensin <[email-masked]> wrote:
>

---

From: John C Klensin <[email-masked]>
Date: Sat, 06 Dec 2014 11:30:52 -0500

Warning/disclaimer: I believe that reclassifying RFC 20 to match
unquestionable reality is so obvious that I'm somewhat annoyed
at having to spend time (and the community's time) discussing
it.  If the IESG feels justified in applying fast track
procedures to documents that are far more controversial (and for
which there is not even five or ten years of successful
operational experience, much less 45), we really shouldn't need
to spend time debating what can or should be done with RFC 20.
If that annoyance comes through below and is unreasonable, my
apologies.

--On Saturday, December 06, 2014 15:02 +0000 "Stewart Bryant

> downref list without being reclassified.

Well, actually, it probably can't.  In addition to what I hope
is a better and more substantive explanation below, this note
concludes with a little protocol-lawyer explanation that could
be the basis for an appeal of any attempt to use that approach.

My understanding, and I think that of the community, was that
the reason for RFC 6410 was to increase the number of document
that we classified as Internet Standards by reducing the
barriers to doing so.  It would be truly unfortunate, IMO, if we
now started introducing additional obstacles such as "this could
be accomplished in some other way".

In addition, there is nothing about the status of RFC 20 that is
unknown.  The way the maturity classification system was
introduced resulted in identifying the status of many early
document with "Unknown" as a short way to say "formal status not
reviewed and assigned", but that is not because we didn't know.
It is because the RFC Editor and the community didn't believe
that going through those documents was worth spending time on.
Those were kinder, gentler, and less procedurally constipated
times; today, the failure to assign a classification to RFC 20
looks in retropsect more like an isolated error in judgment.
The technology the document describes is in very heavy use
(including in the specification that allowed these messages
about it to be sent and interpreted), the RFC itself is
extensively referenced in other IETF specs (not that such
references are a requirement for Internet Standard even under
2026), and it is arguably the most stable RFC-published
specification we still use (all lower-numbered RFCs are either
status reports, associated with the NCP ARPANET, or
documentation specifications that have been replaced by the RFC
Format specification documents  that start with RFC 825 and
cumulate in the recently-approved RFC 7322).  

The Protocol-lawyer speaks:

Strictly speaking and AFAIK, we've got two consensus documents
that modify the RFC 2026 requirements about downrefs.   The
first is RFC 3967.  It is fairly broad about what can be
referenced, but requires an explicit statement in the Last Call
announcement about what is being done.  That can be waived for a
given document, but only if there have been several such Last
Call announcements in the past.  There have been, again AFAIK,
no such announcements for RFC 20, so a single Last Call to
reclassify it would actually involve less work and time than
applying that particular RFC 3967 rule.  More important, RFC
3967 is quite explicit that use of a downref procedure is
inappropriate for cases like this.  The relevant sentence reads:

	"This procedure should not be used if the proper step is
	to move the document to which the reference is being
	made into the appropriate category."

and the text that follows just strengthens that message.

There is also no support in the more flexible RFC 4897 because
it applies only to standards-track documents.  If RFC 20 is not
considered a standards-track document because it is listed with
a status of "Unknown", then 4897 cannot be applied.   If it is
really a standards-track document in spite of that placeholder
designation in the RFC Index, then there is no basis for
treating it as other than a full standard and its listing as
"unknown" is a clerical error that the RFC Editor should be
directed to correct forthwith (with no reclassification
procedure required).

    john

---

From: "John Levine" <[email-masked]>
Date: 6 Dec 2014 17:06:11 -0000

>If it is just for IETF purposes it could be added to the downref list without being reclassified.

I have to agree with John K.  Pick any definition of standard you
want, stable, interoperable, multiple implementations, whatever, and
RFC 20 certainly fits as well as anything else we call a standard.

Just mark it as a standard and be done with it.  Because that's
what it is.

R's,
John

PS: Thought experiment: Let's say we made RFC 20 a full standard.
What Bad Things will happen?

---

From: Stephen Farrell <[email-masked]>
Date: Sat, 06 Dec 2014 17:21:24 +0000

> What Bad Things will happen?

Some people will be upset. Same as if we don't do that:-)

I'm fine with pushing this one along the stds track and
will kick that off next week. I need to go re-read whatever
process stuff is involved, but if someone wants to be the
shepherd for this, (I'm guessing one is needed/handy) then
just mail me.

S.

PS: If Barry or anyone else wants to do this instead that's
fine by me.

---

From: John C Klensin <[email-masked]>
Date: Sat, 06 Dec 2014 12:59:53 -0500

--On Saturday, December 06, 2014 17:21 +0000 Stephen Farrell
<[email-masked]> wrote:

>> PS: Thought experiment: Let's say we made RFC 20 a full
>> standard. What Bad Things will happen?
> 

Based on working in some closely-related areas, the only
legitimate objection I can think of would come from folks who
would claim that ASCII has outlived its usefulness and that we
should drop all references to ASCII, US-ASCII, and RFC 20 in
favor of what I guess would be something like "the Basic Latin
and C0 repertoire of Unicode, represented by code points U+0000
through U+007F, coded in UTF-8".   However, if one takes that
position, then RFC 20 should be moved to Historic, all protocol
specs that we now have that reference ASCII should be viewed as
obsolescent, and we should refuse to accept any new specs that
depend on ASCII unless it is defined in those Unicode terms (see
my previous note and remember that includes almost anything that
depends on ABNF).  If only because it would generate a lot of
basically-useless work, I don't think we want to go there.

While it would affect very few specs in practice, there are also
some subtle differences between ASCII and the Unicode C0+Basic
Latin definition.

> shepherd for this, (I'm guessing one is needed/handy) then

Since I started this and believe that very little is required
(and most of that is putting what has been written already into
shepherd template form), I'm willing to do it unless someone
else volunteers.   

However, if there is anyone around with a little less experience
in this stuff than you, me, or John L and who would like to get
a first-hand introduction to the process of moving/shepherding a
document through the system with me playing advisor, I'd rather
spend my time that way than on template construction.  So, if
you or other IESG members, or any mentor or EDU team members who
happen to be reading this know likely candidates who could use
that bit of education (or someone out there wants to volunteer
themselves), speak up.

> PS: If Barry or anyone else wants to do this instead that's

thanks,
   john

---

From: Stephen Farrell <[email-masked]>
Date: Sat, 06 Dec 2014 18:06:54 +0000

Thanks John,

I'll take you up on that.

It's a good point that this is a good way to dip a toe
in the process stuff, so if someone else for whom that'd
be a good plan mails me I'll let you know that they're
doing it instead.

Cheers,
S.

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 07 Dec 2014 08:08:32 +1300

On 07/12/2014 06:06, John Levine wrote:
...
> What Bad Things will happen?

The worst thing I can imagine is that people will become a bit
confused about whether the Father of the Internet is called
Vint Cerf or Vinf Cert.

(If you don't understand what I'm talking about, check the errata
for RFC 20.)

Which is intended as a whimiscal way of saying: I don't think
any Bad Things will happen. Just do it. It will have the side
benefit of giving Vint credit for the oldest Internet Standard
document, which seems appropriate.

   Brian

---

From: <[email-masked]>
Date: Sat, 6 Dec 2014 21:13:12 +0000

If RFC20 is made a full standard, then references to it must be to a particular paper copy (not necessarily the 'original'), since there's a chicken-and-egg problem in needing to have a
working implementation of RFC20 just to decode and read electronic transcriptions of RFC20.

Better to have a dependency outside the RFC series.

(The silliness of calling anything labelled 'request for comments' a standard is now, alas, traditional.)

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: ietf <[email-masked]> on behalf of John C Klensin <[email-masked]>
Sent: Sunday, 7 December 2014 4:59:53 AM
To: Stephen Farrell; [email-masked]
Cc: John Levine; [email-masked]
Subject: Re: Status of RFC 20 (was: Re: Gen-ART and OPS-Dir review of draft-ietf-json-text-sequence-09)

thanks,
   john

---

From: John C Klensin <[email-masked]>
Date: Sat, 06 Dec 2014 17:15:27 -0500

--On Saturday, December 06, 2014 21:13 +0000 [email-masked]
wrote:

> If RFC20 is made a full standard, then references to it must
> be to a particular paper copy (not necessarily the
> 'original'), since there's a chicken-and-egg problem in
> needing to have a working implementation of RFC20 just to
> decode and read electronic transcriptions of RFC20.

Lloyd, if you want to turn this into an exercise in how pedantic
it is possible to get, I'm pretty sure we will all lose.

However, and perhaps ironically, the very early RFCs were
published and circulated on paper and the normative copies were
paper copies.  You could derive strong hints about this from
reading RFC 10 which (with RFC 16) specifies the distribution
list around the time RFC 20 was written and, notable, gives
postal mail addresses and not network ones.  That is
unsurprising since FTP, much less FTP with anonymous/guest and
mail accounts, didn't come along until over a year later.

If you were to look at the early RFCs, you would find notes on
many of them indicating that they were typed back in or
otherwise reconstructed into machine-readable ASCII page image
form some years later.   Some even contain warnings that the
renditions might not be completely accurate.  In particular, if
you look at the current online version of RFC 20, you will find
a note that reads:

	"[ This RFC was put into machine readable form for 
	entry]
	[ into the online RFC archives by Robbie Bennet 9/99]"

As has already been noted, Robbie made an error in constructing
the document footers (I'd have to dig out my copy of the
original (see below) to even figure out of it had footers --
many of the early documents did not) leaving the author as
someone named "Cert".

In addition, I don't know about RFC 20 (and don't know if Vint
would remember), but many of those early documents were produced
in either NLS or the approximately-proprietary word processing
systems of one environment or another, formats that no one (at
least no one I know of or have talked with about this)
considered appropriate for interchange, much less archival, use.
Some were even produced using an ancient technology that I
believe was called "typewriters".

I've had my hands on a photocopy of the original distribution
version of RFC 20 and not only was it on paper but it was
pretty-printed or at least proportionally spaced, not the
page-format ASCII we consider the norm today (or have until
recently).  Don't know where it is at the moment but I could
find it if sufficiently motivated.

> (The silliness of calling anything labelled 'request for
> comments' a standard is now, alas, traditional.)

I am not an engineer and try to avoid getting sucked into
playing one on television (or elsewhere), but I went to school
with engineers and was taught by engineers.  I had always been
taught that good engineering requires pragmatism and attention
to issues that constrained the problem or solution spaces.  From
that point of view, calling an organization in which it is
necessary to have this type of discussion about the details of
reference stability and availability of 45-year-old documents
the "Internet _Engineering_ Task Force" can equally be justified
more by tradition than anything else.  On the other hand, I
really don't think such discussions are needed or helpful, even
(or especially) in the IETF.

    john

---

From: <[email-masked]>
Date: Sat, 6 Dec 2014 22:49:15 +0000

It's already an exercise in pedantry, and I'm well aware that circulation was on paper copies - hence my comment on 'original', given that it was typed up using some proprietary system. And I'd trust a mimeograph over a newfangled photocopy. All we're doing here is shifting shadows on a wall.

Engineering standards require stable referents; which makes it a valid engineering question.

Security pedants might wonder why there is no easy way to authenticate electronic copies of RFCs, given the vast array of security-related protocols that the IETF has defined. How can I check the integrity of an RFC document and that it hasn't been tampered with? I imagine an MD5sum just won't do.

Lloyd Wood
http://about.me/lloydwood
________________________________________
From: John C Klensin <[email-masked]>
Sent: Sunday, 7 December 2014 9:15:27 AM
To: Wood L  Dr (Electronic Eng)
Cc: [email-masked]
Subject: Re: Status of RFC 20 (was: Re: Gen-ART and OPS-Dir review of draft-ietf-json-text-sequence-09)

    john

---

From: Dave Cridland <[email-masked]>
Date: Sat, 6 Dec 2014 23:37:45 +0000

On 6 December 2014 at 22:49, <[email-masked]> wrote:

> Security pedants might wonder why there is no easy way to authenticate
> electronic copies of RFCs, given the vast array of security-related
> protocols that the IETF has defined. How can I check the integrity of an
> RFC document and that it hasn't been tampered with? I imagine an MD5sum
> just won't do.
>

All the copies I'm reading are properly signed, according to RFC 4637. If
yours aren't, maybe they *have* been tampered with.

Dave.

---

From: Barry Leiba <[email-masked]>
Date: Sat, 6 Dec 2014 19:19:34 -0500

> PS: If Barry or anyone else wants to do this instead that's

I already have; the status-change document is in Last Call Requested state.

Barry

---

From: Patrik Fältström <[email-masked]>
Date: Sun, 7 Dec 2014 10:40:43 +0100

All,

Many definitions in this document have been specified in the form of 1:1 mappings of Unicode code points to single bytes. This include the record separator and for example definitions of the string "false" in the form of five octets.

This is ok if the encoding used for Unicode is UTF-8, and indeed it says that the parser should try to parse the string as if it is a UTF-8 sequence of characters.

But it also reference RFC7159, which doesn't require UTF-8 but instead for some weird reason also allow other encodings of Unicode text. And on top of that it says Byte Order Mark is not allowed.

This together implies that first of all this draft might not lead to stable implementations, secondly one can not store in JSON strings that include the Byte Order Mark, and there are other unspecified situations.

Or, in short, weaknesses, as I see it, in RFC7159 are made even more weak and potentially dangerous with draft-ietf-json-text-sequence.

Yes, I and others should probably not have let RFC7159 through, because there might be where the bugs are.

Suggestion: this draft, draft-ietf-json-text-sequence, should say explicitly that the only "profile" of RFC7159 that is allowed is UTF-8. That should be a MUST.

Reminder for the IETF: having "or" statements is not recommended at all when talking about these kind of things, and RFC7159 include at least one "or" too many. The recommendation from IETF is to use UTF-8 encoding for Unicode (when serializing text).

   Patrik

> On 5 dec 2014, at 15:51, Black, David <[email-masked]> wrote:
>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sun, 7 Dec 2014 09:06:16 -0500

On Sat, Dec 6, 2014 at 12:59 PM, John C Klensin <[email-masked]> wrote:

> through U+007F, coded in UTF-8".

That would be a silly approach. The reason we use ASCII is that our
protocols all deal in byte streams and it is a byte oriented encoding.

One point that we might want to reconsider however is the presentation of
the RFC. At the moment it is in TXT format which leads to the bootstrap
objection. And until recently that was the only option we had for
presenting RFCs.

Presenting a character encoding table is one specific case where an image
is more appropriate. Since that was how the original RFC20 was presented
and we are looking at a transcription into ASCII, it would be more
appropriate to reformat RFC20 in the new presentation format.

That would not require changing the number of course since none of the text
would change.

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 07 Dec 2014 16:47:56 +0100

Phillip Hallam-Baker <[email-masked]> writes:

> it would
> be more appropriate to reformat RFC20 in the new presentation format.

Done:-)

https://www.rfc-editor.org/rfc/pdfrfc/rfc20.txt.pdf

Grüße, Carsten

---

From: Pete Resnick <[email-masked]>
Date: Sun, 7 Dec 2014 10:41:00 -0600

> character, which is explained in RFC 20, Section 5.2.

And as per BCP97, RFC 3967, section 3, this downref was specifically 
called out in the Last Call for this document.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478

---

From: Patrik Fältström <[email-masked]>
Date: Sun, 7 Dec 2014 19:55:41 +0100

> On 7 dec 2014, at 19:05, John Cowan <[email-masked]> wrote:
> 
> Patrik Fältström scripsit:
> 
>> But it also reference RFC7159, which doesn't require UTF-8 but instead
>> for some weird reason also allow other encodings of Unicode text. And
>> on top of that it says Byte Order Mark is not allowed.
> 
> 7159 was meant to tighten the wording of 4627, not to impose additional
> constraints on it.  For that, see the I-JSON draft.

The problem I have is that 7159 is not tight enough as it allows other encodings than UTF-8, which in turn make the encoding not work very well as this draft take for granted each one of the separator characters is one byte each.

I.e. the way I read draft-ietf-json-text-sequence (and I might be wrong), you have specific octet values that act as separators. That only works if the encoding is UTF-8.

See Figure 1:

> possible-JSON = 1*(not-RS); attempt to parse as UTF-8-encoded

Now, if this is NOT UTF-8, then this might be pretty bad situation.

What I am saying is that I would like this draft to explicitly say that the only profile of RFC7159 that can be used is when UTF-8 is in use, i.e. somewhere something like "The encoding MUST be UTF-8, although RFC7159 also allow other encodings, like UTF-16." Then in the security considerations section that "RFC7159 do allow not only UTF-8 encoding but also for example UTF-16, which MIGHT create problems for a parser, all depending on what data is serialized."

I.e. I want this draft to be even more tight than RFC7159.

Let me ask it this way: is there any reason to allow other encodings than UTF-8? If so, how do you handle the encoding of the separators?

>> This together implies that first of all this draft might not lead to
>> stable implementations, secondly one can not store in JSON strings
>> that include the Byte Order Mark, and there are other unspecified
>> situations.
> 
> If by that you mean that a JSON string may not contain U+FEFF, that is
> incorrect, for U+FEFF is recognized as a BOM only when placed at the
> beginning of an entity body, whereas an entity body in JSON format can
> begin only with [ or { classically, or by extension with [0-9"tfn].

Ok, so what you say is that a string in an attribute value in the JSON blob can still start with U+FEFF?

If so, good, and my apologies for not understanding this at my read of the text.

   Patrik

---

From: joel jaeggli <[email-masked]>
Date: Sun, 07 Dec 2014 13:47:40 -0800

> Standard and waste as little more time doing so as possible.  
3967 applies quite effectively

   Once a specific down reference to a particular document has been
   accepted by the community (e.g., has been mentioned in several Last
   Calls), an Area Director may waive subsequent notices in the Last
   Call of down references to it.  This should only occur when the same
   document (and version) are being referenced and when the AD believes
   that the document's use is an accepted part of the community's
   understanding of the relevant technical area.  For example, the use
   of MD5 [RFC1321] and HMAC [RFC2104] is well known among
   cryptographers.

Anyone raising downref issues with rfc 20 is out of their mind.

that said you'll note a rather large gap in citations, given that for
something like 29 of the last 45 years there wasn't an online copy in
the rfc repository.

http://www.arkko.com/tools/allstats/citations-rfc20.html

---

From: John C Klensin <[email-masked]>
Date: Sun, 07 Dec 2014 17:13:57 -0500

--On Sunday, December 07, 2014 13:47 -0800 joel jaeggli
<[email-masked]> wrote:

>> specification, I suggest that the IESG reclassify it to
>> Internet Standard and waste as little more time doing so as
>> possible. 
 
>    Once a specific down reference to a particular document has
> been    accepted by the community (e.g., has been mentioned in
> several Last    Calls), an Area Director may waive subsequent
> notices in the Last    Call of down references to it.  This
> should only occur when the same    document (and version) are
> being referenced and when the AD believes    that the
> document's use is an accepted part of the community's
> understanding of the relevant technical area.  For example,
> the use    of MD5 [RFC1321] and HMAC [RFC2104] is well known
> among    cryptographers.

Except that 3967 requires that the downref be _explicitly_
identified in Last Call announcements and that waiver doesn't
apply unless that has been done.  Approving documents that
contain the downref without an explicit mention in the Last Call
announcement may be fine and sensible but, as 3967 is written,
doesn't count.

One can quibble about the lower case "should not" (and, given
how this conversation has gone, I'm certain someone will), but
the paragraph of 3967 after the one you quoted expressly forbids
the use of the 3967 procedure as a substitute for reclassifying
a document "into the appropriate category".  As John Levine
pointed out, RFC 20 is a standard by any criteria one can use
other than the failure of the RFC Editor and then the IESG to
appropriately identify it.

> Anyone raising downref issues with rfc 20 is out of their mind.

There we agree, but this issue didn't come up because someone
was looking for a way to start trouble.  It came up in a real
review because tools that reviewers are expected to look at and
use called it out.  We could figure out lots and lots of
complicated ways to work around that problem, of which a Last
Call that uses the 3967 mechanism (despite the prohibition)
would be an example, but I need to once again ask why we don't
just fix the problem and reclassify the document to reflect its
obvious status.

> that said you'll note a rather large gap in citations, given
> that for something like 29 of the last 45 years there wasn't
> an online copy in the rfc repository.

(I added Heather to the distribute because of the above)

To the best of my knowledge, there has _never_ been a
requirement that cited documents be available online, and
especially that authoritative copies be available online.
Certainly it is preferred for many reasons, but never has it
been a rule, nor is there a rule that makes RFCs special in that
regard.  If the IESG asked the community for permission to
impose such a rule, it certainly was not within my memory.  As
to the "last 45 years", there simply has not been an online
repository for that long, so that criticism would apply to any
older RFC.

If we want to start inventing new rules about citations to block
progress, I think there are any number of members of the
community who would be happy to contribute to the effort.  More
constructively, April 1 will be here soon.  :-(

  john, probably becoming unbearable

---

From: joel jaeggli <[email-masked]>
Date: Sun, 07 Dec 2014 14:50:27 -0800

On 12/7/14 2:13 PM, John C Klensin wrote:
> apply unless that has been done. 
There are normative citations of RFC 20

Q.E.D.

Whether to include one on the future is a judgment call on the part of
the person writing the last call writeup, e.g. the sponsoring AD.
>  Approving documents that
...
> especially that authoritative copies be available online.
That's not the point, nobody cited it because they couldn't read it.

---

From: "Black, David" <[email-masked]>
Date: Mon, 8 Dec 2014 02:59:44 +0000

I agree with Patrik - this draft assumes UTF-8 encoding and should
state that requirement explicitly.  John's proposed text change below
is in section 2.1 for the decoder; the encoder text in section 2.2
needs a corresponding change:

OLD
   In prose: any number of JSON texts, each preceded by one ASCII RS
   character and each followed by a line feed (LF).
NEW
   In prose: any number of JSON texts encoded as UTF-8, each preceded
   by one ASCII RS character and each followed by a line feed (LF).

Thanks,
--David

> From: John Cowan [mailto:[email-masked]] On Behalf Of John Cowan
> Sent: Sunday, December 07, 2014 4:08 PM
> To: Patrik Fältström
> Cc: Black, David; Nico Williams; General Area Review Team ([email-masked]);
> [email-masked]; [email-masked]; [email-masked]
> Subject: Re: [Json] Gen-ART and OPS-Dir review of draft-ietf-json-text-
> sequence-09
> 
> > I.e. the way I read draft-ietf-json-text-sequence (and I might be
> > wrong), you have specific octet values that act as separators. That
> > only works if the encoding is UTF-8.
> 
> This is a binary representation which has embedded JSON texts represented
> in UTF-8.  Since the first character in a JSON text is necessarily in
> the ASCII repertoire, it is not possible to parse a UTF-16 or UTF-32
> JSON text as UTF-8 and come out with valid JSON.
> 
> However, I grant that mentioning UTF-8 only in an ABNF comment is not
> really prominent enough.  Proposed wording change:
> 
> For:
> 
>    In prose: a series of octet strings, each containing any octet other
>    than a record separator (RS) (0x1E) [RFC0020], all octet strings
>    separated from each other by RS octets.  Each octet string in the
>    sequence is to be parsed as a JSON text.
> 
> read:
> 
>    In prose: a series of octet strings, each containing any octet other
>    than a record separator (RS) (0x1E) [RFC0020], all octet strings
>    separated from each other by RS octets.  Each octet string in the
>    sequence is to be parsed as a JSON text in UTF-8 encoding.
> 
> and add a suitable reference to UTF-8.
> 
> Just so.
> John Cowan          http://www.ccil.org/~cowan        [email-masked]
> As we all know, civil libertarians are not the friskiest group around --
> comes from  forever being on the qui vive for the sound of jack-booted
> fascism coming down the pike.           --Molly Ivins

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 8 Dec 2014 06:42:32 +0100

> On 7 dec 2014, at 22:07, John Cowan <[email-masked]> wrote:
> 

My point is that if you talk about what specific characters or reference RFC20 or what not, then you only get RS if you use UTF-8 encoding. If you use UTF-16, then you neither have RS as one octet (0x1E), nor is RS the only character that do have 0x1E as one of the octets.

I think the problem is that I do not know what "octet string" is. You either have UTF-8 encoded Unicode strings, or... ;-) In this case, you have a series of UTF-8 encoded Unicode Strings, right? Separated by the octet 0x1E, which happen to also be a correctly encoded Unicode character -- the Information Separator Two. This implies the whole thing is a UTF-8 encoded text that is to be parsed like this:

possible-JSON = 1*(not-RS); UTF-8-encoded JSON text
 ; (as specified in RFC7159, but only UTF-8 allowed)

I.e. the blob, to be compliant with this document, MUST be UTF-8 encoded JSON.

Right?

> and add a suitable reference to UTF-8.

I would say that what you have said above is:

This specifies a series of UTF-8 encoded Unicode strings. Each to be interpreted as JSON text. The strings are separated by the octet 0x1E (which is UTF-8 encoding of the Unicode Character U+001E - INFORMATION SEPARATOR TWO). This character because of this must be escaped, for example by using \u001E notation, if it exists in an attribute value.

>> blob can still start with U+FEFF?

Good.

   Patrik

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 8 Dec 2014 06:44:35 +0100

> On 8 dec 2014, at 03:59, Black, David <[email-masked]> wrote:
> 
> OLD

My point is that you do not have to talk about ASCII RS. You can as well talk about the UTF-8 encoding of the unicode character INFORMATION SEPARATOR TWO, U+001E. Much cleaner.

Just say it must be UTF-8 encoded text, done.

   Patrik

---

From: Tim Bray <[email-masked]>
Date: Sun, 7 Dec 2014 23:37:58 -0800

I agree 100% that the fact that JSON Text Sequences MUST BE UTF-8 should be
highlighted.  Both the ASCII and Unicode names of 30/0x1e/00011110 should,
for completeness, be provided.  It would be inappropriate to omit ASCII,
because we picked 0x1e because of its ASCII name.

On Sun, Dec 7, 2014 at 9:44 PM, Patrik Fältström <[email-masked]> wrote:

> My point is that you do not have to talk about ASCII RS. You can as well
> talk about the UTF-8 encoding of the unicode character INFORMATION
> SEPARATOR TWO, U+001E. Much cleaner.
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>
>

-- 
- Tim Bray (If you’d like to send me a private message, see
https://keybase.io/timbray)

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 8 Dec 2014 08:48:08 +0100

FWIW, I think this is a good view. I just felt that given the other discussion about RFC20, you do not HAVE TO talk about ASCII or the name of the code point in the standard. Sure, that is the reason you picked U+001E, but that is more a nice story ;-)

It was not my intention to say that you should not talk about ASCII.

   Patrik

> On 8 Dec 2014, at 08:37, Tim Bray <[email-masked]> wrote:
> 
> I agree 100% that the fact that JSON Text Sequences MUST BE UTF-8 should be highlighted.  Both the ASCII and Unicode names of 30/0x1e/00011110 should, for completeness, be provided.  It would be inappropriate to omit ASCII, because we picked 0x1e because of its ASCII name.
> 
> - Tim Bray (If you’d like to send me a private message, see https://keybase.io/timbray)

---

From: John Cowan <[email-masked]>
Date: Sun, 7 Dec 2014 13:05:28 -0500

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Micropayment advocates mistakenly believe that efficient allocation of
resources is the purpose of markets.  Efficiency is a byproduct of market
systems, not their goal.  The reasons markets work are not because users
have embraced efficiency but because markets are the best place to allow
users to maximize their preferences, and very often their preferences are
not for conservation of cheap resources.  --Clay Shirky

---

From: John Cowan <[email-masked]>
Date: Sun, 7 Dec 2014 16:07:54 -0500

For:

read:

Just so.

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Dec 2014 21:30:57 -0600

[I've been traveling, please excuse my late responses.]

On Sun, Dec 07, 2014 at 07:55:41PM +0100, Patrik Fältström wrote:
> The problem I have is that 7159 is not tight enough as it allows other
> encodings than UTF-8, which in turn make the encoding not work very
> well as this draft take for granted each one of the separator
> characters is one byte each.
> 

Right.  I'll add text to section 2.2 saying that the JSON texts have to
be encoded in UTF-8.

> Now, if this is NOT UTF-8, then this might be pretty bad situation.

Well, you can always fuzz test a parser... :)

But yes, the encoder should use UTF-8.

> What I am saying is that I would like this draft to explicitly say
> that the only profile of RFC7159 that can be used is when UTF-8 is in
> use, i.e. somewhere something like "The encoding MUST be UTF-8,
> although RFC7159 also allow other encodings, like UTF-16." Then in the
> security considerations section that "RFC7159 do allow not only UTF-8
> encoding but also for example UTF-16, which MIGHT create problems for
> a parser, all depending on what data is serialized."
> 

I agree with this.  This was always my intent (as in: I never intended
to support UTF-16 or UTF-32, say, or any other UTF, in any
implementation of mine).

And I agree that this format doesn't work with UTF-16 or UTF-32 EVEN IF
the UTF were part of the MIME type and UTF-specific multi-byte
separators were used: because at least for log-type applications where
atomicity of writes is in question multi-byte separators won't do.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Dec 2014 21:37:13 -0600

>    sequence is to be parsed as a JSON text in UTF-8 encoding.

Agreed.  And a corresponding change to section 2.2, which will now read:

   In prose: any number of JSON texts, each encoded in UTF-8, each
   preceded by one ASCII RS character, and each followed by a line feed
   (LF). Since RS ...

> and add a suitable reference to UTF-8.

Oh, eh, RFC7159 lacked such a thing.  At least this one should be
non-controversial: RFC3629.  (Right? :)

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Dec 2014 21:41:59 -0600

> because we picked 0x1e because of its ASCII name.

Good point.  (It's "Unicode Character 'INFORMATION SEPARATOR TWO' (U+001E)".)

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Dec 2014 22:19:42 -0600

> This is a combined Gen-ART and OPS-Dir review.  Boilerplate for both follows ...

Thanks you.

> of the JSON text that failed to parse.  I suggest:

Good point.

>    the remainder of the sequence from there.

That's a weird way of saying that, wherever the JSON text parse fails,
the sequence parser should then seek forward until the next RS-valued
byte.  But it works for me; I'll take it.

> to parse), and I think that's worth stating.

It's permitted, of course, with the caveat that all incremental parsers
have: the parse can ultimately fail.  I'll add this note:

   Incremental JSON text parsers may be used, though of course failure
   to parse a given text may result after first producing some
   incremental parse results.

to section 2.3.

> instead of the text parse.

I'd wanted to not have to list the characters that are considered
whitespace.  I agree that the "ws" rule is not appropriate though.

> be incomplete because the last character is not LF, e.g.:

A missing terminating LF is not a problem for strings, arrays, or
objects.  I seem to recall that we did discuss this.  We could require
that such texts fail to parse, but perhaps the more important thing is
to require common parser behavior as to such truncations.

You ABNF proposal is certainly more strict than the one in the I-D.  I'm
neutral as to whether this form or the one in the I-D (with the ws issue
fixed) is better.  The stricter form is clearly easier to talk about,
therefore preferable, but it will mean discarding texts where only that
terminating LF is truncated.

One problem we get into with your ABNF is that RFC5234 does not discuss
greediness (this came up on the list).  But this can be noted (see
below).

One nice thing about the stricter rules is that we need not discuss
top-level numbers (or booleans, or null) with normative text, just note
them.

> within a JSON text (and this should be stated).

It will if ABNF matching is greedy and possible-JSON is defined as

   possible-JSON = 1*( 1*(not-RS) LF); ...

Advice as to which form to take?

> considerations and incomplete texts are a relevant consideration for

And surely also for RFC7159.  Besides requiring that they fail to parse
(and the note about incremental parsing), are there any other missing
considerations?  Ah yes, smuggling of data in sequences where parsers
will ignore without warning any octet strings that fail to parse as JSON
texts.

Proposed text:

   As usual, parsers must operate on as-good-as untrusted input. This
   means that parsers must fail gracefully in the face of malicious
   inputs. Note that incremental parsers can produce partial results and
   later indicate failure to parse the remainder of a text. Note that
   texts that fail to parse and are ignored can be used to smuggle data
   past sequence parsers that don't warn about JSON text failures.

> that the author will resolve that during Authors 48 hours.

Sure.

>   ** Downref: Normative reference to an Unknown state RFC: RFC   20

Is this resolved by now?  I can always reference only Unicode.

Thanks for your excellent review,

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Dec 2014 22:27:13 -0600

On Mon, Dec 08, 2014 at 11:20:25PM -0500, John Cowan wrote:
> > This implies the whole thing is a UTF-8 encoded text that is to be
> > parsed like this:
> 
> No, this is a misunderstanding.  There is no requirement that the sequence
> *as a whole* is well-formed UTF-8 text.  For example, if the first JSON
> text is written only in part for whatever reason (system crash, etc.) to
> a log file, the next process can write a 0x1E byte and carry on.

Correct.  But we're splitting hairs: arbitrary octet strings (but for
0x1E) can be attempted to be parsed, though only those that are valid
JSON texts encoded in UTF-8 should be accepted.

The encoder, of course, must produce UTF-8.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Dec 2014 22:52:57 -0600

> yours aren't, maybe they *have* been tampered with.

Maybe each RFC should be like a commit in any modern version control
system, complete with a commit hash binding all past RFCs into each new
RFC.

Of course, that would really bind us to having canonical RFC
representations, and/or new renderings by the RFC-Editor added as
"commits".

Then we could reference RFCs as RFC-af551e0 (short-form) and
RFC-af551e089ca623216a312e475a6837de0aa7995b (long-form) and so on :) in
mailing list discussion, verbally, in RFCs as rendered, in other
documents, ..., and by doing so we'd be embedding the commit hashes of
the entire RFC series deeply into the Internet, in a way that would be
quite difficult to tamper with.

No digital signatures needed, just a decent hash function.

Or at least that's what I think Lloyd was suggesting.

I leave it to others to make a serious proposal along these lines.

(We can't quite adopt a VCS for this: we'd have to standardize it.)

Nico
--

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 9 Dec 2014 06:03:33 +0100

> On 9 dec 2014, at 04:37, Nico Williams <[email-masked]> wrote:
> 

Yes, correct.

   Patrik

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Dec 2014 23:09:25 -0600

On Sun, Dec 07, 2014 at 05:13:57PM -0500, John C Klensin wrote:
> There we agree, [...]

+1, and +1 to moving RFC20 to Standard.

> especially that authoritative copies be available online.

But perhaps there should be one as to new RFCs.  We mind (or should) our
copyright business nowadays so as to ensure such availability.  That
isn't enough to ensure the existence of an online archive, but then the
IETF and the RFC-Editor seem to strive to provide one (I should, but
don't, know whether and which RFC provides for that function, but if
there is none, that seems like a rather funny omission).

> regard.  If the IESG asked the community for permission to

This is certainly true.  We can't really require that external documents
referenced from RFCs be archived online.

> constructively, April 1 will be here soon.  :-(

We shouldn't invent new rules, no, but this seems like a nice place to
segway into proposing a new rule as to online availability of new RFCs.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Dec 2014 23:11:37 -0600

> > non-controversial: RFC3629.  (Right? :)

I received one off-list response that RFC3629 would be inappropriate
because Unicode is a versioned standard.  But here we want a UTF, not
all of Unicode, and surely UTF-8 is stable.

Nico
--

---

From: manning bill <[email-masked]>
Date: Mon, 8 Dec 2014 22:09:23 -0800

this seems silly.  one reason the rocs were widely distributed in both electronic and hardcopy forms was to enable running diffs to check.

/bill
PO Box 12317
Marina del Rey, CA 90295
310.322.8102

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 09 Dec 2014 16:08:50 +0900

> all of Unicode, and surely UTF-8 is stable.

Yes indeed.    Regards,   Martin.

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 9 Dec 2014 08:32:15 +0100

> On 9 dec 2014, at 05:20, John Cowan <[email-masked]> wrote:
> 

Ok, fair.

Then I think the draft itself should state this in one way or another. If nothing else the text you wrote here or a version of it is to go into the security considerations section.

   Patrik

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 9 Dec 2014 13:46:36 +0100

> On 9 dec 2014, at 06:11, Nico Williams <[email-masked]> wrote:
> 

RFC3629 has no limitation based on Unicode version. The limitation is that it only defines UTF-8 for the UTF-16 accessible range, U+0000..U+10FFFF.

   Patrik

---

From: "John Levine" <[email-masked]>
Date: 9 Dec 2014 16:36:04 -0000

>But perhaps there should be one as to new RFCs.

That means nobody can cite a printed book, or an article from an ACM
or IEEE journal.  (To the extent the latter two are online, they're
PDFs behind a paywall.)

This seems to me to be making rules for the sake of making rules, not
because there is a problem that needs to be solved.

R's,
John

---

From: "Black, David" <[email-masked]>
Date: Tue, 9 Dec 2014 16:41:12 +0000

Nico,

Thanks for the comprehensive response.

I've excerpted a few things for further comment - I'm fine with the
proposed actions for anything not mentioned here.

[A] JSON text parse failures

> byte.  But it works for me; I'll take it.

Your alternative wording "whenever the JSON text parse fails, ..." is fine.

[D] Truncation

> terminating LF is truncated.

I concur with both of the above paragraphs - my preference is to detect
incomplete JSON texts at the sequence level via the missing LF rather than
special-casing numbers and relying on failed JSON parses for everything else.
In general, earlier detection of errors increases the options for dealing
with them.

Once the incomplete text is detected, a JSON parse could be attempted,
with the JSON parser knowing that the text is incomplete (e.g., text
may fail to parse, a number at the end of the text must not be produced
as an incremental parse result).

I'll defer to the WG on whether/how to expand the decoder ABNF to better
detect incomplete texts and how to deal with any incomplete texts that are
detected.

As for RFC 20 ...

> Is this resolved by now?  I can always reference only Unicode.

Keep the RFC 20 reference - I have no problem with it.  Moreover, as a
result of all the hubbub around this nit, the IESG has issued a Last Call
to reclassify RFC 20 as an Internet Standard ... so that this never
arises again ...

http://www.ietf.org/mail-archive/web/ietf-announce/current/msg13546.html

Thanks,
--David

> From: Nico Williams [mailto:[email-masked]]
> Sent: Monday, December 08, 2014 11:20 PM
> To: Black, David
> Cc: General Area Review Team ([email-masked]); [email-masked];
> [email-masked]; [email-masked]

---

From: Dave Crocker <[email-masked]>
Date: Tue, 09 Dec 2014 08:45:28 -0800

> because there is a problem that needs to be solved.

Well, there is certainly a higher barrier to access for a paper document
or one behind a paywall than there is for something that is only one
click away.

It's probably reasonable that we document a preference that a 'direct'
link be available, rather than that it be required.  This doesn't mean
refraining from references to paper or paywall docs, but that we'd like
a direct link to be included.

Interestingly, this issue is the other side of our preference /not/ to
cite ephemeral web pages...

So I think this reduces to a combined preference about references:

   1. A reference needs to be to a stable, published copy that is likely
to be available long term.

   2. It is preferred that the citation include an online, direct
reference, independent of the 'primary' publication vehicle for the
document.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Dec 2014 11:01:24 -0600

> or one behind a paywall than there is for something that is only one

Whoops, I wasn't clear!  I meant that new RFCs must be available online
(obviously, old ones that can't be found online... might never be).

Further down in my post this became clear when I pointed out that we
can't require external docs to be available online.  But it should have
been clear here.

Nico
--

---

From: John Cowan <[email-masked]>
Date: Mon, 8 Dec 2014 23:20:25 -0500

> I.e. the blob, to be compliant with this document, MUST be UTF-8 encoded JSON.

The possible-JSON part is either UTF-8 encoded JSON or it's arbitrary junk.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Said Agatha Christie / To E. Philips Oppenheim
"Who is this Hemingway? / Who is this Proust?
Who is this Vladimir / Whatchamacallum,
This neopostrealist / Rabble?" she groused.
        --George Starbuck, Pith and Vinegar

---

From: "Heather Flanagan (RFC Series Editor)" <[email-masked]>
Date: Tue, 09 Dec 2014 04:16:40 -0800

> there is none, that seems like a rather funny omission).

I'm not sure if you're referring to just RFCs or all cited documents? 
In case of the latter, we do check to see when a URI is offered in the
reference whether or not that URI is valid and points to as stable a
place as is practicable.  But I don't see the RFC Editor ever
_requiring_ cited documents to be online.  For one thing, that's not
entirely my call, as it involves the content of the document as approved
by the IESG, IRSG, IAB, or ISE.  I define the format for the reference,
but reject a reference because it doesn't have a URI?  Not so much.  And
for another thing, I wouldn't do that even if I could.  It's a bad idea
given the inherent instability of the web; not all things stay online
forever.

If you're talking about making sure every single RFC is online and
cite-able as such, I point everyone to the RFC Online Project
<http://www.rfc-editor.org/rfc-online-2000.html>.  As for all RFCs
published in the last 27 years, we already do that.

> referenced from RFCs be archived online.

Indeed.  It has been discussed in the past, but that's rather an
enormous job that probably doesn't provide enough value to justify the
costs of storage, backups, and the legal advice that we'd need to do it
properly.

> segway into proposing a new rule as to online availability of new RFCs.

- -Heather

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Dec 2014 11:17:29 -0600

On Tue, Dec 09, 2014 at 04:41:12PM +0000, Black, David wrote:
> > [...]

OK.

> In general, earlier detection of errors increases the options for dealing

And, of course, a streaming/incremental parsers might well output all
there is to output when only the last LF is missing but the top-level
value was properly delimited anyways.  So it's kinda difficult to get a
fool-proof requirement that the trailing LF must be present.

Your review comments included adding this note about incremental
parsing.  There's a conflict here between the two comments that had not
been apparent to me last night.  I now think that fixing the ws problem
is the best way forward.

> as an incremental parse result).

That's so for non-incremental parsers.  (Or when buffering the complete
text instead of handling incrementally, even though one has an
incremental parser.)

Consider one implementation I'm familiar with.  Its JSON text parser is
incremental (but not streaming), so it produces outputs with no need for
extra whitespace when the input text is a string, array, or object, but
for top-level numbers, booleans, and null, it needs to either read one
more byte or reach EOF before it will output them.

So I think we really do need to say something about top-level numbers
(and true, false, and null), namely: that they must be delimited by
whitespace, that '<RS>1234<RS>' is not a valid sequence element because
the number may have been truncated.  (Ditto '<RS>true<RS>', since the
intended text could have been 'trueish', which is invalid of course, but
still.)

> to reclassify RFC 20 as an Internet Standard ... so that this never

Yes, I noticed.  I expect the IETF LC will pass for that.

Thanks,

Nico
--

---

From: "Black, David" <[email-masked]>
Date: Tue, 9 Dec 2014 18:49:35 +0000

> intended text could have been 'trueish', which is invalid of course, but

That would be more robust, as then all JSON texts in a sequence have
delimiters and absence of the closing delimiter clearly indicates
truncation.

Thanks,
--David

> Sent: Tuesday, December 09, 2014 12:17 PM
> To: Black, David

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 9 Dec 2014 17:33:05 -0500

> or one behind a paywall than there is for something that is only one

Yes, and a sensible author would provide the easy access method were
it stable.  But I'm not happy about making a new rule or even
publishing new guidance here, because this creates an opportunity for
yet more ratholes where people have a fight about whether something is
easy enough to get to (do you need to register?  Blah blah) or stable
enough (is online the "primary" publication? &c &c).  I agree with
John that this is making a rule where we don't need one.

A

-- 
Andrew Sullivan
[email-masked]

---

From: John C Klensin <[email-masked]>
Date: Tue, 09 Dec 2014 19:00:23 -0500

--On Tuesday, December 09, 2014 17:33 -0500 Andrew Sullivan
<[email-masked]> wrote:

>> Well, there is certainly a higher barrier to access for a
>> paper document or one behind a paywall than there is for
>> something that is only one click away.
> 
> Yes, and a sensible author would provide the easy access
> method were it stable.  But I'm not happy about making a new
> rule or even publishing new guidance here, because this
> creates an opportunity for yet more ratholes where people have
> a fight about whether something is easy enough to get to (do
> you need to register?  Blah blah) or stable enough (is online
> the "primary" publication? &c &c).  I agree with John that
> this is making a rule where we don't need one.

To add one observation to the latter, while Heather has said
that she will (or needs to) defer to the Streams when they
specify what they want and the various incarnations of the
downref rule were established to preserve the stability and
integrity of standards-track documents, we have traditionally
left these issues up to the discretion of the RFC Editor.  There
is an underlying principle in that which is the assumption that,
at any given time and as thing evolve, the RFC Editor will have
the specialized knowledge and insight to make such decisions (or
will know where to get it and when that is necessary) and good
judgment to apply it.  

The knowledge and perspective involved is something most of us
don't have and I suggest that parts of this "cited document"
discussion illustrate that.   If the RFC Editor doesn't have the
ability to make those decisions and/or we don't trust them to do
so, then the problem would be that we have the wrong person in
those job(s) and need to fix that.

For the record, I think we have the right people in the job and
that Heather, in combination with Sandy, Alice, and the other
Production Center folks, are perfectly capable of developing
reasonable policies and evolving them as appropriate.  If we
really think that new policies are needed, what the rest of us
--including the IAB, RSOC, and IAOC-- need to do, IMO, is to ask
them to review the citation policy and then get and stay out of
their way.

     john

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Dec 2014 18:49:30 -0600

OK.

New section 2.4 text:

   While objects, arrays, and strings are self-delimited in JSON texts,
   numbers, and the values 'true', 'false', and 'null' are not.  Only
   whitespace can delimit the latter four kinds of values.

   Parsers MUST check that any JSON texts that are a top-level number,
   or which might be 'true', 'false', or 'null' include JSON whitespace
   (at least one byte matching the "ws" ABNF rule from RFC7159) after
   that value, otherwise the JSON-text may have been truncated.  Note
   that the LF following each JSON text matches the "ws" ABNF rule.

   Parsers MUST drop JSON-text sequence elements consisting of
   non-self-delimited top-level values that may have been truncated
   (that are not delimited by whitespace).  Parsers can report such
   texts as warnings (including, optionally, the parsed text and/or the
   original octet string).

   For example, '<RS>123<RS>' might have been intended to carry the
   top-level number 123.4, but must have been truncated.  Similarly,
   '<RS>true<RS>' might have been intended to carry the invalid text
   'trueish'.  '<RS>truefales<RS>' is not two top-level values, 'true',
   and 'false'; it is simply not a valid JSON text.

This is the only place where the ws rule comes up, so merely saying "at
least one byte matching" it should suffice.

I'm also adding this following the above, based on your comment about
incremental parsers:

   Implementations may produce a value when parsing '<RS>"foo"<RS>'
   because their JSON text parser might be able to consume bytes
   incrementally, and since the JSON text in this case is a
   self-delimiting top-level value, the parser can produce the result
   without consuming an additional byte.  Such implementations should
   skip to the next RS byte, possibly reporting any intervening
   non-whitespace bytes.

(yes, I think this should be a 'should', not a 'SHOULD').

Nico
--

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 10 Dec 2014 10:52:22 +1000

> (yes, I think this should be a 'should', not a 'SHOULD').
​Not an 'ought to'?​

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Dec 2014 19:06:13 -0600

On Wed, Dec 10, 2014 at 10:52:22AM +1000, Matthew Kerwin wrote:
> Not an 'ought to'?

I'll take that.

---

From: "Black, David" <[email-masked]>
Date: Wed, 10 Dec 2014 02:52:34 +0000

This looks good, and the explanation of what's going on (numbers,
'true', 'false' and 'null' lack delimiters) is a useful addition.

One small nit:

	'<RS>truefales<RS>' is not two top-level values, 'true',
	and 'false'; it is simply not a valid JSON text.

truefales -> truefalse

Thanks,
--David

---

From: John Cowan <[email-masked]>
Date: Wed, 10 Dec 2014 20:31:37 -0500

Patrik Fältström scripsit:

> RFC3629 has no limitation based on Unicode version. The limitation
> is that it only defines UTF-8 for the UTF-16 accessible range,
> U+0000..U+10FFFF.

That is the complete Unicode range.  There are no Unicode scalar values
above U+10FFFF and never will be (as distinct from Unicode scalar values
that are not, or not yet, assigned to characters).  So that is no
limitation at all.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
Let's face it: software is crap. Feature-laden and bloated, written under
tremendous time-pressure, often by incapable coders, using dangerous
languages and inadequate tools, trying to connect to heaps of broken or
obsolete protocols, implemented equally insufficiently, running on
unpredictable hardware -- we are all more than used to brokenness.
                   --Felix Winkelmann

---

From: Julian Reschke <[email-masked]>
Date: Thu, 18 Dec 2014 15:16:17 +0100

> I already have; the status-change document is in Last Call Requested state.

So RFC 20 says it defines a "coded character set". However, in current 
specs (at least in APPS) we frequently talk about "character encoding 
schemes" (<http://tools.ietf.org/html/rfc6365#section-2>), in general 
mapping Unicode code points to octet sequences.

So does RFC 20 define a CES as well? If it does not, should we have an 
additional document taking care of this?

Best regards, Julian

---

From: [email-masked]
Date: Thu, 18 Dec 2014 07:22:09 -0800 (PST)

> mapping Unicode code points to octet sequences.

Actually, we try and talk about charsets, which are mappings from a series of
octets to a series of characters, and avoid all of the complicated ISO
bafflegab.

A CCS is a mapping from characters to integers. A CES is a mapping from one or
more sets of integers to octets. The combination of one or more CCSs with a CES
produces something that is usually (but not always) the same as a charset. The
distinction lies in the fact that a CCS/CES doesn't always fully specify the
meaning of all octet sequences, whereas a charset does.

The supposed utility of the complex CCS/CES approach lay in it's ability to
accomodate very complex CESs. This was thought to be the right way to do things
back in the days when no universal charset existed and thus multiple CCSs had
to be allowed in a single stream of octets.

For example, in X.400 you had the generaltext body part, which used ISO 2022 as
the CES combined with an essentially arbitrary set of CCSs that were specified
both inline and out of band.

But instead we ended up with multiple CESs, which were either profiled subsets
of ISO 2022 or schemes where the hi bit was essentially a CCS flag. It's much
more straightforward to handle such things as charsets, so that's what we did.

See RFC 2978 for additional details.

> So does RFC 20 define a CES as well?

No. Of course there's an obvious CES to associate with it: The mapping of the
128 integer values it defines to octets with the same value. Do that and you
essentially have the US-ASCII charset.

> If it does not, should we have an

I don't see why. The utility of RFC 20 lies in its specification of the meaning
of various characters. If you're using it as a specification for the US-ASCII
charset, you're using it incorrectly because like it or not, it doesn't specify
that. RFC 2046 does that by referencing ANSI X3.4-1986. No doubt it would
have been better to reference RFC 20 for the CCS part, but it wasn't online
at the time.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Thu, 18 Dec 2014 11:09:13 -0500

--On Thursday, December 18, 2014 15:16 +0100 Julian Reschke
<[email-masked]> wrote:

>... 
> So RFC 20 says it defines a "coded character set". However, in
> current specs (at least in APPS) we frequently talk about
> "character encoding schemes"
> (<http://tools.ietf.org/html/rfc6365#section-2>), in general
> So does RFC 20 define a CES as well? If it does not, should we
> have an additional document taking care of this?

With the understanding that RFC 20's being used successfully for
45 years continues to be a very strong argument that it doesn't
need changes or supplemental materials, and noting that many
IETF participants were not reading ANSI/USASA standards (or much
of anything else) 45 years ago,

(1) The terms "coded character set" and "[character] code for
information interchange" were in use for long before Unicode and
its multiple encodings/ representation forms started to redefine
it/them.   In this context, "long" is measured in decades, not
years.

(2) Early versions of ASCII did not specify what we would now
call "encoding" information.  It just specified repertoire and
associated 7 bit CCS.  Late ones, IIR, do specify encoding
information.  That type of difference is one of the reasons we
need to be careful about version numbers or dates when
referencing other people's standards (and why stable references
are important).  For ASCII, the result was that we ended up with
at least two different ways to put those 7 bit characters into
an 8 bit "byte" and at least two different ways to put them into
a 36 bit word.

(3) I assume partially because the encoding issues mentioned in
(2) had most people working on anything resembling applications
on the network to be familiar with the issues, RFC 20 does
specify an on-the-wire encoding for ASCII.  That is one of the
things that makes it more useful than a reference to ASCII
alone: it specifies what we started calling a "charset" in the
early MIME days, i.e., a combination between a CCS and a CES.

So, AFICT, nothing else needed in this area other than getting
on with it and ceasing to embarrass ourselves by needing to drag
out this discussion of a 45-year-old spec of something we've
used heavily and for which there has never been a problem for
what is now called the Basic Latin repertoire.

best,
     john

---

From: John C Klensin <[email-masked]>
Date: Thu, 18 Dec 2014 11:17:27 -0500

+1 to Ned's explanation.  His note crossed mine in the posting
queue and basically address different parts of the issue.
However, one mini-correction that, IMO, just strengthens both of
our points...

> essentially have the US-ASCII charset.

Actually, RFC 20 says, in its very first sentence, "...standard
7-bit ASCII embedded in an 8 bit byte whose high order bit is
always 0".   Unless I'm missing something, that is a mapping
from a CCS (although ASCII defined those integers in Column/Row
form rather than as single integers) and a CES.  So, possibly
modulo references to different versions of ASCII (I don't have
time to check whether the Charset definition for US-ASCII points
to the same version of US-ASCII that RFC 20 does), RFC 20 and
US-ASCII are more than just "essentially" the same".  

    john

---

From: [email-masked]
Date: Thu, 18 Dec 2014 16:05:50 -0800 (PST)

> form rather than as single integers) and a CES.

Yep, it's essentially a CES. The only thing missing is the
definition of the US-ASCII charset name.

>  So, possibly

The CCSs appear identical to me. There may be some subtle difference in how
some control is defined in the ANSI documents versus RFC 20, but that's getting
pretty picky.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Thu, 18 Dec 2014 23:47:36 -0500

--On Thursday, December 18, 2014 16:05 -0800 Ned Freed
<[email-masked]> wrote:

>> Actually, RFC 20 says, in its very first sentence,
>> "...standard 7-bit ASCII embedded in an 8 bit byte whose high
>> order bit is always 0".   Unless I'm missing something, that
>> is a mapping from a CCS (although ASCII defined those
>> integers in Column/Row form rather than as single integers)
>> and a CES.
> 
>> time to check whether the Charset definition for US-ASCII
>> points to the same version of US-ASCII that RFC 20 does), RFC
>> 20 and US-ASCII are more than just "essentially" the same".
> 
> The CCSs appear identical to me. There may be some subtle
> difference in how some control is defined in the ANSI
> documents versus RFC 20, but that's getting pretty picky.

If one pays attention only to the CCS and ignores character
semantics (as you note, especially for  C0 characters in columns
0 and 1), ASCII is unchanged from the time of its first
publication, i.e., stable both before and after RFC 20.

   john