Subject: Re: [Json] secdir review of draft-ietf-jsonbis-rfc7159bis-03

From: Elwyn Davies <[email-masked]>
Date: Sun, 12 Mar 2017 23:07:40 +0000

(with half a Gen-art hat on...)
Does the WG really want to revisit the anguished discussions that resulted in the changes to Section 8.1 of draft-ietf-json-rfc4627bis between versions 07 and 08 back in late November 2013?
See https://www.ietf.org/mail-archive/web/json/current/msg02053.html and many, many messages beore this.
Cheers,Elwyn

Sent from Samsung tablet.
-------- Original message --------From: Peter Cordell <[email-masked]> Date: 12/03/2017  09:06  (GMT+00:00) To: Ned Freed <[email-masked]>, Julian Reschke <[email-masked]> Cc: [email-masked], John Cowan <[email-masked]>, [email-masked], [email-masked], [email-masked], Benjamin Kaduk <[email-masked]> Subject: Re: [Json] secdir review of draft-ietf-jsonbis-rfc7159bis-03 
On 11/03/2017 15:41, Ned Freed wrote:
>> On 2017-03-11 03:08, John Cowan wrote:
>> >
>> > On Thu, Mar 9, 2017 at 12:53 AM, Benjamin Kaduk <[email-masked]
>> > <mailto:[email-masked]>> wrote:
>> >
>> >     If that's what's supposed to happen, it should probably be more
>> >     clear, yes.  (But aren't there texts that have valid
>> interpretations
>> >     in multiple encodings?)
>> >
>> >
>> > Not if the content is well-formed JSON and the only possible encodings
>> > are UTF-8, UTF-16, and UTF-32.  It suffices to examine the first four
>> > bytes of the input.  If there are no NUL bytes in the first four bytes,
>> > it is UTF-8; if there are two NUL bytes, it is UTF-16; if there are
>> > three NUL bytes, it is UTF-32.  This works because the grammar requires
>> > the first character to be in the ASCII repertoire, and the NUL
>> > *character* (U+0000) is not allowed at all.
>
>> Good explanation. Maybe the spec should include it.
>
> +1
>
> This exact issue just came up in a media type review, where someone
> specified a charset parameter because they weren't aware of this algorithm.
>
> It would be very helpful to have this text in the RFC.

Although it does need slightly more detail to take into account 
endian-ness in the case of UTF-16 and -32.

The XML spec may offer some example text:

https://www.w3.org/TR/2008/REC-xml-20081126/#sec-guessing

Pete Cordell
Codalogic Ltd
Read & write XML in C++, http://www.xml2cpp.com

---

From: Julian Reschke <[email-masked]>
Date: Mon, 13 Mar 2017 07:23:20 +0100

On 2017-03-13 00:07, Elwyn Davies wrote:
> Does the WG really want to revisit the anguished discussions that
> resulted in the changes to Section 8.1 of draft-ietf-json-rfc4627bis
> between versions 07 and 08 back in late November 2013?
>
> Cheers,
> Elwyn

No, but on the other hand, we should acknowledge that apparently the 
text both about what's mandatory and how auto detection works is not as 
clear as it should.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 13 Mar 2017 16:56:33 +0100

+1000

Grüße, Carsten

> On 13 Mar 2017, at 08:51, Martin J. Dürst <[email-masked]> wrote:
> 
> On 2017/03/13 15:23, Julian Reschke wrote:
> It looks to me as if at the time of the above message in the WG, the chairs were successful in presenting a consensus, probably at a stage when the participants in the discussion where getting tired.
> 
> It seems that when put in the wider context of the IETF, that compromise now looks somewhat shaky.
> 
> My personal opinion is that we could try to fix this by changing the following:
> 
>>>>> 
>  JSON text SHALL be encoded in UTF-8, UTF-16, or UTF-32 [UNICODE]
>  (Section 3).  The default encoding is UTF-8, and JSON texts that are
>  encoded in UTF-8 are interoperable in the sense that they will be
>  read successfully by the maximum number of implementations; there are
>  many implementations that cannot successfully read texts in other
>  encodings (such as UTF-16 and UTF-32).
>>>>> 
> 
> to something like the following:
> 
>>>>> 
>  JSON text SHOULD be encoded in UTF-8 [UNICODE]
>  (Section 3).  JSON texts that are
>  encoded in UTF-8 are interoperable in the sense that they will be
>  read successfully by the maximum number of implementations.
> 
>  There are
>  many implementations that cannot successfully read texts in other
>  encodings (such as UTF-16 and UTF-32). JSON text MAY be encoded in
>  UTF-16 or UTF-32 [UNICODE] (Section 3) if the sender is sure that
>  the intended recipients can read them.
>>>>> 
> 
> That should then go together with a MIME registration that only lists UTF-8.
> 
> Regards,   Martin.
> 
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Stefan Hagen <[email-masked]>
Date: Mon, 13 Mar 2017 17:01:52 +0100

Yes, tired is a good match && I second the proposed change.
"Stefan"

On 13/03/17 08:51, Martin J. Dürst wrote:
> It looks to me as if at the time of the above message in the WG, the
> chairs were successful in presenting a consensus, probably at a stage
> when the participants in the discussion where getting tired.
>

---

From: Peter Cordell <[email-masked]>
Date: Mon, 13 Mar 2017 16:33:33 +0000

>    the intended recipients can read them.

My only thought is to perhaps reflect that JSON isn't only transmitted, 
and JSON can be used for file based configuration etc, (even if this 
isn't strictly IETFs concern).  So perhaps s/sender/encoder/ in the last 
sentence, plus a few other tweaks yielding something like:

     There are many implementations that cannot successfully read texts
     in other encodings (such as UTF-16 and UTF-32 [UNICODE]).  JSON
     text MAY be encoded in other encodings if the encoder is sure that
     the intended recipients can read them.

Pete.
-- 
---------------------------------------------------------------------
Pete Cordell
Codalogic Ltd
http://json-content-rules.org/
---------------------------------------------------------------------

---

From: "Matthew A. Miller" <[email-masked]>
Date: Mon, 13 Mar 2017 11:54:31 -0600

>     the intended recipients can read them.

/me doffs hat

I like this change myself.

/me dons hat

As I recall, the table was removed mostly because the vast majority of
implementations did not support any encoding other than UTF-8, and no
one (that I recall) reported implementing the detection table.

- m&m

Matthew A. Miller
< http://goo.gl/LM55L >

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 13 Mar 2017 20:07:56 +0100

On 13 Mar 2017, at 18:54, Matthew A. Miller <[email-masked]> wrote:
> no

There even was one that implemented something for UTF-16/32: Jackson.
The author said:

"Most other
libraries just punt the problem by requiring caller to pass in Reader that
does byte->char decoding”

(He did not say whether his code was ever exercised outside of his test cases :-)

He also mentioned that, by contrast, most XML libraries do implement auto-detection (as they should).

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Mon, 13 Mar 2017 20:51:27 +0100

On 2017-03-13 18:54, Matthew A. Miller wrote:
> ...
> ...

Well, if we allow UTF-16 and UTF-32 - even if we discourage their use - 
we should say how to detect those...

Alternatively we could forbid them, but that would be a normative change 
from RFC 7159.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 13 Mar 2017 20:57:00 +0100

On 13 Mar 2017, at 20:51, Julian Reschke <[email-masked]> wrote:
> 
> Well, if we allow UTF-16 and UTF-32 - even if we discourage their use - we should say how to detect those...

Consistency is the hobgoblin of the little minds.
(Apparently, not having that nonsense in there did not hurt for 7159?
If people want to implement something useless and cannot figure this stuff out, should we help them?)

Putting back any form of apparent support for UTF-16 and UTF-32 is a regression.

Grüße, Carsten

---

From: John Cowan <[email-masked]>
Date: Tue, 14 Mar 2017 09:58:39 -0400

On Mon, Mar 13, 2017 at 3:57 PM, Carsten Bormann <[email-masked]> wrote:

The actual quotation says "foolish consistency".  But not all consistency
is foolish.

> Putting back any form of apparent support for UTF-16 and UTF-32 is a

In my opinion, we should either take all references to UTF-16 or UTF-32
out, or add back a correct detection algorithm. UTF-8 is what is actually
interoperable, and interoperability is what RFCs are supposed to be all
about.

--