Subject: Re: i18n requirements (was: Re: NF* (Re: PKCS#11 URI slot attributes & last call))

From: Nico Williams <[email-masked]>
Date: Thu, 1 Jan 2015 21:01:35 -0600

On Wed, Dec 31, 2014 at 10:41:28AM -0500, John C Klensin wrote:
> [...]

In the case of PKCS#11 there's not a lot in the way of security
considerations regarding normalization: all the "devices" in question
are trusted, and the user is supposed to be in physical possession of
them.  As usual, we assume local security.  Therefore there's no
question of an attacker trying to fool a user into entering their PINs
into fake tokens.

This makes the security considerations regarding normalization simpler
in this case.

I think we could use some text like this:

   PKCS#11 does not specify a canonical from for UTF-8 string slots in
   the API.  This presents the usual false negative and false positive
   (aliasing) concerns that arise when dealing with unnormalized
   strings.  Because all PKCS#11 items are local and local security is
   assumed, these concerns are mainly about usability.

   In order to improve the user experience, applications that create
   PKCS#11 objects or otherwise label tokens, SHOULD normalize labels to
   NFC.  For the same reason PKCS#11 libraries, slots (token readers),
   and tokens SHOULD normalize their names to NFC.  When listing
   libraries, slots, tokens, or objects, an application SHOULD normalize
   their names to NFC.  When matching PKCS#11 URIs to libraries, slots,
   tokens, and/or objects, applications may use form-insensitive Unicode
   string comparison for matching, as the objects might pre-date these
   recommendations).

Then later in the security considerations section, add something like:

   PKCS#11 does not authenticate devices to users; PKCS#11 only
   authenticates users to tokens.  Instead, local and physical security
   are demanded: the user must be in possession of their tokens, and
   system into whose slots the users' tokens are inserted must be
   secure.  As a result, the usual security considerations regarding
   normalization do not arise.  For the same reason, confusable script
   issues also do not arise.  Nonetheless, it is best to normalize to
   NFC all strings appearing in PKCS#11 API elements.

Nico
--

---

From: Patrik Fältström <[email-masked]>
Date: Fri, 2 Jan 2015 07:57:18 +0100

> On 2 jan 2015, at 04:01, Nico Williams <[email-masked]> wrote:
> 

For the record, I like what is suggested the below.

   Patrik

---

From: John C Klensin <[email-masked]>
Date: Fri, 02 Jan 2015 09:30:22 -0500

> For the record, I like what is suggested the below.

Wfm, but please note the other comments in my prior note.  To
summarize, normalization is not the only issue.  Even for some
normalization cases, NFC is not the cure.  For example, NFKC is
needed to rationalize characters of various widths but causes
problems elsewhere.  So, I would suggest some additional words
that say, more or less, that, until PKCS#11 is revised to be
clear about how to handle characters outside the ASCII
repertoire, those who discover a need to use such characters
should be cautious, conservative, and expend extra effort to be
sure they know what they are doing and that failure to do so
will create both operational and security risks.

>>   PKCS#11 does not specify a canonical from for UTF-8 string
>>   slots in the API.  This presents the usual false negative
>>   and false positive (aliasing) concerns that arise when
>>   dealing with unnormalized strings.  Because all PKCS#11
>...

---

From: Patrik Fältström <[email-masked]>
Date: Fri, 2 Jan 2015 16:19:32 +0100

> On 2 jan 2015, at 15:30, John C Klensin <[email-masked]> wrote:
> 
> 
> 
> normalization cases, NFC is not the cure.

This is why I think the way NFC is mentioned is correct.

> For example, NFKC is

Having this clarification explicit and not only implicit would make the text even better.

   Patrik

---

From: Nico Williams <[email-masked]>
Date: Fri, 2 Jan 2015 11:05:58 -0600

> normalization cases, NFC is not the cure.  For example, NFKC is

Indeed, which is why the proposed text also mentions confusable scripts.
(I didn't include references, but they could be added, all as
informative given the special circumstances here).

> will create both operational and security risks.

I'm not opposed.  I just don't think this will really add value.

Thanks for your and Patrick's help BTW (my apologies if my earlier reply
to you seemed caustic; I tend to get annoyed when told that the IETF
"doesn't do APIs" :/).

Happy new year,

Nico
--

---

From: Jan Pechanec <[email-masked]>
Date: Sat, 3 Jan 2015 18:42:25 -0800 (PST)

On Wed, 31 Dec 2014, Nico Williams wrote:

>> Alternate suggestion in the interest of getting this out and
>> recognizing that this is mostly a PKCS#11 problem (presumably
>> ITU and/or RSA, but what do I know) and shouldn't be an IETF one:
>
>RSA.

	for the record, PKCS#11 management has been transitioned to 
OASIS in 2013.  There is a recent draft out for version 2.40 and 
supposedly it should become a new version of the standard soon.  2.30 
draft from 2009 never made it so the last version is still 2.20 from 
2004.

	I do not see anything about UTF-8 normalization and/or i18n in 
2.40.  I will reach out to them and ask whether that was a deliberate 
decision.  However, I think we can proceed with the URI draft 
independently to that if it is updated as discussed here.

	I'm just going through this whole thread.

	regards, Jan.

-- 
Jan Pechanec <[email-masked]>

---

From: Jan Pechanec <[email-masked]>
Date: Sat, 3 Jan 2015 19:58:52 -0800 (PST)

On Wed, 31 Dec 2014, Nico Williams wrote:

>I'm in favor of saying "do form-insensitive comparison".  I'd settle for
>"always normalize to NFC when creating PKCS#11 resources" because, after
>all, that's what IRIs need anyways, it's just that a PKCS#11 URI-using
>app might not be in a position to make "normalize on create" happen.

	we could recommend the normalize-before-matching approach.  
Even for objects (keys), the application could use only non-UTF-8 
value attributes in the PKCS#11 search template.  Then, go through all 
returning objects and for UTF-8 value attributes, do NFC normalization 
first before matching them.

	however, I think that "SHOULD" would be too strong.  I think 
it could be mentioned side by side to the warning text based on what 
John noted about situations with a need to use non-ASCII characters.

	Jan

---

From: Nico Williams <[email-masked]>
Date: Sat, 3 Jan 2015 22:55:43 -0600

> John noted about situations with a need to use non-ASCII characters.

"RECOMMEND" and "SHOULD" mean the same thing.  Even if you try to use
"recommend" outside RFC2119 usage, it still kinda means "SHOULD", but
"should", "ought to" -- these can be taken to mean "not quite SHOULD".

Nico
--

---

From: Jan Pechanec <[email-masked]>
Date: Sat, 3 Jan 2015 21:58:17 -0800 (PST)

On Thu, 1 Jan 2015, Nico Williams wrote:

	Nico, many thanks for the drafted text and also to Patrik and 
John for discussing it.

	I've updated the draft in sections on URI matching guidelines,
URI comparision, added a new section on I18n, and added a new paragraph
to the Security considerations.  Individial diffs inline, a draft for
new draft 18 attached (draft-pechanec-pkcs11uri-18-v1.txt).

>   string comparison for matching, as the objects might pre-date these

	I've created "Internationalization Considerations" section and 
put the text above there after I slightly modified it.  I wanted to
mention CK_UTF8CHAR type so that it's clear what is discussed.

768	6.  Internationalization Considerations

770	   The PKCS#11 specification does not specify a canonical form for
771	   strings of characters of the CK_UTF8CHAR type.  This presents the
772	   usual false negative and false positive (aliasing) concerns that
773	   arise when dealing with unnormalized strings.  Because all PKCS#11
774	   items are local and local security is assumed, these concerns are
775	   mainly about usability.

777	   In order to improve the user experience, applications that create
778	   PKCS#11 objects or label tokens, SHOULD normalize labels to NFC.  For
779	   the same reason PKCS#11 libraries, slots (token readers), and tokens
780	   SHOULD normalize their names to NFC.  When listing PKCS#11 libraries,
781	   slots, tokens, and/or objects, an application SHOULD normalize their
782	   names to NFC.  When matching PKCS#11 URIs to libraries, slots,
783	   tokens, and/or objects, applications MAY use form-insensitive Unicode
784	   string comparison for matching, as those might pre-date these
785	   recommendations.  See also Section 3.5.

	in section 3.5 on URI Matching Guidelines, I've added the
following as the last paragraph of the section (it was based on John's
note from his last email).  This paragraph might not be necessary there
and the first part could be moved to the I18N section but I think it's
good to put it to where attribute matching is discussed so that it is
not easily overlooked.

513	   As noted in Section 6, the PKCS#11 specification is not clear about
514	   how to normalize UTF-8 encoded Unicode characters [RFC2279].  Those
515	   who discover a need to use characters outside the ASCII repertoire
516	   should be cautious, conservative, and expend extra effort to be sure
517	   they know what they are doing and that failure to do so may create
518	   both operational and security risks.  It means that when matching
519	   UTF-8 string based attributes (see Table 1) with such characters,
520	   normalizing all UTF-8 strings before string comparison may be the
521	   only safe approach.  For example, for objects (keys) it means that
522	   PKCS#11 attribute search template would only contain attributes that
523	   are not UTF-8 strings and another pass through returned objects is
524	   then needed for UTF-8 string comparison after the normalization is
525	   applied.

>   NFC all strings appearing in PKCS#11 API elements.

	I've added the following to the Security Considerations 
section (again, slightly modified, I'd rather not use "PKCS#11" as 
an alias for the specification):

807	   The PKCS#11 specification does not provide means to authenticate
808	   devices to users; it only allows to authenticate users to tokens.
809	   Instead, local and physical security are demanded: the user must be
810	   in possession of their tokens, and system into whose slots the users'
811	   tokens are inserted must be secure.  As a result, the usual security
812	   considerations regarding normalization do not arise.  For the same
813	   reason, confusable script issues also do not arise.  Nonetheless, it
814	   is best to normalize to NFC all strings appearing in PKCS#11 API
815	   elements.  See also Section 6.

	on top of that, I've added the following sentence to 3.6. PKCS#11 URI
Comparison section:

532	   strictly avoiding false positives.  When working with UTF-8 strings
533	   with characters outside the ASCII character sets, see important
534	   caveats in Section 3.5 and Section 6.

	the attribute Table 1 now also states which attributes are 
UTF-8 strings so that it's clear without consulting the spec.

	thank you, Jan.

---

From: Jan Pechanec <[email-masked]>
Date: Sun, 4 Jan 2015 06:53:10 -0800 (PST)

On Sun, 4 Jan 2015, Jaroslav Imrich wrote:

>Hello Jan,
>
>the text mentions "NFC" but I was unable to find any pointer or reference
>to its definition. I was not familiar with the term before so I think
>reference in the text would be helpful.

	hi Jaroslav, it's defined in the Unicode standard which I can 
reference directly but you would still need to search for the term 
rather than read the whole standard.

	please google "unicode NFC", all first hits lead to the right 
information.

	Jan.

>
>Regards, Jaroslav
>
>
>On Sun, Jan 4, 2015 at 6:58 AM, Jan Pechanec <[email-masked]>
>wrote:
>
>> _______________________________________________
>> saag mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/saag
>>
>>
>

---

From: Christian Huitema <[email-masked]>
Date: Mon, 5 Jan 2015 01:04:07 +0000

> rather than read the whole standard.

Of course it is defined in the context of Unicode. But then, in a generic networking context, NFC also stands for near-field communication. So it is definitely good practice to spell the acronym out in the text on first use.

-- Christian Huitema

---

From: Jan Pechanec <[email-masked]>
Date: Sun, 4 Jan 2015 20:13:24 -0800 (PST)

On Mon, 5 Jan 2015, Christian Huitema wrote:

>Of course it is defined in the context of Unicode. But then, in a 
>generic networking context, NFC also stands for near-field 
>communication. So it is definitely good practice to spell the acronym 
>out in the text on first use.

	hello Christian, yes, I will spell it.  I can see that I can 
also reference only Annex 15 that defines the normalization forms.  
The text now says:

777	   In order to improve the user experience, applications that create
778	   PKCS#11 objects or label tokens SHOULD normalize labels to
779	   Normalization Form C (NFC) [UAX15].  For the same reason PKCS#11

	with the new reference:

854	   [UAX15]    Davis, M., Ed., Whistler, K., Ed., and Unicode Consortium,
855	              "Unicode Standard Annex #15 - Unicode Normalization Forms,
856	              Version Unicode 7.0.0", June 2014.

	v2 of upcoming draft 18, draft-pechanec-pkcs11uri-18-v2.txt, 
is attached.

	thank you, Jan.

---

From: Jaroslav Imrich <[email-masked]>
Date: Sun, 4 Jan 2015 12:40:22 +0100

Hello Jan,

Regards, Jaroslav

---

From: Jan Pechanec <[email-masked]>
Date: Wed, 7 Jan 2015 11:16:05 -0800 (PST)

On Sat, 3 Jan 2015, Jan Pechanec wrote:

	hi, I haven't received any other comments on the draft 
recently (I know the LC already ended on Dec 29 though) so I think I 
can file changes discussed and drafted in this thread as draft 18 on 
Friday.  Thank you all for feedback, I really appreciate it.

	one more change for the draft 18 (v2 attached) is to spell 
"NFC" and reference the Unicode Annex on normalization based on 
comments from Jaroslav and Christian.

    In order to improve the user experience, applications that create
-   PKCS#11 objects or label tokens, SHOULD normalize labels to NFC.  For
-   the same reason PKCS#11 libraries, slots (token readers), and tokens
...
+   PKCS#11 objects or label tokens SHOULD normalize labels to
+   Normalization Form C (NFC) [UAX15].  For the same reason PKCS#11
+   libraries, slots (token readers), and tokens SHOULD normalize their
...

+   [UAX15]    Davis, M., Ed., Whistler, K., Ed., and Unicode Consortium,
+              "Unicode Standard Annex #15 - Unicode Normalization Forms,
+              Version Unicode 7.0.0", June 2014.

	best regards, Jan.

---

From: Jan Pechanec <[email-masked]>
Date: Thu, 8 Jan 2015 10:45:38 -0800 (PST)

On Thu, 8 Jan 2015, John C Klensin wrote:

>> recently (I know the LC already ended on Dec 29 though) so I
>> think I  can file changes discussed and drafted in this thread
>> as draft 18 on  Friday.  Thank you all for feedback, I really
>> appreciate it.
>> 
>> "NFC" and reference the Unicode Annex on normalization based
>> on  comments from Jaroslav and Christian.
>>...
>
>Jan,
>
>I don't have a lot of time to spend on this and am not an expert
>on either X.509 or PKCK (#11 or otherwise).  At least the first
>may be unfortunate, but it is what it is.
	
	hi John, I very much appreciate time you already spent on 
this.  Please see my comments inline.

>While I think the changes you have made are definitely
>improvements, this i18n stuff is complicated.  As with Security,
>there is a completely inadequate supply of magic pixie dust that
>can be thrown at problems to make them go away.  "Normalize to
>NFC" (with spelling-out and references) is a vast improvement or
>"use [valid] UTF-8" but there are many other issues.  You have
>noted some and omitted others.  For example, case-independent
>matching is a very simple and completely deterministic issue for
>ASCII (one essentially just masks off one bit within a certain
>range), it can get very messy if one tries to be sensitive to
>different locales that have different conventions about what to
>do with diacritical marks when lower-case characters are
>converted to upper case.  There are Unicode "CaseFold" rules

	I understand from the previous discussion that the topic is a 
very complex one and that my draft needed to acknowledge that.

<...>

>I don't know how far in explaining this your document should go.
>I would urge, as I think I did before, some fairly strong
>warnings that, at least until the issues are clarified in
>PKCS#11 itself, one should be very certain one knows what one is
>doing (and what the consequences of one's choices will be) if
>one decides to move beyond the safety and general understanding
>of the ASCII/ ISO 646/ IA5 letter and digit repertoire.  That
>sort of warning should supplement your NFC language, not replace
>it-- neither is a substitute for the other.   Whether you
>incorporate it or not, your I-D should not assume that, by
>saying "NFC", you have somehow resolved the full range of issues
>in this area, any more than saying "UTF-8" did. 

	I understand that.  The note about spelling NFC was on top of 
the first changes I incorporated.  I don't know if you saw those, I 
know there were many emails and your time you could spend on this is 
very limited.  So, in section on URI matching, I tried to be very 
explicit and based the warning I added on one of your comments from 
the previous discussion:

+   As noted in Section 6, the PKCS#11 specification is not clear about
+   how to normalize UTF-8 encoded Unicode characters [RFC3629].  Those
+   who discover a need to use characters outside the ASCII repertoire
+   should be cautious, conservative, and expend extra effort to be sure
+   they know what they are doing and that failure to do so may create
+   both operational and security risks.  It means that when matching
+   UTF-8 string based attributes (see Table 1) with such characters,
+   normalizing all UTF-8 strings before string comparison may be the
+   only safe approach.  For example, for objects (keys) it means that
+   PKCS#11 attribute search template would only contain attributes that
+   are not UTF-8 strings and another pass through returned objects is
+   then needed for UTF-8 string comparison after the normalization is
+   applied.

	do you suggest a stronger warning than that?

	more on that was incorporated into a new Internationalization 
Condiderations section, based on new text drafted by Nico:

+6.  Internationalization Considerations
 
+   The PKCS#11 specification does not specify a canonical form for
+   strings of characters of the CK_UTF8CHAR type.  This presents the
+   usual false negative and false positive (aliasing) concerns that
+   arise when dealing with unnormalized strings.  Because all PKCS#11
+   items are local and local security is assumed, these concerns are
+   mainly about usability.
+
+   In order to improve the user experience, applications that create
+   PKCS#11 objects or label tokens SHOULD normalize labels to
+   Normalization Form C (NFC) [UAX15].  For the same reason PKCS#11
+   libraries, slots (token readers), and tokens SHOULD normalize their
+   names to NFC.  When listing PKCS#11 libraries, slots, tokens, and/or
+   objects, an application SHOULD normalize their names to NFC.  When
+   matching PKCS#11 URIs to libraries, slots, tokens, and/or objects,
+   applications MAY use form-insensitive Unicode string comparison for
+   matching, as those might pre-date these recommendations.  See also
+   Section 3.5.

	and a new paragraph was also added to the existing Security 
Considerations section:

+   The PKCS#11 specification does not provide means to authenticate
+   devices to users; it only allows to authenticate users to tokens.
+   Instead, local and physical security are demanded: the user must be
+   in possession of their tokens, and system into whose slots the users'
+   tokens are inserted must be secure.  As a result, the usual security
+   considerations regarding normalization do not arise.  For the same
+   reason, confusable script issues also do not arise.  Nonetheless, it
+   is best to normalize to NFC all strings appearing in PKCS#11 API
+   elements.  See also Section 6.

	I think these new paragraphs convey the message that users 
should very careful when using characters outside ASCII, and what to 
do to mitigate problems that can arise from such use.  Do you think 
more should be said in the draft itself?

>For more information, you might have a look at some of the
>PRECIS work, notably draft-ietf-precis-framework.
>
>I also remain convinced that the best place to fix this is in
>the PKCS#11 spec itself.  One is always at a disadvantage when
>trying to work around an inadequate specification in a different
>specification that has to depend on it and your work is no
>exception.  I wish there were whatever liaison arrangements
>between the IETF and others (presumably notably RSA) to be sure
>that happened or at least there was clear awareness on the PKCS
>side of the deficiencies.

	last week I did contact OASIS PKCS 11 TC which is where 
PKCS#11 moved to since 2013.  However, even if the issue is going to 
be fixed there I don't think it will be in new version 2.40 which is 
close to be published.

	Happy New Year to you, too.

	best regards, Jan.

---

From: John C Klensin <[email-masked]>
Date: Thu, 08 Jan 2015 13:01:17 -0500

Jan,

that are at least self-consistent but which contain wjat amount
to exceptions for some language contexts (e.g., for dotless "i")
but they are wildly unpopular in some places.

We used to joke that, every time we tried to carefully examine a
new script and set of languages for IDN-related purposes, it was
like turning over rocks with vipers hiding under them.  Each new
script or language context turned up a different set of
difficult issues -- the only surprise what what sort of
creatures crawled out, not whether there would be creatures
there.  The joke has gone out of fashion, but the realities that
inspired it survive.

Part of this is an inherent problem with trying to create a
universal character set -- languages and writing systems are
diverse enough that any "one size fits all" model or set of
decision rules is guaranteed to be deeply problematic and
upsetting for some people (and legitimately so) while developing
too many script-specific (or language-specific) rules or
exceptions is almost certain to upset those who feel a need for
simpler approaches that can be incorporated into generic
software.  For your reading pleasure,
draft-klensin-idna-5892upd-unicode70 discusses one set of cases
in which application of different rules and criteria led to a
conclusion that may be just right for some communities but that
is definitely problematic for others.

Happy New Year,
    john