Subject: Comments on draft-dusseault-caldav-15 and draft-newman-i18n-comparator-14

From: Julian Reschke <[email-masked]>
Date: Sat, 23 Sep 2006 11:20:04 +0200

Hi,

I notice that draft-dusseault-caldav-15 now delegates part of the text 
matching semantics to draft-newman-i18n-comparator-14, which probably is 
a good thing. As far as I can tell, this change was made in following up 
to Sam H's comments in 
<https://datatracker.ietf.org/public/pidtracker.cgi?command=view_comment&id=55575>:

> Section 9.6 contains a facility for doing substring matching on
> internationalized strings but contains no discussion of normalization,
> limited discussion of case folding, and no discussion of other
> matching issues that have been problematic for LDAP, IMAP and other
> IETF technologies.  This seems quite inadequate.  If the
> internationalization community within the IETF is quite sure that this
> is OK, I'll remove the discuss.  I just don't see why text matching in
> caldav should be so much easier than IDN or LDAP.

But as a matter of fact, draft-newman-i18n-comparator-14 doesn't define 
any collations that would actually solve the Unicode NF issue, so it's 
not really clear how this helps CalDAV (except that it now uses a 
framework in which the solution may become available in the future).

Maybe the set of initial registrations in 
<http://tools.ietf.org/html/draft-newman-i18n-comparator-14#section-9> 
needs to be extended?

Best regards, Julian

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Sat, 23 Sep 2006 12:56:08 +0200

> <http://tools.ietf.org/html/draft-newman-i18n-comparator-14#section-9> 

Two more used to be there, but I split them off for various reasons and 
will them as submit them as separate drafts. As it is, the document 
establishes a framework and registers the legacy collations from 2244.

I expect that i;basic (the UCA, 
http://www.unicode.org/reports/tr10/tr10-14.html) will go to RFC 
quickly and that i;nameprep will expire as an internet drafts.

Arnt

---

From: Sam Hartman <[email-masked]>
Date: Sun, 24 Sep 2006 14:33:15 -0400

>>>>> "Julian" == Julian Reschke <[email-masked]> writes:

    Julian> But as a matter of fact, draft-newman-i18n-comparator-14
    Julian> doesn't define any collations that would actually solve
    Julian> the Unicode NF issue, so it's not really clear how this
    Julian> helps CalDAV (except that it now uses a framework in which
    Julian> the solution may become available in the future).

Yep, but at least it points to a solution.  I'd be much happier if
there was a mandatory colation that was a good fit for international
calendar usage.  I understand thogh that we have to balance forward
progress against forcing people to solve everything all at once.

---

From: Lisa Dusseault <[email-masked]>
Date: Mon, 25 Sep 2006 11:07:32 -0700

On Sep 23, 2006, at 2:20 AM, Julian Reschke wrote:

>
> But as a matter of fact, draft-newman-i18n-comparator-14 doesn't  
> define any collations that would actually solve the Unicode NF  
> issue, so it's not really clear how this helps CalDAV (except that  
> it now uses a framework in which the solution may become available  
> in the future).
>
> Maybe the set of initial registrations in <http://tools.ietf.org/ 
> html/draft-newman-i18n-comparator-14#section-9> needs to be extended?

Yes, I agree. That's one of the next steps and why a registry was  
created (so we could do it outside the base comparator draft).

Last week Ted & I were discussing whether one could define a Very  
Liberal Comparator (VLC) for general use.  It would be handy to have  
one which matched e with E, é, è É... and matched o with O, ø, ô, and  
so on.  That would help in calendar searching use cases, e.g. a user  
who can't type in accents (or doesn't know how) wants to find the  
invitation from André by searching for "andre".  It would probably be  
useful in many other cross-language or unknown-language situations too.

Such a comparator would be most useful for exact and substring  
matches; I don't know offhand how it would best do ordering so it  
might not be as useful for ordering.

I believe Arnt intends to continue working on this general problem,  
for which I'm very grateful, and other contributions would be most  
welcome.

Lisa

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 25 Sep 2006 20:42:53 +0200

Lisa Dusseault writes:
> invitation from André by searching for "andre".

You're talking about i;basic;level=1 ;)

You may also be talking about i;basic: During its life as part of 
draft-newman-i18n-comparator the default level was partly 1 and partly 
3 (IIRC and don't hold me to it). I don't know which level is the best 
default (which was one of the reasons for splitting it out).

Comments welcome (to me personally, please): Which level is the right 
default? http://unicode.org/reports/tr10/#Multi_Level_Comparison lists 
the possibilities.

Arnt

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 26 Sep 2006 13:09:34 +0200

Lisa Dusseault wrote:
 
> one could define a Very Liberal Comparator (VLC) for general
> use.  It would be handy to have one which matched e with E,
> é, è É... and matched o with O, ø, ô, and so on.

Good idea.  I'm thinking of something in that direction for my
mail filters:  At the moment any typo in the local part could
meet spamcop without any questions asked, clearly not optimal.

Frank

---

From: John C Klensin <[email-masked]>
Date: Tue, 26 Sep 2006 08:57:57 -0400

>> But as a matter of fact, draft-newman-i18n-comparator-14
>> doesn't   define any collations that would actually solve the
>> Unicode NF   issue, so it's not really clear how this helps
>> CalDAV (except that   it now uses a framework in which the
>> solution may become available   in the future).

Please watch for the final version of draft-iab-idn-nextsteps
(probably to be posted as RFC 4690 within the next few days) and
for draft-???-idnabis-issues-00 (soon).  Neither "solves" the NF
problem, but they may help make it more clear why the NF problem
is not solvable in any general case.  It can be solved for
particular languages or, more specifically, particular
orthographies of particular languages.  But, as long as we are
operating at the "Unicode" level, without specific
language-identifying information transmitted in-band every time
we transmit a string, there is no general solution.

Fortunately, I don't believe that issue is in the critical path
of the base comparator document.

>> <http://tools.ietf.org/ 
> Yes, I agree. That's one of the next steps and why a registry
> was  created (so we could do it outside the base comparator
> draft).
> 
> Last week Ted & I were discussing whether one could define a
> Very  Liberal Comparator (VLC) for general use.  It would be
> handy to have  one which matched e with E, é, è É... and
> matched o with O, ø, ô, and  so on.  That would help in
> calendar searching use cases, e.g. a user  who can't type in
> accents (or doesn't know how) wants to find the  invitation
> from André by searching for "andre".  It would probably be

Arggh.  The difficulty here is that, for some scripts and
languages a "decorated" version of a base characters can be, by
convention or the natural properties of the language, replaced
by an undecorated version.  For others, the decorations actually
form different characters, with different phonetics, names, and
other properties (Unicode character names do not consistently
reflect this distinction, partially because it is impossible to
do to.  To pull two examples out of idn-nextsteps, a very
liberal comparator should let "ö" and "ø" match for some
well-known Scandinavian languages and neither should match "o".
But, in German, "ö" should generally match "oe", but not
vice-versa.  Perhaps it should match "o" as well, but that would
be controversial.

This set of problems actually gets worse as on moves outside
Roman-derived scripts, even though the Roman-derived scripts
probably have the richest collection of characters whose glyph
forms are decorated versions of other characters.

So, by all means do this if you think it is useful -- and I
agree that it might be-- but please give it a value-neutral
name, not, e.g., "very liberal".

Again, not in the critical path for the comparator document, IMO.

> matches; I don't know offhand how it would best do ordering so
> it  might not be as useful for ordering.

Ordering is even more tightly tied to the "different character"
versus "decorated version of existing character where the
decorations are semi-optional" distinction.  I suggest that
trying to do it in a general way will lead either to frustration
or to serious errors.

> I believe Arnt intends to continue working on this general
> problem,  for which I'm very grateful, and other contributions
> would be most  welcome.

I very much appreciate his efforts in the area, wish him luck,
and  hope that the community will be tolerant of efforts that
meet specific needs and are clearly identified with those needs.

    john

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 26 Sep 2006 16:14:29 +0200

I've got a "very liberal" comparator in production at the Linux Counter 
project.

It mostly works, because the  people who assign names tend to not assign 
names that collide under the comparator.
But the strings that result from the comparator are quite distant from 
"expected" strings in many cases - in particular, "asciifying" Japanese 
with a heuristic that works OK for Chinese Mandarin gives output that's 
simply meaningless.

Better Than Nothing Matching......

            Harald

---

From: Sam Hartman <[email-masked]>
Date: Tue, 26 Sep 2006 16:44:28 -0400

>>>>> "John" == John C Klensin <[email-masked]> writes:

    John> --On Monday, 25 September, 2006 11:07 -0700 Lisa Dusseault

    >>> solution may become available in the future).

    John> Please watch for the final version of
    John> draft-iab-idn-nextsteps (probably to be posted as RFC 4690
    John> within the next few days) and for
    John> draft-???-idnabis-issues-00 (soon).  Neither "solves" the NF
    John> problem, but they may help make it more clear why the NF
    John> problem is not solvable in any general case.  It can be
    John> solved for particular languages or, more specifically,
    John> particular orthographies of particular languages.  But, as
    John> long as we are operating at the "Unicode" level, without
    John> specific language-identifying information transmitted
    John> in-band every time we transmit a string, there is no general
    John> solution.

    John> Fortunately, I don't b

I understand the IAB may be sold on this belief and it may even be
true.  However, we do not ye.yet have an IETf consensus that this is
the case and we need to get there one way or another.

Since the publication of stringprep, the security community has (and
continues) to go full speed ahead on the assumption that we are
developing one stringprep profile for a protocol or group of related
protocol and that this will solve our comparison issues for most of
our protocol.

Changing that for security protocols is going to be significantly more
problematic than changes proposed for IDN.

So, people who have worked in relatively closed groups to come to
these conclusions for the specific domain of DNS really need to start
building a broader consensus--understanding the needs of the rest of
the IETF and working with the rest of the IETF to explain their
conclusions.

I don't say you are wrong--simply that you need to start the consensus
building and that I'm alarmed at your result.

In particular, in the security area, it would probably be strongly
desirable for us to have a discussion in San Diego.

---

From: [email-masked]
Date: Mon, 25 Sep 2006 18:28:05 +0000

> for which I'm very grateful, and other contributions would be most  

	http://author.handalak.com/archives/042003/000078

--bill