Subject: Re: New Last Call: 'Tags for Identifying Languages' to BCP

From: Bruce Lilly <[email-masked]>
Date: Sun, 12 Dec 2004 13:10:00 -0500

>  Date: 2004-12-11 10:48
>  From: "Peter Constable" <[email-masked]>
>  To: [email-masked], [email-masked]
>  
> > -----Original Message-----
> > From: [email-masked] [mailto:ietf-languages-
> > [email-masked]] On Behalf Of Bruce Lilly

> > My comments are in response to the "New Last Call" made on
> > the ietf-announce list.  They are in response to the text which
> > accompanied that new last call and the text of
> > draft-phillips-langtags-08.txt dated November 2002.  The
> > specific claim that accessibility has been a problem was made in
> > the text accompanying the new last call
> 
> I don't know where the statement accompanying the announcement came from,

According to the "New Last Call" issued by the IESG Secretary,
the text is "Author's discussion of drivers for this work".

> You singled out that one point to comment on as though it were the main factor.

I mentioned a matter which was repeatedly indicated as a
factor for existing implementations and with which I
strongly disagree.

There are points with which I do not necessarily disagree,
and there are points with which I have not yet had time to
study in detail, due to the surprise of the announcement
of an impending decision (I do not understand why no
announcement of work on an RFC 3066 replacement was made
to the ietf-822 list, especially as the core Internet
protocols discussed there are affected by this draft),
the shortness of the time before a decision (deadline for
comments was given as 5 Jan 2005), and the impending
holidays.

[regarding the proposed registry vs. internationally-
standardized ISO lists for subtag definitions]
> It is certainly the case that only it should be consulted for determining what sub-tags are valid with what denotation, which was the intent.

That is a problem for existing implementations of RFC 3066
tags, which can obtain official, internationally agreed
descriptions of the codes in two languages.

> By looking in the sub-tag registry. If ISO changed the meaning of "US" to something other than what it is now, its meaning for purposes of use in an IETF language tag would not change, because it would remain stable in the sub-tag registry. You would be fairly well protected against the whim of politicians.

OK, continuing your hypothetical example and its relationship
to language, suppose that there is another civil war and
that what now corresponds to "US" is split into Blue America
and Red America.  Further suppose that in due course ISO
assigns some other code to one of those countries and retains
"US" for the other, and that that happens after the proposed
registry is set up with a definition for "US" and some
description referring to the "old" use.  Now suppose that one
wishes to produce an appropriate language tag for the text
"moral values" (which clearly has different meaning in Blue
America (telling the truth, admitting to mistakes, etc.) and
in Red America (imposing totalitarian control over others)).
How specifically would the proposed registry handle such a
change in the meaning of "US", and how would the registry
help differentiate the meaning of a 1990's "en-us" tag to
that of the hypothetical time described?

I suspect that it won't help, and I recommend review of
how another artifact of politics (viz. time zones) are
handled by the (unofficial) database of time zones
maintained at ftp://elsie.nci.hih.gov/pub/tzdata2004g.tar.gz.
The format used handles multiple changes in definitions
that went into effect at different times, something that
the proposed registry doesn't appear to handle.
  
> > > But if the proposed new registry's description of "CS" says
> > "foo" and the ISO standard code list says "bar", what's
> > an implementor supposed to present to a user as *the*
> > description associated with "CS"?
> 
> The *meaning* of the sub-tag is determined by the sub-tag registry. If you want human-readable descriptors,

The draft says that the proposed registry will contain a
description, in English (only).

> you already have to look beyond the ISO standards for anything more than English and French

But existing RFC 3066 implementations can get official
descriptions in *both* of those languages; the proposal
would adversely affect those existing implementations by
eliminating the French description.

Of course, it is a more serious defect of the proposal
that it would fail to reflect internationally-agreed
codes and would fail to keep pace with changes...

> it would not be new that you have to look beyond the registry itself to decide what human-readable descriptors you should provide in a product.  

It would be new that one could not find a standard
(i.e. official) French-language description in the
list of codes.

> > One possibility would be two description fields.  But the
> > registry would need a charset closer to ISO-8859-1 than
> > to ANSI X3.4 as currently specified.  Or an encoding
> > scheme.
> 
> Personally, I don't see the value in something like that. Given the intent to have a registry that can be machine-readable, changing its charset from ANSI X3.4 in order to gain descriptors in just one more language is not worth it IMO. 

Fine, use utf-8, which encompasses ANSI X3.4 and
ISO-8859-1 (plus others).  The point is that ANSI
X3.4 is inadequate.
 
> Speaking at least for Microsoft, we're interested in having descriptors in far more than two languages, and we certainly would not blindly base the descriptors we present to our customers solely on what a registry provides, no matter what its charset.

Surely in going from two (the current situation per
RFC 3066) to "more than two" indicates that decreasing
to one (as in the draft proposal) is heading in the
wrong direction.  It certainly invalidates the claim
that the proposal is compatible with existing
implementations, at least one of which does make use
of the descriptions currently provided in both
languages in the ISO lists specified by RFC 3066.
 
> As you know, the 'grandfathered' production is loose in the ABNF given in the draft, but is very tightly constrained elsewhere in the draft: it is limited to only items registered under RFC 1766 or RFC 3066 up to the date of acceptance of this proposed spec. (In fact, only a subset of those, all explicitly identified in the sub-tag registry.) On the date of acceptance, you will be able to know precisely what the valid tags that fit under the 'grandfathered' production are and will forever be, and it is 100% guaranteed that none of them will have any of the forms that seem to concern you.

I see no guarantee that a future revision won't use
such forms on the basis that they are permitted
by the ABNF.

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Bruce Lilly <[email-masked]>
Date: Sun, 12 Dec 2004 14:27:12 -0500

>  Date: 2004-12-11 11:53

> Our disagreement amounts to a basic question of whether parsers should be written based on the ABNF alone, or based on the ABNF plus other constraints provided in the spec. Clearly, I think anyone writing a parser should consider other constraints as well.

No, I agree that a parser should take normative text
into account, but I feel that there should be a
reasonable effort made to make the ABNF agree with
that normative text -- otherwise there's little
point in providing ABNF.

> As mentioned, the limit is imposed by other tight constraints on 'grandfathered'; you have already identified that the longest registered tag under RFC 3066 is 11 octets in length, therefore a 'grandfathered' tag can be at most 11 octets in length.

But the constraints probably aren't as tight as you
believe; the draft specifically permits a future
revision to allow a primary subtag longer than
8 octets, or not purely alphabetic, etc.

> a de-facto upper limit of 11 (subject to change if new tags are registered before the proposed spec is accepted).

We're agreed on that, for the present draft, but
apparently Mark Davis disagrees.  And I am concerned
about the loophole left for future revisions.

> > > We could impose some upper limits on these things...
> 
> > That leaves the extension portions' length at up to
> > 25 * (1 + 1 + 8 * 9) = 1850 octets, not taking any other parts
> > of a tag into account!   That's way too long (the RFC 2047
> > limit for an encoded-word is 75 octets, including charset tag,
> > some text, and some syntactic glue in addition to the language
> > tag).
> 
> The problem already exists in RFC 3066. Even apart from private-use tags, tomorrow someone could request a registration for a tag that's 87 octets long, and there's nothing in RFC 3066 that would prohibit acceptance.

One would hope that under RFC 3066 rules, that the
reviewer, a list subscriber, or an Applications Area
Director would recognize the conflict with RFCs 2047/2231
and would object.  If indeed that were to happen
literally tomorrow, I am quite sure that an objection
would be made.  The situation is quite different
under the draft proposal, where registration of a
complete tag is not required, and where there are
no upper bounds on length of a tag.

> > > So, I think Bruce has identified a valid issue here. I personally would
> > > not have characterized it as greatly exacerbating, though,
> > 
> > IMO, an increase from 11 octets worst-case, which is tolerable
> > for constructing RFC 2047/2231 encoded-words, to >> 1850
> > octets, which exceeds by a large margin what can be handled
> > in a Content-Language or Accept-Language message header
> > field, constitutes "greatly exacerbated".
> 
> Repeating my previous point, RFC 3066 doesn't stop a registered tag from being 10^100 octets in length.

RFC 3066 provides a registration mechanism that can be
trusted to prevent that; in particular, the Applications
Area Directors are supposed to look out for issues affecting
the core Internet applications protocols.

> I suggest that wording be added to the draft giving a strong recommendatation to users that they not use tags the complete length of which exceeds 75 characters.

75 octets would be too large for a language-tag used in
an encoded word (perhaps different limits could be
specified for different uses, but one would have to be
careful about implicit re-use between applications). An
encoded-word has the form:
  =?<charset>*<language-tag>?<encoding>?<text>?=
and is limited to a total of 75 octets. Eliminating the
syntactic glue (7 octets, unbracketed above) leaves a
total of at most 68 octets for text, charset, encoding,
and language-tag.  There are at present two encodings,
specified with 1-octet tags.  Assuming that longer
encoding tags are not required, that leaves 67 octets
for charset, language-tag, and text.  The text must be
at least four octets in order to accommodate B encoded
text, leaving 63 octets at most for charset and
language-tag (ideally, one would prefer to leave more
room than that for text).  It is guaranteed (in theory,
if not in practice) that there will be a charset name
of no more than 40 octets for each charset, but that is
not necessarily the preferred name (there has been some
discussion about possibly reducing that limit). That
leaves about 23 octets for a language-tag as an upper
bound for use in an encoded-word.  Obviously that
hasn't been a problem in practice to date; the longest
registered language tag is less than half that length.

> > By deferring to the bilingual ISO lists for language and country
> > tags, 3066 at least provided a minimal degree of internationalization.
> > By explicitly limiting description fields to English and restricting
> > the charset to US-ASCII, the draft proposal takes a giant leap
> > backwards.
> 
> The US-ASCII limitation existed in RFC 3066, so is not new. 

No, I'm talking about the character set of the description,
which currently resides in the ISO lists, and is certainly
not limited to ANSI X3.4 in those lists.  Under the draft
proposal, the description is limited to ANSI X3.4, which
is a problem for the description for UN region 248, whose
description includes an A-ring character, which is not in
X3.4.  I note that BCP 18 section 3.1 specifies that it MUST
be possible to use the UTF-8 charset, so the specification
of the registry as solely X3.4 appears to violate that
provision of BCP 18.
 
> On the more general point, I believe you are mistaking i18n concerns with localization concerns:

No, I am concerned about changing what is currently
internationalized (to an admittedly small extent) into
something that is strictly monolingual in a severely
restricted charset.

> > > I don't quite understand what the critique is here: what is there to
> > > internationalize about language tags?
> > 
> > There should probably be a reference (at least informative)
> > pointing to BCP 18 and mentioning that the language tags
> > defined provide a means of labeling the language of text,
> 
> Have you not read the abstract in the draft?
[...]
> Or the introduction?

I have; neither mentions BCP 18 or the core Internet
protocols.

> > The draft (if/when approved) should also indicate that
> > it updates BCP 18, which refers to RFC 1766.
> 
> Is this right? This draft is not a replacement for RFC 2277, or an addendum to it. RFC 2277 also refers to RFC 1958, which was updated by RFC 3439, but surely RFC 3439 doesn't state that it updates BCP 18? (RFC 227 does have a section with significant overlap in topic, though, so perhaps this makes sense. I'm not well-enough versed in IETF document process to know.)

N.B. "update" != "replacement".  If the draft obsoletes
3066, which obsoletes 1766, then it affects 2277.  3066
should probably have so indicated also...
 
> > Given the divergence noted above from RFC 3066's use
> > of multilingual reference lists, the Internationalization
> > considerations section should include a synopsis of the
> > approach chosen (viz. to restrict description to English) and
> > the rationale for that choice (see BCP 18 section 6).
> 
> Again, this is a localization issue, not an internationalization issue. I do not consider this necessary or even appropriate.

No, it's relevant to the extent that BCP 18 specifies
that text strings are subject to internationalization,
and the description field in the draft-proposed registry
protocol certainly appears to be a text string (although
the draft does not clearly state whether it is a text
string or a protocol element).

> > > >     implications (ISO 8601 date format parsing).
> > >
> > > As mentioned above, this really is a non-issue.
> > 
> > It's an issue (esp. in light of the finger pointing regarding
> > accessibility to ISO 639/3166).
> 
> As has been pointed out, there is no such finger-pointing in the draft.

The finger-pointing accompanied the new last call and was
used as justification for replacement of RFC 3066 with the
proposed scheme.  If indeed accessibility is a non-issue,
then the justification for the proposed scheme, in whole
or in part, rests solely on other considerations, such as
they might be.

> > Again, it is an issue that imposes requirements on language
> > tag parsers.  What you've shown is that the ABNF is not
> > consistent with what was desired to be expressed, and
> > that makes it an issue that needs to be addressed.
> 
> Again, I believe the bigger issue is not getting the ABNF to express what was desired

No it is a concern because of a loophole left for future
revisions to incorporate syntax which is not currently
permitted by 3066, but which is inexplicably permitted by
the draft's proposed ABNF.

> > > The maximal length issue exists just as much
> > > in RFC 3066 due to private-use tags; it is a technical concern that
> > > might worth reviewing in RFC 3066bis, however; but it is not
> > > insurmountable, and not a new problem.
> > 
> > Private-use carries its own considerable baggage; aside from
> > that, the draft proposal increases the length of non-private
> > tags that affect both protocol design and implementations
> > from a worst case maximum of 11 octets under RFC 3066...
> 
> Worst case at present; a month from now it could be unlimitedly larger.

No, the current registration review process in conjunction
with the requirement would prevent that.  The draft proposal
decouples use from registration, which directly leads to
unlimited length, and changes the review process (in unclear
ways which I have not yet had time to review fully).

> But I've accepted that it would be an improvement to add constraints on overall length. 

That's a start.

---

From: Bruce Lilly <[email-masked]>
Date: Sun, 12 Dec 2004 14:52:19 -0500

>  Date: 2004-12-11 11:59
>  From: "JFC (Jefsey) Morfin" <[email-masked]>
> Gentlemen,
> I see several points discussed here which are/are not of the same order and 
> seem confusing the issue.
> 
> 1. the discussion creeps from Harald's RFC 3066 to Multilingual Internet. 
> It seems strange to discuss byte oriented details without having first a 
> Multilingual framework telling what is the scope of the discussion and its 
> implications (which are certainly major) on the whole Internet 
> architecture. I submit that an IAB guidance is first necessary. Before 
> going any further a true WG-Multilingualism should be created and open to 
> everyone (a private IETF-Language lists should be an interim situation 
> towards such a WG)

There is in fact an "ietf-languages" list; RFC 3066 and the
draft under discussion give its submission mailbox as
"[email-masked]", which makes finding the real
list an exercise since IANA's web site makes no mention
of any mailing lists.  I made an educated guess that I
might find the list at alvestrand.no, and indeed the
list submission mailbox is "[email-masked]",
and the list archive is available at
http://www.alvestrand.no/mailman/listinfo/ietf-languages

Neither RFC 3066 nor the draft provide any instruction
for joining the mailing list, and from the remarks above
it should be clear that IANA's web site provides no clear
clue either.
 
> 2. I see quoted "RFC 3066bis" as a document. The RFC Editor seems to ignore 
> that RFC? Where can I find it?

It is apparently an unofficial term for the Phillips draft
mentioned in the new last call and to which you have
repeated the URI.
 
> 3. there are at least four different levels:
> 
> - what is Multilingualism vs. vernacularism (there are 6000 human languages 
> but a standard should be able to support non scripted and computer 
> generated and past languages, what may lead to millions of references).

One should then consider different types of tags for
different uses -- a tag for a non-scripted language
makes no sense in an RFC 2047/2231 encoded-word,
which is strictly text.
 
> - vernacular granularity has nothing to do with geography and countries. 

True in general; but can we reverse the precedent
set by RFC 1766?

> The way this inserts into the general digital convergence (is the IANA the 
> proper register?).
[...]
> The same as the IANA is not in the business of defining 
> countries (Jon Postel, RFC 1591) it should not be in the business of 
> defining languages.

The draft in question apparently seeks to get IANA into the
business of defining countries (and languages), usurping
those roles from ISO (as also noted in RFC 1591).

> I also submit that IANA is not the proper place anymore to support such a 
> Register. Experience shown that IANA (now a function of ICANN) is subject 
> to controversies in this or in parallel real life areas: ccTLD delegation, 
> ccTLD entries in the root file, accepted MINC reaction to the Polish non 
> concerted introduction of Arabic, Russian and Hebraic tables, ICANN 
> strategy for internationalized rather than multilingual TLDs, etc. I also 
> submit that UNESCO, MPEG or other standard/cultural organizations involved 
> in the daily reality (universities, editors, posts, governments, 
> copyrights, WIPO, etc. etc.) are more concerned and may make their own 
> standard prevail after an unnecessary and harassing dispute. It seems that 
> any semantic able to support open sub-tags whatever they originate from, is 
> useful. Going any further would push in favor of a less and less 
> [unilingual or internationalized] network centric market against a 
> market  evolution toward user centric [multilingual/multiulcural] networked 
> relations [P2P, VoIP, NAT, coreboxes, OPES, etc.].

Good points all, though I do sympathize with the concern about
loss of information about what a tag meant at a given time
due to changes in the ISO lists.  I would support provision for
a definitive time-stamped registry of changes of some sort;
ideally that would be provided by ISO as part of (or a supplement
to) the lists, and I would be quite surprised if ISO were not
receptive to such a suggestion if made appropriately and with
a clear indication of the problems.

---

From: John Cowan <[email-masked]>
Date: Sun, 12 Dec 2004 15:54:33 -0500

> list submission mailbox is "[email-masked]",

Both addresses seem to work for posting to this list.  Not all
IETF lists have ietf.org or iana.org mailing addresses anyhow;
consider [email-masked], which is not a W3C mailing list but an IETF one.

> those roles from ISO (as also noted in RFC 1591).

This is doubly incorrect.  To begin with, ISO defines neither countries
nor languages.  UNSD defines country-like objects for its purposes and
assigns them numeric codes, specifying them using English and French
names.  Then ISO assigns alphabetic identifiers to the names.  Languages
are not defined at all; ISO assigns alpha and numeric identifiers to
certain words which it believes to be the names of languages, without
always specifying exactly which language among those so named is meant.
Note the titles of the ISO 3166 and 639 standards.

The proposed registry will merely serve to stabilize the ISO mappings,
making it less likely that they will be gratuitously changed, because
it will not be under the control of an MA with unfettered discretion to
make changes.

-- 
On the Semantic Web, it's too hard to prove     John Cowan    [email-masked]
you're not a dog.  --Bill de hOra               http://www.ccil.org/~cowan