Subject: Re: New Last Call: 'Tags for Identifying Languages' to BCP

From: Bruce Lilly <[email-masked]>
Date: Sun, 12 Dec 2004 20:59:45 -0500

>  Date: 2004-12-12 17:34
>  From: "Mark Davis" <[email-masked]>
>  To: [email-masked], [email-masked]
>  CC: [email-masked]
>  
> > Are you claiming that
> >
> > sr-CS-891-boont-gaulish-guoyu-boont-gaulish-guoyu-boont-gaulish-guoyu
> 
> > is nonconformant per some specification in the draft
> > proposal?
> 
> Clearly not. But
> 
>   x-sr-CS-891-boont-gaulish-guoyu-boont-gaulish-guoyu-boont-gaulish-guoyu

So what? A private-use tag has to be agreed to by the communicating
parties; in this case they'll find that such an unwieldy tag is
unusable in an encoded-word and will have to agree to use something
more manageable.  That's a problem for the parties involved and
nobody else, since it doesn't affect the rest of us.  That's a
different matter from a public tag that everybody is expected to
be able to use.

> is already absolutely conformant with the current RFC 3066. And the current
> RFC 3066 clearly permits the registration of something as long as
> 
>   sr-CS-891-boont-gaulish-guoyu-boont-gaulish-guoyu-boont-gaulish-guoyu
> 
> (although of course this particular combination would certainly never get
> in).

I agree that that would never be registered -- because of the
review process which is part of RFC 3066.  But the draft under
discussion has no mechanism to prevent it, unlike 3066. 

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "Mark Davis" <[email-masked]>
Date: Fri, 10 Dec 2004 18:22:53 -0800

> So, I think Bruce has identified a valid issue here. I personally would
> not have characterized it as greatly exacerbating, though, as the issue
> was present in RFC 3066: private-use tags did not need to be registered
> in RFC 3066, so there was no way in implementation could be written with
> certain knowledge that tags beyond some given length would not be
> encountered.

As you say, the situation is no different now than in RFC 3066 in terms of
the possible length. I am somewhat sympathetic to the idea of having some
total limit (except for the late date for the proposed change). However, we
got considerable pushback on having RFC 3066bis make any previously valid
RFC3066 tag be invalid, and any length restriction would do that.

‎Mark

----- Original Message ----- 
From: "Peter Constable" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Sent: Friday, December 10, 2004 17:03
Subject: RE: New Last Call: 'Tags for Identifying Languages' to BCP

Resuming my comments:

> -----Original Message-----
> From: [email-masked] [mailto:ietf-languages-
> [email-masked]] On Behalf Of Bruce Lilly

[snip]

> Specifically, the draft allows, and RFC 3066 disallows:
>    subtags more than 8 octets in length
>    hyphens which do not separate subtags
>    zero-length subtags
>    primary tags which are not purely alphabetic
> Curiously, all of those are permitted by the draft ABNF
> production "grandfathered"...

The "grandfathered" production in the current draft is

grandfathered   = ALPHA *(alphanum / "-")

which does permit the sequences claimed by Bruce (except for
not-purely-alphabetic primary sub-tags), syntactically; but the set of
tags available for use is constrained by more than the ABNF syntax
alone: the acceptable productions for each sub-tag must either be taken
from one of the source standards or be registered. This is no different
from RFC 3066, so it is no more of a problem in this specification than
it was in RFC 3066.

It might be that the wording in 2.2 could be tightened up to eliminate
any possible question regarding the source for "grandfathered"
productions. Maybe it's not as obvious to someone coming to this cold as
it for us who have been discussing it for the past year.

Alternately, there's no reason why the "grandfathered" production
shouldn't be composed exactly to match what was used in RFC 3066:

grandfathered = 1*8ALPHA *("-" 1*8alphanum)

So, perhaps there is room for technical improvement, but there are not
any serious problems IMO -- certainly nothing as serious as the tone of
Bruce's conveyed.

> I see no reason for the ABNF to permit such content as is
> forbidden by RFC 3066; the actual ABNF for what RFC 3066
> permits is contained within 3066, and could have been directly
> incorporated rather than producing a "grandfathered"
> production which opens up several cans of worms.

This vastly overstates the problem. There is no can of worms unless it
exists in tags currently available under RFC 3066.

> One defect related to tag length in RFC 3066 is not remedied
> by the draft; indeed the problem is greatly exacerbated...

> Unfortunately, a language- tag's length is unlimited by
> the ABNF in RFC 3066 (due to an unlimited number of subtags)
> and in the draft...

> In particular, tags other than private-use tags with more than
> two subtags require registration under RFC 3066 rules, and it
> is a trivial matter to determine the longest registered tag.
> The draft, however, encourages use of more subtags as well as
> removal of the subtag length upper bound; moreover, it permits
> infinite numbers of subtags without requiring registration of
> the resulting complete tag.

Bruce states incorrectly that there is no upper bound on the length of
sub-tags. His other concern, on the overall length of complete tags, is
valid, however: in terms of the ABNF syntax for both RFC 3066 and RFC
3066bis, infinite-length productions are possible, but RFC 3066 would
require registration of complete non-private-use tags while RFC 3066bis
does not.

There are three open doors for infinite-length productions in the ABNF
of the current draft:

- unlimited extlang sub-tags
- unlimited variant sub-tags
- the number of possible extensions is limited to 25, but the length of
extensions is unlimited

We could impose some upper limits on these things; e.g.

Language-Tag = ... *8("-" extlang) ... *8("-" variant) ... 1*25("-"
extension)
...
extension = singleton 1*8("-" 2*8alphanum)

If we also imposed limits on the length of private-use tags and defined
the grandfathered production in a way that made clear there was an upper
limit for those, then we could end up eliminating an issue that had
existed in RFC 3066.

So, I think Bruce has identified a valid issue here. I personally would
not have characterized it as greatly exacerbating, though, as the issue
was present in RFC 3066: private-use tags did not need to be registered
in RFC 3066, so there was no way in implementation could be written with
certain knowledge that tags beyond some given length would not be
encountered.

> > The new registry provides a complete,
> > easily parseable file which provides the precise the contents of
valid tags for
> > any point in time.
>
> That is the first time I have ever heard ISO 8601 date
> format described as "easily parseable".  Perhaps the draft
> authors meant to say that a specific subset of the tortuously
> complex ISO 8601 date format is used, but that is not what
> the draft states...

It seems very clear that the authors intended only a specific subset:
YYYY-MM-DD. This is a minor technical issue that the authors can very
easily remedy.

> I am absolutely shocked that a draft dealing with language
> lacks an "Internationalization considerations" section as
> recommended by RFC 2277 (a.k.a. BCP 18).

No more or less shocking than for RFC 3066, regarding which I'm not
aware of any complaints.

I don't quite understand what the critique is here: what is there to
internationalize about language tags? They are symbolic identifiers that
have no culture-specific content. The only possible consideration is the
charset, which for this spec involves ALPHA, DIGIT and "-" only. It's
true that ALPHA and DIGIT are not defined and that it would be better to
do so; it couldn't hurt to have a section for i18n considerations
(wouldn't need to be long). These are very minor concerns, and hardly
"shocking".

> Perhaps even more disturbing is the content of the "IANA
> Considerations" section; the draft predicts that certain things
> will happen ("IANA will"[...]), but doesn't actually direct
> (e.g. "IANA shall") IANA to do anything.  The placement of that
> section does not correspond to current RFC-Editor guidelines
> (it should appear after Security Considerations); also on that
> point, Appendices should precede References.

There is a process issue here, but I have assumed that the authors have
dealt with IANA on that. Otherwise, these are editorial issues -- "even
more disturbing" seems to me to be somewhat overstated.

> Many of the references are obsolete (e.g. RFCs 1327,
> 1521)... and at least one reference ([19])
> gives a bracketed URI rather than the correctly formatted
> RFC reference.  Although reference is made to the "Accept-
> Language" header field, RFC 3282 (the defining RFC for that
> field) is not listed among the references...

> The formatting of the draft is atrocious

All editorial.

> there is no differentiation between normative and
> informative references,

A valid concern.

> I am extremely surprised that the draft has been published
> at least nine times in such a state of poor formatting and
> poor attention to editorial content (e.g. obsolete and
> missing references), and that it progressed as far as IESG
> last call in such a state, with no Internationalization
> considerations section, etc.

In fairness to the authors, page-oriented plain text is not exactly
conducive to authoring and revising a long document, and a lot of energy
was spent focusing on details that have far more consequence than
formatting. And, as mentioned above, the lack of an i18n-concerns
section is hardly without precident, and not particularly significant in
the case of this spec. This really feels like nit-picking, IMO. I'm left
wondering if Bruce has been looking for nits to pick because he is...

> ... particularly concerned about the implementation
> ramifications of the proposed changes, especially (as
> noted in detail above):
> 1. the apparent contradiction between the stated
>     objectives w.r.t. accessibility of relevant ISO data and
>     standards and the reality of the proposal's
>     implications (ISO 8601 date format parsing).

As mentioned above, this really is a non-issue.

> 2. the clear contradiction between the claims about
>     ABNF compatibility with RFC 3066 and the factual
>     incompatibility of certain provisions in the grammar.

The main concern was with the "grandfathered" production, but I've shown
that that is a non-issue. The maximal length issue exists just as much
in RFC 3066 due to private-use tags; it is a technical concern that
might worth reviewing in RFC 3066bis, however; but it is not
insurmountable, and not a new problem.

Peter Constable
Microsoft Corporation
_______________________________________________
Ietf-languages mailing list
[email-masked]
http://www.alvestrand.no/mailman/listinfo/ietf-languages

---

From: "Peter Constable" <[email-masked]>
Date: Fri, 10 Dec 2004 17:03:05 -0800

[snip]

valid tags for

 

All editorial.

A valid concern.

Peter Constable
Microsoft Corporation

---

From: "Peter Constable" <[email-masked]>
Date: Sat, 11 Dec 2004 08:53:26 -0800

> > not-purely-alphabetic primary sub-tags),
> 
> No exception.  "alphanum" is ALPHA / DIGIT.

My mistake; again, I had on my mind constaints beyond the ABNF.

> > syntactically; but the set of
> > from one of the source standards or be registered.
> 
> So what? The ABNF is an expression of the grammar that
> describes the set of all valid tags.

It is *part* of the expression of the grammar. Even in RFC 3066 this is the case: you know that t-abc is not valid under RFC 3066, but not because that is constrained by the ABNF of RFC 3066.

I will accept that the ABNF of draft should be changed to better reflect what the form of grandfathered productions can be, which, as I stated in my previous message, would be the equivalent of the ABNF of RFC 3066:

I think that's an improvement, though technically I don't think it changes anything.

> If
> one doesn't intend to impose such requirements, the
> ABNF specifying the grammar should be changed
> accordingly.
> 
> > This is no different
> It is a very different grammar from RFC 3066, imposing
> very different requirements on parsers.

Our disagreement amounts to a basic question of whether parsers should be written based on the ABNF alone, or based on the ABNF plus other constraints provided in the spec. Clearly, I think anyone writing a parser should consider other constraints as well.

> > sub-tags.
> Look again at the draft definition of "grandfathered" -- now
> show me where there's a limit in that production on subtag
> length.

As mentioned, the limit is imposed by other tight constraints on 'grandfathered'; you have already identified that the longest registered tag under RFC 3066 is 11 octets in length, therefore a 'grandfathered' tag can be at most 11 octets in length.

> > - the number of possible extensions is limited to 25
...
> > , but the length of
> > extensions is unlimited
> 
> You have missed several others:
> 
> 1. "privateuse" length is unlimited (either tacked on
>     after "lang" etc., or directly as an alternative in
>     "Language-Tag")

I disregarded this since it is identical to the case for RFC 3066, and you were, after all, charging that the draft creates problems that were worse than for RFC 3066.

> 2. "grandfathered", which as already discussed
>     permits unlimited length.

But as already stated is very tightly constrained, with a de-facto upper limit of 11 (subject to change if new tags are registered before the proposed spec is accepted).

> > We could impose some upper limits on these things...

> That leaves the extension portions' length at up to
> 25 * (1 + 1 + 8 * 9) = 1850 octets, not taking any other parts
> of a tag into account!   That's way too long (the RFC 2047
> limit for an encoded-word is 75 octets, including charset tag,
> some text, and some syntactic glue in addition to the language
> tag).

The problem already exists in RFC 3066. Even apart from private-use tags, tomorrow someone could request a registration for a tag that's 87 octets long, and there's nothing in RFC 3066 that would prohibit acceptance.

> > not have characterized it as greatly exacerbating, though,
> 
> IMO, an increase from 11 octets worst-case, which is tolerable
> for constructing RFC 2047/2231 encoded-words, to >> 1850
> octets, which exceeds by a large margin what can be handled
> in a Content-Language or Accept-Language message header
> field, constitutes "greatly exacerbated".

Repeating my previous point, RFC 3066 doesn't stop a registered tag from being 10^100 octets in length. Of course, all of us know that such a tag wouldn't be useful. At some point, we have to engage common sense, even for RFC 3066. The draft would allow a tag 

en-boont-boont-boont-boont-boont-boont-boont-boont-boont-boont-boont-boont-boont

(over 75 octets), but common sense tells us it doesn't make sense (and that anyone who uses such a thing deserves whatever they get). 

Now, we could try to revise the ABNF to constrain for such things, just as the ABNF of RFC 3066 could have been constrained further. It's not easy to express common-sense constraints in ABNF, however.

I suggest that wording be added to the draft giving a strong recommendatation to users that they not use tags the complete length of which exceeds 75 characters.

> By deferring to the bilingual ISO lists for language and country
> tags, 3066 at least provided a minimal degree of internationalization.
> By explicitly limiting description fields to English and restricting
> the charset to US-ASCII, the draft proposal takes a giant leap
> backwards.

The US-ASCII limitation existed in RFC 3066, so is not new. 

On the more general point, I believe you are mistaking i18n concerns with localization concerns: you are looking for strings to be used in UI for different local markets. Apart from charset, RFC 1766, RFC 3066 or RFC 3066bis do not have *internationalization* concerns.

> > internationalize about language tags?
> 
> There should probably be a reference (at least informative)
> pointing to BCP 18 and mentioning that the language tags
> defined provide a means of labeling the language of text,

Have you not read the abstract in the draft?

<quote>
   This document describes the structure, content, construction, and
   semantics of language tags for use in cases where it is desirable to
   indicate the language used in an information object.
</quote>

Or the introduction?
<quote>
   One means of indicating the language used is by labeling the
   information content with a language identifier...

   This document specifies an identifier mechanism...
</quote>

How much clearer does it need to be?

> The draft (if/when approved) should also indicate that
> it updates BCP 18, which refers to RFC 1766.

Is this right? This draft is not a replacement for RFC 2277, or an addendum to it. RFC 2277 also refers to RFC 1958, which was updated by RFC 3439, but surely RFC 3439 doesn't state that it updates BCP 18? (RFC 227 does have a section with significant overlap in topic, though, so perhaps this makes sense. I'm not well-enough versed in IETF document process to know.)

> Given the divergence noted above from RFC 3066's use
> of multilingual reference lists, the Internationalization
> considerations section should include a synopsis of the
> approach chosen (viz. to restrict description to English) and
> the rationale for that choice (see BCP 18 section 6).

Again, this is a localization issue, not an internationalization issue. I do not consider this necessary or even appropriate.

> > It's
> > true that ALPHA and DIGIT are not defined
> 
> Non-sequitur aside, those terms are defined in RFC 2234.

Of course I meant "not defined *within this document*".

> It's an issue (esp. in light of the finger pointing regarding
> accessibility to ISO 639/3166).

As has been pointed out, there is no such finger-pointing in the draft.

> Admittedly it can be
> resolved without much difficulty (but then that could
> have been done earlier, couldn't it?).

I think the authors and those of us who have been reviewing thought that the intent was quite clearly YYYY-MM-DD, so didn't see a concern. That's why last calls are announced to a much wider audience.

> > that that is a non-issue.
> 
> Again, it is an issue that imposes requirements on language
> tag parsers.  What you've shown is that the ABNF is not
> consistent with what was desired to be expressed, and
> that makes it an issue that needs to be addressed.

Again, I believe the bigger issue is not getting the ABNF to express what was desired, but rather whether parsers are written to consider only the ABNF or the ABNF plus other specified constraints as well.

> > The maximal length issue exists just as much
> Private-use carries its own considerable baggage; aside from
> that, the draft proposal increases the length of non-private
> tags that affect both protocol design and implementations
> from a worst case maximum of 11 octets under RFC 3066...

Worst case at present; a month from now it could be unlimitedly larger. But I've accepted that it would be an improvement to add constraints on overall length.

Peter Constable
Microsoft Corporation

---

From: "Peter Constable" <[email-masked]>
Date: Sun, 12 Dec 2004 12:33:38 -0800

> The point is that under RFC 3066,
> the bilingual ISO language and country code lists are
> considered definitive.

That is nowhere stated or even suggested in RFC 3066.

Peter Constable
Microsoft Corporation

---

From: "Peter Constable" <[email-masked]>
Date: Sun, 12 Dec 2004 12:55:29 -0800

> > I don't know where the statement accompanying the announcement came from,
> 
> According to the "New Last Call" issued by the IESG Secretary,
> the text is "Author's discussion of drivers for this work".
> 
> > You singled out that one point to comment on as though it were the main
> factor.
> 
> I mentioned a matter which was repeatedly indicated as a
> factor for existing implementations and with which I
> strongly disagree.

You have not responded to the point that accessibility of source ISO standards is supposed to be a major factor, yet the draft itself clearly indicates otherwise.

> [regarding the proposed registry vs. internationally-
> standardized ISO lists for subtag definitions]
> > It is certainly the case that only it should be consulted for
> determining what sub-tags are valid with what denotation, which was the
> intent.
> 
> That is a problem for existing implementations of RFC 3066
> tags, which can obtain official, internationally agreed
> descriptions of the codes in two languages.

Descriptions (language names) are beyond the scope of RFC 3066. It is a non sequitor to claim that this draft creates a problem for existing implementations of RFC 3066 on this basis.

> > By looking in the sub-tag registry. If ISO changed the meaning of "US"
> to something other than what it is now, its meaning for purposes of use in
> an IETF language tag would not change, because it would remain stable in
> the sub-tag registry. You would be fairly well protected against the whim
> of politicians.
> 
> OK, continuing your hypothetical example and its relationship
> to language, suppose that there is another civil war and
> that what now corresponds to "US" is split into Blue America
> and Red America.  Further suppose that in due course ISO
> assigns some other code to one of those countries and retains
> "US" for the other, and that that happens after the proposed
> registry is set up with a definition for "US" and some
> description referring to the "old" use.

That is a scenario that has been well considered: it would be very bad IT practice to redefine a metadata tag "US" to have a narrower denotation than it previously did, as that immediately breaks an unknown amount of existing data. If ISO were to make such a change in the meaning of "US", then IT implementations *absolutely should not* follow suit; the ID "US" must retain it's prior, broader meaning.

> Now suppose that one
> wishes to produce an appropriate language tag for the text
> "moral values" (which clearly has different meaning in Blue
> America (telling the truth, admitting to mistakes, etc.) and
> in Red America (imposing totalitarian control over others)).
> How specifically would the proposed registry handle such a
> change in the meaning of "US", and how would the registry
> help differentiate the meaning of a 1990's "en-us" tag to
> that of the hypothetical time described?

It would leave "US" with it's historic meaning, so that existing data is left intact. (You wouldn't want a document containing "moral values" created on the eve of the cival war by someone supporting the Blue America side of the divide to suddenly get assigned an interpretation of 'imposing totalitarian control over others'.) New identifiers would be assigned for use in IT applications to designate the two new countries.

> > you already have to look beyond the ISO standards for anything more than
> English and French
> 
> But existing RFC 3066 implementations can get official
> descriptions in *both* of those languages; the proposal
> would adversely affect those existing implementations by
> eliminating the French description.
> 
> Of course, it is a more serious defect of the proposal
> that it would fail to reflect internationally-agreed
> codes and would fail to keep pace with changes...
> 
> > it would not be new that you have to look beyond the registry itself to
> decide what human-readable descriptors you should provide in a product.
> 
> It would be new that one could not find a standard
> (i.e. official) French-language description in the
> list of codes.

Incorrect. The registry for RFC 3066 did not provide a language/country name in *any* language for any ISO 639 or ISO 3166 identifier. Tags registered under RFC 3066 included an English-language name and an ASCII-transcription of the indigenous name; they did not contain French-language names.

Again, you are trying to impose UI-localization concerns that have always been out of scope for the RFC 1766/3066/... sequence of specifications.

> > > One possibility would be two description fields.  But the
> > > registry would need a charset closer to ISO-8859-1 than
> > > to ANSI X3.4 as currently specified.  Or an encoding
> > > scheme.
> >
> > Personally, I don't see the value in something like that. Given the
> intent to have a registry that can be machine-readable, changing its
> charset from ANSI X3.4 in order to gain descriptors in just one more
> language is not worth it IMO.
> 
> Fine, use utf-8, which encompasses ANSI X3.4 and
> ISO-8859-1 (plus others).  The point is that ANSI
> X3.4 is inadequate.

There is no point changing the charset to support something that is out of scope for the specification.

> > Speaking at least for Microsoft, we're interested in having descriptors
> in far more than two languages, and we certainly would not blindly base
> the descriptors we present to our customers solely on what a registry
> provides, no matter what its charset.
> 
> Surely in going from two (the current situation per
> RFC 3066) to "more than two" indicates that decreasing
> to one (as in the draft proposal) is heading in the
> wrong direction.  It certainly invalidates the claim
> that the proposal is compatible with existing
> implementations, at least one of which does make use
> of the descriptions currently provided in both
> languages in the ISO lists specified by RFC 3066.

Incorrect; you are making false claims about what is specified in RFC 3066. 

Peter Constable
Microsoft Corporation

---

From: "Peter Constable" <[email-masked]>
Date: Sun, 12 Dec 2004 17:57:24 -0800

> > That is not at all the aim here wrt stability; rather, the aim is
that a
> > symbolic identifier used for metadata in IT systems not change
because
> > some government on a whim says, "We would now prefer to use 'yz'
rather
> > than 'xy' to designate our country."
> 
> If by international agreement, 'yz' becomes the designation
> for that country, then it is rather silly to stick one's
> fingers in one's ears and shout "NA-NA-NA-NA-NA I don't want
> to hear you".

That misses the point entirely. The point is that IDs used by political
administrations may change for any number of reasons, and those
admministrations may have no qualms with such changes; but in IT
systems, we cannot afford changes that break existing implementations
and data. If for whatever reason ISO and the UN decided that "US" should
be used to designate the country of France, I doubt you'd expect every
software vendor to update all of their deployed installations to use
"fr-US" instead of "fr-FR", and for every user to go through every data
repository they manage to make such changes in their data.

The people that maintain time zone definitions may have their means for
changing times; that's fine for them. They are not dealing with the same
concerns as we are dealing with. The group here that has focused
specifically on language-tagging issues for several years has evaluated
issues that affect language tags and the impact of changes and has
decided what is best practice for *this* domain, and it is to maintain
stability of data rather than cater to whims of political
administrations.

> "Designed" or not, country codes *are* read by humans; they
> appear in top-level domain names.  Currently the ISO 639
> 2-letter codes mean the same thing as the last component of
> a domain name

I think you mean ISO 3166 2-letter codes.

> and as the second component of a language-tag.
> It's rather silly to change that correspondence simply because
> a few people are piqued that international agreement has been
> reached to change a few 2-letter codes.

The usability flaw in treating ISO 639 and ISO 3166 as human-readable is
evident in the confusion between ja and JP (or is it jp and JA?), and GB
vs UK. As for what is silly, if the UN country ID for Canada changed to
CN (and that for PRC changed to something else), I'm sure it would cause
far greater problems for users to have to change the last two letters in
domain names than for them to keep doing what they always did. In fact,
I would have thought it would create a rather significant problem on the
Internet if such a change were made. (URIs don't come with versioning
dates for domain names, so how would a DNS server know what the "cn"
meant?)

> > Neither RFC 1766 or RFC 3066 has ever presented "official"
translations;
> 
> Both defer to the ISO lists for definitions (not "translations")
> of the various codes.

Definitions; not language names for display use.

> > this is no different for RFC 3066bis.
> 
> It is very different; under the proposed draft, there is only
> an English definition, somebody wishing to provide a French
> definition finds that he has none and must resort to an
> unofficial translation.

The more you press this, the more silly it seems. RFC 3066 does not
anywhere discuss display names; localization data is beyond its scope.
The registry it defines does not give provision for French language
names. The source ISO standards are every bit as accessible as they ever
were, and just as RFC 3066 gave the user no option but to refer to the
source ISO standard, so users should and can continue to do so.

After this response, I will not waste my time any further on this
foolishness.

> I'm willing to postpone the discussion
> (other problems with the proposed registry format dictate
> a broader solution which could easily have provision for
> an arbitrary number of descriptions).

I strongly object to the suggestion that progress on this draft be
delayed to deal with this non issue that caters to implementation issues
that are well beyond the scope of either RFC 3066 or its proposed
replacement.

> No, you are overlooking the fact that a set of codes with
> no corresponding definitions is useless.  RFC 3066 defers
> the code/definition pairs to ISO, which provides multilingual
> definitions. The proposed draft would remove that multilingual
> characteristic.

What if the registry provide no name, just the ID? Then people would
have to refer to the source ISO standard as they did in the past, and we
would be able specify which ISO IDs were or were not valid. That would
achieve the goal that we had wrt stability while eliminating the concern
that English-only annotations for some reason apparently create for you.
Personally, I think the English annotation is helpful, but it seems that
the real solution you're looking for is to remove any annotation
whatsoever so that the situation is closer to what we have under RFC
3066.

> > Display names for languages and countries are not within the scope
of
> > RFC 1766 or RFC 3066. It is preposterous to suggest that this draft
is
> > not compatible with existing implementations of RFC 3066 on that
basis.
> 
> On the contrary, it is preposterous to suggest that codes
> will be attached to text by magic; some human somewhere,
> somehow is going to have to indicate the language to
> something, and it certainly isn't going to be by way of
> a 2- or 3-letter code without some reference to what those
> codes *mean*.  And at the present time, the meaning of
> those codes is defined -- bilingually -- in the ISO
> lists.

RFC 3066 did not even discuss let alone provide a means for attaching
display text to codes. It *is* preposterous to suggest that this draft
is incompatible with RFC 3066 on that basis. Again, the more you press
this, the more silly it seems.

> > But
> > you are simply adding localization requirements to a spec for i18n
> > infrastructure, and I consider that not at all appropriate.
> 
> No, I am complaining about removal of internationalized
> definitions associated with language tag components.

No definitions are removed. The draft points to the source ISO standards
just as RFC 3066 does.

> "Localization" would be translation of the French definition
> into some other language.  That is not my concern. My concern
> is the elimination of the French definition in the first place.

No, you have not commented on definitions; you have repeatedly commented
on stings to present to users. Please accept that your arguments on this
matter are empty.

> > > One part of my claim is that non-private-use RFC 3066 tags
> > > up to the present time are no longer than 11 octets in length.
> >
> > Only co-incidently at the present time.
> 
> As mentioned, under RFC 1766/3066 review/registration rules,
> excessively long tags would certainly raise objections. That's
> no coincidence -- it's an intentional design feature.

But excessive is not defined anywhere in RFC 1766/3066, and if there was
a very good reason presented why a tag of x characters long were needed,
it would have to be considered.

> > And so that limit would be a constraint applying for all time to the
> > 'grandfathered' production which concerned you so much.
> 
> And so it can easily be incorporated into that ABNF production.

The productive thing would be for you to provide a suggested revision of
the ABNF to the authors.

Peter Constable
Microsoft Corporation

---

From: "Peter Constable" <[email-masked]>
Date: Sun, 12 Dec 2004 18:00:35 -0800

> RFC 3066 section 2.2 states, in part:
> 
>    - All 2-letter subtags are interpreted according to assignments
found
>      in ISO standard 639, "Code for the representation of names of
>      languages" [ISO 639], or assignments subsequently made by the ISO
>      639 part 1 maintenance agency or governing standardization
bodies.
> 
> and has a similar statement regarding ISO 3166.
> 
> "interpreted according to assignments found in" certainly
> sounds as if the ISO lists are considered definitive for
> their respective categories of subtags, since their
> interpretation is specified as that given in those lists.
> I don't see how the RFC 3066 text can be interpreted
> otherwise.

RFC 3066 indicates that the *interpretation* is determined by the source
ISO standards. You were discussing display names. (Though, now that I've
shown that display names are out of scope, you appear to be attempting
to change things as though you had been discussing definitions.)

Peter Constable
Microsoft Corporation

---

From: "Peter Constable" <[email-masked]>
Date: Sun, 12 Dec 2004 20:13:00 -0800

> > What is silly is saying that every language tag has to have a
date/time
> > attribute associated with it so that computer software managing that
> text
> > knows the language of that text.
> 
> In the specific cases of the core Internet protocols that
> I have mentioned, there *is* a date/time attribute in the
> form of an RFC [2]822 Date field.  If we're talking about
> some file stored on some machine, every OS that I know of
> has a date/time stamp associated with that file.  If you
> have something else in mind, a concrete description and/
> or example might help.

That is not sufficient for many other implementations of RFC 3066. For
instance, an XML document may well be stored in a file system that has
date/time stamps associated with the file; it might also be stored in a
content manangement system that does not report creation dates when
returning content. And elements from within that XML document may be
returned as the result of an X-Path query or a call into a DOM API, and
those surely cannot be assumed to have creation date/time stamps, though
one certain must assume that they can have RFC 3066 tags as xml:lang
attributes.

> I'm not "eager to abolish" "uniqueness".  There never was
> any guarantee that codes would never change. Both RFCs
> 1766 and 3066 specifically mention changes as a fact of
> life.

Some of us consider that fact and the instability particularly of ISO
3166 to be a serious problem. That (not accessibility) was one of the
key reasons for this revision.

> > > SO where are the French definitions?
> >
> > Ask a person who is bilingual in English and French to provide one.
> 
> That would lack definitiveness which characterizes the
> ISO lists.

You started out this thread by talking about display names, not
definitions; hence Mark's suggestion. Now you have switched to talking
about definitions. The draft clearly indicates where one finds the
definitions:

"   o  All 2-character language subtags were defined in the IANA
registry
      according to the assignments found in the standard ISO 639..."

I.e. the definition is provided in the registry on the basis of what is
defined in ISO 639; hence if what is indicated in the registry is for
any reason insufficient for your purposes, you consult the definitive
source, the ISO standard.

Peter Constable
Microsoft Corporation

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 15 Dec 2004 09:15:03 -0500

>  Date: 2004-12-13 04:37
>  From: Mark Crispin <[email-masked]>

> Silliness aside, the file may well have embedded language tags in the text 
> of the file.  Have you forgotten Plane 14?

No, but I note that its introduction strongly discouraged
its general use (specifically mentioning ACAP as the
intended scope of usage, IIRC); the current version of the
Unicode document continues that strong discouragement and
further reinforces it by emphasis via italics.

Another issue is that both RFC 3066 and the draft proposal
call for language tags to be expressed in a subset of ANSI
X3.4, corresponding to a subset of the first half of a
particular Unicode plane -- and not plane 14.  There may
be an ambiguity as to whether such deprecated Unicode 3.x
tags are in fact compliant with 3066 or the draft under
discussion.

> That's what's now being fixed.

No the problem will remain. Currently sr-CS has a specific
meaning under RFC 3066; it has had for some time.  For that
meaning to remain stable, it will be necessary to take any
change in the (current) meaning of the "-CS" part into
account. I.e. for a future parse of language tags to do the
right thing, it will have to recognize sr-CS generated under
the RFC 3066 rules per the 3066/639 definitions.
 
> Why is this vestige of colonialism important in the IETF context?

You seem to be making an incorrect assumption, one which
renders your question meaningless.

> What magic attribute is there to French that provides "definitiveness" 
> that is absent in English, or Mandarin, or Hindi, all of which are far 
> more significant languages to the world?

No such attribute of the language was claimed.  It is the
attribute of being used in the official ISO lists that
provides the characteristic.
 
> A mandatory French translation to an English definition does not 
> significantly increase the information content, and certainly does not 
> double it.

You are again making incorrect assumptions.  The languages
used in ISO documents are considered "separate but equal",
not "a mandatory [...] translation of" some other language.
That is in fact why ISO is called "ISO" and not "OIN" or
"OIS" -- you might wish to visit the ISO web site for
details.
 
[more nonsense about "mandatory translation" elided]

> > You have not explained how the code came to be "embedded
> > within the text itself" -- surely the author didn't say
> > (or write, or sign) "this text is in language QZ"; most
> > likely the language was indicated by name, or by some proxy
> > representing the name (such as a locale).
> 
> Plane 14.
> 
> HTML and other markups.

That provides no explanation of how a *code* came to be
embedded in text -- authors in general do not refer to
language by codes, and codes do not embed themselves by
magic.

---

From: "Peter Constable" <[email-masked]>
Date: Wed, 15 Dec 2004 11:09:49 -0800

> Currently sr-CS has a specific
> meaning under RFC 3066; it has had for some time.

The meaning "Serbia and Montenegro" was introduced relatively recently
(a little more than a year ago), was immediately received with alarm by
many in the IT sector. There were vain attempts to get it reversed, and
that failure was an impetus to introduce protection against such changes
in the revision of RFC 3066. I am not aware of "CS" being used in the IT
sector with the new meaning, though cannot guarantee that.

Peter Constable
Microsoft Corporation

---

From: "Peter Constable" <[email-masked]>
Date: Wed, 15 Dec 2004 11:41:15 -0800

> > By reading both RFC 2047 and RFC 2231, one
> > finds that they assume that a language tag must be at most 64
characters
> > long...

> > - the shortest charset names are 2 characters long (e.g. "IT")
> 
> Not all charsets have 2-character names...

In determining the longest language tag permitted, one must identify the
shortest possibilities for all other components. 

> > - the minimum encoded-text length is 1 character long
> 
> That is strictly only true for text that meets all of the
> following conditions...

Hey, I just said what the EBNF said.

> > An encoded-word must contain at least 11 characters that are not
part of
> > the language tag and have a total length of no more than 75
characters.
> > Therefore, an upper bound on language tags that can be used in an
RFC
> > 2047/2231 encoded-word production is 64 characters.
> 
> That is a best case upper bound...

I identified it as such.

> The worst case appears to be the charset named
> Extended_UNIX_Code_Fixed_Width_for_Japanese (43 characters)...

> As mentioned, use of an encoded-word
> plus the necessary whitespace around it to represent a
> single character is rather wasteful, so a brief language tag
> is indicated; fortunately "ja" suffices for text likely to
> be used with that charset.

Of course, the length limitations must be balanced between the charset
tag, the language tag and the encoded-word itself.

> > I see no reason why limits must be added as a
> > constraint in a revision of RFC 3066.
> 
> The primary reason for specifying limits is due to the
> proposed removal of the review/registration process
> which currently limits the length of non-private-use
> tags.

The review/registration process for RFC 3066 registrations does not
impose pre-defined limits that implementers of RFC 3066 can assume in
their parsers.

> > It would be a good idea, however,
> > to point out in section 2.1 of the draft that some applications of
this
> > specification may impose limits on the length of accepted language
tags,
> > and perhaps to cite RFC 2231 as an example.
> 
> As a general principle, that's fine, however I would point
> out that given the inability of experts to be able to
> accurately point out the limits quickly...  I do
> not think it is sufficient merely to state the fact that
> there are limits, with or without a pointer to RFC 2231 as
> an example.  Some indication of the magnitude of worst-case
> restrictions is at least advisable...

How is it possible to identify what is the worst-case bound assumed in
implementations that are out there?

How is it possible to predict ahead of time what is the worst-case
length for a RFC3066-registered language tag?

Neither is possible. In light of that, I think it best to make sure
implementers of the revised RFC 3066 be reminded that some
implementations may impose limits (whether those implementers be
constructing tags or passing them from one process to another), and for
implementers to incorporate robustness into their implementations so
that they can respond gracefully if an unexpectedly-long tag is
encountered -- after all, no matter what limit could be imposed in a
revision to RFC 3066, there's no way to stop malware from sending bad
data.

(How *do* encoded-word parsers react if a bogus charset or language tag
that's 2k octets long is encountered? The encoded-word spec already
allows for segmenting long strings; could it not also be revised to
allow segmenting for the parameters, which would also make it more
robust?)

Peter Constable
Microsoft Corporation

---

From: "Peter Constable" <[email-masked]>
Date: Wed, 15 Dec 2004 11:45:49 -0800

> [email-masked]] On Behalf Of John Cowan

> But absolutely nothing except his good sense prevents Michael from
> registering
>
en-the-dialect-spoken-on-the-bowery-between-1933-and-1945-by-alcoholic-
> drug-users-who-live-in-flophouses.

Sub-tags can be at most 8 chars long, so Michael would ask for it to be
changed to something like
en-the-dialect-spoken-on-the-bowery-between-1933-and-1945-by-alcoholc-dr
ug-users-who-live-in-flophses. :-)

Peter Constable
Microsoft Corporation

---

From: "Peter Constable" <[email-masked]>
Date: Thu, 16 Dec 2004 11:36:34 -0800

found
bodies.
> I don't see how the RFC 3066 text can be interpreted

You're now quoting things so far removed from their context that they
are no longer being evaluated fairly. I believed we were talking about
the specific strings, as you had made reference to implementers of
bilingual products not having access to that data. Perhaps I
misunderstood you, but whether or not, the relevant facts are that RFC
3066 referred to ISO source standards to establish the denotation of
identifiers drawn from those standards, and the proposed revision does
the same.

Peter Constable
Microsoft Corporation

---

From: Bruce Lilly <[email-masked]>
Date: Fri, 17 Dec 2004 09:13:09 -0500

>  Date: 2004-12-14 13:02
>  From: John Cowan <[email-masked]>
>  To: "Addison Phillips [wM]" <[email-masked]>
>  CC: [email-masked]
>  
> Addison Phillips [wM] scripsit:
> 
> > The IETF process is not really my concern. I will note that many IETF and
> > non-IETF standards folks have participated in the process of developing and
> > reviewing draft-langtags, though. 
> 
> Actually, we're all IETF people.  If you're on an IETF mailing list discussing
> an IETF item of work like an RFC, you're part of the IETF.  The process
> is designed to serve us, not vice versa.

It's not quite that simple; IETF process has several specific requirements
(as spelled out in RFC 2026) -- an IETF Working group requires a charter
with a well-defined scope, specific milestones, etc.  There is an official
list of IETF working groups.

---

From: Mark Crispin <[email-masked]>
Date: Sun, 19 Dec 2004 10:53:35 -0800 (Pacific Standard Time)

On Sat, 18 Dec 2004, Brian Rosen wrote:
> I don't have any comment on the issue of language tags, but speaking as a
> reasonably avid ABNF hacker, I agree with Sam, and would not want to
> establish a convention that ABNF in IETF RFCs is expected to be precise.

The counter-argument is the all-too-frequent occurance when you deal with 
willful cretins who will *insist* that the specification says 
such-and-such when it really says the opposite, and will leap upon the 
most bizarre interpretation of text in order to bolster their arguments.

This is unavoidable; however, it helps a lot if the ABNF firmly comes down 
on the side of the good guys.

I've spent entirely too much of my life in the past few years fending off 
cretins, to agree knowingly to anything that makes me more vulnerable to 
them in the future.

Nor do "gentlemen's agreements" work any more.  We may all be (ladies and) 
gentlemen here, but "out there" there are individuals who are not.

As painful as the process may be, I believe that the ABNF should be as 
tight as possible, preferably by ABNF rules but at least through ABNF 
comments.

However, be careful about comments.  I had one cretin insist that "between 
n and m inclusive" (where n and m were integers) had an implied 
restriction that n <= m, and that when n > m it meant an empty set of 
values.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Bruce Lilly <[email-masked]>
Date: Tue, 21 Dec 2004 08:36:56 -0500

>  Date: 2004-12-18 23:37
>  From: "Doug Ewell" <[email-masked]>
>  To: [email-masked]
>  
> Bruce Lilly <blilly at erols dot com> wrote:
> 
> >> If you can write a reasonable "grandfathered" production in ABNF that
> >> will allow this set of tags and no others, such that the ABNF can be
> >> used without also referring to the prose, then I salute you.
> >
> > If there really are only 24 items of less than 11 octets each,
> > a trivial solution is to simply list them (with the usual ABNF
> > syntax) as literal strings.  That should take no more than a
> > half-dozen lines.
> 
> Listing the 24 literal strings doesn't seem like a particularly elegant
> solution.

Perhaps it doesn't meet your subjective criteria for elegance.
But it is a *reasonable* production that meets specific criteria,
and that is what you asked for.  A list of specific literal
strings is not unusual (e.g. RFC 3464 sect. 2.3.3, RFC 3798
sect. 3.2.6, RFC 2156 (summarized in Appendix E)).

> Look, RFCs 1766 and 3066 both had ABNF that was insufficient to describe
> the range of valid language tags, and AFAIK they were not greatly
> criticized for this. [...] The same is true for RFC 3066bis.

A crucial difference is that RFC 3066 and 1766 required
registration before use, and community review before
registration.  If a tag were proposed that failed to meet
some criteria not adequately detailed in the ABNF, the
reviewer, the community, and the Area Director could
explain the issue *before* the darned thing went into use.
As that safety mechanism is being removed, it is more
important that the specification be clear and precise and
consistent.

> RFC 2231, which you have mentioned often in this thread, has the
> following as part of its ABNF:
> 
> -----begin pasted material-----
>    charset := <registered character set name>
> 
>    language := <registered language tag [RFC-1766]>
> -----end pasted material-----
> 
> If this type of syntax specification is good enough for RFC 2231, why
> wouldn't it be good enough here?

RFC 2231 isn't BCP and doesn't obsolete BCP; it does not
remove any registration requirements.  While it obsoletes
another RFC (2184), it does not attempt to incorporate
content of the obsoleted RFC or artifacts of its use by a
vague reference.  Reference to (unaffected) external
specifications is fine; the draft uses RFC 2234 productions,
for example, and that is not a problem.

---

From: Bruce Lilly <[email-masked]>
Date: Tue, 21 Dec 2004 15:59:21 -0500

>  Date: 2004-12-21 00:57

> The RFC 3066bis approach involves creating a registry of all the pieces
> that can make, or be combined to make, a language tag.  This is much
> easier to implement and understand than chasing down the various
> standards and their history, and it permits stability that cannot exist
> if ISO maintenance agencies change their codes.

Substituting a Numbers Authority for a Maintenance Agency
might not solve the problem; indeed it may bring new problems.
IANA isn't infallible, and has botched some registry entries.
See http://mail.apps.ietf.org/ietf/charsets/msg01477.html
for an example.
 
> Vernon Schryver [...] characterized debating RFC 3066bis (for over a year!)
> within the IETF-Languages group, and only presenting it to other groups
> during the Last Call period, as a "process problem,"

OK.

> and charged this 
> group with engaging in "lawyerly talk such as whether 'accounts' is more
> appropriate than 'account'" even though no such exchange ever took place
> (I checked the archives back to January 2002).

No, he was referring to concurrent discussions on the
IETF mailing list.
 
> Now Bruce wants us to wait a few more days before rolling out his
> suggestions to fix these perceived problems.
> 
> This is a filibuster, an attempt to stall RFC 3066bis out of existence.

I also (i.e. in addition to JFC) find that characterization
offensive.  I am responding to an IETF New Last Call in
accordance with established procedures, and within the time
period established.  I had at one time entertained an
informal approach to addressing the procedural issues, but
given such an accusation, I am now inclined to use the
formal procedure outlined in RFC 2026 section 6.5.2.