Subject: Re: New Last Call: 'Tags for Identifying Languages' to BCP

From: Bruce Lilly <[email-masked]>
Date: Sun, 12 Dec 2004 11:46:52 -0500

>  Date: 2004-12-10 22:37
>  From: John Cowan <[email-masked]>
>  
> Bruce Lilly scripsit:
> 
> > It's not clear to me that the proposal will provide protection
> > against the whims of politicians.  If the definition of "CS" as
> > a country code changes again under the proposed scheme,
> > how is one to determine specifically what some archived
> > language-tag referred to at some point in time?  I'm not
> > particularly concerned about that problem, as I am resigned
> > to instability associated with anything specified by politicians
> > (and that includes the UN region codes).
> 
> The U.N. Statistics Division are only "politicians" in the sense
> that IETF WG members are.  They are, in fact, statisticians.
> Their track record for stability is considerably longer than the
> IETF's.

I hope that I need not repeat any of the well-known remarks
about "statistics".  Nor that I need point to the many uses
by politicians of statistics (and statisticians) for
political purposes.

Moreover, the point is that countries do change, and that use
of country codes (as provided for in RFC 3066 and in the
proposed draft) carries with it the inherent instability
which is characteristic of politics.  A quest for "stability"
of countries seems Quixotic and oxymoronic.  According to the
principle of stability as that term is used in defense of the
draft, I suppose we're all intended to refer to Malawi as
"Rhodesia" because that's what it (in part) was called 50 years
ago, or that we're supposed to ignore the breakup of the USSR,
Yugoslavia, etc., the reunification of Germany, etc.

A related problem with the use of country codes in language
tags is that there is not necessarily an inherent relationship
between language and country borders.  The borders of Germany
have changed many, many times.  If one is referring to the
German language as spoken by inhabitants of Alsace, using
country codes would imply that that same language spoken by
the same people would have been tagged at various times as
de-DE and de-FR according to where the France-Germany border
happened to have been determined by politicians of the time.
That strikes me as being a rather silly way to tag language,
but that's the precedent set by RFC 1766.  As far as I can tell,
the draft doesn't really deal with the issue of changing borders
or changing country names -- it merely pretends that these
things don't happen by attempting to declare a snapshot of the
status at some point in time as being valid for all time.

> > But if the proposed new registry's description of "CS" says
> > "foo" and the ISO standard code list says "bar", what's
> > an implementor supposed to present to a user as *the*
> > description associated with "CS"?
> 
> The former.  That's the whole point of having a registry.

But the user has indicated that he speaks French, and the
proposed registry contains a description in English only.
Where is the implementor supposed to get the *official*
translation for display?  N.B. under the current (RFC 3066)
situation, the definitive ISO lists provide an official
description in French.
 
> > One possibility would be two description fields.  
> 
> Why two?

There are now two in the ISO lists (and, as noted, in the
UN list).  I have no objection to more, but I object to
a reduction.  The text accompanying the new last call
states:

"This specification addresses each of these issues with a simple, elegant design
that is compatible with existing language tags and implementations."
and
"One concern that is crucial to acceptance of the new language tag design is how
it works with existing implementations of RFC 3066 and how existing
implementations will interact with implementations of the newer language tags."
and
"It is important to recognize that all language tags that were valid under the
existing RFC 3066 will remain valid, with their meanings intact, under this
specification."

I have an implementation which (in accordance with RFC 3066)
uses the official ISO lists. It has provision for displaying
ISO 639 language tags with their descriptions in either of the
two languages supported by the official 639 lists, and likewise
for the ISO 3166 country codes.  The specification of the
draft is *NOT* compatible with that existing implementation
because it removes the existing functionality of official
descriptions in French of language and country codes. As a
result of that incompatibility,  the newly proposed
specification does not work with (at least that one)
existing implementation (but I agree that that is a crucial
concern).

Language tags remaining valid, I presume that the tag "sr-CS"
will continue to mean Serbian as used in Serbia and Montenegro
(officially equivalent to Serbe par Serbie et Monténégro) as that
is a valid RFC 3066 language tag and its corresponding meaning...
but I can see no evidence of that in the draft -- indeed it
appears that the draft would change that meaning significantly.

> There are 6000 languages spoken on Earth, of which 
> perhaps 600 have a standard written form.

ISO 639 lists about 650, not precisely 6000.

It might be worthwhile considering the differences in the
way languages tags are used, by whom they are used, and for
what purpose.  There may well be a substantial difference
between use of a tag to represent an obscure dialect of a
dead language in a research paper vs. tagging a piece of
text in one of the core Internet protocols such as SMTP.
The draft seems to ignore the needs of the core Internet
protocols (e.g. unbounded tag length which is incompatible
with those protocols).

> What is supposed to 
> be privileged about English and French?

They happen to be the languages in which international
standards (q.v. the ISO and UN lists) are published. If
one is going to use those standards (or a snapshot of them)
as a basis for subtags, then one ought to preserve the
standardized descriptions in the offcial languages of those
standards rather than discarding all but one of them in a
fit of Anglo-centrism.

> > Eliminating bilingual descriptions for the language,
> > country (and UN region) codes leaves implementors
> > in a quandary.
> 
> Only for those implementers to whom English and French, but
> no other language, is essential.

Implementors of RFC 3066, where the relevant standards
provided official bilingual descriptions of the country
and language codes.  Something which the "new last call"
text states that the draft proposal "is compatible with",
but which is not evident in the substance of the proposal. 
 
> > ABNF from the draft:
> 
> You're technically right, but your underlying claim (that RFC 3066 tags are
> bounded in length) is false, as has been shown

One part of my claim is that non-private-use RFC 3066 tags
up to the present time are no longer than 11 octets in length.
As the draft, if/when approved, would close that registration
process, that limit (unless a longer tag is registered in
the interim) would apply for all time.  The other part of my
claim is that under the proposed scheme, non-private-use tags
become unbounded in length, and that is incompatible with
existing Standards Track RFCs (821, 822, 2821, 2822, 2047,
2231, 3282 among them) and the core Internet protocols
which they specify.

> and the "grandfathered" 
> production is only used to match certain existing registered RFC 3066
> tags as they appear in the registry.

Then the ABNF for that production should match those
"certain existing registered RFC 3066 tags as they appear
in the registry" , and not match unbounded-length subtags,
non-alphabetic primary subtags, zero-length subtags, dangling
hyphens, etc.; I don't want that ABNF to be used as an excuse
for a future revision to introduce such constructs officially
on the basis that they are permitted by that ABNF.

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Bruce Lilly <[email-masked]>
Date: Sun, 12 Dec 2004 12:16:33 -0500

>  Date: 2004-12-11 00:52
>  From: "Mark Davis" <[email-masked]>
>  To: [email-masked], [email-masked]
>  CC: [email-masked]
>  
> > The ABNF is an expression of the grammar that
> describes the set of all valid tags.
> 
> No, this is simply incorrect. You cannot expect that any implementation that
> simply does the ABNF is conformant.

I made no such claim.  I do claim that if the ABNF
contradicts the normative text, as is the case in
your draft w.r.t. acceptance of several constructs
not permitted by RFC 3066 ABNF, that there is an
error in either the normative text or the ABNF.

> There are a great many constraints on 
> the tags that are not in the ABNF grammar, that are clearly required in any
> reading of the text. Most of these *cannot* be encompassed in any ABNF
> grammar.

If your claim is that the ABNF cannot express a
grammar consistent with the RFC 3066 ABNF, that
is clearly false.

> There are a few that could be expressed in the ABNF; some at little 
> cost, some with a great deal of complication.

Are you claiming that it is unduly difficult to
make the ABNF match RFC 3066's?

> This is not a technical 
> problem for the draft.

It is a problem due to the conflict between the
ABNF and the text.  It is a problem because it
opens a loophole for future revisions to formalize
content which is incompatible with RFC 3066
implementations.
 
> > as reasonable as the current worst-case of 11 octets.
> Also simply untrue. You seem not to be reading all the messages on this
> subject. Look at the ABNF for RFC 3066. There is *no* limit in the ABNF
> there!

The draft proposes closing RFC 3066-style registrations.
Show me a registered RFC 3066 language tag longer than
11 octets.  Show me a general-use (i.e. not private-use)
RFC 3066 language tag which is too long to be used in an
RFC 2047/2231 encoded-word.  Show me a general-use RFC
3066 language tag which is too long to fit on an RFC
2822/3282 Content-Language header field line.

---

From: John Cowan <[email-masked]>
Date: Sun, 12 Dec 2004 15:34:44 -0500

Bruce Lilly scripsit:

> of countries seems Quixotic and oxymoronic.  

Of course countries change, and then the numeric country codes change
as well.  The point is that the alpha codes change for political reasons
when there has been *no* change in the underlying country:  Romania's
3-alpha code changed from ROM to ROU without any change in Romania at all.
The CS case is particularly gratuitous, as its denotation changed from
"Czechoslovakia" (a no longer existent country) to "Serbia and Montenegro"
(a newly created country).

> between language and country borders.  

Of course not.  But for the most part, variations in orthography
do tend to follow national boundaries, since orthography in many
languages is either de jure or de facto a national matter.

> As far as I can tell,

No, it attempts to freeze the code-to-country mapping at a single
point.  New countries or changes in old countries should involve only the
additions of codes, not the reuse of old codes.

> translation for display?  

I don't know.  Where is the implementor supposed to get the
official German, or Catalan, or Mandarin translations?
Not in the ISO registry, for sure.  To say nothing of the
cases where no official translations exist.

> ISO 639 lists about 650, not precisely 6000.

ISO 639-2 is deliberately incomplete.  The current draft of ISO 639-3,
which is not yet an IS, lists over 7000 languages.

> text in one of the core Internet protocols such as SMTP.

That count does not include dead languages.  Whether it includes
dialects is a matter of terminology.

-- 
Deshil Holles eamus.  Deshil Holles eamus.  Deshil Holles eamus.
Send us, bright one, light one, Horhorn, quickening, and wombfruit. (3x)
Hoopsa, boyaboy, hoopsa!  Hoopsa, boyaboy, hoopsa!  Hoopsa, boyaboy, hoopsa!
  -- Joyce, Ulysses, "Oxen of the Sun"       [email-masked]

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 12 Dec 2004 10:00:49 -0800

> Show me a general-use RFC

Your claim was that RFC 3066bis (the informal name we've been using for the
new draft) permits language tags that are longer than those permitted by RFC
3066. That is clearly false, as many people have pointed out. Any subsequent
niggling that particular *types* of language tags can be longer or not is
not relevant to the conformance implications of the two documents for
language tags. The new draft neither extends nor contracts the maximum
length of language tags conformant to RFC 3066.

Your claim that the RFC 3066 ABNF itself has a restriction in length is also
clearly false. I will quote that again since you seem somehow not to have
seen it:

"
   The syntax of this tag in ABNF [RFC 2234] is:
    Language-Tag = Primary-subtag *( "-" Subtag )
    Primary-subtag = 1*8ALPHA
    Subtag = 1*8(ALPHA / DIGIT)
"

Both documents establish many further limitations on the contents of
language tags in the text of each document. Ignoring those stated
limitations will, in both documents, result in nonconformant language tags.

‎Mark

----- Original Message ----- 
From: "Bruce Lilly" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Sent: Sunday, December 12, 2004 09:16
Subject: Re: New Last Call: 'Tags for Identifying Languages' to BCP

> Ietf-languages mailing list
> [email-masked]
> http://www.alvestrand.no/mailman/listinfo/ietf-languages
>

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 12 Dec 2004 10:00:49 -0800

"

‎Mark

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 12 Dec 2004 12:30:03 -0800

There is a fundamental misunderstanding on two points.

1. Of course countries go in and out of existence, and change their borders;
nobody disputes that. That is not the stability problem in question; it is
where the meaning of tags changes so drastically as to refer to a completely
different country. One can't willy-nilly change data that has significant
effects on databases all over the world; when someone's birthplace is
indicated by a stored country code, for example, it mustn't suddenly
designate a different country! For more, see
http://www.unicode.org/consortium/positions.html.

2. The fact that the 3066 registry is not in multiple languages (either
currently or in the new draft) has nothing to do with any alleged
discouragement of any language, French included. The names in the registry
are simply to distinguish and identify the subtags, not to provide
recommended localizations.

The registry, and for that matter the ISO 639/3166 standards, are the wrong
place for localization data. The language coverage (only 2!) is a very small
fraction of what is really needed for any real product development -- and
even for those languages that are present, the names used there are not
optimal for user interfaces since they are sometimes not the customary form.
For an example of a data repository that is designed for localization of
language/region names, see http://www.unicode.org/cldr/.

‎Mark

----- Original Message ----- 
From: "Bruce Lilly" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Sent: Sunday, December 12, 2004 08:46
Subject: Re: New Last Call: 'Tags for Identifying Languages' to BCP

"This specification addresses each of these issues with a simple, elegant
design
that is compatible with existing language tags and implementations."
and
"One concern that is crucial to acceptance of the new language tag design is
how
it works with existing implementations of RFC 3066 and how existing
implementations will interact with implementations of the newer language
tags."
and
"It is important to recognize that all language tags that were valid under
the
existing RFC 3066 will remain valid, with their meanings intact, under this
specification."

---

From: "Mark Davis" <[email-masked]>
Date: Sun, 12 Dec 2004 12:30:03 -0800

‎Mark