Subject: Re: New Last Call: 'Tags for Identifying Languages' to BCP

From: Bruce Lilly <[email-masked]>
Date: Sun, 12 Dec 2004 18:44:27 -0500

>  Date: 2004-12-12 15:31
>  From: "Peter Constable" <[email-masked]>
>  To: [email-masked], [email-masked]
>  
> > From: [email-masked] [mailto:ietf-languages-
> > [email-masked]] On Behalf Of Bruce Lilly
> 
> 
> > Moreover, the point is that countries do change, and that use
> > of country codes (as provided for in RFC 3066 and in the
> > proposed draft) carries with it the inherent instability
> > which is characteristic of politics.  A quest for "stability"
> > of countries seems Quixotic and oxymoronic.  According to the
> > principle of stability as that term is used in defense of the
> > draft, I suppose we're all intended to refer to Malawi as
> > "Rhodesia" because that's what it (in part) was called 50 years
> > ago, or that we're supposed to ignore the breakup of the USSR,
> > Yugoslavia, etc., the reunification of Germany, etc.
> 
> That is not at all the aim here wrt stability; rather, the aim is that a
> symbolic identifier used for metadata in IT systems not change because
> some government on a whim says, "We would now prefer to use 'yz' rather
> than 'xy' to designate our country."

If by international agreement, 'yz' becomes the designation
for that country, then it is rather silly to stick one's
fingers in one's ears and shout "NA-NA-NA-NA-NA I don't want
to hear you".  A more rational approach would be to say that
before such-and-such a date/time the designation was 'xy' and
after that date/time (until further notice) it is 'yz'. As I
have pointed out, politicians change the definitions of time
zones frequently, and those who have to deal with time zone
issues have found a way to cope with such change without
trying to declare international standardization organizations
irrelevant.

> Sure, there will be changes that we need to deal with; but there's no
> reason to subject all implementations, users and data to changes that
> are purely cosmetic changes to things that are not designed to be read
> by humans.

"Designed" or not, country codes *are* read by humans; they
appear in top-level domain names.  Currently the ISO 639
2-letter codes mean the same thing as the last component of
a domain name and as the second component of a language-tag.
It's rather silly to change that correspondence simply because
a few people are piqued that international agreement has been
reached to change a few 2-letter codes.
  
> > A related problem with the use of country codes in language
> > tags is that there is not necessarily an inherent relationship
> > between language and country borders.
> 
> That is not what country IDs within a language tag is intended to
> suggest. In fact, if there were inherent relationships, we probably
> would never have needed to use country IDs in a language tag.

I submit that it was never a good idea. Language evolves
over time, even in a given place.

> > The borders of Germany
> > have changed many, many times.  If one is referring to the
> > German language as spoken by inhabitants of Alsace, using
> > country codes would imply that that same language spoken by
> > the same people would have been tagged at various times as
> > de-DE and de-FR according to where the France-Germany border
> > happened to have been determined by politicians of the time.
> > That strikes me as being a rather silly way to tag language,
> > but that's the precedent set by RFC 1766.
> 
> I agree that that's a silly way to tag that language; I disagree that
> RFC 1766 suggests I should tag it that way. 

RFC 1766 (and 3066) leave you little choice; if you wish
to indicate a region, you either have to do it with ISO
639 codes or you have to register a separate tag (no
separate tag for German as spoken in Alsace exists). Never
mind the shortcomings of that particular example; consider
"de-DE" -- does that mean Germany as it exists today, West
Germany as it existed 25 years ago, Germany as it existed
in the 1930s, the 1900s, ...?
 
> > As far as I can tell,
> > the draft doesn't really deal with the issue of changing borders
> > or changing country names -- it merely pretends that these
> > things don't happen by attempting to declare a snapshot of the
> > status at some point in time as being valid for all time.
> 
> That may be your reading of the situation, but it is not how it is seen
> by those of us who have been working on this spec and examining these
> issues closely.

As far as I can tell, the draft pretends that the meaning
of "CS" hasn't changed, and would in fact change the meaning
of the currently valid RFC 3066 language tag "sr-CS".

> > But the user has indicated that he speaks French, and the
> > proposed registry contains a description in English only.
> > Where is the implementor supposed to get the *official*
> > translation for display?  N.B. under the current (RFC 3066)
> > situation, the definitive ISO lists provide an official
> > description in French.
> 
> Neither RFC 1766 or RFC 3066 has ever presented "official" translations;

Both defer to the ISO lists for definitions (not "translations")
of the various codes.

> this is no different for RFC 3066bis.

It is very different; under the proposed draft, there is only
an English definition, somebody wishing to provide a French
definition finds that he has none and must resort to an
unofficial translation.

> Under RFC 3066, one is pointed to 
> ISO 639-1 and ISO 639-2 to get the alpha-2 and alpha-3 IDs, but it does
> not anywhere state that implementors should use the English and French
> language names in those ISO standards;

It defines the correspondence between the codes and
names as agreed to by international agreement.

> exactly the same situation holds 
> for RFC 3066bis.

SO where are the French definitions?

> (Note, btw, that the names listed by ISO 639-1/-2 have 
> no particular "official" status; they are normative in those standards
> to the extent that the indicate what language variety a given ID
> denotes, but they do not claim that the particular form of the language
> names have any particular status.) 

Well, sure. But the name is an important thing by itself.
It is rather pointless to ask a user to indicate the
language of a piece of text by selecting from a list "AB, ACE,
ACH,..., ZHA, ZUL, ZUN" -- the user doesn't normally refer to
languages by codes. It's quite a different matter to ask the
user to select from "Abkhaze, Aceh, Acoli,..., Zhuang (Chuang),
Zoulou, Zuni".

> > > > One possibility would be two description fields.
> > >
> > > Why two?
> > 
> > There are now two in the ISO lists (and, as noted, in the
> > UN list).  I have no objection to more, but I object to
> > a reduction.
> 
> If anything, I am inclined to object to two:

We now have objections to one and to two; would anybody
care to try for three :-).

> Note that the RFC 3066 specifies a registry that does not include French
> language names. I suggest that this issue should be dropped.

Yes, the current IANA registry has that problem for
the non-ISO-based tags only. If the registry is to be
changed to subsume ISO codes as well, that defect should
be remedied.  I'm willing to postpone the discussion
(other problems with the proposed registry format dictate
a broader solution which could easily have provision for
an arbitrary number of descriptions).
 
> > I have an implementation which (in accordance with RFC 3066)
> > uses the official ISO lists. It has provision for displaying
> > ISO 639 language tags with their descriptions in either of the
> > two languages supported by the official 639 lists, and likewise
> > for the ISO 3166 country codes.  
> 
> RFC 3066 *does not at any point* suggest let alone state that
> implementations should use ISO 639 language names or ISO 3166 country
> names for UI purposes. IMO, you are creating an issue where none exists.

No, you are overlooking the fact that a set of codes with
no corresponding definitions is useless.  RFC 3066 defers
the code/definition pairs to ISO, which provides multilingual
definitions. The proposed draft would remove that multilingual
characteristic.
 
> > The specification of the
> > draft is *NOT* compatible with that existing implementation
> > because it removes the existing functionality of official
> > descriptions in French of language and country codes. As a
> > result of that incompatibility,  the newly proposed
> > specification does not work with (at least that one)
> > existing implementation (but I agree that that is a crucial
> > concern).
> 
> Display names for languages and countries are not within the scope of
> RFC 1766 or RFC 3066. It is preposterous to suggest that this draft is
> not compatible with existing implementations of RFC 3066 on that basis.

On the contrary, it is preposterous to suggest that codes
will be attached to text by magic; some human somewhere,
somehow is going to have to indicate the language to
something, and it certainly isn't going to be by way of
a 2- or 3-letter code without some reference to what those
codes *mean*.  And at the present time, the meaning of
those codes is defined -- bilingually -- in the ISO
lists.

> > It might be worthwhile considering the differences in the
> > way languages tags are used, by whom they are used, and for
> > what purpose.  There may well be a substantial difference
> > between use of a tag to represent an obscure dialect of a
> > dead language in a research paper vs. tagging a piece of
> > text in one of the core Internet protocols such as SMTP.
> > The draft seems to ignore the needs of the core Internet
> > protocols (e.g. unbounded tag length which is incompatible
> > with those protocols).
> 
> IETF language tags are used in a wide variety of applications. The
> parties involved in development of this spec (the authors and others)
> have examined these issues for the past several years and have arrived
> at this architecture.

Then why is there no provision for limiting the length of tags
to a range appropriate for encoded-words (N.B. we're not talking
about some obscure little-used protocol; we're talking about
email, which is widely recognized as one of the core Internet
application protocols!)?  Why, when I pointed out that the
length limit for an encoded-word, including charset name,
text, encoding, and syntactic glue, did you still assume that
limiting language-tags to 75 octets (leaving no room at all
for the text to be tagged (!!!) let alone the charset,
encoding, and necessary overhead?  Something tells me that
there has been insufficient attention paid to the implications
for core Internet protocols.

> But this spec is not an ISO standard; it is an IETF
> standard.

One that seeks to replace use of ISO standards with something
less functional.

> But
> you are simply adding localization requirements to a spec for i18n
> infrastructure, and I consider that not at all appropriate.

No, I am complaining about removal of internationalized
definitions associated with language tag components.
"Localization" would be translation of the French definition
into some other language.  That is not my concern. My concern
is the elimination of the French definition in the first place.

> > One part of my claim is that non-private-use RFC 3066 tags
> > up to the present time are no longer than 11 octets in length.
> 
> Only co-incidently at the present time.

As mentioned, under RFC 1766/3066 review/registration rules,
excessively long tags would certainly raise objections. That's
no coincidence -- it's an intentional design feature. 

> > As the draft, if/when approved, would close that registration
> > process, that limit (unless a longer tag is registered in
> > the interim) would apply for all time. 
> 
> And so that limit would be a constraint applying for all time to the
> 'grandfathered' production which concerned you so much.

And so it can easily be incorporated into that ABNF production.

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Bruce Lilly <[email-masked]>
Date: Sun, 12 Dec 2004 19:01:43 -0500

>  Date: 2004-12-12 15:33
>  To: [email-masked], [email-masked]
>  
> > The point is that under RFC 3066,
> > the bilingual ISO language and country code lists are
> > considered definitive.
> 
> That is nowhere stated or even suggested in RFC 3066.

RFC 3066 section 2.2 states, in part:

   - All 2-letter subtags are interpreted according to assignments found
     in ISO standard 639, "Code for the representation of names of
     languages" [ISO 639], or assignments subsequently made by the ISO
     639 part 1 maintenance agency or governing standardization bodies.

and has a similar statement regarding ISO 3166.

"interpreted according to assignments found in" certainly
sounds as if the ISO lists are considered definitive for
their respective categories of subtags, since their
interpretation is specified as that given in those lists.
I don't see how the RFC 3066 text can be interpreted
otherwise.

---

From: Bruce Lilly <[email-masked]>
Date: Sun, 12 Dec 2004 19:35:04 -0500

>  Date: 2004-12-12 15:34
>  From: John Cowan <[email-masked]>

> Of course countries change, and then the numeric country codes change
> as well.  The point is that the alpha codes change for political reasons
> when there has been *no* change in the underlying country:  Romania's
> 3-alpha code changed from ROM to ROU without any change in Romania at all.
> The CS case is particularly gratuitous, as its denotation changed from
> "Czechoslovakia" (a no longer existent country) to "Serbia and Montenegro"
> (a newly created country).

There is a limited supply of 2-letter codes and the supply
of 3-digit codes is only slightly greater.  Reassignment of
codes from such a limited supply is inevitable.  Better to
deal with the fact of tides than to try to command the tide
not to flow in.

> No, it attempts to freeze the code-to-country mapping at a single
> point.  New countries or changes in old countries should involve only the
> additions of codes, not the reuse of old codes.

Too late. King Canute commands the tide not to come in, but
his feet still get wet.  Better to deal with such change
appropriately rather than commanding countries (or
international standards bodies) not to change.
 
> I don't know.  Where is the implementor supposed to get the
> official German, or Catalan, or Mandarin translations?
> Not in the ISO registry, for sure.  To say nothing of the
> cases where no official translations exist.

But I'm not concerned with translations, but with the
definitions. And currently the definitions are available
in French and English.

> That count does not include dead languages.  Whether it includes
> dialects is a matter of terminology.

Fine. The point is that the draft provides for language tags
that are so long that they cannot be used with the core
Internet protocols. A tag associated with audio media doesn't
need a means to indicate script or other orthography -- they're
irrelevant for spoken material.  RFC 3066's provision for
registry worked well. Removing that requirement -- as the
draft would do -- necessitates a specific upper bound on
tag length that will work with existing core protocols, to
replace the reviewer, Area Director, and community review
process that ensure that current registered tags work with
those protocols.

---

From: Bruce Lilly <[email-masked]>
Date: Sun, 12 Dec 2004 20:35:18 -0500

>  Date: 2004-12-12 15:55

> You have not responded to the point that accessibility of source ISO standards is supposed to be a major factor, yet the draft itself clearly indicates otherwise.

The source for the statement claiming accessibility as a
major factor has been indicated to be the author or
authors of the draft.  I can't explain why it says what
it says; I suggest that you direct that question to
the author(s).
 
> > That is a problem for existing implementations of RFC 3066
> > tags, which can obtain official, internationally agreed
> > descriptions of the codes in two languages.
> 
> Descriptions (language names) are beyond the scope of RFC 3066. It is a non sequitor to claim that this draft creates a problem for existing implementations of RFC 3066 on this basis.

3066 refers to "interpretation" of the codes and defers
that interpretation to that given by the ISO lists. One
cannot have an "interpretation" based on the lists without
the natural language definitions which are paired with
the codes.  It is a fact that those definitions are
available in two languages in the ISO lists, and that
the proposed replacement for the ISO lists would eliminate
one of those languages.
 
> > OK, continuing your hypothetical example and its relationship
> > to language, suppose that there is another civil war and
> > that what now corresponds to "US" is split into Blue America
> > and Red America.  Further suppose that in due course ISO
> > assigns some other code to one of those countries and retains
> > "US" for the other, and that that happens after the proposed
> > registry is set up with a definition for "US" and some
> > description referring to the "old" use.
> 
> That is a scenario that has been well considered: it would be very bad IT practice to redefine a metadata tag "US" to have a narrower denotation than it previously did, as that immediately breaks an unknown amount of existing data. If ISO were to make such a change in the meaning of "US", then IT implementations *absolutely should not* follow suit; the ID "US" must retain it's prior, broader meaning.

So long as it is known what definition of "US" applied at
the time, there is no problem.  This is dealt with in IT
all the time; "EST" has had many definitions in terms of
exact offset from UTC, and when it goes into and out of
effect (likewise for other time zones).  Yet we manage to
be able to state with precision the offset and effective
times of "EST" well into the past, and without declaring
that a single value must hold true for all time. I have
provided a URI to the time zone data; a similar mechanism
could be used to track historical values for ISO language
and country codes.  Given the existence of such proven
technology, there is no need for the incompatible approach
outlined in the draft.

---

From: John Cowan <[email-masked]>
Date: Mon, 13 Dec 2004 07:58:48 -0500

Bruce Lilly scripsit:

> to hear you".  

Actually, 'yz' doesn't designate the country in the ISO standard,
as I explained yesterday.  Rather, it designates the *name* of the
country, which is of course subject to change *without* international
agreement.  In RFC 1766/3066, we attempt to use it to designate
the country, which requires some straining of the concept.

> As I have pointed out, politicians change the definitions of time
> zones frequently, and those who have to deal with time zone issues
> have found a way to cope with such change without trying to declare
> international standardization organizations irrelevant.

Ah, but you kick the ball through your own goalposts here.  The
Olsen time zone system is excellent -- but it becomes so only by totally
ignoring the customary names of time zones and inventing its own!
(Thus U.S. Eastern time is named "America/New_York", e.g.)  The
customary names are carried only as time zone abbreviations such as
"EST", which are not unique, are English-only, and most of which are
also made up.  (Countries with a single time zone generally don't
bother with an official name for it, with some obvious exceptions.)

> reached to change a few 2-letter codes.

Not much of an international agreement, really.

-- 
Samuel Johnson on playing the violin:           John Cowan
"Difficult do you call it, Sir?                 [email-masked]
 I wish it were impossible."                    http://www.ccil.org/~cowan

---

From: John Cowan <[email-masked]>
Date: Mon, 13 Dec 2004 08:06:22 -0500

Bruce Lilly scripsit:

> codes from such a limited supply is inevitable.  

In the very long run, yes; but even the 75-octet limit probably won't
stand in the *very* long run.  Countries and languages, as opposed
to codes for them, don't come and go like IETF protocols: many of
them have centuries of history, or half a century in the case of the
post-colonialist countries; the events of 1991-93 were historically
anomalous.

> his feet still get wet.  

Canute was making a moral object lesson about the limitations of
kingship, not acting like an idiot.

> definitions. And currently the definitions are available

What of it?  In what case does the provision of a French name
significantly tighten the definition provided by the English
name (or for that matter vice versa)?

> Removing that requirement [for registration] -- as the draft would do
> -- necessitates a specific upper bound on tag length that will work
> with existing core protocols, to replace the reviewer, Area Director,
> and community review process that ensure that current registered tags
> work with those protocols.

Michael, I assume you're ignoring this kerfuffle, and rightly so.
But for the record, have you ever been given cause to take into
account a hard limit in the length of language tags?

-- 
Here lies the Christian,                        John Cowan
        judge, and poet Peter,                  http://www.reutershealth.com
Who broke the laws of God                       http://www.ccil.org/~cowan
        and man and metre.                      [email-masked]

---

From: Mark Crispin <[email-masked]>
Date: Sun, 12 Dec 2004 16:20:43 -0800 (PST)

> fingers in one's ears and shout "NA-NA-NA-NA-NA I don't want

What is silly is saying that every language tag has to have a date/time 
attribute associated with it so that computer software managing that text 
knows the language of that text.

But that is precisely what you are advocating.

> reached to change a few 2-letter codes.

It's bad enough that TLDs get recycled.

It is a disaster for language identifiers to get recycled.  Something has 
to make those identifiers unique.  Your notion will force the inclusion of 
a date/time stamp in language tags, to restore the uniqueness that you are 
so excruciatingly eager to abolish.

> Never

For the 98% case, it does not matter at all.

But it does matter if, one day, "DE" becomes Denmark.

> of the currently valid RFC 3066 language tag "sr-CS".

No, it restores the previous meaning of sr-CS.

> definition finds that he has none and must resort to an

Why is the situation for French different from someobody wishing to 
provide a Lower Slobbobian definition?

> SO where are the French definitions?

Ask a person who is bilingual in English and French to provide one.

> user to select from "Abkhaze, Aceh, Acoli,..., Zhuang (Chuang),

Abkhaze, Aceh, Acoli,..., Zhuang (Chuang), Zoulou, and Zuni are not 
language tags.  So what's your point?

> be remedied.

Why is it a problem?  Why is it a defect?

> will be attached to text by magic

Here is where you are misled.  Many of these tags are embedded within the 
text itself.  That text may long outlive its author in an archive.

> My concern

Why is this a problem?

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: "Jo Wilkes" <[email-masked]>
Date: Mon, 13 Dec 2004 01:31:34 +1

Bruce Lilly <[email-masked]> wrote on 2004-12-12 T 18:44:27 -0500

(...)
> in the 1930s, the 1900s, ...?

As far as I can tell, this is about about /language/ tags, not about the 
tagging of borders or nationalities.
These last two have certainly influence on the use of language, but should not 
on the name and tagging of the language itself. Unless there is some forced 
change of language, which needs to be reflected in tagging.

(...)
> something, 

Tagging may also be done by some software instead of a human being. 
Whether you consider that "magic" is up to you, but I think both ways of 
finding a tag for a given text are possible.

> and it certainly isn't going to be by way of
> codes *mean*. 

I beg to differ. Many humans do not know the lists, not even their name, and 
yet they use the codes on a daily basis. They simply recall the codes they've 
seen and found relevant to them, like the TLD ones they are used to; or they 
are told by somebody "use this tag when using language A and that tag when 
using language B". This information may be wrong or outdated, of course.

Please do not assume tags will be assigned only by humans who have a recent 
list of the code(s) at hand.

Just my 0.02€.

Best regards,
J. Wilkes