Subject: Objection to Last Call - draft-ietf-eai-utf8headers-09.txt

From: Charles Lindsey <[email-masked]>
Date: Tue, 18 Mar 2008 15:13:00 GMT

This is a formal objection to the Last Call of
     draft-ietf-eai-utf8headers-09.txt [EAI-utf8header].
Insofar as fixing the problem I shall describe would have repercussions in
     draft-ietf-eai-smtpext-11.txt [EAI-SMTP-extension] and in
     draft-ietf-eai-dsn-06.txt [EAI-dsn]
(especially the latter), it should be construed as a formal objection to
those as well.

Essentially, my concern is that a new MIME subtype message/global is
defined, and that this new message subtype is in breach of RFC2045, which
currently has the status of Draft Standard.

Introduction and Terminology
----------------------------

[EAI-utf8header] describes, for Experimental use, a new class of Internet
Email Messages, which we may term "Global Messages" which it is hoped will,
in the long term, become a replacement for the messages defined in RFC2822
(plus the MIME standards RFC 204[56]), and which we may term "Current
Messages".

Global Messages are a superset of Current Messages, defined by adding extra
features (notably the use of UTF-8 within headers) by way of extensions to
RFC 2822 and RFC204[56], plus associated extensions to RFC2821 in
[EAI-SMTP-extension]. They are intended to be created, transported and read
by suitably augmented MTAs and MUAs which will advertise the capability
"UTF8SMTP". Whenever a Global Message is offered to an agent which does not
support UTF8SMTP, it MUST either be rejected, or alternatively "downgraded"
to become a valid Current Message capable of being transported over the
"Curent Network" of installed MTAs (possibly being upgraded after final
delivery).

I have no problem with defining extensions to RFC 2822 and even to
RFC204[56] (e.g. to permit UTF-8 in places where those current documents do
not allow it). However, this proposal goes beyond that by introducing
objects that are intended to be transported over the Current Network in a
manner at variance with RFC2045.

Message/global
--------------

[EAI-utf8header] states, in Section 1.2:

   This document updates section 6.4 of RFC 2045.  It removes the
   blanket ban on applying a content-transfer-encoding to all subtypes
   of message/, and instead specifies that a composite subtype MAY
   specify whether or not a content-transfer-encoding can be used for
   that subtype, with "cannot be used" as the default.
 
   This document also updates [RFC2822] and MIME, and the fact that an
   experimental specification updates a standards-track spec means that
   people who participate in the experiment have to consider those
   standards updated.

   Allowing of use a content-transfer-encoding on subtypes of messages
   is not limited to transmissions, which are authorized by the SMTP
   extension specified in [EAI-SMTP-extension].  Message/global permits
   use of a content-transfer-encoding.

The third paragraph of that explicitly seeks to permit the extension
specified in the first paragraph to be used in messages handled by agents
which do not support UTF8SMTP as defined in [EAI-SMTP-extension].

More specifically, Section 4.6 describes the new MIME subtype message/global
which is intended to encapsulate Global Messages in the same way that
message/rfc822 is intended to encapsulate Current Messages. The Template for
this new MIME subtype includes the words:

   Encoding considerations:  Any content-transfer-encoding is permitted.
      The 8-bit or binary content-transfer-encodings are recommended
      where permitted.

And furthermore it is claimed that, even without downgrading, this new MIME
subtype will interoperate with the installed base:

   Interoperability considerations:  The media type provides
      functionality similar to the message/rfc822 content type for email
      messages with international email headers.  When there is a need
      to embed or return such content in another message, there is
      generally an option to use this media type and leave the content
      unchanged or downconvert the content to message/rfc822.  Both of
      these choices will interoperate with the installed base, but with
      different properties.  Systems unaware of international headers
      will typically treat a message/global body part as an unknown
      attachment, while they will understand the structure of a message/
      rfc822.  However, systems which understand message/global will
      provide functionality superior to the result of a down-conversion
      to message/rfc822.  The most interoperable choice depends on the
      deployed software.

Now message/global needs to be able to pass unscathed through a chain of
agents such as the following:

        A                   B                   C                   D
   Global MTA    ----> Current MTA ---->   Current MTA  ---->  Current MUA
    supporting          supporting         not supporting 
   both UTF8SMTP         8BITMIME            8BITMIME
   and 8BITMIME

and [EAI-utf8header] proposes to allow A to pass a message/global to B
as-is, in the expectation that B will then apply a Content-Transfer-Encoding
such as Quoted-Printable or Base64 before passing it to C, which would then
be decoded by D.

The problem
-----------

However, RFC2045, with which B and C as part of the "installed base" are
supposed to comply, contains the following (Section 6.4):

   Certain Content-Transfer-Encoding values may only be used on certain
   media types.  In particular, it is EXPRESSLY FORBIDDEN to use any
   encodings other than "7bit", "8bit", or "binary" with any composite
   media type, i.e. one that recursively includes other Content-Type
   fields.  Currently the only composite media types are "multipart" and
   "message".  All encodings that are desired for bodies of type
   multipart or message must be done at the innermost level, by encoding
   the actual body that needs to be encoded.

The Encoding Consideration quoted above, which allows B to apply that
Encoding, is therefore in direct violation of that "EXPRESSLY FORBIDDEN".

Now I might well agree that that restriction is unfortunate and misguided,
and even that it ought to be changed. But RFC2045 is a Draft Standard, and
it is entirely outside the remit of the EAI WG to attempt to change what is
in a Draft Standard.

Section 5.2.4 of RFC2046 (which is also a Draft Standard) contains the
following:

   MIME implementations must in general treat unrecognized subtypes of
   "message" as being equivalent to "application/octet-stream".

   Future subtypes of "message" intended for use with email should be
   restricted to "7bit" encoding. A type other than "message" should be
   used if restriction to "7bit" is not possible.

and on that basis it is argued in [EAI-utf8header] (4.6) that existing
software such as B or C would therefore treat the unrecognized message
global as an "application/octet-stream" and therefore generate (C) or accept
(D) Quoted-Printable or Base64 accordingly. However, the second paragraph
quoted above makes it clear that the encoding of any future message subtype
should be "7bit", thus supporting that "EXPRESSLY FORBIDDEN". Moreover,
RFC2046 Section 4.5.1 discourages any further processing of an
application/octet-stream:

   The recommended action for an implementation that receives an
   "application/octet-stream" entity is to simply offer to put the data
   in a file, with any Content-Transfer-Encoding undone, or perhaps to
   use it as input to a user-specified process.

Furthermore, if the interpretation of RFC2046 Section 5.2.4 suggested in
[EAI-utf8header] is correct, it follows that RFC2046 is inconsistent with
RFC2045. Inconsistency between two Draft Standards is a serious matter, but
one which is to be resolved by the IESG rather than by an individual WG
such as EAI.

Consequences for the Installed Base
-----------------------------------

Now I might have been persuaded that this violation of existing standards
should be allowed to pass if it were the case that the "installed base" had
already taken that particular interpretation and was already setup, when
asked to pass an unknown message subtype to an agent which did not advertise
8BITMIME, to encode it in the manner that would have been appropriate for an
application/octet-stream. But such is patently not the case.

It has been established, for example, that none of Sendmail, Exim or Qmail
take that approach, and between them these three account for a very
substantial part of the installed base. Sendmail, in particular, takes great
pains to follow the current standards as written. I asked repeatedly on the
WG for examples of current implementations that would behave in the manner
expected, but was told of only one (M-Switch, by Isode, which is used by
only a fraction of current sites), and it was not even certain whether even
that one would do so.

It can only be concluded, therefore, that the current installed base is
unsuited to handle the new message/global in the manner envisaged.

What to do about it?
--------------------

My proposed solution to the problem would be to declare an extension
allowing Global Messages to be encapsulated within the present
message/rfc822 subtype , but only when passed between agents supporting these
extensions, and for it to be downgraded to an unextended message/rfc822
before passing into the Current Network. This would require a recursive
downgrading of all the Global objects within it, but current agents already
do something similar (or are supposed to do so) when passing messages to
non-8BITMIME systems, and so they already contain the necessary basic
framework for that purpose.

It should be noted that [EAI-dsn] requires this problem to be adequately
solved, since DSNs regularly include the full message which is the subject
of the Notification, and even when this is a Global Message the DSN may
still need to pass through (perhaps several) Current MTAs. [EAI-dsn]
currently proposes to use the new message/global for this purpose, but this
simply will not work for the reasons given.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5
_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: "Frank Ellermann" <[email-masked]>
Date: Wed, 19 Mar 2008 09:09:22 +0100

Charles Lindsey wrote:

> fixing the problem I shall describe would have repercussions in
> (especially the latter), it should be construed as a formal
> objection to those as well.

[procedural nit]
IMO it is slightly beside the point to dispute the right to start
a "Last Call".  If it doesn't get out of hand ADs can "Last Call"
whatever and whenever they like.  Figuring out what the community
wants before stating what the consensus is is the purpose of this
exercise, a good reason for objections could be when the alleged
consensus is dubious.  

[technical]
Of course the EAI drafts to varying degrees depend on each other,
but I'm confident that UTF8SMTP (the SMTP extension) makes sense,
therefore I posted a draft with a normative UTF8SMTP reference
yesterday, <http://tools.ietf.org/html/draft-ellermann-spf-eai>.

From my POV the DSN draft is also in excellent shape.  I'd like
it better to reference BCP 137, because the DSN draft ended up
with using an in theory problematic "unicode escape", but that's
perfectionism, at best an editorial nit.

> my concern is that a new MIME subtype message/global is
> defined, and that this new message subtype is in breach
> of RFC2045, which currently has the status of Draft Standard.

I strongly support to introduce message/global, an EAI message
is *NOT* an ordinary US-ASCII message/rfc822, an EAI message
uses UTF-8 "as is" in its header.  Simplified message/global
vs. message/rfc822 is like IRI vs. URI, or IDNAbis vs. LDH, and
it is very important to have these differences clear.

Taking the message/global idea as given the details of its
definition are indeed fascinating.  I had a long chat with Ned
(the co-author of RFC 2045) some days ago on the SMTP list in
a thread about "I18N considerations" for the email-arch draft,
mostly we discussed message/global and the history of MIME,
see <http://thread.gmane.org/gmane.ietf.smtp/6678/focus=6697>

Highly recommended for reading, I dare not summarize it here.

> I have no problem with defining extensions to RFC 2822 and
> even to RFC204[56] (e.g. to permit UTF-8 in places where
> those current documents do not allow it).

I'd prefer to stay away from MIME, for the job at hand (EAI)
it is unnecessary to "embrace and extend" MIME version 1.0.
There are "only" two places in the EAI drafts where MIME is
touched:  

1 - The use of a "nested" CTE B64 if all else fails to send
    DSNs to an EAI sender with a 7bit bit hop on the route.
2 - The use of UTF-8 in MIME version 1.0 part headers, this
    can require to parse the MIME structure of a message to
    figure out if it's a message/global or a message/rfc822,
    instead of only looking at the message header.

I'm concerned about (2), different from your concerns.  And I
considered (1) as negligible, maybe a 2045 erratum.  Nobody -
now also including Ned - liked my erratum idea, admittedly it
was formalistic, it would be "editorial", not "technical".

[...skipping your problem description...]

> I might well agree that that restriction is unfortunate and
> misguided, and even that it ought to be changed. But RFC2045
> is a Draft Standard, and it is entirely outside the remit of
> the EAI WG to attempt to change what is in a Draft Standard.

Oddly you don't have this problem with (1), and I don't have
it with (2).  For both issues I think they could be fixed in
MIME, and I don't insist on an erratum or 2045bis for (2), or
on a MIME versin 1.1 for (1).  And for you issue (= my 2) it
might be precisely the purpose of an "IETF experiment" to see
if it works out in practice.

I could say the same about (1), but I'd hate it if (1) turns
out to be critical and kills the EAI effort, because (1) is
not really necessary to experiment with "Email Address I18N".

Arguably (2) is also not really necessary, but Ned told me
that application/message ideas to solve this problem are old
and never made it, and without something in the direction of
application/message (2) is necessary.  Somehow those DSNs to
an EAI sender with 7bit hops on the return path must make it.

This should be a rare situation, what does an old 7bit relay
do on the return path to an UTF8SMTP sender, and accepting it
as given oddity all UTF8SMTP senders can be expected to know
how to "unpack" a B64 message/global part in a DSN.

Of course the security considerations need to be very clear.
B64 message/global is not designed for forward paths, it is
no ersatz-downgrading, it is a loophole for rare cases where
the alternative would be to drop DSNs.

> Furthermore, if the interpretation of RFC2046 Section 5.2.4
> suggested in [EAI-utf8header] is correct, it follows that
> RFC2046 is inconsistent with RFC2045. Inconsistency between
> two Draft Standards is a serious matter, but one which is
> to be resolved by the IESG rather than by an individual WG
> such as EAI.

For my purposes I "resolved" it in a discussion about I18N in
email-arch.  I didn't know that apparently technical details
in MIME were actually results of "political" decisions at the
time when 204[56] were published (1996, twelve years ago).

 Frank

---

From: "Frank Ellermann" <[email-masked]>
Date: Wed, 19 Mar 2008 09:32:42 +0100

Apologies, I confused (1) and (2):
  
 [...]
>     instead of only looking at the message header.
[...]
| Oddly you don't have this problem with (2), and I don't have
| it with (1).  For both issues I think they could be fixed in
| MIME, and I don't insist on an erratum or 2045bis for (1), or
| on a MIME version 1.1 for (2).  And for your issue (= my 1) it
| might be precisely the purpose of an "IETF experiment" to see
| if it works out in practice.
 
| I could say the same about (2), but I'd hate it if (2) turns
| out to be critical and kills the EAI effort, because (2) is
> not really necessary to experiment with "Email Address I18N".
 
| Arguably (1) is also not really necessary, but Ned told me
> and never made it, and without something in the direction of
| application/message (1) is necessary.  Somehow those DSNs to
> an EAI sender with 7bit hops on the return path must make it.
[...]

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 20 Mar 2008 10:46:31 +1300

Hi Charles,

On a procedural point:

On 2008-03-19 04:13, Charles Lindsey wrote:

> But RFC2045 is a Draft Standard, and

This draft does lack a couple of headers:

Updates: 2045 (if approved)
Intended status: Experimental

and equivalent text in its Abstract.

I'd be very surprised if EAI didn't need to update a whole bunch
of earlier standards. The idea of Experimental RFCs updating
standards-track RFCs is a tricky one though. It can't reasonably
be avoided in some cases, but it seems like something that should
be very carefully considered by the IESG, to check operational
compatibility.

Regards
   Brian Carpenter
   University of Auckland

---

From: Charles Lindsey <[email-masked]>
Date: Sat, 22 Mar 2008 21:13:35 GMT

On Wed, 19 Mar 2008 09:32:42 +0100

 [...]
>>     instead of only looking at the message header.

Unfortunately, the threading on the [email-masked] archive had got all
screwed up, and you were not replying to the message you thought you were
replying to.  Indeed the message you were replying to is listed in the
archive as [no subject] and has no headers. So I have recovered it and
quoted it in full below so that I may answer it. I have copied the
References header from your last message, so this should now get threaded
somewhere in the right place, but since none of the MUAs available to me
allow manual tinkering with References (and, worse, the ietf archives
don't show Message-ID headers at all), I am constructing this message by
hand and will have to send it using sendmail -t :-( .

Anyway, here goes.

Sometime on Mar 19th, (archived as [no subject])

>it is very important to have these differences clear.

I might not have had any problem with message/global if it had only been
intended to be seen inside the "global sandbox". But, unfortunately it is
expected to be passed around _outside_ that sandbox, on the presently
deployed network and, in spite of some wishful thinking on the EAI WG,
that just will not work (as I showed).
>Highly recommended for reading, I dare not summarize it here.

I have read that thread, and it is clear that Ned would take a very dim
view of disobeying that "EXPRESSLY FORBIDDEN", especially any attempt to
do so on the existing network. It may, with hindsight, have been the wrong
decision, but it is there and that is what lots of agents have
implemented. Moreover, as I showed at the start of this thread, even RFC
2046 does not permit what EAI is trying to do (yes, there is a sentence
about treating unknown stuff as application/octet-stream, but the very
next sentence following that makes it clear that does not extend to using
Q-P or Base64 as a CTE).
>I'd prefer to stay away from MIME, for the job@hand (EAI)
>    instead of only looking@the message header.

Folks should note this is where Frank confused (1) and (2). I have
annotated his remarks with what he meant to say, inside [..].
>Oddly you don't have this problem with (1[2]), and I don't have
>it with (2[1]).  For both issues I think they could be fixed in
>on a MIME versin 1.1 for (1[2]).  And for you issue (= my 2[1]) it

It doesn't need any experiment. It is already known that 3 widely used
MTAs will not cope with it, and I am only aware of one (not so widely
used) that will. As I said above, no problem if (1) happens inside the
global sandbox, but it is explicitly intended to work outside it.
>
>I could say the same about (1[2]), but I'd hate it if (1[2]) turns
>out to be critical and kills the EAI effort, because (1[2]) is

But EAI is trying to do far more than fix "Email Address I18N".
>
>Arguably (2[1]) is also not really necessary, but Ned told me
>application/message (2[1]) is necessary.  Somehow those DSNs to
There are two ways to fix (1). Either an application/message, or else
insist the message/global is downgraded before letting it loose on the
current network (in which case you might as well continue to call it (an
extended) message/rfc822.

>how to "unpack" a B64 message/global part in a DSN.
It may well be rare, but the IETF places great store on not breaking
existing implementations, however ancient. You and I might agree that is
an unnecessary straightjacket, but there it is :-( .

>the alternative would be to drop DSNs.

But that is NOT what the utf8headers draft says.
>in MIME were actually results of "political" decisions@7)
the
>time when 204[56] were published (1996, twelve years ago).

No, you didn't resolve it. What I think we really need is for Ned to join
this discussion and to read my objection. I believe he will uphold my
position but, if not, then I might be prepared to shut up. Cc to Ned.

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 25 Mar 2008 02:23:12 +0100

Charles Lindsey wrote:

> none of the MUAs available to me allow manual tinkering with
> References (and, worse, the ietf archives don't show Message-ID
> headers at all)

From my POV <news://news.gmane.org/[email-masked]>
was still okay apart from my unnecessary (1) and (2) confusion,
same as <http://article.gmane.org/gmane.ietf.general/29670/raw>,
whatever happened was between the list and the IETF archive.

> I might not have had any problem with message/global if it had
> only been intended to be seen inside the "global sandbox". But,
> unfortunately it is expected to be passed around _outside_ that
> sandbox, on the presently deployed network and, in spite of
> some wishful thinking on the EAI WG, that just will not work 
> (as I showed).

I never really believed in the "sandbox" approach, any MIME type
can be the Content-Type of a message/rfc822, or a part of a MIME
multipart/*, if that's an image/x-never-heard-of, message/global,
or any other MIME object.  My MUA would hopefully treat them all
like an application/octet-stream, or IOW as "dunno", I can save
it as unidentified file, and maybe tackle it with a hex. editor
or similar.

OTOH nobody has a reason to send me any image/x-never-heard-of
or message/global in MIME parts.  Unless it is in a DSN, because
I originally sent an EAI message, and in that case my MUA should
know what a message/global is.  

What could go wrong is that somebody gets an EAI message/global,
and sends it as an "attachment" of a message/rfc822 to somebody
else.  Ideally the MUA could say "don't do this", but if MUAs
try to be smart it usually gets worse, I'm not optimistic that
this part of the "sandbox" approach works.

>> <http://thread.gmane.org/gmane.ietf.smtp/6678/focus=6697>
 
> I have read that thread, and it is clear that Ned would take
> a very dim view of disobeying that "EXPRESSLY FORBIDDEN", 
> especially any attempt to do so on the existing network.

> It may, with hindsight, have been the wrong decision, but it
> is there and that is what lots of agents have implemented.

I don't see how existing software could have implemented any
special treatment of new unknown message/* subtypes, they can't
know what it is, they are forced to handle it as "dunno", no
matter what the CTE is.  Actually they should know what B64 or
QP is, and unpack the unknown object so far when the user wants
to save it as file.  And the B64 or QP case is limited to rare
cases where a message/global needs to make it over a 7bit hop.

Or the mentioned message/global "attachment" case, but that's
not better or worse than image/x-never-heard of.  Insisting on
"EXPRESSLY FORBIDDEN" would mean that the MUA of the receiver
refuses to unpack it when the user tries to save it as file.

I don't know what MUAs do with a B64 encoded message/unknown,
but the spec. clearly says "treat as application/octet-stream",
IMO refusing to unpack B64 (or QP) would be just stupid.

Of course MIME tries to avoid the situation, it is designed to
cause as less pain as possible for MIME unaware MUAs, i.e. old
MUAs with no idea what B64 or QP is.  Violating the "EXRESSLY
FORBIDDEN" can hurt MIME-unaware MUAs, users won't see as much
as possible of what's inside the message/global.

But "fixing" this by using application/message, where B64 and
QP is allowed, won't improve the result for MIME-unaware MUAs,
they still have no clue how to unpack it, and how to show the
hidden structure.  IMO the whole point of "EXPRESSLY FORBIDDEN"
is to have a visible internal structure for MIME-unaware MUAs,
users can then at least read any "internal" text/* etc. parts.

> It is already known that 3 widely used MTAs will not cope
> with it, and I am only aware of one (not so widely used)
> that will.

MTAs have no business to look into the body of a mail unless
they are gateways from say 8BITMIME to 7bit.  Whatever they do
today with an 8bit message/unknown in this situation, they'd
also do it with an 8bit message/global while they don't know
what it is.  If they can't handle an 8bit message/unknown they
also can't handle an 8bit message/global in this scenario, but
at this point the CTE is 8bit, not B64 or QP.  I don't see why
permitting QP or B64 makes it worse as it is.  The gateway is
broken, if it can't handle 8bit message/unknown it should not
have accepted it for delivery.

> EAI is trying to do far more than fix "Email Address I18N".

Yes, that is the part I don't like, EAI trying to do only EAI
is interesting enough for the experiment.

> There are two ways to fix (1). Either an application/message,
> or else insist the message/global is downgraded before 
> letting it loose on the current network

For the case with a MIME-unaware MUA application/message isn't
better.  For a broken gateway accepting 8bit message/unknown
without a clue what to do with it offering a general solution
not limited to message/global as application/message can help,
but it still requires to fix the broken gateway (in that case
to use B64 application/message subtype=global or similar).

The WG didn't like this idea, and limited to message/global
using B64 message/global has the same effect.  In both cases
the broken gateway has to be fixed.  No idea why EAI didn't
want a general solution.

For your other proposal, insist on downgrading message/global
at the UTF8SMTP to 8BITMIME gateway (before the hard case 7bit
enters the picture at the same or a later hop), I don't know...

When we talked about it IIRC we both agreed that MTAs still not
supporting 8BITMIME are anyway doomed.  I didn't look into any
downgrading draft for ages, the last state I know was not good.

> But that is NOT what the utf8headers draft says.

The MIME registration template does, sort of:
| 8-bit or binary content-transfer-encodings are recommended
| where permitted.

Well, this stuff could be clearer, it's spread over three drafts,
the point where message/global is explicitly required is in the
DSN draft, the point where it's implicity used in in the smtp-ext
UTF8SMTP draft (explaining reject vs. forward), and finally the
definition is in utf8headers.  

IMO pretending that other cases (neither UTF8SMTP nor DSN) are
"impossible" won't fly, those mad "attachments" are at least in
theory possible, no matter what the drafts say.  Adding a DON'T
somewhere is an idea, but I would not believe that it works to
get a real "message/global sandbox".

 Frank