Subject: The 'failure' of SMTP RE: DNS Choices: Was: [ietf-dkim] Re: Last Call: 'DomainKeys

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Wed, 22 Nov 2006 07:37:00 -0800

> From: [email-masked] 
> [mailto:[email-masked]] 
> Sent: Wednesday, November 22, 2006 7:41 AM
> To: [email-masked]
> Subject: Re: DNS Choices: Was: [ietf-dkim] Re: Last Call: 'DomainKeys
> 
> > > And since SMTP has been an utter and complete failure in 
> operations, 
> > > I find that to  be a dubious point.
> > Anything used by close to a billion people can't be classed 
> a complete 
> > failure.
> 
> The failure is not that it is ignored but that it is so 
> difficult to operate. Both the end users and the server 
> operators are unhappy with what they get from the email 
> system based around SMTP, POP, SUBMIT and IMAP.

But these failures have nothing to do with the DNS or the MX record. In fact the MX record is one of the conspicuous successes of the mail system, it allows a remarkable degree of fault tolerance and is the reason the system runs at all at this stage.

The DNS is a conspicuous success. Most global naming schemes fail. X.500 is dead, RealNames is dead and there are other schemes being perpetrated today that will go the same way. DNS did not have to succeed and it suceeded despite some unnecessary complications.

The problem with the mail system has nothing to do with the protocol performance. The problems are caused by PEOPLE.

In particular the protocols do not anticipate what is necessary to deal with a population of a billion users. That is a problem but not an operational problem, the problem is architectural. But moving to a different directory scheme (Hello John) or network architecture (Hello David) won't help unless the new architecture takes account of the real issue - people.  

Fortunately it is possible to retrofit infrastructure for dealing with people into the legacy systems which turn out to be rather better than the councils of despair would imply.

The early SMTP system held together because there was ACCOUNTABILITY. There were few limits on what you could do but if you messed up there were consequences.

The problem with SMTP is the lack of accountability in the billion user Internet. The accountability mechanisms of the NSFNET did not scale.

And so now we have ad-hoc measures in place that attempt to sort out the 'good email' from the 'bad email'. These don't work too well because computers don't have the faintest idea what good or bad is and its pretty hard to teach them.

What we can achieve is to sort email according to whether the sender can be held accountable for their actions or not. If someone sends me anonymous email it is unfortunately going to go straight to the bit bucket. I receive 3000 emails every day of which 300 are legitimate. I do not read all my email from people I know. I don't have time to read 2,700 advance fee frauds and ads for viagra.

Knowing who sent an email with a high degree of confidence is the first step towards knowing whether they can be held accountable.

SMTP does its intended job. DKIM adds a layer that is also foundational for achieving accountability.

Accountability in this case probably means 'if you sent me mail in the past which was authenticated and did not turn out to be spam I will accept this mail on more favorable terms, if on the other hand people complained about the mail you sent as spam then modulo the possibility that they might have lied this next mail you sent is also going to the bit bucket'.

I know that there are people who really do not like the concept of being held accountable (who does) but this is no different from what is going on today only at the moment you are being held accountable for mail you did not send.

Accountability is not incompatible with anonymity either. The Internet does not really provide true anonymity as many Internet criminals find to their cost. The Internet actually provides multiple levels of pseudonymity, you have an identity but one that cannot necessarily be correlated to your other identities.

It is certainly possible to have accountability with pseudonymity, the Slashdot karma system is an existence proof.

It is even possible to have accountability with complete anonymity if you use trustworthy hardware (OB Disclosure, patent is pending).

> It has nothing to do with software and everything to do with 
> architecture. IM networks have less problems because all the 
> participants share a relationship with the IM service 
> providers. Nobody has yet tried to build an open-ended email 
> network based on a chain of trust between participants. 
> Instead we have the flat SMTP protocol open to all comers and 
> two client protocols that do NOT support sending an email message.

On the contrary, I get calls from a new VC-backed startup touting exactly that type of scheme roughly every three months.

The proposal here is to lay the groundwork to allow the phased trasnsition of the existing email system to allow networks of mutually trusted parties to exchange email on an accountability basis. For example Yahoo and Gmail both send vast quantities of email to each other. Both companies impose internal velocity limits that are an effective block on spammers using their system to originate mail. Both companies are willing to trust the effectiveness of the other's controls. Both companies sign their email with DKIM. 

So there is actually an existence proof (or will be when the companies filter on DKIM data) for what you propose but using legacy SMTP and legacy DNS.

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: [email-masked]
Date: Wed, 22 Nov 2006 16:34:15 +0000

> The DNS is a conspicuous success. 

Exactly my point. The domain naming system is a
conspicuous success. Fiddling with the DNS is not
going to make that success rub off on SMTP et al.

> The problem with the mail system has nothing to do with the protocol
> performance. The problems are caused by PEOPLE.

SMTP, POP, IMAP and SUBMIT just don't fit the use cases
that these PEOPLE require. That means that the problems
with the protocols are caused by inadequate engineering.

> In particular the protocols do not anticipate what is necessary to 
> deal with a population of a billion users. That is a problem but not
> an operational problem, the problem is architectural. 

Agreed. The problem is with the *EMAIL* architecture. In the
beginning there was apparently only a very simple architectural
idea that involved leveraging IP networks to transfer the files
which were sitting in mail server spool directories. SMTP did
that job well but, even with band-aids applied, does not suit
the email architectural needs of the present day.

> a different directory scheme (Hello John) or network architecture 
> (Hello David) won't help unless the new architecture takes account 
> of the real issue - people. 

Agreed. A new email architecture must take into account the current
use-cases which involve large numbers of people, multiple languages
with no lingua-franca, primarily non-technical users, financially
motivated criminal gangs attempting to leverage weaknesses in the
architecture, and the knowledge that hierarchy tames complexity.

> Fortunately it is possible to retrofit infrastructure for dealing 
> with people into the legacy systems which turn out to be rather 
> better than the councils of despair would imply.

I don't disagree that band-aids can make things better for
a while. Or that huge efforts to create new band-aids are
better than tossing the whole thing out. But I also believe
that the band-aids are not a long term solution and that 
a series of band-aids is an extremely expensive way to reach
a satisfactory email architecture.

As far as I can see there is no good reason why the IETF
shouldn't undertake work to define a new email architecture
and then define the protocols required to create that new
email architecture.

> The early SMTP system held together because there was 
> ACCOUNTABILITY. There were few limits on what you could do but if 
> you messed up there were consequences.

Yes, there was accountability, but it was not in the email
architecture and it was wholly unconnected to the email
protocols and services. It turns out that a reliable email
system needs accountability imbedded in it, not tacked on 
as an afterthought. Here is where the lessons of hierarchy
taming complexity can be applied. Instead of a flat system
where millions of hosts talk SMTP to random other hosts,
we could have an architecture where hosts play a defined
role in a hierarchy of roles. There would be protocols
use for hosts to communicate with peers at the same level
of the hierarchy and others for end-hosts to communicate
up the hierarchy. And a CHAIN OF TRUST to bind them all
together in a single cooperative architecture.

Mail servers will still exchange messages with known and
trusted peers. A new mail server operator will have to
arrange a trusted peer relationship with one or more 
existing operators at some point in the hierarchy. A mail
user will have a trusted relationship with a local server
operator. Many messages will have to be relayed because
there is no one-level trust relationship between sender
and recipient. Mail will flow along the chain of trust.
And everybody will be motivated to keep those chains
intact because when they break, messages stop flowing.

> Knowing who sent an email with a high degree of confidence is the 
> first step towards knowing whether they can be held accountable.

Even knowing the last email hop with a high degree of
confidence will allow accountability. In order to know 
this with a high degree of confidence, the organizations
must have a relationship, probably contractual, and must
be in contact. The fact that a chain-of-trust email architecture
encourages organizations to maintain functional contacts
is a very good thing from an operational point of view.

> On the contrary, I get calls from a new VC-backed startup touting 
> exactly that type of scheme roughly every three months.

VC-backed schemes cannot solve such a large problem which
requires many organizations to participate in building a
chain of trust.

--Michael Dillon

---

From: Paul Robinson <[email-masked]>
Date: Wed, 22 Nov 2006 17:22:24 +0000

Ah, it feels like *ages* since we had a "SMTP is broken" thread on  
[email-masked] - I was wondering if some of you were dead or  
something. :-)

On 22 Nov 2006, at 15:37, Hallam-Baker, Phillip wrote:

> The problem with the mail system has nothing to do with the  
> protocol performance. The problems are caused by PEOPLE.

And to be fair, most of the problems with DNS are caused by people.  
It is not unique to DNS: it applies to virtually all protocols.

Protocols in the application/presentation space could be improved  
upon architecturally in their design to reflect that they are there  
to serve human operators and users, not to be academically perfect in  
a lab and fail when they get popular or in the hands of people who  
don't read RFCs. I think the IETF used to be quite good at this, but  
recent RFCs have left me a little nervous in this regard. Looking  
back, and I know this might be controversial, the tipping point seems  
to have started when Postel went to /dev/null.

> better than the councils of despair would imply.

I agree, however the infrastructure a lot of people have wanted to  
introduce in the past is brain-dead on multiple levels. That's why  
they're not rolled out yet.

However, I think the SPF 'hack' is a creative invention that does not  
seriously degrade SMTP in the slightest. I think DKIM is a clever use  
of existing protocols to help build accountability. But even  
combined, they are not enough. We need more ideas like that, and  
fewer that involve us handing over money or re-writing every MTA, MDA  
and MUA going. Even SPF and DKIM require serious deployment and  
development considerations and will struggle to get widely adopted.

> you messed up there were consequences.

There still are. Set up an open SMTP relay on your network. Leave it  
four weeks. Let's see if you still have an Internet connection. If  
you do, let's see if you can still send e-mail to the majority of MXs  
out there.

> The problem with SMTP is the lack of accountability in the billion  
> user Internet.

That is also its primary strength. The fact I can send you an e-mail  
out of the blue from my phone is a powerful factor in the growth of  
popularity of the medium.

> And so now we have ad-hoc measures in place that attempt to sort  
> out the 'good email' from the 'bad email'. These don't work too  
> well because computers don't have the faintest idea what good or  
> bad is and its pretty hard to teach them.

Actually, I find Bayesian classification extremely good.

We have plenty of tools for fighting spam, we're just not deploying  
them far and wide enough. We could be trying to formalise those anti- 
spam techniques that we know are so successful when deployed. There  
are lots of things can be done, and we should really question why  
they're not being done - I think that is an architectural problem  
too, just within the IETF itself.

This isn't a complaint: just an observation.

DKIM is a great step in the right direction, but it needs wide-scale  
adoption to be successful in reducing network-wide levels of spam. It  
is also likely that spammers will do as they have done in the past  
and find a work-around. I also fear it won't make it to wide  
deployment because most admins running a recursive resolver will see  
the overhead the cache will need and maybe, just maybe, panic a  
little bit.

Personally, I also think DKIM looks like a great way to DoS a DNS  
server with lots and lots and lots of keys, but maybe I'm not reading  
the drafts in enough detail because I'm some way off needing to  
implement it anywhere.

That said, I hope the obstacles are worked around, the problems  
associated with humans being able to play with this are catered for  
(much like the human problems of SMTP) before it gets widely  
deployed, and it achieves its aims.

> What we can achieve is to sort email according to whether the  
> sender can be held accountable for their actions or not.

Users won't accept such a protocol in a World where they are quickly  
tiring of the "authorities" being able to do what they claim a legal  
right to do.

If users won't adopt it, you're wasting your time, c.f. IPv6.

> If someone sends me anonymous email it is unfortunately going to go  
> straight to the bit bucket.

I think you mean 'pseudonymous' which is completely different to  
'anonymous'.

If I send you mail, and I'm not in your address book, am I anonymous?  
DKIM doesn't get around this: if I mail you from a gmail account, and  
the header is there signed by Google saying "yup, he's one of ours"  
it doesn't stop the fact that I might be using a pseudonym and  
attempting to spam you.

> first step towards knowing whether they can be held accountable.

You are welcome to create your own whitelists, of course. Just don't  
expect the rest of the planet to change their habits around your vision.

> SMTP does its intended job. DKIM adds a layer that is also  
> foundational for achieving accountability.

Its a piece of the puzzle, yes. However, it is not the magic piece  
some might expect it to be. It will reduce/eliminate phishing if  
widely adopted by banks and major e-commerce sites, it will allow  
some levels of accountability, but it will not eliminate spam.

All DKIM gets you fundamentally is SPF with the ability for an MTA to  
determine "you are who you say you are, but some people think you're  
a prick". That doesn't help as much as you think it will.

> I know that there are people who really do not like the concept of  
> being held accountable (who does) but this is no different from  
> what is going on today only at the moment you are being held  
> accountable for mail you did not send.

No I'm not. I have never had somebody e-mail me saying "why did you  
spam me with some viagra stuff?" because they will quickly know it  
wasn't me.

What's more, bayesian classifiers are *dramatically* reducing the  
amount of spam users see. A lot of the figures we look at when  
determining the spam problem is based on the amount of messages  
thrown across port 25 and don't take into account what happens in the  
MUA. Add in to the mix that third-party groups such as spamhaus are  
thankfully covering our backs and doing things the IETF WGs should be  
patching over with new services, the amount of spam a user actually  
gets in their inbox to read is minimal: the majority just see spam go  
straight to their junk folder.

> Accountability is not incompatible with anonymity either.

Yes it is. I must get authorisation for a set of actions for somebody  
to say "this person is accountable to us for the production of this e- 
mail", and whilst authorisation is not the same as authentication, to  
protect the former, I need to engage in the latter. If I need to  
authenticate myself I must by definition lose some aspect of anonymity.

If a system does not require authentication to protect authorisation,  
the accountability assigned to that authorisation is undermined and  
becomes meaningless.

In other words, if I opened up an open relay but DKIM signed every  
outgoing mail, I will have created the only accountable but truly  
anonymous e-mail system in the World. If I want my users to be  
accountable in a meaningful way, I can not afford them the luxury of  
any sense of anonymity. This is problematic.

> The Internet does not really provide true anonymity as many  
> Internet criminals find to their cost.

Many criminals are never caught. There are likely thousands of crimes  
going on right this very second online that will never be detected,  
never mind the perpetrator caught.

> It is certainly possible to have accountability with pseudonymity,  
> the Slashdot karma system is an existence proof.

You're not being serious are you?

> exactly that type of scheme roughly every three months.

I hope you told them to give the money back.

--
Paul Robinson
http://vagueware.com

---

From: Tony Finch <[email-masked]>
Date: Wed, 22 Nov 2006 18:06:08 +0000

On Wed, 22 Nov 2006, [email-masked] wrote:
>
> Mail servers will still exchange messages with known and trusted peers.
> A new mail server operator will have to arrange a trusted peer
> relationship with one or more existing operators at some point in the
> hierarchy. A mail user will have a trusted relationship with a local
> server operator. Many messages will have to be relayed because there is
> no one-level trust relationship between sender and recipient. Mail will
> flow along the chain of trust. And everybody will be motivated to keep
> those chains intact because when they break, messages stop flowing.

This sounds like the Usenet architecture, or the BGP architecture.

Usenet did not escape spam. Spammy usenet servers were not reliably cut
off - certainly the trust relationships between server operators did not
provide an effective way to stop spam. Your last sentence above is the
reason why: keeping legitimate communication working is more important
than the inconvenience of spam.

You can apply the same logic at the level of BGP routing: there
are trust relationships between networks, some of which are clean and some
of which are infested with criminals. The latter spoil it for the rest of
us but they are still not cut off.

For a third example of reluctance to punish the innocent, look at the
hatred directed at DNS blacklists that deliberately block people who are
unlucky enough to be too close in network space to spammers.

Given this, your proposed architecture is just as vulnerable to botnets as
the open SMTP architecture. There are always going to be enough admins who
don't cut off infected machines and who also have enough legitimate
customers that their upstreams won't cut the whole network off. This will
be enough to poison the well.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
BAILEY: CYCLONIC BECOMING NORTHWESTERLY SEVERE GALE 9 TO VIOLENT STORM 11,
OCCASIONALLY HURRICANE FORCE 12 IN SOUTH, DECREASING 7 TO SEVERE GALE 9 LATER.
HIGH OR VERY HIGH. RAIN OR SQUALLY SHOWERS. MODERATE OR POOR.

---

From: Douglas Otis <[email-masked]>
Date: Wed, 22 Nov 2006 10:51:56 -0800

On Nov 22, 2006, at 9:22 AM, Paul Robinson wrote:

> All DKIM gets you fundamentally is SPF with the ability for an MTA  
> to determine "you are who you say you are, but some people think  
> you're a prick". That doesn't help as much as you think it will.

While greatly reduces false-positive filtering of phishing attempts,  
DKIM does _not_ identify the MTA (SMTP client).  While there is often  
a desire to associate various email related domains with SMTP clients  
when gauging acceptance, SPF does not offer a safe method for this.   
Associations using name comparisons rather than address lists can be  
much safer using small and simple answers.

The answer satisfying SPF makes address-path authorization both  
impractical and highly dangerous.  Currently SPF scripts may invoke  
100 DNS targeted transactions per each email-address resolution; for  
more than one per message, and more than once along the delivery  
path.   While most will disable scripts found within an anonymous  
email, how is executing SPF scripts stored in DNS any different?   
Surely script stored in DNS does not make it safe.

-Doug

---

From: Dave Cridland <[email-masked]>
Date: Wed, 22 Nov 2006 20:22:28 +0000

> requires many organizations to participate in building a

Given the numbers of millionaires and contented shareholders that 
existing "chains of trust" such as TLS certificates provide, I can 
well imagine that VCs will happily back such a solution.

Now, tell me, how much do I have to pay to send my email under your 
new system, and who shall I make the cheque out to?

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade