Subject: Re: Email messages: How large is too large?

From: Bob Braden <[email-masked]>
Date: Tue, 14 Dec 1999 18:46:38 GMT

*> 
  *> It's actually worse than that.  Instead of demanding a way to move large
  *> files, they demand a way to move large email messages, and they cannot
  *> understand the distinction.
  *> 

Vernon,

As I recall, the reason that Mime was developed was precisely to allow
email to substitute for many file transfers.  Before Mime, it was
always a bit of an annoyance/embarassment that email could not be used
in place of FTP for binary files.

But I guess we forgot to take the next big step, redesigning email to
properly scale to handling arbitrarily large messages in a relatively
graceful manner when necessary.

Bob Braden

  *> 
  *> ] If the user would like to transfer 28 MB we should make it possible
  *> ] (there is always somebody who is in front of the big group, so I do not
  *> ] say that just because one person wants it we have to make it possible).
  *> 
  *> There are scaling problems.  Consider how many disk drives an ISP would
  *> need to dedicate to mail directories to support many users moving 28 MByte
  *> messages.  A 32 GByte drive can hold only about 1000 such messages.
  *> Consider an ISP with 1,000,000 customers.   Remember the bad old days
  *> before the ESMTP extension when a second SMTP hop would run out of space
  *> in its spool directory, and the message would try to bounce.
  *>

---

From: Ned Freed <[email-masked]>
Date: Tue, 14 Dec 1999 13:11:45 -0800 (PST)

> in place of FTP for binary files.

Actually, the motivation for developing MIME was internationalization of email.
The ability to send content of types other than text was a secondary
consideration.

> graceful manner when necessary.

I remain to be convinced that problems handling large messages have
much if anything to do with the modern ESMTP protocol. It seems to me
that it has a lot more to do with implementation and deployment.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Tue, 14 Dec 1999 13:11:45 -0800 (PST)

Ned

---

From: Dave Crocker <[email-masked]>
Date: Tue, 14 Dec 1999 13:45:45 -0800

>that it has a lot more to do with implementation and deployment.

Adding to Ned's response --

In fact ESMTP is getting quite good at supporting large file transfers:

(0.  Pipelining isn't really relevant for bulk transfer, but it doesn't 
hurt and is at least worth mentioning explicitly so no one thinks it was 
forgotten.)

1.  ESMTP Checkpoint/restart permits recovering from interrupted sessions; 
too bad it's not well deployed

2.  ESMTP Message Size Declaration permits the receiving server to declare 
its limit

3.  MIME Message/Partial permits fragmenting (and reassembling...) large 
messages so that the Size Declaration can be honored for over-sized 
messages; too bad reassembly is not well deployed

4.  End-user application-to-application negotiation work being done in the 
Conneg and Fax working groups is emerging; it will permit recipients to 
inform senders of preferences and constraints.

Since it is a highly asynchronous channel, with potentially very high 
latencies, email is a bit of a challenge for dealing with arbitrary message 
sizes.  The existing and emerging specifications appear to be adequate to 
the task, but implementation has been slow.  That is almost certainly a 
question of lack of market pull.

d/

=-=-=-=-=
Dave Crocker  <[email-masked]>
Brandenburg Consulting  <www.brandenburg.com>
Tel: +1.408.246.8253,  Fax: +1.408.273.6464
675 Spruce Drive,  Sunnyvale, CA 94086 USA

---

From: Dave Crocker <[email-masked]>
Date: Tue, 14 Dec 1999 13:45:45 -0800

d/

=-=-=-=-=

---

From: George Michaelson <[email-masked]>
Date: Wed, 15 Dec 1999 10:17:26 +1000

Amen!

A few observations:

Many places depend on mailers which operate as 'parallel to serial' gateways
taking apparently decoupled, asynchronous mail submission and queueing them
into a single linear pass. Therefore while the size of an object for the
submitter has no apparent 'cost' in delay terms, if more than 1 hop exists
between the sender and recipient, then for disinterested third parties there
can be quite substantive issues:
	
	o big mail delays other peoples service
	o big mail impacts on service providers in the wider sense who have
	  no 'contractual obligations' or SLA/QoS signoff with the sender or
	  the recipient.

End users have a low conciousness of list size. Therefore sending large
attachments to 'mail' can mean a n->oo expansion (ok, not infinity, but
certainly better than 2*n expansion) of data in flow. This has to be set
against sending a 'pointer to pickup' which will only cause m*n expansion
based on <total list size actually caring, and fetching the data, coupled
to any cache effects for local recipients who are lists, or otherwise can
optimize the fetchpath.

	o you can't easily track or manage the inefficiencies of sending
	  large mail to lists

	o the scaling effects of sending attachments to lists and of
	  choosing to preference pointers to objects are pretty directly
	  (inversely?) related to each other in their effect on the network

Bandwith may be getting fatter for some people, but since the total population
on the net is growing worldwide, the distribution of users to bandwidth is
not shifting up the scale at the same rate (modem speeds are lumpy increase,
post-modem access is mired in issues worldwide so its not all cable yet etc)
therefore for a priviledged few @work or @home senders, the cost of submitting
is out of all scale to the cost of reading.

	o you can't easily constrain the receiver GUI to avoid d/l of large
	  objects so you can predict for a list that some % of recipients
	  are going to be screwed by an apparently benign act

	o intermediate mail systems like HotMail impose size limits as a 
	  function of scaling management and commercial imperitives, so
	  you have a really nasty DOS attack here by blitzing the naieve
	  with content. This one is real: I support a user on the MPEG
	  standards track and she lost all mail by (a) going offsite and
	  (b) .forward-ing to hotmail and (c) being mailed large attachments
	  from the standards bodies

I think this is about education AND operational deployment:

	o Internet driving licences may seem a bit naff, but there
	  is value in requiring people to migrate to a power-user
	  status by at least trying to teach them that there are
	  consequences to using tools in distributed communications

	o If there are simple, deployable codechanges which can make
	  mailers drop in a {reference to object} in place of 
	  {entire content of object} then there is a really really
	  good reason to try and deploy them.

	o There probably is an 80:20 rule here which can be thought
	  up and promulgated like:

		If you know its going to less than 10 people and you
		know the datapaths, you can do risky things but if
	  	you think it may go to 100 people and you don't know
	   	who they are, its better to be cautious

	o Coupling mail-to-web for archive encourages people to shift
	  from direct mail access to web is viable for catchup which
	  leads naturally to using the web as the REAL data repositary
	  and using mail to point to it. In a short while, you can maybe
	  shift some practices to 'its easier' instead of 'they require it'

Actually, I think the best example I know of the 'right way' is the
old 'new RFC' email Joyce Reynolds used to post, which had the two variant
MIME attachments for fetch it via FTP and fetch via the web. We just need
more people to be that sensible!

cheers
	-George
--
George Michaelson         |  DSTC Pty Ltd
Email: [email-masked]    |  University of Qld 4072
Phone: +61 7 3365 4310    |  Australia
  Fax: +61 7 3365 4311    |  http://www.dstc.edu.au

---

From: George Michaelson <[email-masked]>
Date: Wed, 15 Dec 1999 10:17:26 +1000

Amen!

A few observations:

cheers
	-George

---

From: Jon Knight <[email-masked]>
Date: Wed, 15 Dec 1999 09:43:07 +0000 (GMT)

> 	  consequences to using tools in distributed communications

Just to point out that there appears to be something called the European
Computer Driving License (see <URL:http://www.wlv.ac.uk/pers/csdpages/
ecdl.htm> for instance).  I've no idea what sort of Internet training this
provides to end users but I would guess its more the "which button to
press in IE 5" type of training course.  I'll find out early next year as
my girlfriend who is a public librarian is going to have to go on the
course.

User education is a tricky subject.  One of my collegues in the Networks
Team is currently on the phone explaining to an end user that sending an
email with a large Word attachment to all 15000 users on campus isn't a
good idea as our mail servers will melt.  Despite our email training
courses telling people this, its a regular request, especially from
non-academic departments who are used to doing paper based mass mailings
to students.  Funny thing is that depite us offering to put the Word
document on a web page and then send a small email pointing at it,
they might well just decide to flood the campus postal mail still (has
happened before).  Managing this even on a single campus is a headache.

Tatty bye,

Jim'll

---

From: Jon Knight <[email-masked]>
Date: Wed, 15 Dec 1999 09:43:07 +0000 (GMT)

Tatty bye,

Jim'll

---

From: Jon Crowcroft <[email-masked]>
Date: Wed, 15 Dec 1999 10:21:00 +0000

>>ecdl.htm> for instance).  I've no idea what sort of Internet training

one minor difference between the internet and roads
is that on roads, most people are ok drivers, whereas on the internet,
you have to follow Postel's rule - assume everyone else is a psycopath
_and_ behave like a philanthropist

there's a special course in +defensive+ driving you can do which is
close - if you've ever tried going above 120mph on 101, you'll know
exasctly what i mean
 >>they=A0might well just decide to flood the campus postal mail still (has

 cheers

   jon

---

From: Jon Crowcroft <[email-masked]>
Date: Wed, 15 Dec 1999 10:21:00 +0000

cheers

   jon

---

From: [email-masked]
Date: Wed, 15 Dec 1999 13:23:59 -0500

> more people to be that sensible!

We don't need more people to be that sensible.

We need MUAs that make it easy to be that sensible, and we need ubiquitous
storage for the external-body parts.

I've received large e-mail attachments from people on Macs who don't
have a web server handy.  I'l even admit to attaching large documents,
even though my MUA makes it relatively easy to do so(*), and I even
have a webserver running on this machine.

Why did I do it?  Because I *know* I'll forget to go remove the file from
the webserver directories after the recipient has retrieved it, and I've
been too lazy to set up an automated "kill after 2 weeks" command...

/Valdis

(*) And of course, I decided to double-check this before hitting 'send',
and discovered that it (a) does allow building anon-FTP references but
doesn't do http: references and (b) there's a bug because somebody forgot
to overhaul some code.. ;)

---

From: [email-masked]
Date: Wed, 15 Dec 1999 13:23:59 -0500

/Valdis