Subject: RE: Email messages: How large is too large?

From: "Mason, Shane" <[email-masked]>
Date: Thu, 16 Dec 1999 18:39:49 -0500

What's the SOLUTION???  Watching this thread is like watching a dog chase
it's tail.  The distinguished left hand keeps saying that "E-mail wasn't
designed to do this."  The down and dirty right hand keeps asserting "User's
will keep using it if there is no good alternative."

Well I have news for all of you.  You are all correct!

Quit arguing and let's come up with a solution.

Shall a new WG be created to deal with this issue??

ICMan

-----Original Message-----
From: Danny Iacovou [mailto:[email-masked]]
Sent: Thursday, December 16, 1999 5:57 PM
To: J. Noel Chiappa; [email-masked]
Subject: RE: Email messages: How large is too large?

> -----Original Message-----
> From: J. Noel Chiappa [mailto:[email-masked]]
> Sent: Thursday, December 16, 1999 3:30 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: Email messages: How large is too large?
> 
> 
>     > From: Jon Knight <[email-masked]>
> 
>     > sending an email with a large Word attachment to all 
> 15000 users on
>     > campus isn't a good idea as our mail servers will melt. 
> ... especially
>     > from non-academic departments who are used to doing 
> paper based mass
>     > mailings to students. ... depite us offering to put the 
> Word document
>     > on a web page and then send a small email pointing at it
> 
> 
> The caveat about using the email system to transfer things 
> from one user to
> another is that it is, after all, an *email* system, and 
> engineered for that,
> not a generic bulk-data-transfer system. To use a real-world 
> analogy, the
> post-office may take parcels, but above a certain size, you 
> have to switch to
> a shipping company, which is set up to deal with larger 
> items. In a similar
> vein, the email system was designed for certain 
> characteristic payloads,
> particularly size - i.e. *email*.
 
  Speaking of University campuses:

  The University of Minnesota has a great looking mall. They spend a 
  lot of time and money maintaining the trees, shrubs, and grass. The 
  sidewalks running between buildings were a great idea. A lot of people
  used them and stayed off the grass. But you know, a lot of other people 
  did the math and the shortest distance between 2 points is a straight 
  line. Too bad the sidewalks weren't designed to run along that line. 
  So a lot of nice pretty grass turned to ugly mud as more and more people 
  stomped over the grass to get were they needed to be in the most "optimal"
  way.

  This practice continued for a long time. Sure, eventually signs went up
  telling people not to step on the grass. But that didn't work.

  Eventually the University put chain fences on every corner of the mall. 
  The chain fences look nice. The grass is well maintained now. I guess they

  decided not to add sidewalks.

  E-mail should either be able to handle a message of any size or the user 
  should be told right away that the message can't be sent. People have 
  already mentioned the GUI aspect of this problem. I agree to a large
extent
  with them.

  So the problem essentially comes down to money. When are message sizes 
  causing too many problems for admins to cost justify their efforts on? At 
  that point we either decide to put up nice fences or we build sidewalks.

  The post-office has a nice fence in place. They didn't want to build a 
  sidewalk. I think eventually we are going to opt for the sidewalk.

----------------------------------------------------------------------------
----
Neophytos Iacovou                                              Ancept Inc.
[email-masked]                                             400 First Ave
N.
www.ancept.com                                                 Suite 450

---

From: Randy Bush <[email-masked]>
Date: Thu, 16 Dec 1999 16:10:47 -0800

> What's the SOLUTION?

    :
    * ^Subject:.*How large is too large
    /dev/null

procmail is your friend

randy

---

From: Randy Bush <[email-masked]>
Date: Thu, 16 Dec 1999 16:10:47 -0800

:

procmail is your friend

randy

---

From: "Tim Dierks" <[email-masked]>
Date: Thu, 16 Dec 1999 16:21:41 -0800

OK, I'll make a suggestion.

When the sending user injects the message into the mail system, the SMTP
server could detect large e-mail attachments and replace them with reference
pointers; it would then cache the attachment someplace; possibly in local
storage, but an ISP or an enterprise with multiple SMTP servers could have a
seperate set of file storage servers. The reference pointer would refer to
the storage location of the attachment and would be obscurely chosen in a
manner to make it difficult to index and/or retrieve attachments without
access to the e-mail message containing the reference link.

When the receiving user's mail agent sees such a reference, it can directly
contact the caching server and download the attachment. This retrieval can
happen immediately or it can wait until the user explicitly requests it. The
retrieval protocol should support interruption and restart of the download
process.

In addition, SMTP servers which handle the e-mail message between the
initial server and the end user could choose to fetch the attachment on
behalf of the user and reassemble the e-mail message into a monolithic
whole. They would presumably do this based on some local policy.

The attachment would be deleted from the SMTP server's cache based on some
set of criteria, which could include:
 - After some period of time
 - After being completely fetched by the intended recipient[s]
 - Some combination of the above.

If the initial SMTP server doesn't have enough caching space, the user agent
could automatically upload the attachment to some other caching server
(possibly just using the user's local machine, if it has appropriate access
to the Internet and sufficient availability) and edit the outgoing message
to include a reference rather than the inline attachment.

Regardless, compression should be added to MIME and integrated into user
agents to automatically compress and decompress large attachments.

While it's non-trivial to implement and deploy this proposal, it would seem
to offer a more flexible mechanism for large message delivery without
forcing users to unduly conform their usage to the limitations of
implementations.

 - Tim

Tim Dierks
Chief Technology Officer, Certicom
[email-masked]
510.780.5409 [Hayward] -- 905.501.3791 [Mississauga]

> From: Mason, Shane [mailto:[email-masked]]
> Sent: Thursday, December 16, 1999 3:40 PM
> To: [email-masked]

---

From: "Tim Dierks" <[email-masked]>
Date: Thu, 16 Dec 1999 16:21:41 -0800

- Tim

Tim Dierks

---

From: "Fred Marshall" <[email-masked]>
Date: Thu, 16 Dec 1999 17:31:35 -0800

Tim's suggestion has real merit.  It's an embodiment of what I suggested in
an earlier post - without being the inventor - that something as easy as
email and as flexible as ftp might be a really good idea - if we agree
there's a real problem.  How does one proceed with this in IETF?

To summarize much of what's been said and to wholeheartedly throw my
(crrent) bias into it:

I continue to reject the notions:
1) People who are clueless are pathetic examples of internet humanity.  At
least training should be required.
2) We need to have rules or guidelines for email.

I continue to support the ideas:
1) Users are the public now - they don't understand all the underlying
technical details.  Accept it and deal with the reality.  They will send
files that somebody (with good reason) will deem are "too big".
2) Focus on the real issues - not the things that happen at 3 sigma.  Are
download rejections and other disruptions at 1 sigma, 3 sigma, 6 sigma?
What's an acceptable rate?
3) Users and providers alike will respond to real problems - either with
larger capabilities or smaller message content.
4)  Hard drives are quite cheap.  Populations are rising rapidly.  File
sizes have reason to be going up as well.  Pipes are getting fatter.  Right
now storage spaces are limiting and pipes/access (therefore transport times)
are limiting.

So, if we need to do something, let's support at least the spirit of Tim's
idea and make something happen.  First thing on the agenda should be to
decide if we need to do something - which will include at what cost?  to
what benefit?  for how long (i.e. will the need disappear before the
solution appears)?  I'm not suggesting a flurry of emails to address these
questions right here and now.  I'm suggesting a bit more deliberate process
to deal with it.

Fred Marshall
Mission Systems, Inc.
http://fcmarshall.home.mindspring.com
http://expert-market.com/pengroup/missionsystems/index.html

---

From: "Fred Marshall" <[email-masked]>
Date: Thu, 16 Dec 1999 17:31:35 -0800

Fred Marshall
Mission Systems, Inc.

---

From: George Michaelson <[email-masked]>
Date: Fri, 17 Dec 1999 11:58:02 +1000

:
  
  procmail is your friend
  
  randy

Great solution for the last mile. Shame it makes some people pay to import
what they will throw away...

-George 
--
George Michaelson         |  DSTC Pty Ltd
Email: [email-masked]    |  University of Qld 4072
Phone: +61 7 3365 4310    |  Australia
  Fax: +61 7 3365 4311    |  http://www.dstc.edu.au

---

From: George Michaelson <[email-masked]>
Date: Fri, 17 Dec 1999 11:58:02 +1000

:
  
  procmail is your friend
  
  randy

-George

---

From: Reto Lichtensteiger <[email-masked]>
Date: Fri, 17 Dec 1999 00:04:46 -0500 (EST)

Fred Marshall wrote:

<> I continue to support the ideas:

<> 4)  Hard drives are quite cheap.  Populations are rising rapidly.  File
<> sizes have reason to be going up as well.  Pipes are getting fatter.  Right
<> now storage spaces are limiting and pipes/access (therefore transport times)
<> are limiting.

<Holds breath><counts slowly from 0 to 100 and then back to 0>

You seem to have entirely missed or ignored what has been stated a
number of times.  Just because you have more disk and fatter pipe
doesn't mean the whole "internet community" does.  One place I spend
a lot of time in Vermont, I'd be lucky to get 24.4 ... more likely
to get 9.6kbits. And never mind frame relay or ISDN, the squirrel
powering the switch can't handle the load.

And as was so eloquently pointed out before, that 250 buck 9gig drive
is two months pay in some not so minor parts of the world.

Please stop architecting, and thinking, in an affluent societal
context.  What we decide will end up being used +everywhere+

I think that Tim Dierk's idea has merit and should get some attention.
It's somewhat cumbrous, but than a lot of protocols coming out of IETF
are <g>

Reto L.
-- 
R A Lichtensteiger	 [email-masked] -or- [email-masked]
			 http://www.meitca.com/ITA/People/rali
    "Yes, you're doing things right, but are you doing the right things?"
    "Nope.  I'm just doing something dumb fast."

---

From: Reto Lichtensteiger <[email-masked]>
Date: Fri, 17 Dec 1999 00:04:46 -0500 (EST)

Fred Marshall wrote:

Reto L.

---

From: Jon Crowcroft <[email-masked]>
Date: Fri, 17 Dec 1999 08:52:29 +0000

>>It's somewhat cumbrous, but than a lot of protocols coming out of IETF
 
Reto 

hmm - yes....thinks...
 
in the AVT and MMUSIC world we developed some tools to do session
initiation - some of them allow a user to register their existence,
and some allow one to advertise a session with performance parameters
(we're not talking about resource reservation, just advisory info) -
if a mailserver for a user was configured with this info, then you
could _dynamically_ select when to split large messages into things
cached on a server site + message with URL, and when to just send the
whole thing, based in some hinted at access line capacity  the
receiver had (and they don't have to be _always on_ for this to work)-

so when you register with a dialup ISP, maybe one could indicate disk
and modem rate, and preferences...

 cheers

   jon

---

From: Jon Crowcroft <[email-masked]>
Date: Fri, 17 Dec 1999 08:52:29 +0000

Reto 

hmm - yes....thinks...

 cheers

   jon

---

From: "Dick St.Peters" <[email-masked]>
Date: Mon, 20 Dec 1999 01:02:03 -0500 (EST)

> powering the switch can't handle the load.

While I firmly agree with the basic point of this (and have argued it
here myself), this is just a bit overstated.  I live full-time only 20
miles west of Vermont and serve an area running the length of the NY
side of the 200-mile NY-VT border; it's not quite that bad in most of
this region, although some places do come close.

Except in places with independent phone companies (NY has 35 of them),
frame relay is available anywhere in NY or VT.  ISDN is available most
places close enough to a telco CO, although in many places it's
imported "virtual" ISDN, and it's not cheap.  The many farms and about
half the suburbs of small cities and towns are too far out for ISDN.

Many of the more remote dwellings get phone service that's multiplexed
to use one wirepair to carry two phone lines.  When both lines are in
use, these lines won't support speeds higher than 16.8Kbps, but I've
yet to find anywhere that can get only 9.6Kbps.  Where people do have
private wirepairs, they often come from an old SLC-96 "slick" and have
a hard limit of 26.4Kbps, making that the most common connection
speed.  Getting more bandwidth than that is going to be a major
challenge for a lot of people

That's still only looking at people here in the US.

--
Dick St.Peters, [email-masked] 
Gatekeeper, NetHeaven, Saratoga Springs, NY
Saratoga/Albany/Amsterdam/BoltonLanding/Cobleskill/Greenwich/
GlensFalls/LakePlacid/NorthCreek/Plattsburgh/...
    Oldest Internet service based in the Adirondack-Albany region

---

From: "Dick St.Peters" <[email-masked]>
Date: Mon, 20 Dec 1999 01:02:03 -0500 (EST)