Subject: Was it foreseen that the Internet might handle 242 Gbps of traffic for Oprah's Book Club webinars?

From: Dan York <[email-masked]>
Date: Fri, 7 Mar 2008 15:11:24 -0500

IETFers,

Here's your Friday afternoon bit of humor - as you all who have been  
around for a while were designing this set of Tubes known as the  
Internet, did you imagine that someday it might be used for 242 Gbps  
of traffic related to 500,000 people joining a web collaboration  
session for... Oprah's Book Club?

If you aren't aware of what I'm talking about, Oprah is hosting 10  
weekly (Monday night) web collaboration sessions to discuss Eckhart  
Tolle's new book "A New Earth".  Over 750,000 people have signed up  
and this past Monday 500,000 tried to participate and basically  
crashed the system. I have some links to various posts about it here  
in my blog:  http://www.disruptivetelephony.com/2008/03/the-oprah- 
tizat.html

The statement from her company, Harpo Productions, about the event is  
here: http://event.oprah.com/videochannel/ondemand/od_main.html and  
I'll include this text:
--------
Monday night's webcast was one of the largest single online events in  
the history of the Internet. More than 500,000 people simultaneously  
logged on to watch Oprah Winfrey and Eckhart Tolle live, resulting in  
242 Gbps of information moving through the Internet. Unfortunately,  
some of our users experienced delays in viewing the webcast. We are  
working to identify the specific causes for the problems experienced  
and will work diligently to rectify them.

Harpo Productions, Inc., Move Networks and Limelight Networks  
recognize that interactive Internet broadcasting to a mass audience  
is still an emerging medium, and we're proud to have been pioneers in  
pushing the industry forward.
--------

The plan is for the sessions to go on for the next 9 Monday nights.    
So on Monday night at 9pm Eastern US at IETF-71, if anyone wants to  
watch traffic patterns on the Internet, it might make for some  
interesting metrics...  ;-)

See you in Philly,
Dan

-- 
Dan York, CISSP, Director of Emerging Communication Technology
Office of the CTO    Voxeo Corporation     [email-masked]
Phone: +1-407-455-5859  Skype: danyork  http://www.voxeo.com
Blogs: http://blogs.voxeo.com  http://www.disruptivetelephony.com

Bring your web applications to the phone.
Find out how at http://evolution.voxeo.com

_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: Joel Jaeggli <[email-masked]>
Date: Sat, 08 Mar 2008 07:02:40 -0800

10 years ago I would have thought audiences of this scale or larger 
would be supported with multicast...

joelja

Dan York wrote:
> Internet, did you imagine that someday it might be used for 242 Gbps of 
> traffic related to 500,000 people joining a web collaboration session 
> for... Oprah's Book Club?
> 
> Tolle's new book "A New Earth".  Over 750,000 people have signed up and 
> this past Monday 500,000 tried to participate and basically crashed the 
> system. I have some links to various posts about it here in my blog: 
>  http://www.disruptivetelephony.com/2008/03/the-oprah-tizat.html
> 
> here: http://event.oprah.com/videochannel/ondemand/od_main.html and I'll 
> include this text:
> --------
> 242 Gbps of information moving through the Internet. Unfortunately, some 
> of our users experienced delays in viewing the webcast. We are working 
> to identify the specific causes for the problems experienced and will 
> work diligently to rectify them.
> 
> Harpo Productions, Inc., Move Networks and Limelight Networks recognize 
> that interactive Internet broadcasting to a mass audience is still an 
> emerging medium, and we're proud to have been pioneers in pushing the 
> industry forward.
> --------
> 
> The plan is for the sessions to go on for the next 9 Monday nights.   So 
> on Monday night at 9pm Eastern US at IETF-71, if anyone wants to watch 
> traffic patterns on the Internet, it might make for some interesting 
> metrics...  ;-)
> 
> See you in Philly,
> Dan
> 
> -- 
> <mailto:[email-masked]>
> ------------------------------------------------------------------------
>

---

From: Marshall Eubanks <[email-masked]>
Date: Sat, 8 Mar 2008 11:36:28 -0500

> would be supported with multicast...

And indeed they could be. Most consumer facing ISP's, however, say  
that they would rather not
deploy multicast (to save deployment money and gather content  
provider revenues) while
they complain about the high cost of supporting Peer to Peer. Some  
even pay good money to annoy their
customers by blocking said P2P traffic (TCP resets are the favored  
mechanism for this right now.)

If this seems broken, well, I think that it is.

I have to wonder, though, if any ISP will have the guts to block Oprah.

Regards
Marshall

>> recognize

---

From: Theodore Tso <[email-masked]>
Date: Sat, 8 Mar 2008 15:05:02 -0500

> I have to wonder, though, if any ISP will have the guts to block Oprah.

Heh.  and if they tried, I'm just imagining what would happen when she
tells all of her faithful audience to call up their congresscritters
to support Net Neutrality....   :-)

						- Ted

---

From: Patrik Fältström <[email-masked]>
Date: Sat, 8 Mar 2008 17:16:22 -0500

> I have to wonder, though, if any ISP will have the guts to block  

Why would they? Each one of the 750k "friends" of Oprah is paying for  
their IP packets, and the server hosting people also pay for their IP  
packets.

What is then wrong?

That the usage is "different" from what the access providers expected  
when they calculated the price people pay as compared to what it cost  
to produce? Or because some negotiations have failed (worked?)  
regarding transit?

I.e. to some degree I feel the same story but with different words  
where used in early 1990th when we saw the first gif images inline on  
web pages. "Here comes the wolf!"

I think "The Internet" very well can handle all 750k friends of Oprah,  
if just each friend and Oprah herself buy good Internet access.

Market economy at work...that is my view.

    Patrik

P.S. And if multicast is in use, or unicast or some othercast, that is  
from my point of view part of the "innovation" the ISPs have to do  
(and will do) to ensure that the production cost is as low as possible  
so that their margin is maximized.

---

From: Jeroen Massar <[email-masked]>
Date: Sat, 08 Mar 2008 23:43:21 +0100

Patrik Fältström wrote:
[..]
> so that their margin is maximized.

I actually see a bit of a problem here as multicast would lower the usage of 
links, as such, they can't charge as much as with link that is saturated with 
unicasted packets. Thus to lower the use in the internal network one would use 
multicast, but the client would then still have to get unicast so that for every 
listener they are actually paying...

Greets,
  Jeroen

---

From: Marshall Eubanks <[email-masked]>
Date: Sat, 8 Mar 2008 17:51:58 -0500

On Mar 8, 2008, at 5:43 PM, Jeroen Massar wrote:

> Patrik Fältström wrote:
> [..]
>> P.S. And if multicast is in use, or unicast or some othercast,  
>> that is  from my point of view part of the "innovation" the ISPs  
>> have to do  (and will do) to ensure that the production cost is as  
>> low as possible  so that their margin is maximized.
>
> I actually see a bit of a problem here as multicast would lower the  
> usage of links, as such, they can't charge as much as with link  
> that is saturated with unicasted packets. Thus to lower the use in  
> the internal network one would use multicast, but the client would  
> then still have to get unicast so that for every listener they are  
> actually paying...
>

I am afraid that this is the sort of reasoning that has lead to P2P  
having such widespread use.

Regards
Marshall

---

From: Patrik Fältström <[email-masked]>
Date: Sat, 8 Mar 2008 18:09:14 -0500

> having such widespread use.

Is not one of the problems of exchanging multicast packets that  
someone that receive a multicast packet do not know how much bandwidth  
in the internal network that packet in reality will take? If the  
incoming packet is a unicast packet, there is a 1:1 relationship  
between incoming and outgoing packets. With multicast, one might have  
to send >1 packet out over the egress after receiving a packet?

If so, could not new models of charging be that if A send multicast  
packet to B, "the number of packets sent" are the number of packets  
going _out_ from B, not in to B? If it was possible to do such  
accounting...

But I should keep my mouth shut, I should not discuss such low levels  
of the stack...I am just seeing here some issues being discussed that  
are discussed above level 7...so I dived down. Now back to the normal  
business.

    Patrik

---

From: Jeroen Massar <[email-masked]>
Date: Sun, 09 Mar 2008 00:23:19 +0100

Patrik Fältström wrote:
[..]
>> having such widespread use.

I fully agree, (unicast) P2P is a godsend for Transit operators.

The fun with "p2p" is though, that HTTP is also peer to peer and actually 
anything unicast is p2p from a design point of view ;)

> Is not one of the problems of exchanging multicast packets that someone 
> that receive a multicast packet do not know how much bandwidth in the 
> internal network that packet in reality will take? If the incoming 
> packet is a unicast packet, there is a 1:1 relationship between incoming 
> and outgoing packets. With multicast, one might have to send >1 packet 
> out over the egress after receiving a packet?

Generally a network is a tree, unicast will mean you get a packet per branch, 
multicast you get 1 packet for all branches. As such your traffic is less. 
Though indeed, if you get 1 packet from your transit (thus at the root of your 
network), it takes up 1 packet everywhere else in your network. But you only pay 
your transit for 1 packet, not the zillion of branches that you might have and 
thus not for a zillion of packets.

AFAIK the biggest problem with multicast is management though. Not evening going 
onto the subject of it being available in hardware for most platforms...

> going _out_ from B, not in to B? If it was possible to do such 

Multicast account is simple: you do it at the place where you measure, same as 
unicast.

Thus if you have:
                        /----> H
                       /-----> I
                /---> E------> J
A ---> B ---> C ---> F....
        \       \---> G....
         \
          \
           \---> D....

For multicast, if you have clients everywhere, at A you see 1 packet and at H 
you see 1 packet, but that same packet is also at I J etc.

For unicast you see <clients> times packets at A and 1 packet at each client.

As such, an ISP (B) who has clients C,D,E,F,G..., will pay Transit A, in case of 
Multicast 1 packet, while for unicast he would pay Transit A for <clients> 
packets. In both cases though ISP B will charge his clients for that single 
packet. As such, multicast makes money for B, but not for A.

Transits thus don't like it, ISP's don't get it.

Greets,
  Jeroen

(This all reminds me to put working multicast on the list again for SixXS...)
u

---

From: Fred Baker <[email-masked]>
Date: Sat, 8 Mar 2008 15:44:26 -0800

yes. those that built the integrated services model felt that it was  
appropriate for internet telephony to have a way to test the capacity  
available for a real time data stream, and if capacity wasn't  
available, to say "no". Those who have worked in ieprep have pointed  
out that absent such a capability, in times when everyone is trying  
to use such services, there are some people and some services that by  
policy need to be enabled to have access anyway, and the system  
should be engineered accordingly.

Making the basic point that this particular crash so obviously  
illustrates has been very difficult in both the IETF and the operator  
communities. For the life of me, I don't understand why.

On Mar 7, 2008, at 12:11 PM, Dan York wrote:

> IETFers,
>
> Here's your Friday afternoon bit of humor - as you all who have  
> been around for a while were designing this set of Tubes known as  
> the Internet, did you imagine that someday it might be used for 242  
> Gbps of traffic related to 500,000 people joining a web  
> collaboration session for... Oprah's Book Club?
>
> crashed the system. I have some links to various posts about it  
> here in my blog:  http://www.disruptivetelephony.com/2008/03/the- 
> oprah-tizat.html
>
> The statement from her company, Harpo Productions, about the event  
> is here: http://event.oprah.com/videochannel/ondemand/od_main.html  
> and I'll include this text:
> --------
> Monday night's webcast was one of the largest single online events  
> in the history of the Internet. More than 500,000 people  
> simultaneously logged on to watch Oprah Winfrey and Eckhart Tolle  
> live, resulting in 242 Gbps of information moving through the  
> Internet. Unfortunately, some of our users experienced delays in  
> viewing the webcast. We are working to identify the specific causes  
> for the problems experienced and will work diligently to rectify them.
>
> is still an emerging medium, and we're proud to have been pioneers  
> in pushing the industry forward.
> --------
>
> The plan is for the sessions to go on for the next 9 Monday  
> nights.   So on Monday night at 9pm Eastern US at IETF-71, if  
> anyone wants to watch traffic patterns on the Internet, it might  
> make for some interesting metrics...  ;-)
>
> See you in Philly,
> Dan
>
> --

---

From: "Greg Shepherd" <[email-masked]>
Date: Sat, 8 Mar 2008 17:59:46 -0800

2008/3/8 Jeroen Massar <[email-masked]>:
>  AFAIK the biggest problem with multicast is management though.

The problem with Internet Multicast is that it's an all-or-nothing
delivery solution - every router in the path must support AND have PIM
enabled or you don't get the bits. And there are too many parties
involved to coordinate deployment globally. If IGMP had been designed
as a multi-hop protocol from the beginning, early adopters would have
been able to benefit from multicast content delivery to the edge of
their domain, or their multicast providers domain. But beyond that
there are just too many edge networks to coordinate or incent to
deploy multicast for someone else's content.

Today we have AMT as a way to get multicast content from the edge of
the multicast boundary over the unicast edge network to the receivers.
I've often heard people say that the future is VOD/P2P and multicast
has missed it's opportunity. Seems to me Opra may have a different
opinion. Do you think Opra would add
draft-ietf-mboned-auto-multicast-08.txt as recommended reading to her
book club? ;-)

> Not evening going

Any modern router replicates multicast traffic in hardware - some
better than others.

Greg

---

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Sat, 8 Mar 2008 18:18:26 -0800

The problem with multicast in this application is that it only works if all the clients are accepting the same data stream and viewing it live.

That's not how people tend to view Web video, there might be 50% of the crowd watching it as Oprah speaks but the rest are likely to be time shifted from a few secs to hours or even days.

An application layer architecture with in built caching would be more efficient. Similar to the peer to peer schemes in use today but with the caches installed by the local ISPs who are complaining about their bandwidth getting swamped. 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Jeroen Massar
> Sent: Saturday, March 08, 2008 5:43 PM
> To: Patrik Fältström
> Cc: IETF discussion list
> Subject: Re: Was it foreseen that the Internet might handle 
> 242 Gbps of trafficfor Oprah's Book Club webinars?
> 
> Patrik Fältström wrote:
> [..]
> I actually see a bit of a problem here as multicast would 
> lower the usage of links, as such, they can't charge as much 
> as with link that is saturated with unicasted packets. Thus 
> to lower the use in the internal network one would use 
> multicast, but the client would then still have to get 
> unicast so that for every listener they are actually paying...
> 
> Greets,
>   Jeroen
> 
>

---

From: Dan York <[email-masked]>
Date: Sun, 9 Mar 2008 07:20:13 -0400

On Mar 8, 2008, at 9:18 PM, Hallam-Baker, Phillip wrote:

> The problem with multicast in this application is that it only  
> works if all the clients are accepting the same data stream and  
> viewing it live.
>
> That's not how people tend to view Web video, there might be 50% of  
> the crowd watching it as Oprah speaks but the rest are likely to be  
> time shifted from a few secs to hours or even days.

FYI, in this case of Oprah's Book Club, it was 500,000 people trying  
to view the webinar *at the same time*.

The web collaboration sessions are then archived on her site so that  
people can view them at a later time.  So some % of viewers will  
watch them at some later time, but 500,000 were trying to participate  
in the "interactive" and "real-time" web collab session.

Dan

---

From: Patrik Fältström <[email-masked]>
Date: Sun, 9 Mar 2008 09:49:04 -0400

> time shifted from a few secs to hours or even days.

As others have said, this time 500k people (out of 750k) really wanted  
to view this at the same time...

> An application layer architecture with in built caching would be  
> more efficient.

I completely agree with this. Caching/storage etc of video and other  
distributed data have during the years moved closer and closer to the  
end user (consumer of the data).

At first, we only had direct transmission of TV. Directly from the TV  
camera to the viewer. Then we had caching in the form of tapes at the  
TV channel. Now we have DVRs at peoples homes, which imply even TV  
series that are produced long in advance could be distributed  
beforehand and cached on peoples DVRs (encrypted -- so at the time of  
the "live" transmission, the data is unlocked). How many things/hours  
are actually live, *really* live?

And this "pre-distribution" could be done over IP-multicast.

    Patrik

---

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Mon, 10 Mar 2008 06:52:43 -0700

To expand on this in response to comments in the meeting:

1. Design for Deployment

Anyone who wants to change the Internet infrastructure needs to consider the problem of deployment as their single biggest concern. The Internet now has a billion users. Changing the Internet, even in a small way takes a huge amount of time and effort.

Anyone who wants to rely on one of the big platform providers to deploy an infrastructure needs to talk to some of the IETF participants who work for those companies talk about their own difficulties getting their own company to adopt their own proposals. Such decisions are made by product managers, not engineers. 

In particular some people need to stop thinking that anyone other then themselves has a duty to evangelize for their work product. You build a product for the Internet you have, not the Internet you might want. Getting an RFC through the IETF is the beginning of the process, not the end.

I would like to see a deployment statement included in all RFCs and architecture statements that require infrastructure changes. This should state why the authors think that the parties who are required to make changes have an incentive to do so.

2. There has to be a path from A to B

A lot of IETF protocols suffer from the problem that they work well in the state where they are ubiquitously deployed but there is no practical path from where we are to where we want to be. 

Often there are people who consider preserving the architectural purity of the design to be more important than developing a path from A to B. For several years I have been telling people that we have to embrace NAT if there is going to be the slightest chance of deploying IPv6. In the plenary we are going to have a demonstration of why that is essential, the IPv6 only Internet worketh not without at least two NAT conversions being built into the stacks.

3. There has to be an economic incentive to travel the path from A to B.

Ten years later the multicast world is still failing to make an economic case for deployment. I cannot obtain multicast on my home network and my provider has no plans to support it. And even if it did I have no reason to expect that my local network would support it.

That is one reason I prefer the application level cache approach, there are no infrastructure preconditions. It can be deployed in the current Internet infrastructure.

But note that as PAF points out, once you have an infrastructure of caching application layer distribution points you can use multicast to distribute from the upstream feed.

One approach here would be for the multicast group to attempt to tell the cache infrastructure group that they are required to make multicast a precondition for deployment. Unless the cache group is able to refuse this will inevitably kill the killer application. 

Now consider what happens if the cache distribution points are designed so that they will function on the Internet as is but allow use of multicast as an optimization. Suddenly the ISPs have an incentive to deploy multicast - it has a dollar return and its dollars that the ISPs care about saving, not packets.

I make a much longer version of this argument in my book, The dotCrime Manifesto. The principal problem in securing the Internet is not design of the cryptography. Crypto is important and useful of course but crypto alone is not enough. There has to be a business model for deployment and a business model for use. 

We now have a conference series on the economics of computer crime. Considering the economics of deployment is a critical part of deploying the solution. Ross Anderson has some resources on this topic.

---

From: Peter Sherbin <[email-masked]>
Date: Mon, 10 Mar 2008 08:31:26 -0700 (PDT)

> Ten years later the multicast world is still failing to make an economic case for
> deployment.

The main reason for deployment failures is the Internet (unsustainable) pricing
model originating from early ARPA days. By definition IETF does not comment on the
"proper" economic model. Alas alternatives are scarce simply because there is not
enough traffic expertise outside of IETF. Please see one of the options in the
attached, which aims at CFOs via CTOs participating in IETF.

Thanks,

Peter 

--- "Hallam-Baker, Phillip" <[email-masked]> wrote:

> Anyone who wants to change the Internet infrastructure needs to consider the
> problem of deployment as their single biggest concern. The Internet now has a
> billion users. Changing the Internet, even in a small way takes a huge amount of
> time and effort.
> 
> Anyone who wants to rely on one of the big platform providers to deploy an
> infrastructure needs to talk to some of the IETF participants who work for those
> companies talk about their own difficulties getting their own company to adopt
> their own proposals. Such decisions are made by product managers, not engineers. 
> 
> In particular some people need to stop thinking that anyone other then themselves
> has a duty to evangelize for their work product. You build a product for the
> Internet you have, not the Internet you might want. Getting an RFC through the
> IETF is the beginning of the process, not the end.
> 
> I would like to see a deployment statement included in all RFCs and architecture
> statements that require infrastructure changes. This should state why the authors
> think that the parties who are required to make changes have an incentive to do
> so.
> 
> 
> A lot of IETF protocols suffer from the problem that they work well in the state
> where they are ubiquitously deployed but there is no practical path from where we
> are to where we want to be. 
> 
> Often there are people who consider preserving the architectural purity of the
> design to be more important than developing a path from A to B. For several years
> I have been telling people that we have to embrace NAT if there is going to be the
> slightest chance of deploying IPv6. In the plenary we are going to have a
> demonstration of why that is essential, the IPv6 only Internet worketh not without
> at least two NAT conversions being built into the stacks.
> 
> 
> Ten years later the multicast world is still failing to make an economic case for
> deployment. I cannot obtain multicast on my home network and my provider has no
> plans to support it. And even if it did I have no reason to expect that my local
> network would support it.
> 
> That is one reason I prefer the application level cache approach, there are no
> infrastructure preconditions. It can be deployed in the current Internet
> infrastructure.
> 
> But note that as PAF points out, once you have an infrastructure of caching
> application layer distribution points you can use multicast to distribute from the
> upstream feed.
> 
> One approach here would be for the multicast group to attempt to tell the cache
> infrastructure group that they are required to make multicast a precondition for
> deployment. Unless the cache group is able to refuse this will inevitably kill the
> killer application. 
> 
> Now consider what happens if the cache distribution points are designed so that
> they will function on the Internet as is but allow use of multicast as an
> optimization. Suddenly the ISPs have an incentive to deploy multicast - it has a
> dollar return and its dollars that the ISPs care about saving, not packets.
> 
> 
> I make a much longer version of this argument in my book, The dotCrime Manifesto.
> The principal problem in securing the Internet is not design of the cryptography.
> Crypto is important and useful of course but crypto alone is not enough. There has
> to be a business model for deployment and a business model for use. 
> 
> We now have a conference series on the economics of computer crime. Considering
> the economics of deployment is a critical part of deploying the solution. Ross
> Anderson has some resources on this topic.
> 
> 
> 
> To: Jeroen Massar; Patrik F�ltstr�m
> Cc: IETF discussion list
> The problem with multicast in this application is that it only works if all the
> clients are accepting the same data stream and viewing it live.
> 
> That's not how people tend to view Web video, there might be 50% of the crowd
> watching it as Oprah speaks but the rest are likely to be time shifted from a few
> secs to hours or even days.
> 
> 
> An application layer architecture with in built caching would be more efficient.
> Similar to the peer to peer schemes in use today but with the caches installed by
> the local ISPs who are complaining about their bandwidth getting swamped. 
> 
> > To: Patrik F�ltstr�m
> > Patrik F�ltstr�m wrote:

      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs

---

From: "Tony Hain" <[email-masked]>
Date: Mon, 10 Mar 2008 22:44:49 -0400

Hallam-Baker, Phillip wrote:
> The problem with multicast in this application is that it only works if
> all the clients are accepting the same data stream and viewing it live.
> 
> That's not how people tend to view Web video, there might be 50% of the
> crowd watching it as Oprah speaks but the rest are likely to be time
> shifted from a few secs to hours or even days.

You clearly don't understand the point of this event. There were 500k+
wanting to simultaneously watch it live. The other 250k+ that signed up
would be watching a time shifted replay. Due to the problems, significantly
more people ended up doing the time shifted version than had intended to. 

Tony