Subject: TCP bandwidth usage was: Yahoo is not using ESMTP

From: "Michel Py" <[email-masked]>
Date: Mon, 15 Nov 2004 21:42:40 -0800

> Fred Baker wrote:
> But frankly, every sample I have seen in the past decade placed
> SMTP in the "noise" category, a single digit percentage or less.

There might be a handful of exceptions (people in the middle of nowhere
with a dial-up worth of bandwidth and/or relying exclusively on pricey
satlink with a currency not worth anything, etc) but elsewhere this is
what I hear too.

> Estimates I have heard from ISPs and seen on measurement sites
> available to me have consistently placed HTTP/HTTPS at better
> than 2/3 of total traffic in the late 1990's, and in the past
> few years have either estimated HTTP/HTTPS as neck-and-neck
> with file sharing traffic or have put file sharing well ahead.

You forgot FTP, but I also concur. The reason I am following on this the
part of file sharing traffic: a recent entry in the list of "big" p2p
protocols is BitTorrent. Figures of some 30% of the total Internet
bandwidth being used by BT were discussed recently. Read ahead.

Contrary to the other big ones (eDonkey(eMule), DirectConnect(Kaaza),
WinMX(WinMX), Gnutella(Shareaza)), BT has been underestimated by many
people (beginning with me :-). It's not as easy as classify traffic as
it used to be.

BitTorrent is becoming _the_ choice of hardcore file swappers with a
darn good reason: download speed is basically unlimited. I did not
believe until I actually installed it at home, and short of a good
distributed probe not available yet, here it is on a $37/mo residential
3mbit down / 384kbit up aDSL:
http://home.pacbell.net/arn-py/photos/bt_bandwidth.jpg
Fast enough? (this requires downloading from anything between 50 and 100
peers simultaneously).

However, to get this kind of speed there's a catch, and this catch is
called private trackers and private torrents. Generic torrent sites such
as suprnova are not too shabby, but can't compare in terms of speed with
private trackers (a generic torrent will rarely yield more than
50kByte/s). Basically, what we have failed to see coming is a large and
growing number of private tracker sites (a lot being based on
ByteMonsoon) that both a) require a custom port number and b) enforce
UL/DL ratios. 

What's annoying with these:

a) custom port numbers (read rationale below)
Before, life was simple:
Direct Connect 411 - 413
Kazaa 1214
eDonkey 4662
Gnutella 6346 - 6347
WinMX 6699
(with some minor and obvious variations, such as 5662 for
some eDonkey variants)

Now, you can forget about port-based measuring. Even though BitTorrent
is supposedly 6881 - 6889, most private trackers have something like
this clearly stated:

> Why do I get a "rejected by tracker - Port xxxx is blacklisted" error?
> Your client is reporting to the tracker that it uses one of the
default
> bittorrent ports (6881-6889) or any other common p2p port for incoming
> connections.
> joeblow's tracker (name altered to protect the guilty) does not allow
> clients to use ports commonly associated with p2p protocols. The
reason
> for this is that it is a common practice for ISPs to throttle those
> ports (that is, limit the bandwidth, hence the speed).
> The blocked ports list include, but is not necessarily limited to, the
> following: Direct Connect 411 - 413 Kazaa 1214 eDonkey 4662  Gnutella
> 6346 - 6347  BitTorrent 6881 - 6889 
> In order to use use our tracker you must configure your client to use
> any port range that does not contain those ports (a range within the
> region 49152 through 65535 is preferable, cf. IANA). Notice that some
> clients, like Azureus 2.0.7.0 or higher, use a single port for all
> torrents, while most others use one port per open torrent. The size
> of the range you choose should take this into account (typically less
> than 10 ports wide. There is no benefit whatsoever in choosing a wide
> range, and there are possible security implications). 
> These ports are used for connections between peers, not client to
> tracker. Therefore this change will not interfere with your ability
> to use other trackers (in fact it should increase your speed with
> torrents from any tracker, not just ours). Your client will also still
> be able to connect to peers that are using the standard ports. If your
> client does not allow custom ports to be used, you will have to switch
> to one that does.
> Do not ask us, or in the forums, which ports you should choose. The
> more random the choice is the harder it will be for ISPs to catch on
> to us and start limiting speeds on the ports we use. If we simply
> define another range ISPs will start throttling that range also.
> Finally, remember to forward the chosen ports in your router and/or
> open them in your firewall, should you have them. See the Why am I
> listed as not connectable?  section and links therein for more
> information on this.

In short: as of today, forget port-based stats. As of tomorrow, tough
luck with stats at all as the traffic not only being on a random port
will also be encrypted so you can't inspect the contents of the packet.

b) ratios:
> Download Ratios Policies
> With a download between:
> 5.0 GB and 7.5 GB you must have a ratio of at least 0.2
> 7.5 GB and 10 GB you must have a ratio of at least 0.3
> Above 10 GB you must have a ratio of at least 0.4
> Just remember: Who ever you are, What ever your status is,
> How many millions you did donate .. IF YOUR RATIO DROPS
> BELOW 0.4 (w/10gigs or more d/l'ed) AND WE SEE NO ACTIVITY
> FROM YOU TO GET IT UP OR IF YOU THINK YOU HAVE A RIGHT TO
> INSULT PEOPLE OR THIS SITE IN GENERAL.. WE WILL BAN YOU!!***

In short: you can download at 300kBytes per second, _IF_ you leave your
machine open so your buddies can download too. Simple mathematics: this
is unicast traffic so if you don't maintain a 1/1 overall ratio the
system does not work. Corollary: if your upstream is 10 times slower
than your downstream you need to leave your upstream open 10 times as
long as you download.

This is where it kills: non-hardcore file swappers that would otherwise
open their P2P clients only occasionally now leave them on 24/7 to pump
up their UL/DL ratio: "I maintain my ratio above 2.0 at all times, so no
problem if I decide to download that 55 GigaByte double-layer DVD
StarWars 6-episode set plus collateral material over the weekend".

<proverb>
Small torrents make big rivers.
</proverb>

Welcome to file sharing.

Michel.

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Paul Hoffman / VPNC <[email-masked]>
Date: Tue, 16 Nov 2004 08:18:03 -0800

What Michael said. This is probably below the radar for now because, 
so far, BitTorrent trading of large files (regular TV shows and 
pirated movies) is more popular in Asia and Europe than in the US. 
Even on the well-known BitTorrent ports, there is a *huge* amount of 
trading going on. To make things more difficult for the law, a good 
chunk of that trading is completely legitimate, allowed-use music 
trading of artists who actively encourage trading of live show 
recordings.

--Paul Hoffman, Director
--VPN Consortium

---

From: "Michel Py" <[email-masked]>
Date: Tue, 16 Nov 2004 21:14:13 -0800

> Paul Hoffman / VPNC wrote:
> What Michael said. This is probably below the radar for now
> because, so far, BitTorrent trading of large files (regular TV
> shows and pirated movies) is more popular in Asia and Europe
> than in the US. 

Indeed, I made that very point on Nanog a couple months ago. But it will arrive in America sooner or later and I don't see anything stopping that freight train. Plenty of overseas torrent sites (such as www.torrentbits.org, located in the NL) are in English, no language barrier here.

> Even on the well-known BitTorrent ports, there is a *huge*
> amount of trading going on.

Approaching a petabyte per day (that's right, 1K terabytes) me thinks. And it might get worse soon: double-layer DVD burners are out (I bought one for $79 last week), in a short while we will see blank double-layer DVDs at the current DVD-R prices ($0.40 a pop) and images that are 8.7 GBytes instead of 4.7 GBytes. Christmas 2005: blue ray DVDs :-( $DEITY help ISPs.

<conspiracy theory>
Cisco is supporting 10+ Gbyte file swap to sell more CRS-1s.
Juniper is supporting 10+ Gbyte file swap to sell more T-640s.
</conspiracy theory>

> To make things more difficult for the law, a good chunk of
> that trading is completely legitimate, allowed-use music
> trading of artists who actively encourage trading of live
> show recordings.

And some that encourage distributing software and/or software updates that way. I foresee small players publishing a torrent instead of having their FTP site, and seed it only with a few low-bandwidth machines. BitTorrent was not originally designed to swap illegal content.

Technically speaking, hard to take it down too. Last week, I successfully tested resuming multiple downloads with re-published torrents pointing to a different tracker than the original one that was taken down. No problem.

And it's not such a big deal to run a big site, apparently:
> TorrentBits.org is situated on a dedicated server in the Netherlands.
> For the moment we have monthly running costs of approximately € 213.

Private torrent site for 100,000 users....

Michel.

---

From: Paul Hoffman / VPNC <[email-masked]>
Date: Wed, 17 Nov 2004 08:46:50 -0800

At 9:14 PM -0800 11/16/04, Michel Py wrote:
>  > For the moment we have monthly running costs of approximately ¤ 213.

Another popular music torrent site (not based in the US, of course) 
meticulously prohibits commercially-available music *and* only allows 
lossless encodings (which are about 5 times as large as typical MP3s, 
or about half the size of uncompressed music). It has 90,000 users, 
20,000 of which are active at the moment, half of them seeding. The 
site is causing a distribution speed of 150 MB/sec. Their running 
costs so far are about US$5,000, and they have already been donated 
twice that much completely voluntarily.

And, of course, the software they're using for all of this is 
open-source and free. Setting up a torrent site is about as difficult 
as setting up a BBS was 20 years ago, it just causes about 4 orders 
of magnitude more traffic.

--Paul Hoffman, Director
--VPN Consortium