Subject: Re: IP network address assignments/allocations information?

From: Sean Doran <[email-masked]>
Date: Thu, 9 Dec 1999 18:08:59 +0100 (CET)

Christian -

| Sean, the appropriate answer to your question is RTFM (as in, read the
| manual.) The top 48 bits of IPv6 addresses are anything but flat. The flat
| space is initially limited to the top 16 bits, 3 of which are used as a
| format prefix, leaving a "flat space" table size of 8K entries.

I was merely using the wording Harald Alvestrand put forward:

> Message-Id: <[email-masked]>
> 
> A /48 leaves 16 bits for subnetting, before you hit the 64 bits of flatspace.

I think by "flatspace" he meant the _lower_ 64 bits here.

| These
| entries are supposed to be assigned to "top level" providers -- the
| equivalent of v4's default-free ISPs. There is indeed a concern that 8K may
| not be enough in the long run, which has lead to place the next 8 bits in
| reserve. Each of these "top level" prefixes thus contains 24 bits of
| address space for organization or delegation by the provider. 

So, forgive me for asking a stupid question, but RFC 2374 is
full of oddities, especially in s3.2.   Are you of the belief that
as a matter of policy, everyone but "top level" providers will have
addresses from a "top level" provider, with no exceptions?

Do you also beleive that for inter-TLA routing information-exchange 
purposes, with respect to the destination address, ONLY the 13 (to
21) TLA (+ RES) bits you mention should ever be considered by a
router in the core of the global network, except where two
directly-peering TLAs agree to exchange some NLA information?

	Sean.

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 09 Dec 1999 12:40:47 -0600

Sean Doran wrote:
...
>   Are you of the belief that

This is the plan. We start out with no holes in BGP and we try to
keep it that way.

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 09 Dec 1999 12:40:47 -0600

Sean Doran wrote:
...

   Brian

---

From: Christian Huitema <[email-masked]>
Date: Thu, 09 Dec 1999 18:14:18 -0500

>addresses from a "top level" provider, with no exceptions?

Let's put it this way: the registries are instructed that only top level
providers should get one of these addresses. Everyone who does not qualify
supposedly get a delegation from a TLA, or several delegations in the case
of multi-homed networks. Who qualifies is indeed an interesting matter of
debate for the policy councils of the registries.

>directly-peering TLAs agree to exchange some NLA information?

Yes, absolutely. There should not be a requirement that inter-TLA routing
carries anything else, except for bilateral agreements on a voluntary
basis. It should be entirely within the specs for a backbone provider to
ignore all announcements that are more specific than the TLA bits.
-- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Thu, 09 Dec 1999 18:14:18 -0500

---

From: Keith Moore <[email-masked]>
Date: Thu, 09 Dec 1999 19:36:28 -0500

> of multi-homed networks. 

of course, this requires that sending hosts or applications make 
intelligent decisions about which destination address to use
(and which source address to use with a particular destination
address), usually in the absence of any information which might
inform the decision.

it's not at all clear that this can work well enough to be a general
purpose multihoming mechanism, at least not without adding a fair
amount of extra infrastructure and complexity - i.e. a mechanism
which hosts or applications can use to query the network to determine
relative proximity of several different addresses.  if it does turn 
out to work it will probably be because all of the available prefixes 
for both the source and destination host are so reliable and have so 
much available bandwidth that most of the time that it doesn't matter 
which of the available addresses you use.  (it's tempting to say that 
multihoming will work quite well for those cases where you don't need 
multihoming...  but that is a bit of an exaggeration)

to be fair, "traditional" multihoming doesn't scale well enough 
to use that approach either.

maybe we just need for Internet service to be as reliable as telephone
service.

Keith

---

From: Keith Moore <[email-masked]>
Date: Thu, 09 Dec 1999 19:36:28 -0500

Keith

---

From: Jessica Yu <[email-masked]>
Date: Fri, 10 Dec 1999 11:38:05 -0500

>relative proximity of several different addresses. 

There is also a potential scaling issue of using multiple addresses 
as general purpose multihomging mechanism. This is because if this 
is the case, most of the Internet hosts will end up with multiple 
addresses. 

Based on the current assignment strategy, only top-tier ISPs (networks)
will be assigned with TLAs. tier-2 ISPs will get allocation address
space from the TLAs of their connected tier-1 ISPs. If a tier-2 ISP 
is multihomed, the ISP will be assigned with two IP addresses from two 
different TLAs. All the downstream customers of this tier-2 ISP will 
get multiple addresses for each host. Since most of the tier-2 ISP 
(if not all) will be multihomed to tier-1 ISPs for redundancy purpose, 
most of the Internet hosts will end up with multiple IP addresses. 
Only tier-1 ISPs' direct connected single-homed customers will not 
have multiple addresses for their hosts.

It's possible that some multihomed sites will have to assign 4 or even 
more ip addresses per host, depend on what kind of ISPs they multihoming 
with. E.g. a site that happen to multihome to two tier-2 ISPs, each 
multihomed with two different tier-1 ISPs, each host in this multihomed
site will have 4 IP addresses in order to get full benefit of redundancy. 
One can image even more complicated case.

So I think we need to explore different ways of doing multihoming.

					--Jessica 

Date:    Thu, 09 Dec 1999 19:36:28 EST
To:      Christian Huitema <[email-masked]>
cc:      Sean Doran <[email-masked]>, [email-masked]

From:    Keith Moore <[email-masked]>
Subject: Re: IP network address assignments/allocations information? 

Sender:  [email-masked]
Content-Length: 1465

Keith

---

From: Jessica Yu <[email-masked]>
Date: Fri, 10 Dec 1999 11:38:05 -0500

--Jessica 

Sender:  [email-masked]
Content-Length: 1465

Keith

---

From: Bill Sommerfeld <[email-masked]>
Date: Fri, 10 Dec 1999 12:32:18 -0500

> is the case, most of the Internet hosts will end up with multiple 

I don't see why this is inherently a problem.

> One can image even more complicated case.

I believe that it should be fairly straightforward to extend
implementations to deal with this case; some of the proposals floating
about actually significantly simplify the API's used by real
applications.

					- Bill

---

From: Bill Sommerfeld <[email-masked]>
Date: Fri, 10 Dec 1999 12:32:18 -0500

- Bill

---

From: Keith Moore <[email-masked]>
Date: Fri, 10 Dec 1999 12:49:57 -0500

> I don't see why this is inherently a problem.

it's a problem because it's essentially asking the sending host to do
routing in the absence of any routing information.  

if multiple addresses are available for a host, the chances are good 
that the paths associated with some of those addresses are significantly
better, or worse, than others.  with IPv4 multihoming, the routing system 
sorts out which path to use.  this doesn't work perfectly but at least 
the decision is made in light of some information about the nature and 
current state of those paths.  with IPv6 multihoming, the sending host is 
just guessing.  it's difficult to believe that this will work well.

---

From: Keith Moore <[email-masked]>
Date: Fri, 10 Dec 1999 12:49:57 -0500

---

From: Bill Sommerfeld <[email-masked]>
Date: Fri, 10 Dec 1999 13:00:25 -0500

> just guessing.  it's difficult to believe that this will work well.

That's why you add some feedback-based smarts to the address-order
selection scheme, and overlap the connection attempts (perhaps
separating them by an RTT or so).  Given typical distributed apps,
feedback selection based on end-to-end latency will be the right
answer in almost all cases; bandwidth-based feedback can also
potentially be done.

Given how hard it is to get an ISP do to anything special for you
these days, I really can't see a routing-system-based multihoming
actually scale down to, say, individual SOHO networks being
multihomed, while multiple-address-based multihoming doesn't require
anything special out of the ISP's..

					- Bill

---

From: Bill Sommerfeld <[email-masked]>
Date: Fri, 10 Dec 1999 13:00:25 -0500

- Bill

---

From: Murali K Viswanathan <[email-masked]>
Date: Fri, 10 Dec 1999 12:00:43 -0600

Hi Group,
I am a new member of this group and am a Masters student at Iowa state
Univ, working in the field of networking. Could someone let me know
what does the TCP standard say about response to frames received in error?
1. Is the packet just discarded?
2. Is a duplicate acknowledgmenet sent for the previous frame?
3. Or maybe it depends on individual implementations?

Thanks,
Mur

Murali K Viswanathan
[email-masked]

---

From: Murali K Viswanathan <[email-masked]>
Date: Fri, 10 Dec 1999 12:00:43 -0600

Hi Group,

Thanks,
Mur

Murali K Viswanathan
[email-masked]

---

From: John Stracke <[email-masked]>
Date: Fri, 10 Dec 1999 14:58:11 -0500

Bill Sommerfeld wrote:

> > with IPv4 multihoming, the routing system
> separating them by an RTT or so).

I agree that this is not difficult (I've done it before, for weird reasons).
But it doesn't solve the problem of what happens if the route you chose
becomes worse (or unavailable) during the lifetime of your connection, or if
the other one becomes better.

> multihomed

It happens today: if I buy service from a multihomed ISP, I get the benefits
of their multihoming for free.  (If you're talking about multiple links to my
home from different providers, then it's irrelevant; in that case, I'd have to
handle the multihoming on site anyway.)

--
/==============================================================\
|John Stracke    | http://www.ecal.com |My opinions are my own.|
|Chief Scientist |=============================================|
|eCal Corp.      |A successful tool is one that was used to do |
[email-masked]|something undreamed of by its author.        |
\==============================================================/

---

From: John Stracke <[email-masked]>
Date: Fri, 10 Dec 1999 14:58:11 -0500

---

From: Jessica Yu <[email-masked]>
Date: Fri, 10 Dec 1999 15:01:24 -0500

>I don't see why this is inherently a problem.

        This is paradigm shift in the Internet from majority of hosts 
	with single IP address to the majority of the hosts with 
	multiple IP addresses.  Many existing support mechanisms such as 
	routing (see Keith's message), DNS name look up, traffic engineering
	network managment,etc. may not be adequate. It may also break the 
	things that we have not even thought of.  And do not forget about 
	operational complexity issues. Are we really ready for such a
	major shift?

	So I would not say so quickly that it's not a problem.

						--Jessica

---

From: Jessica Yu <[email-masked]>
Date: Fri, 10 Dec 1999 15:01:24 -0500

--Jessica

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 10 Dec 1999 15:45:36 -0600

Jessica,

>         operational complexity issues. Are we really ready for such a

I really think this is close to FUD. Routing "may not be adequate." In what way?
We will still be using BGP4(+) with highly aggregated addressing.
DNS lookup "may not be adequate." Which aspect? Are you worried about the AAAA
record or the A6 record, and what are the specific problems caused by multiple 
prefixes? Traffic engineering "may not be adequate."  What aspect of TE is
different with multiple prefixes per site, and why is it any harder than
with single prefixes? Network management "may not be adequate." Again, specifically
what is harder with multiple prefixes (apart from modifying network management
databases and GUIs to support longer and multiple prefixes)? 

I'm not sure that multiple prefixes actually increase operational complexity; 
from some viewpoints they simplify things- at least for operations staff,
the source and destination prefixes will be a big clue to how the traffic
should be routed. We've never had that clue before.

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 10 Dec 1999 15:45:36 -0600

Jessica,

   Brian

---

From: Christian Huitema <[email-masked]>
Date: Fri, 10 Dec 1999 16:45:06 -0500

>routing in the absence of any routing information.  

Well, it is a problem that you have to solve anyhow, as soon as the same
service is available from multiple servers accross the Internet. What is
the difference, from a host point of view, between choosing among interface
addresses aa, ab, ac, ad of server A and choosing between interface
addresses aa, bb, cc and dd of servers A, B, C and D? If you wanted to
solve the latter at the routing level, then you should allocate an anycast
address to every group of servers that provide the same service (e.g. a
copy of the same DNS domain, of the same Web sites, or a gateway to the
same mail systems.) Now *that* would be an interesting routing problem --
BGP tables with approximately 3 million entries...

Let's face it: the odds of solving a scaling problem from the edges of the
network, from the hosts, are much better than the odds of scaling the
routing system.
-- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Fri, 10 Dec 1999 16:45:06 -0500

---

From: Daniel Senie <[email-masked]>
Date: Fri, 10 Dec 1999 18:51:05 -0500

> routing in the absence of any routing information.

First thing that came to my mind was "great, looks like token-ring
source-routing." I'd hoped we learned our lesson on that one. Having end
nodes attempt to make network path decisions is a monumentally BAD idea.

> just guessing.  it's difficult to believe that this will work well.

I can imagine that a company which is multihomed to two or more ISPs,
each of whom is tier-2 or below, may well have a dozen addresses to deal
with, per host. Now somebody remind me of why we wanted all this extra
address space? It was so we could give every machine a dozen addresses?

Exactly how the end nodes are to know which of these addresses to use,
especially when the decision point in the topology is several layers
above (at the tier2 to tier1 provider attachment point) is going to be
interesting.

Token-ring source routing allowed relatively dumb bridges to be used.
This protected the "core" of the network from having to have extra
processing horsepower. Of course we soon saw Ethernet bridges and
switches which could handle all of the needed decisions without the
involvement of the end stations. We started down the path of MPLS to
protect the core routers which were going to melt under the traffic
load. Then vendors showed us hardware which could route packets at wire
speed. Now we're busy pushing technology to ensure routing table sizes
won't increase because the present generation of hardware can't handle
larger routing tables. Are we being a bit short-sighted?

-- 
-----------------------------------------------------------------
Daniel Senie                                        [email-masked]
Amaranth Networks Inc.            http://www.amaranthnetworks.com

---

From: Daniel Senie <[email-masked]>
Date: Fri, 10 Dec 1999 18:51:05 -0500

---

From: Keith Moore <[email-masked]>
Date: Fri, 10 Dec 1999 19:40:04 -0500

> separating them by an RTT or so).  

given TCP's inability to do congestion-sensitive backoff during
connection establishment, I'm not so sure that increasing the number
of concurrent connection attempts is a good idea.

> Given typical distributed apps,

in my experience with automatic selection of ftp mirrors based
on latency estimtes, the selection of the server with the apparent
lowest latency was often a good one (i.e. significantly better than 
a random selection) but rarely was that selection the optimal one, 
and it certainly didn't work well "in almost all cases".  
in a significant number of cases (~10%) the choice was 
very pessimal, the variability of time to load a web page
over ftp *increased* when mirror selection was used,
and the users we tried this out on were often less satisified 
with the results than with a browser that just used the ftp server 
in the URL.  so I doubt that latency alone is a good metric
for server selection.  latency over a highly-loaded path is highly 
variable, and packet loss rates have a significant effect on how much
bandwidth you get out of TCP.  (to be fair, variations in load
between different servers were also a factor in our experiments, 
which would presumably not be the case for a single server host 
with multiple addresses)

> anything special out of the ISP's..

no, I can't see how a routing-system-based multihoming
will scale down to SOHO networks either.  but neither do I 
think that DNS based multihoming will work well, at least,
not without developing a lot more infrastructure than is
even on the drawing board at present.  and making this work 
well in practice is at least a few years away.  

Keith

---

From: Keith Moore <[email-masked]>
Date: Fri, 10 Dec 1999 19:40:04 -0500

Keith

---

From: Keith Moore <[email-masked]>
Date: Fri, 10 Dec 1999 19:50:44 -0500

> service is available from multiple servers accross the Internet.

agreed that it's highly desirable to solve the problem.  what I'm not
so sure of is whether it's highly desirable to make a solution to 
this problem part of the critical path for IPv6 deployment.

> What is
> addresses aa, bb, cc and dd of servers A, B, C and D?

in the former case, you're just concerned about choosing the best path.
in the latter case, you're also concerned with server load.

> If you wanted to
> same mail systems.)

I don't think you can solve the latter problem at the routing level,
unless you want to somehow convert server load factors into routing 
metrics.  Also, an anycast based solution would appear to preclude
any possibility for server selection algorithms tuned to the needs
of the application - say, if for a particular application bandwidth 
or low packet loss are more important than latency.

> network, from the hosts, are much better than the odds of scaling the

I like the general idea of doing more routing work at the periphery, but 
it's not clear to me that this is strictly an edge-vs-core tradeoff.
And I do see merit in DNS-based server selection, but I don't think it's
going to be effective enough in the near term to replace traditional 
multihoming.

Keith

---

From: Keith Moore <[email-masked]>
Date: Fri, 10 Dec 1999 19:50:44 -0500

Keith

---

From: Jessica Yu <[email-masked]>
Date: Mon, 13 Dec 1999 14:58:44 -0500

You seem to disagree with my statement of "routing may not be adequate." 
in the Internet where majority of the hosts with multiple IP addresses.
 
Do you really believe that there is an adequate support for the routing 
issue described below in majority-hosts-with-multiple-ip-addresses
environment? 

						--Jessica 

Date:    Fri, 10 Dec 1999 12:49:57 EST
To:      Bill Sommerfeld <[email-masked]>
cc:      Jessica Yu <[email-masked]>, Keith Moore <[email-masked]>,
         Christian Huitema <[email-masked]>,
         Sean Doran <[email-masked]>, [email-masked]

Sender:  [email-masked]
Content-Length: 913

---

From: Jessica Yu <[email-masked]>
Date: Mon, 13 Dec 1999 14:58:44 -0500

--Jessica 

Sender:  [email-masked]
Content-Length: 913

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 13 Dec 1999 16:05:48 -0600

Jessica,

I don't think this is a problem for the routing system. It will deliver to
whatever destination address is in the packet. The problem is for hosts
to choose which destination address to use when there is a choice,
and ditto for the source address.

I don't agree with those who see this as asking the host to guess at
routing decisions. However, it's a complex topic that is already 
an active work item in the IPNG WG.

   Brian

Jessica Yu wrote:
> Return-Path: [email-masked]
> X-URI:   http://www.cs.utk.edu/~moore/
> In-reply-to: Your message of "Fri, 10 Dec 1999 12:32:18 EST."
>          <[email-masked]>
> Sender:  [email-masked]
> Content-Type: text
> Content-Length: 913
>

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 13 Dec 1999 16:05:48 -0600

Jessica,

   Brian

---

From: Randy Bush <[email-masked]>
Date: Mon, 13 Dec 1999 15:38:44 -0800

> I don't think this is a problem for the routing system.

indeed, you seem to be pushing a lot of routing decisions to the host.  and
i am worried about where in the host's stack those decisions are supposed to
be made.

randy

---

From: Randy Bush <[email-masked]>
Date: Mon, 13 Dec 1999 15:38:44 -0800

randy

---

From: Kent Crispin <[email-masked]>
Date: Mon, 13 Dec 1999 16:02:54 -0800

> i am worried about where in the host's stack those decisions are supposed to

Application layer.

-- 
Kent Crispin                               "Do good, and you'll be
[email-masked]                           lonesome." -- Mark Twain

---

From: Kent Crispin <[email-masked]>
Date: Mon, 13 Dec 1999 16:02:54 -0800

Application layer.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 14 Dec 1999 12:28:46 +0100

> > i am worried about where in the host's stack those decisions are 
Not in my 5-line client, it's not.
Source/destination address choice inside whois? inside finger? NO.

(NOTE: I've changed the subject line. The old one was getting irrelevant.)

            Harald

--
Harald Tveit Alvestrand, EDB Maxware, Norway
[email-masked]

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 14 Dec 1999 12:28:46 +0100

Harald

---

From: Kent Crispin <[email-masked]>
Date: Tue, 14 Dec 1999 06:19:40 -0800

> Source/destination address choice inside whois? inside finger? NO.

That was a flip remark for entertainment purposes, and it begs the question
of what you mean, precisely, by "application layer" in a non ISO world. 

That being said, it would be straightforward to bury something like this in
libraries, so your 5 line application just loads with the multihoming
library code, which would be, in practical terms, an integrated
resolver/socket library that cached performance metrics for IP addresses 
associated with a particular domain name, and automatically switched to 
different IP addresses when performance became bad.  You would want to 
have it periodically ping the other addresses, as well.

I'm not proposing from a position of advocacy, you understand, but from a
pure implementation point of view, it seems straightforward, and you could
easily write 5 line clients that used it.

---

From: Kent Crispin <[email-masked]>
Date: Tue, 14 Dec 1999 06:19:40 -0800

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 14 Dec 1999 20:36:41 +0100

>of what you mean, precisely, by "application layer" in a non ISO world.

to me, it's where different purposes do different things.

>have it periodically ping the other addresses, as well.

Socket library/DLL, I can live with.

>easily write 5 line clients that used it.

Understood.

                      Harald

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 14 Dec 1999 20:36:41 +0100

Understood.

                      Harald