Subject: [Anima] Comments on draft-behringer-anima-autonomic-control-plane-01.txt

From: Brian E Carpenter <[email-masked]>
Date: Mon, 02 Mar 2015 11:28:12 +1300

Hi,

I went through this more carefully than the previous version, so I have
a number of questions and comments:

> 1.  Introduction
...
> The Autonomic Control Plane serves several purposes:

It would be useful to state explicitly that these purposes can co-exist.

...
> The ACP therefore supports directly Autonomic Networking functions, 
> as described in [I-D.behringer-anima-reference-model].

 ... and it can, for example, directly carry GDNP traffic.

> 2.  Problem Statement
> 
> An "Autonomic Control Plane" (ACP) provides a solution to some of 
> today's operational challenges.  These fall into three broad 
> categories:
> 
> o  Bootstrapping a network while devices are not yet configured. 
> Bootstrapping a new device typically requires all devices between
> the controller and the new device to be completely and correctly 
> addressed, configured and secured.  Therefore, bootstrapping a 
> network happens in layers around the controller.

This bullet has a strong SDN flavour due to the word "controller".
While SDN is certainly a valid use case, I'm not sure we want to
emphasise it here.

I think there are a couple of missing problems at the end of this section:

o Running an autonomic networking protocol such as GDNP when the rest
of the network apart from the ACP is out of order.

o Need for a stable substrate for management traffic when the
network as a whole is variable due to dynamic virtualization
of any kind.

> 3.3.  Authenticating Neighbors
...
> Authentication could be a function of a generic Adjacency Discovery 
> protocol, for example the "Generic Discovery and Negotiation 
> Protocol" GDNP described in [I-D.carpenter-anima-gdn-protocol].

I'm not sure about that statement, on reflection. It's certainly
on the table to leave authentication to a secure transport layer.

> 3.5.  Channel Establishment
> 
> After authentication and capability negotiation autonomic nodes 
> establish a secure channel towards their direct AN neighbors with
> the above negotiated parameters.  In order to be independent of 
> configured link addresses, these channels can be implemented in 
> several ways:
> 
> o  As a secure IP tunnel (e.g., IPsec, DTLS, etc.)

I would like TLS to be listed too. UDP+DTLS is a fairly complex beast
with startup costs and MTU issues, and I think we seriously need to
consider TLS instead.

...
> Since channels are established between adjacent neighbors, the 
> resulting overlay network does hop by hop encryption.  Each node 
> decrypts incoming traffic from the ACP, and encrypts outgoing traffic
> to its neighbors in the ACP.  Routing is discussed in Section 3.8.

I have several issues here (and the routing discussion doesn't solve them).

1. It's well understood that we need a link-local mode because of
bootstrapping, fault conditions, etc. But...

2. When distant nodes need to communicate, that simply doesn't work,
which is why we give them ULAs and need a routing protocol.

3. So there is something missing in your description. How does a packet
from Node A, which isn't a router, get to Node B, which isn't a router,
on some other link?

4. In a large network I expect the ACP to be reasonably busy. So
hop by hop encryption and decryption seems like an unreasonable overhead.

Certainly in GDNP we are assuming normal layer 3 end-to-end connectivity
between Node A and Node B, with e2e security.

> 3.7.  Addressing in the ACP
> 
> The channels explained above only establish communication between two
> adjacent neighbors.  In order for the communication to happen across
> multiple hops, the autonomic control plane requires internal network
> wide valid addresses and routing.  Each autonomic node must create a
> loop back interface with a network wide unique address inside the ACP
> context mentioned in Section 3.6.

The second sentence is actually in contradiction with what I just
quoted from section 3.5. But I agree with it. However, on the third
sentence and its implications you lose me. Why would you do something
as bizarre as inventing a fictitious interface instead of deploying
a ULA prefix across the ACP? (I must have read the previous version
carelessly not to spot this.) Making the Global ID of the ULA prefix
a hash of the domain ID is a fine idea. But I would much prefer the
subnet prefixes to be real ones without the consequent risk of clashes,
and conventional RAs and SLAAC within the ACP on each link. The
interface IDs should probably follow draft-ietf-6man-default-iids.

> 3.8.  Routing in the ACP
> 
> Once ULA address are set up all autonomic entities should run a 
> routing protocol within the autonomic control plane context.

I don't like that. I don't think that nodes which aren't already
routers should be obliged to run a routing protocol. Please remember
that in a large carrier network you might easily be asking for
100000 nodes to operate a routing protocol. It just isn't necessary;
the ACP nodes that happen to be routers can do this.

...
> The routing protocol inside the ACP should be light weight and highly
> scalable to ensure that the ACP does not become a limiting factor in
> network scalability.  We suggest the use of RPL as one such protocol
> which is light weight and scales well for the control plane traffic.

I think it's way too soon to select a protocol. For now we can sit back
and watch the homenet debate, but the scaling issues are very different.
My gut feeling is that different types of autonomic network will
need different solutions.

> 3.9.  Connecting a Controller / NMS system

Doesn't this section overlap with draft-eckert-anima-stable-connectivity?

> 4.  Self-Healing Properties
...
> o  If an existing device gets revoked, it will automatically be 
> denied access to the ACP as its domain certificate will be validated 
> against a Certificate Revocation List during authentication.

Excuse my ignorance, but will that instantly kill any open IPsec,
TLS or DTLS associations?

...
> After a network partition, a merge will just establish the previous 
> status,

Nit: s/merge/re-merge/ to make the context clear.

> 5.  Self-Protection Properties
...
> The ULA addresses are only reachable inside the ACP context, 
> therefore unreachable from the data plane.

Nit: 'data plane' is a bit vague IMHO. I think it would more precise to say:

As long as there is no leakage from the ACP routing protocol, the ULA
addresses are only reachable inside the ACP context, therefore
unreachable from the user environment or any external point of attack.

> 6.  Use Cases for the ACP

Hmm. To me, this section seems completely redundant with Section 2.

Regards
   Brian Carpenter

---

From: Markus Stenberg <[email-masked]>
Date: Mon, 2 Mar 2015 08:10:25 +0200

(Mostly lurking)

> TLS or DTLS associations?

Nope. Some implementation-specific magic might do this, but standards essentially do it only at connection opening time (IKEv2 for IPsec, TLS/DTLS handshake).

Cheers,

-Markus

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 03 Mar 2015 08:21:25 +1300

> Nope. Some implementation-specific magic might do this, but standards essentially do it only at connection opening time (IKEv2 for IPsec, TLS/DTLS handshake).

Ahah. So that leaves an attack window open, if a node is suborned and
revoked,
as long as it has open security associations. That suggests that there
should
be an arbitrary lifetime for all SAs.

    Brian

---

From: "Michael Behringer (mbehring)" <[email-masked]>
Date: Wed, 4 Mar 2015 18:27:42 +0000

Hi Brian, thanks for your comments. Inline...

> -----Original Message-----
> From: Anima [mailto:[email-masked]] On Behalf Of Brian E
> Carpenter
> Sent: 01 March 2015 23:28
> To: Anima WG
> Subject: [Anima] Comments on draft-behringer-anima-autonomic-control-
> plane-01.txt
> 
> Hi,
> 
> I went through this more carefully than the previous version, so I have a
> number of questions and comments:
> 
> > 1.  Introduction
> ...

Will be done in next version. 

> ...
> > The ACP therefore supports directly Autonomic Networking functions, as
> > described in [I-D.behringer-anima-reference-model].
> 

Indeed. However, GDNP can also run in the data plane, for example to negotiate the tunnel type for the ACP. 

> > Bootstrapping a new device typically requires all devices between the
> > controller and the new device to be completely and correctly
> > addressed, configured and secured.  Therefore, bootstrapping a network
> > happens in layers around the controller.
> 
> While SDN is certainly a valid use case, I'm not sure we want to emphasise it
> here.

I think we do want to emphasise this. To the "quick" reader there is the potential misunderstanding that autonomics and SDN are "either or", and this is clearly not the case. Those two paradigms complement each other nicely. It is important to remind readers about this. 

> o Running an autonomic networking protocol such as GDNP when the rest of
> the network apart from the ACP is out of order.

Hmmmm... This section evolved from a use case discussion, ie, how the operator sees it. Now of course it's more generic, where such a statement would make sense. I'll add something along those lines. Good point. 

> o Need for a stable substrate for management traffic when the network as a
> whole is variable due to dynamic virtualization of any kind.

That is covered in the second bullet? If not, then I don't understand your point? 
 
> > protocol, for example the "Generic Discovery and Negotiation Protocol"
> > GDNP described in [I-D.carpenter-anima-gdn-protocol].
> 
> I'm not sure about that statement, on reflection. It's certainly on the table to
> leave authentication to a secure transport layer.

OK, I'll take that paragraph out for now. 
 
> > establish a secure channel towards their direct AN neighbors with the
> > above negotiated parameters.  In order to be independent of configured
> > link addresses, these channels can be implemented in several ways:
> >
> I would like TLS to be listed too. UDP+DTLS is a fairly complex beast with
> startup costs and MTU issues, and I think we seriously need to consider TLS
> instead.

Well, those are examples, and don't preclude any other choice. But I can list it if you like, no strong feelings ;-) 

> from Node A, which isn't a router, get to Node B, which isn't a router, on
> some other link?

The ACP is an overlay network, essentially. Each node maintains a VRF (or similar), and the tunnels mentioned above get plugged into this VRF. Inside the VRF you define a loopback address, and route through those tunnels. The net effect is that you have an overlay network, which only depends on link local addresses in the data plane, yet provides network wide reachability. 

So, please tell me: 
- does this explanation still not answer your question? In which case I don't understand your concern. Or: 
- does it, in which case I need to make the text clearer. 

> 4. In a large network I expect the ACP to be reasonably busy. So hop by hop
> encryption and decryption seems like an unreasonable overhead.

That depends entirely what you want to do with it. And "unreasonable" is objective; all our current experiences indicate that while this looks really odd at first, it works actually quite well. 

One key feature of the ACP is that the traffic inside doesn't need to be further protected. You could of course use non-encrypted tunnels, and secure all protocols through it separately. In the end that's a tradeoff. 

> between Node A and Node B, with e2e security.

That is perfectly ok, and works with the currently described ACP. 
 
> The second sentence is actually in contradiction with what I just quoted
> from section 3.5. 

... then we're not being clear enough. 

> But I agree with it. However, on the third sentence and its
> implications you lose me. Why would you do something as bizarre as
> inventing a fictitious interface instead of deploying a ULA prefix across the
> ACP? (I must have read the previous version carelessly not to spot this.)
> Making the Global ID of the ULA prefix a hash of the domain ID is a fine idea.
> But I would much prefer the subnet prefixes to be real ones 

Define "real ones" please - I don't think I understand...

> without the
> consequent risk of clashes, and conventional RAs and SLAAC within the ACP
> on each link. The interface IDs should probably follow draft-ietf-6man-
> default-iids.

In our model, the ACP only has globally routable addresses on the loopbacks. There is no link addressing (inside). We don't need it. Simplicity. I'll point that out. 

> I don't like that. I don't think that nodes which aren't already routers should
> be obliged to run a routing protocol. Please remember that in a large carrier
> network you might easily be asking for
> 100000 nodes to operate a routing protocol. It just isn't necessary; the ACP
> nodes that happen to be routers can do this.

Well, if the network is hop by hop, then there is no choice, each hop has to run a routing protocol.  Of course you need a really lightweight protocol which scales enormously. 
 
> I think it's way too soon to select a protocol. For now we can sit back and
> watch the homenet debate, but the scaling issues are very different.
> My gut feeling is that different types of autonomic network will need
> different solutions.

Sure, this is open to debate. We were trying to come up with a solution that works in all contexts, and we believe this to be true (bold statement - for debate ;-)  

> Doesn't this section overlap with draft-eckert-anima-stable-connectivity?

There is some overlap. We need to sort that, true. 
 
> > o  If an existing device gets revoked, it will automatically be denied
> > access to the ACP as its domain certificate will be validated against
> > a Certificate Revocation List during authentication.
> 
> Excuse my ignorance, but will that instantly kill any open IPsec, TLS or DTLS
> associations?

That was discussed on another thread, and I think is answered. 

> Nit: s/merge/re-merge/ to make the context clear.

Will do. 
 
> > The ULA addresses are only reachable inside the ACP context, therefore
> > unreachable from the data plane.
> 
> addresses are only reachable inside the ACP context, therefore unreachable
> from the user environment or any external point of attack.

OK, maybe our wording is vague, but I doubt that "user environment" is any clearer. :-)  
I'll think about that one...

> Hmm. To me, this section seems completely redundant with Section 2.

Yes, unless we follow your suggestion from above and include protocol discussions here, because then it's not a use case any more. But I see your point and will try to address it in the next version. 

BTW, it may be useful for discussions on certain points above that have the risk to take longer, to start a separate mail thread, for ease of use. 

Thanks for your comments. Please try to clarify where I wasn't clear, I'll try to get a new version out before the deadline. 

Michael

> Regards
>    Brian Carpenter
> 
> _______________________________________________
> Anima mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/anima

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 05 Mar 2015 13:24:17 +1300

(Starting a separate thread for each debatable issue;
this is the first one...)

> I think we do want to emphasise this. To the "quick" reader there is the potential misunderstanding that autonomics and SDN are "either or", and this is clearly not the case. Those two paradigms co…

Well, I could agree, but in that case I would be even more explicit
and write something like:
  Bootstrapping a new device typically requires all devices between a
  controlling node (such as an SDN controller)...

 Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 05 Mar 2015 13:31:51 +1300

...
...
> That is covered in the second bullet? If not, then I don't understand your point? 

That bullet refers to "misconfiguration and routing issues" which seems
to mean faults of some kind. I was thinking of scenarios where dynamic
reconfiguration of routing was intentional, not an error condition,
but we need management traffic to run during the reconfiguration.
In other words we use the stable ACP as the tool for reconfiguring
routing for the tenant networks.

I agree, this could be included in the second bullet.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 05 Mar 2015 13:52:51 +1300

> - does this explanation still not answer your question? In which case I don't understand your concern. Or: 

My concern is that I don't see why you would do this rather than just
running a native ULA prefix and a regular security protocol. What does
the overhead of a VRF, with routing support in every autonomic node, buy
you? What failure modes and threats does it exclude, in exchange for the
overhead and scaling issues?

> - does it, in which case I need to make the text clearer.

I think it does need some extra words, because at the moment it only
becomes clear on the second or third reading of the document as
a whole. Mentioning a VRF helps a lot.

> One key feature of the ACP is that the traffic inside doesn't need to be further protected. You could of course use non-encrypted tunnels, and secure all protocols through it separately. In the end…

Yes. And with my GDNP hat on, I believe that we must assume that
there are bad actors on the layer below. We'd end up with double
encryption.

> ... then we're not being clear enough. 

That's why I need the extra text about VRF.

> Define "real ones" please - I don't think I understand...

Just a plain, normal domain-wide ULA /48 prefix, with longer prefixes
delegated to routers that send regular RAs to hosts that perform
regular SLAAC. IPv6 out of the box.

As I said, I understand your model, I just don't understand
why it's required.

(But this comes from someone who believes that VLANs are a really
bad idea, too.)

    Brian

---

From: "Michael Behringer (mbehring)" <[email-masked]>
Date: Thu, 5 Mar 2015 09:47:47 +0000

> > The ACP is an overlay network, essentially. Each node maintains a VRF (or
> similar), and the tunnels mentioned above get plugged into this VRF. Inside
> the VRF you define a loopback address, and route through those tunnels.
> The net effect is that you have an overlay network, which only depends on
> link local addresses in the data plane, yet provides network wide
> reachability.
> >
> > So, please tell me:
> My concern is that I don't see why you would do this rather than just running
> a native ULA prefix and a regular security protocol. What does the overhead
> of a VRF, with routing support in every autonomic node, buy you? What
> failure modes and threats does it exclude, in exchange for the overhead and
> scaling issues?

The "separate context" as we call it (maybe we should call it differently) provides separation of error domains. Only with the logical separation can the ACP operation be mostly independent of the data plane (the other contexts). 

And, the separation isn't the resource issue, really. You could argue (and I think you are saying exactly this) that you could run a second addressing and routing instance right in the same data plane, thereby avoiding the additional context, but then you'd be subject to more error conditions, for example address overlaps. 

So, to your question "what does it buy me?" - you can mess up all addressing, routing, AAA, etc, on a device and still have access to it through this ACP. 
 
> becomes clear on the second or third reading of the document as a whole.
> Mentioning a VRF helps a lot.

OK. I'll look at that, and I'm FULLY aware that this concept is NOT easy to convey. It *is* very unusual. Any help in how we can make it easier to understand, please really let me know! 

Maybe I'll add a section "overview" with an ascii art graphics. 

> >> 4. In a large network I expect the ACP to be reasonably busy. So hop
> >> by hop encryption and decryption seems like an unreasonable overhead.
> >
> > That depends entirely what you want to do with it. And "unreasonable" is
> objective; all our current experiences indicate that while this looks really
> odd at first, it works actually quite well.
> >
> > One key feature of the ACP is that the traffic inside doesn't need to be
> further protected. You could of course use non-encrypted tunnels, and
> secure all protocols through it separately. In the end that's a tradeoff.
> 
> Yes. And with my GDNP hat on, I believe that we must assume that there are
> bad actors on the layer below. We'd end up with double encryption.

I would say that you *can* run protocols insecurely inside the ACP. Now you can argue that for any protocol that you need also in the global context (which is the case for GDNP), you may want to secure that protocol anyway. I think that's ok.

> >> Certainly in GDNP we are assuming normal layer 3 end-to-end
> >> connectivity between Node A and Node B, with e2e security.
> >
> >>> loop back interface with a network wide unique address inside the
> >>> ACP context mentioned in Section 3.6.
> >>
> >> quoted from section 3.5.

OK, let me work on an "overview" section. I agree with you, additional clarity would definitely help. 
 
> >> inventing a fictitious interface instead of deploying a ULA prefix
> >> across the ACP? (I must have read the previous version carelessly not
> >> to spot this.) Making the Global ID of the ULA prefix a hash of the domain
> ID is a fine idea.
> delegated to routers that send regular RAs to hosts that perform regular
> SLAAC. IPv6 out of the box.

This is essentially what we're doing, minus the RA/SLAAC part. The goal is that addressing is static for a device, so that I can ALWAYS reach a device, independently on where it is right now in the topology. The details are to some extent up for debate, and we should have that debate. 

> As I said, I understand your model, I just don't understand why it's required.
>
> (But this comes from someone who believes that VLANs are a really bad
> idea, too.)

:-)   Now, I'd like to hear more about that, but that's probably a dinner discussion! 

Thanks for these substantial comments Brian! I know this model is hard to explain/understand, and really appreciate your help. Your comments really help us understand how to improve the draft. 

Michael

---

From: "Michael Behringer (mbehring)" <[email-masked]>
Date: Thu, 5 Mar 2015 10:17:17 +0000

> From: Brian E Carpenter [mailto:[email-masked]]
> Sent: 05 March 2015 01:32
> To: Michael Behringer (mbehring); Anima WG
> Subject: Problem statement [Comments on draft-behringer-anima-
> autonomic-control-plane-01.txt]
> 
> >> o Need for a stable substrate for management traffic when the network
> >> as a whole is variable due to dynamic virtualization of any kind.
> >
> That bullet refers to "misconfiguration and routing issues" which seems to
> mean faults of some kind. I was thinking of scenarios where dynamic
> reconfiguration of routing was intentional, not an error condition, but we
> need management traffic to run during the reconfiguration.
> In other words we use the stable ACP as the tool for reconfiguring routing
> for the tenant networks.

OK, now I understand you, thanks for clarifying. This is actually the intention of the 3rd bullet. And re-reading it, I really think we are addressing this point. Routing changes are even explicitly mentioned there. Can you please re-read too and if you disagree, let us know how we could improve it? 

Michael

---

From: "Michael Behringer (mbehring)" <[email-masked]>
Date: Thu, 5 Mar 2015 10:20:33 +0000

> > I think we do want to emphasise this. To the "quick" reader there is the
> potential misunderstanding that autonomics and SDN are "either or", and
> this is clearly not the case. Those two paradigms complement each other
> nicely. It is important to remind readers about this.
> 
> Well, I could agree, but in that case I would be even more explicit and write
> something like:

Fixed it. Thanks!

Michael

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 06 Mar 2015 08:37:47 +1300

> OK, now I understand you, thanks for clarifying. This is actually the intention of the 3rd bullet. And re-reading it, I really think we are addressing this point. Routing changes are even explicitl…

Well, two points about the 3rd bullet: it reads almost as though all
networks will be SDN, which is far from universally accepted, and it says
the ACP will "simplify" changes, which I think undersells the benefit
considerably. It's actually transformational compared to either an inband
approach or a traditional separate control plane.

(BTW I assume that the ACP would be very valuable for managing SFC too.)

I think we are in violent agreement about the intent here.

   Brian

---

From: "Michael Behringer (mbehring)" <[email-masked]>
Date: Fri, 6 Mar 2015 13:39:48 +0000

> >> That bullet refers to "misconfiguration and routing issues" which
> >> seems to mean faults of some kind. I was thinking of scenarios where
> >> dynamic reconfiguration of routing was intentional, not an error
> >> condition, but we need management traffic to run during the
> reconfiguration.
> > OK, now I understand you, thanks for clarifying. This is actually the
> intention of the 3rd bullet. And re-reading it, I really think we are
> addressing this point. Routing changes are even explicitly mentioned there.
> Can you please re-read too and if you disagree, let us know how we could
> improve it?
> 
> Well, two points about the 3rd bullet: it reads almost as though all networks
> will be SDN, which is far from universally accepted, and it says the ACP will
> "simplify" changes, which I think undersells the benefit considerably. It's
> actually transformational compared to either an inband approach or a
> traditional separate control plane.

OK, we've now re-written the entire section, and combined sections 2 and 6. :-)   Will be out later today. 
 
> (BTW I assume that the ACP would be very valuable for managing SFC too.)

Absolutely. There are many uses for the ACP. 
 
> I think we are in violent agreement about the intent here.

Thanks. I'm still concerned about making the draft easier to understand, but the new version should hopefully be clearer, with an "overview" section. 

Michael