Subject: Re: Gen-art LC review: draft-mm-netconf-time-capability-05

From: Tal Mizrahi <[email-masked]>
Date: Sat, 1 Aug 2015 18:32:01 +0000

Hi Robert,

Thanks for the comments.

We have submitted an updated version of the draft, which addresses the comments we received from you and other reviewers in IETF last call.
https://tools.ietf.org/html/draft-mm-netconf-time-capability-06 

Our responses to your comments can be found below.
Please let us know if you have further comments or questions.

Thanks,
Tal and Yoram.

>-----Original Message-----
>From: ietf [mailto:[email-masked]] On Behalf Of Robert Sparks
>Sent: Thursday, July 09, 2015 12:40 AM
>To: General Area Review Team; [email-masked]; draft-mm-netconf-time- 
>[email-masked]
>Subject: Gen-art LC review: draft-mm-netconf-time-capability-05
>
>I am the assigned Gen-ART reviewer for this draft. For background on 
>Gen- ART, please see the FAQ at
>
><http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>.
>
>Please resolve these comments along with any other Last Call comments 
>you may receive.
>
>Document: draft-mm-netconf-time-capability-05
>Reviewer: Robert Sparks
>Review Date: 8 Jul 2015
>IETF LC End Date: 29 Jul 2015
>IESG Telechat date: not yet scheduled
>
>Summary: This draft has open issues to address before publication
>
>This draft adds two separable concepts to netconf
>* Asking for and receiving knowledge of when a command was executed
>* Requesting that a command be executed at a particular time
>
>The utility of the first is obvious, and I have no problems with the 
>specification of that part of this extension. Would it be better to 
>pull these apart and progress them separately?
>

We believe there is a great benefit to defining these two feature together, although each of them can be used independently. The second certainly gains from the first, since the execution-time provides feedback to the client about the actual time of execution compared to the scheduled time of execution.

>The utility of the second would be more obvious if the draft didn't 
>limit the time to be "near future scheduling". It punts on most of the 
>hard problems with scheduling things outside a very tight range (15 
>seconds in the future by default), without motivating the advantages of 
>saying "wait until 5 seconds from now before you do this".
>
>So:
>
>Why was 15 seconds chosen? Could you add a motivating example that 
>shows why being able to say "now is not good, but 5 seconds from now is 
>better" is useful? (Something like having a series of things happen as 
>close to simultaneously without the network delay of sending the 
>requests impacting how they are separated perhaps?)
>

Point well taken. We have added the following example, motivating why near future scheduling (<15 seconds) can be useful:

A typical example of using near-future scheduling is a coordinated commit; a client needs to trigger a commit at n servers, so that the n servers perform the commit as close as possible to simultaneously. Without the time capability, the client sends a sequence of n commit messages, and thus each server performs the commit at a different time. By using the time capability, the client can send commit messages that are scheduled to take place at time Ts, which is 5 seconds in the future, causing the servers to invoke the commit as close as possible to time Ts.

We have also added an explanation of why 15 seconds were chosen as the default value:

The default value of sched-max-future is defined to be 15 seconds. This duration is long enough to allow the scheduled RPC to be sent by the client, potentially to multiple servers, and in some cases to send a cancellation message, as described in Section ‎3.2. On the other hand, the 15 second duration yields a very low probability of a reboot or a permission change.

>Given the punt, why isn't there a statement that sched-max-future MUST 
>NOT be configured for more than some small value (twice the default, or
>30 seconds, perhaps), especially while this is targeted for 
>Experimental? Without something like that, I think the document needs 
>to talk about more of the issues it is trying to avoid with longer term 
>scheduling, even if it doesn't solve those issues. (If I have a fast 
>pipe, I can make a server keep a lot of queued requests, eating a lot 
>of state, even if the window is only 15 seconds. Pointing to how 
>netconf protects against state-exhaustion abuse might be useful).
>

Note that we did not define a maximal value for sched-max-future, since one of the goals was to define a generic tool that can be used for various different environments. The draft clearly states the intention of using near-future-scheduling, but the requirements and constraints of different environments may require the sched-max-future to have a different value, potentially higher than 30 seconds. Hence, we prefer not to define a maximal value. Indeed, in the draft 06 there is a more detailed discussion about the issues we are trying to prevent by using near-future scheduling (Section 3.6).

>The security considerations section talks about malicious parties 
>attempting to cause sched-max-future to be configured to "a small 
>value". Could you more clearly characterize  "small", given that the 
>default is 15 seconds?
>

Agreed.
We rephrased this paragraph to be more clear about the "small" value:

This YANG module defines <sched-max-future> and <sched-max-past>, which are writable/creatable/deletable. These data nodes may be considered sensitive or vulnerable in some network environments. An attacker may attempt to maliciously configure these parameters to a low value, thereby causing all scheduled RPCs to be discarded. For instance, if a client expects <sched-max-future> to be 15 seconds, but in practice it is maliciously configured to 1 second, then a legitimate scheduled RPC that is scheduled to be performed 5 seconds in the future will be discarded by the server.

>Even with the near-future limit, there are issues to discuss introduced 
>with the ability to cancel a request:
>
>* What prevents a 3rd party from cancelling a request? I think it's 
>only that the 3rd party would have to obtain the right id to put in the 
>cancel message. If so, the document should talk about how you keep 
>eavesdroppers from seeing those ids, and that the servers that generate 
>them should make ids that are hard to guess.
>

We understand this needs further clarification. As noted by Andy Bierman in a corresponding mail:

>>Since the scheduled rpc event is sent to every client that is 
>>listening for notifications, there is no possibility for security 
>>through hard-to-guess token, as is done with the "persist-id"  for cancelling a confirmed-commit.

We rephrased the paragraph to clarify these issues:

This YANG module defines the <cancel-schedule> RPC. This RPC may be considered sensitive or vulnerable in some network environments. Since the value of the <schedule-id> is known to all the clients that are subscribed to notifications from the server, the <cancel-schedule> RPC may be used maliciously to attack servers by canceling their pending RPCs. This attack is addressed in two layers: (i) security at the transport layer, limiting the attack only to clients that have successfully initiated a secure session with the server, and (ii) the authorization level required to cancel an RPC should be the same as the level required to schedule it.

>* Especially given the near-future limitation, you run a high risk that 
>the cancel arrives after the identified request has been executed. It's 
>not clear in the current text what the server should do. I assume you 
>want the server to reply to the cancel with a "I couldn't cancel that"
>rather than to do something like try to undo the request. The document 
>should be explicit.
>
>* The document should explicitly disallow adding <scheduled-time> to 
><cancel-schedule>
>

Agreed.
We have addressed these two comments by adding the following paragraph:

A cancel-schedule message MUST NOT include the scheduled-time parameter. A server that receives a cancel-schedule should try to cancel the schedule as soon as possible. If the server is unable to cancel the scheduled RPC, for example because it has already been executed, it should respond with an rpc-error [RFC6241], in which the error-type is 'protocol', and the error-tag is 'operation-failed'.

>One editorial comment: It would help to move the concept of the 
>near-future limitation much earlier in the document, perhaps even into 
>the introduction and abstract.
>

Agreed.
We added the following to the introduction:

The NETCONF time capability is intended for scheduling RPCs that should be performed in the near future, allowing to coordinate simultaneous configuration changes, or to specify an order of configuration updates. Time-of-day-based policies and far-future scheduling, e.g., [Cond], are outside the scope of this memo.

[Cond]                          Watsen, K., "Conditional Enablement of Configuration Nodes", draft-kwatsen-conditional-enablement-00 (expired), 2013.

>And for the shepherding AD: The document has no shepherd or shepherd 
>writeup. While a writeup is not required, one would have been useful in 
>this case to discuss the history of (lack of) discussion of the 
>document on the group's list and the group's reaction to progressing as 
>Experimental as an Individual Submission.

---

From: Andy Bierman <[email-masked]>
Date: Sat, 1 Aug 2015 18:22:39 -0700

On Sat, Aug 1, 2015 at 11:32 AM, Tal Mizrahi <[email-masked]> wrote:

> Hi Robert,
>
> Thanks for the comments.
>
> We have submitted an updated version of the draft, which addresses the
> comments we received from you and other reviewers in IETF last call.
> We believe there is a great benefit to defining these two feature
> together, although each of them can be used independently. The second
> certainly gains from the first, since the execution-time provides feedback
> to the client about the actual time of execution compared to the scheduled
> time of execution.
>
>

IMO returning the execution-time is not needed.
How far from the requested time do you expect the server to be?
Maybe a few milli-seconds?

> Point well taken. We have added the following example, motivating why near
> future scheduling (<15 seconds) can be useful:
>
>

I don't understand the arbitrary 15 sec. limit.
What problems magically disappear if the timeout is 14 sec vs. 16 sec.?

> A typical example of using near-future scheduling is a coordinated commit;
> a client needs to trigger a commit at n servers, so that the n servers
> perform the commit as close as possible to simultaneously. Without the time
> capability, the client sends a sequence of n commit messages, and thus each
> server performs the commit at a different time. By using the time
> capability, the client can send commit messages that are scheduled to take
> place at time Ts, which is 5 seconds in the future, causing the servers to
> invoke the commit as close as possible to time Ts.
>
>
You should pick a different example because the NETCONF confirmed-commit
procedure is designed to be loose-coupled.  The default timeout is 10
minutes.
Since the client needs sessions open with all servers involved in the
network-wide
commit, there is no advantage in staging the <commit> operations 15 sec. in
advance,
to make sure the servers are reachable.

I thought the synchronized <get> on operational state was a good use-case.

Andy

> The default value of sched-max-future is defined to be 15 seconds. This
> duration is long enough to allow the scheduled RPC to be sent by the
> client, potentially to multiple servers, and in some cases to send a
> cancellation message, as described in Section ‎3.2. On the other hand, the
> 15 second duration yields a very low probability of a reboot or a
> permission change.
>
>
> Note that we did not define a maximal value for sched-max-future, since
> one of the goals was to define a generic tool that can be used for various
> different environments. The draft clearly states the intention of using
> near-future-scheduling, but the requirements and constraints of different
> environments may require the sched-max-future to have a different value,
> potentially higher than 30 seconds. Hence, we prefer not to define a
> maximal value. Indeed, in the draft 06 there is a more detailed discussion
> about the issues we are trying to prevent by using near-future scheduling
> (Section 3.6).
>
>
> This YANG module defines <sched-max-future> and <sched-max-past>, which
> are writable/creatable/deletable. These data nodes may be considered
> sensitive or vulnerable in some network environments. An attacker may
> attempt to maliciously configure these parameters to a low value, thereby
> causing all scheduled RPCs to be discarded. For instance, if a client
> expects <sched-max-future> to be 15 seconds, but in practice it is
> maliciously configured to 1 second, then a legitimate scheduled RPC that is
> scheduled to be performed 5 seconds in the future will be discarded by the
> server.
>
>
> This YANG module defines the <cancel-schedule> RPC. This RPC may be
> considered sensitive or vulnerable in some network environments. Since the
> value of the <schedule-id> is known to all the clients that are subscribed
> to notifications from the server, the <cancel-schedule> RPC may be used
> maliciously to attack servers by canceling their pending RPCs. This attack
> is addressed in two layers: (i) security at the transport layer, limiting
> the attack only to clients that have successfully initiated a secure
> session with the server, and (ii) the authorization level required to
> cancel an RPC should be the same as the level required to schedule it.
>
>
>
> A cancel-schedule message MUST NOT include the scheduled-time parameter. A
> server that receives a cancel-schedule should try to cancel the schedule as
> soon as possible. If the server is unable to cancel the scheduled RPC, for
> example because it has already been executed, it should respond with an
> rpc-error [RFC6241], in which the error-type is 'protocol', and the
> error-tag is 'operation-failed'.
>
>
>
>
> The NETCONF time capability is intended for scheduling RPCs that should be
> performed in the near future, allowing to coordinate simultaneous
> configuration changes, or to specify an order of configuration updates.
> Time-of-day-based policies and far-future scheduling, e.g., [Cond], are
> outside the scope of this memo.
>
> [Cond]                          Watsen, K., "Conditional Enablement of
> Configuration Nodes", draft-kwatsen-conditional-enablement-00 (expired),
> 2013.
>
>
>
>

---

From: Tal Mizrahi <[email-masked]>
Date: Sun, 2 Aug 2015 06:59:24 +0000

Hi Andy,

Thanks for the prompt response.

>Maybe a few milli-seconds?

The scheduled-time refers to the *start time* of the RPC, whereas the execution-time refers to the *completion time* of the RPC.
The difference between these two values is affected by: (i) How accurately the server is able to *start* the RPC compared to its scheduled time, and (ii) The elapsed time of execution of the RPC.
Yes, we expect the typical difference between the execution-time and the scheduled-time to be hundreds of microseconds to a few milliseconds, depending on the RPC type, and on the server’s processing power. However, in some cases, especially when the server is heavily utilized, this difference can be as high as hundreds of milliseconds, or even more than a second. In such cases it is important for the client to know that the RPC was executed a long time after it was scheduled to be performed.

>What problems magically disappear if the timeout is 14 sec vs. 16 sec.?

Instead of ‘15 seconds’, it should actually be read ‘a few seconds’.

The idea is that ‘a few seconds’ is *long enough* to allow the client to:

1.       Send the scheduled RPC, potentially to multiple servers.

2.       Receive notifications or rpc-error messages from all the servers (or wait for a timeout and decide that if no response has arrive then something is wrong).

3.       [If necessary] Send a cancellation message, potentially to multiple servers.

At the same time, ‘a few seconds’ is *short enough* to guarantee a low probability of something going wrong, for example:

-          If a server is restarted due to a fault, the procedure of the fault detection + restart will typically take more than a few seconds.

-          A change in the authorization level of the client is usually an event that is triggered by a human operation, and therefore takes more than a few seconds.

Having said that, it is still possible (with a low probability) that something *will* go wrong during those ‘few seconds’. However, this short potentially hazardous period is not significantly worse than in conventional (unscheduled) RPCs, as even a conventional RPC may in some cases be executed a few seconds after it was sent by the client.

15 seconds was selected as a default value. Please let us know if you think the value should be different.
We expect that this number may vary in different environments.

>>a client needs to trigger a commit at n servers, so that the n servers perform
>>the commit as close as possible to simultaneously. Without the time capability,
>>the client sends a sequence of n commit messages, and thus each server
>>performs the commit at a different time. By using the time capability, the client
>>can send commit messages that are scheduled to take place at time Ts, which is
>>5 seconds in the future, causing the servers to invoke the commit as close as
>>possible to time Ts.

>procedure is designed to be loose-coupled.  The default timeout is 10 minutes.
>Since the client needs sessions open with all servers involved in the network-wide
>commit, there is no advantage in staging the <commit> operations 15 sec. in advance,

Right, confirmed-commit is loose-coupled. But the example quoted above (Example 1 in the draft) is not intended to replace the confirmed commit. The purpose in this example is different: the client wants the commit RPCs to be executed at the same time in all servers.
The confirmed-commit serves a different purpose, which is to make sure that everyone either commits or rolls back. BTW, a confirmed commit can be sent with the scheduled-time element, allowing to enjoy the best of both worlds.

>I thought the synchronized <get> on operational state was a good use-case.

Please let us know if you have further comments.
Thanks,
Tal.

From: Andy Bierman [mailto:[email-masked]]
Sent: Sunday, August 02, 2015 4:23 AM
To: Tal Mizrahi
Cc: Robert Sparks; General Area Review Team; [email-masked]; [email-masked]
Subject: Re: Gen-art LC review: draft-mm-netconf-time-capability-05

On Sat, Aug 1, 2015 at 11:32 AM, Tal Mizrahi <[email-masked]<mailto:[email-masked]>> wrote:
Hi Robert,

Thanks for the comments.

Thanks,
Tal and Yoram.

>From: ietf [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Robert Sparks
>To: General Area Review Team; [email-masked]<mailto:[email-masked]>; draft-mm-netconf-time-
>[email-masked]<mailto:[email-masked]>

You should pick a different example because the NETCONF confirmed-commit
procedure is designed to be loose-coupled.  The default timeout is 10 minutes.
Since the client needs sessions open with all servers involved in the network-wide
commit, there is no advantage in staging the <commit> operations 15 sec. in advance,
to make sure the servers are reachable.

Andy

Agreed.

Agreed.

Agreed.

---

From: Robert Sparks <[email-masked]>
Date: Mon, 03 Aug 2015 17:40:08 -0500

Inline

> A typical example of using near-future scheduling is a coordinated commit; a client needs to trigger a commit at n servers, so that the n servers perform the commit as close as possible to simultan…
I'm interested in your response to Andy's point on this paragraph.
> The default value of sched-max-future is defined to be 15 seconds. This duration is long enough to allow the scheduled RPC to be sent by the client, potentially to multiple servers, and in some cas…
I'm not finding the explanation terribly persuasive, but it's at least 
_some_ explanation - thanks for that.  I'll leave it to the ADs and 
other reviewers in the field to see if it's sufficient for an 
experimental protocol.
> Note that we did not define a maximal value for sched-max-future, since one of the goals was to define a generic tool that can be used for various different environments. The draft clearly states t…
Without a maximal value, I think you need more of a discussion guiding 
the choice of sched-max-future. Otherwise, you are just waiving your 
hands at not addressing the problems with far-future scheduling, and 
potentially well-meaning but uninformed people are going to go step in 
them anyway. There was a point to choosing the near-future limit. 
Enforce it or explain it with more vigor please.
> This YANG module defines the <cancel-schedule> RPC. This RPC may be considered sensitive or vulnerable in some network environments. Since the value of the <schedule-id> is known to all the clients…
To help me along, point me to the specifics of what you use to set and 
verify such an authorization level?
> A cancel-schedule message MUST NOT include the scheduled-time parameter. A server that receives a cancel-schedule should try to cancel the schedule as soon as possible. If the server is unable to c…
looks good - thanks

---

From: Tal Mizrahi <[email-masked]>
Date: Tue, 4 Aug 2015 16:19:27 +0000

Hi Robert,

Thanks for the comments.

>>can send commit messages that are scheduled to take place at time Ts, which 
>>is 5 seconds in the future, causing the servers to invoke the commit as close as 
>>possible to time Ts.

And here is our response from 02-Aug-2015:

>Right, confirmed-commit is loose-coupled. But the example quoted above (Example 
>1 in the draft) is not intended to replace the confirmed commit. The purpose in this 
>example is different: the client wants the commit RPCs to be executed at the same 
>time in all servers.
>The confirmed-commit serves a different purpose, which is to make sure that everyone 
>either commits or rolls back. BTW, a confirmed commit can be sent with the scheduled-time 
>element, allowing to enjoy the best of both worlds.

Please let us know if you have further concerns about this point.

>>The default value of sched-max-future is defined to be 15 seconds. This duration 
>>is long enough to allow the scheduled RPC to be sent by the client, potentially to 
>>multiple servers, and in some cases to send a cancellation message, as described 
>>in Section ‎3.2. On the other hand, the 15 second duration yields a very low probability 
>>of a reboot or a permission change.

>other reviewers in the field to see if it's sufficient for an 

(*) Please see comment (**) below.

>>Note that we did not define a maximal value for sched-max-future, since one 
>>of the goals was to define a generic tool that can be used for various different 
>>environments. The draft clearly states the intention of using near-future-scheduling, 
>>but the requirements and constraints of different environments may require the 
>>sched-max-future to have a different value, potentially higher than 30 seconds. Hence, 
>>we prefer not to define a maximal value. Indeed, in the draft 06 there is a more detailed 
>>discussion about the issues we are trying to prevent by using near-future scheduling (Section 3.6).

>Enforce it or explain it with more vigor please.

(**) Your point is well taken. What we suggest, regarding this point and the previous point (*), is that we add more text explaining the factors that affect sched-max-future to Section 3.6 .

Here is the new text we suggest. Please let us know if this addresses your comment:

The challenge in far future scheduling is that during the long period between the time at which the RPC is sent and the time at which it is scheduled to be executed the following erroneous events may occur:
- The server may restart.
- The client's authorization level may be changed.
- The client may restart and send a conflicting RPC.
- A different client may send a conflicting RPC.

In these cases if the server performs the scheduled operation it may perform an action that is inconsistent with the current network policy, or inconsistent with the currently active clients.

Near future scheduling guarantees that external events such as the examples above have a low probability of occurring during the sched-max-future period, and even when they do, the period of inconsistency is limited to sched-max-future, which is a short period of time.

Hence, sched-max-future should be configured to a value that is high enough to allow the client to:
1. Send the scheduled RPC, potentially to multiple servers.
2. Receive notifications or rpc-error messages from the server(s), or wait for a timeout and decide that if no response has arrive then something is wrong.
3. If necessary, send a cancellation message, potentially to multiple servers.

On the other hand, sched-max-future should be configured to a value that is low enough to allow a low probability of the erroneous events above, typically on the order of a few seconds. Note that even if sched-max-future is configured to a low value, it is still possible (with a low probability) that an erroneous event will occur. However, this short potentially hazardous period is not significantly worse than in conventional (unscheduled) RPCs, as even a conventional RPC may in some cases be executed a few seconds after it was sent by the client.

>>This YANG module defines the <cancel-schedule> RPC. This RPC may 
>>be considered sensitive or vulnerable in some network environments. 
>>Since the value of the <schedule-id> is known to all the clients that are 
>>subscribed to notifications from the server, the <cancel-schedule> RPC 
>>may be used maliciously to attack servers by canceling their pending RPCs. 
>>This attack is addressed in two layers: (i) security at the transport layer, 
>>limiting the attack only to clients that have successfully initiated a secure 
>>session with the server, and (ii) the authorization level required to cancel 
>>an RPC should be the same as the level required to schedule it.

>verify such an authorization level?

Indeed, there is a need for an authorization scheme, which is able to set and verify the authorization level.
NETCONF (RFC 6241) does not explicitly define an authorization scheme, and it is probably not within the scope of the current draft to define such a scheme either.
Quoting RFC 6241:

   This document does not specify an authorization scheme, as such a
   scheme will likely be tied to a meta-data model or a data model.
   Implementors SHOULD provide a comprehensive authorization scheme with
   NETCONF.
   ...
   Different environments may well allow different rights prior to and
   then after authentication.  Thus, an authorization model is not
   specified in this document.  When an operation is not properly
   authorized, a simple "access denied" is sufficient. 

Please let us know if you have further comments or concerns about any of the issues above.

Thanks,
Tal.

---

From: Robert Sparks <[email-masked]>
Date: Tue, 04 Aug 2015 13:00:55 -0500

> - A different client may send a conflicting RPC.
Well, those are just a subset of the things that could change in 
command's context that would cause the command to be erroneous or even 
damaging if it were run, and you're not addressing the other security 
issues that come with very long scheduling (overflowing buffers, or 
having lots of time to schedule a massive number of commands to all try 
to happen at once). I suspect there are other things that pressured 
adding the "near future" restriction that haven't been captured well yet.
> The default value of sched-max-future is defined to be 15 seconds. This duration is long enough to allow the scheduled RPC to be sent by the client, potentially to multiple servers, and in some cas…
I still think, especially while this as at experimental, you should 
scope this with an absolute max. But I'm just one reviewer. Work it out 
with your AD.

>     authorized, a simple "access denied" is sufficient.
I think you're saying that in production deployments today, the 
authorization policy is "the peer was able to send me a packet". Is that 
wrong?

---

From: Tal Mizrahi <[email-masked]>
Date: Tue, 4 Aug 2015 23:41:29 +0000

Hi Robert,

Thanks again for the prompt responses.

>Well, those are just a subset of the things that could change in command's
>context that would cause the command to be erroneous or even damaging if
>it were run, and you're not addressing the other security issues that come
>with very long scheduling (overflowing buffers, or having lots of time to
>schedule a massive number of commands to all try to happen at once). I
>suspect there are other things that pressured adding the "near future"
>restriction that haven't been captured well yet.

Well, the thing is that 15 seconds (or 'a few seconds' for that matter) is a long enough time to send thousands (or more) of scheduled RPCs, so I am not sure the sched-max-future mitigates the buffer overflow threat. Generally speaking, Section 3.6 discusses erroneous scenarios, and not security threats.

I would suggest to add some text to the security considerations section, which discusses the overflow attack you mentioned here. Would this address your concern?

>authorization policy is "the peer was able to send me a packet". Is that 

I can't comment about what is deployed in production today, although I am sure there are operators out there who can comment about that. RFC 6536, which defines a NETCONF access control model, is cited by 6 other RFCs, so I do not think access control has been overlooked by the community. Nevertheless, I believe that (much like RFC 6241) the access control specifics are not within the scope of the current draft.

Thanks,
Tal.

>From: Robert Sparks [mailto:[email-masked]]
>Sent: Tuesday, August 04, 2015 9:01 PM
>To: Tal Mizrahi
>Cc: [email-masked]; General Area Review Team; draft-mm-netconf-time-
>[email-masked]
>>>> A typical example of using near-future scheduling is a coordinated
>>>> commit; a client needs to trigger a commit at n servers, so that the
>>>> n servers perform the commit as close as possible to simultaneously.
>>>> Without the time capability, the client sends a sequence of n commit
>>>> messages, and thus each server performs the commit at a different
>>>> time. By using the time capability, the client can send commit
>>>> messages that are scheduled to take place at time Ts, which is 5
>>>> seconds in the future, causing the servers to invoke the commit as close
>as possible to time Ts.
>>>> You should pick a different example because the NETCONF
>>>> confirmed-commit procedure is designed to be loose-coupled.  The
>default timeout is 10 minutes.
>>>> Since the client needs sessions open with all servers involved in
>>>> the network-wide commit, there is no advantage in staging the
>>>> <commit> operations 15 sec. in advance, to make sure the servers are
>reachable.
>>> 1 in the draft) is not intended to replace the confirmed commit. The
>>> purpose in this example is different: the client wants the commit
>>> RPCs to be executed at the same time in all servers.
>>> The confirmed-commit serves a different purpose, which is to make
>>> sure that everyone either commits or rolls back. BTW, a confirmed
>>> commit can be sent with the scheduled-time element, allowing to enjoy
>the best of both worlds.
>>
>>>> The default value of sched-max-future is defined to be 15 seconds.
>>>> This duration is long enough to allow the scheduled RPC to be sent
>>>> by the client, potentially to multiple servers, and in some cases to
>>>> send a cancellation message, as described in Section ‎3.2. On the
>>>> other hand, the 15 second duration yields a very low probability of a
>reboot or a permission change.
>>> I'm not finding the explanation terribly persuasive, but it's at
>>> least _some_ explanation - thanks for that.  I'll leave it to the ADs
>>> and other reviewers in the field to see if it's sufficient for an
>>> experimental protocol.
>>>> Note that we did not define a maximal value for sched-max-future,
>>>> since one of the goals was to define a generic tool that can be used
>>>> for various different environments. The draft clearly states the
>>>> intention of using near-future-scheduling, but the requirements and
>>>> constraints of different environments may require the
>>>> sched-max-future to have a different value, potentially higher than
>>>> 30 seconds. Hence, we prefer not to define a maximal value. Indeed, in
>the draft 06 there is a more detailed discussion about the issues we are trying
>to prevent by using near-future scheduling (Section 3.6).
>>> Without a maximal value, I think you need more of a discussion
>>> guiding the choice of sched-max-future. Otherwise, you are just
>>> waiving your hands at not addressing the problems with far-future
>>> scheduling, and potentially well-meaning but uninformed people are
>>> going to go step in them anyway. There was a point to choosing the near-
>future limit.
>> (**) Your point is well taken. What we suggest, regarding this point and the
>previous point (*), is that we add more text explaining the factors that affect
>sched-max-future to Section 3.6 .
>>
>> The challenge in far future scheduling is that during the long period between
>the time at which the RPC is sent and the time at which it is scheduled to be
>executed the following erroneous events may occur:
>Well, those are just a subset of the things that could change in command's
>context that would cause the command to be erroneous or even damaging if
>it were run, and you're not addressing the other security issues that come
>with very long scheduling (overflowing buffers, or having lots of time to
>schedule a massive number of commands to all try to happen at once). I
>suspect there are other things that pressured adding the "near future"
>restriction that haven't been captured well yet.
>>
>> In these cases if the server performs the scheduled operation it may
>perform an action that is inconsistent with the current network policy, or
>inconsistent with the currently active clients.
>>
>> Near future scheduling guarantees that external events such as the
>examples above have a low probability of occurring during the sched-max-
>future period, and even when they do, the period of inconsistency is limited
>to sched-max-future, which is a short period of time.
>>
>> 2. Receive notifications or rpc-error messages from the server(s), or wait for
>a timeout and decide that if no response has arrive then something is wrong.
>> On the other hand, sched-max-future should be configured to a value that is
>low enough to allow a low probability of the erroneous events above, typically
>on the order of a few seconds. Note that even if sched-max-future is
>configured to a low value, it is still possible (with a low probability) that an
>erroneous event will occur. However, this short potentially hazardous period
>is not significantly worse than in conventional (unscheduled) RPCs, as even a
>conventional RPC may in some cases be executed a few seconds after it was
>sent by the client.
>>
>duration is long enough to allow the scheduled RPC to be sent by the client,
>potentially to multiple servers, and in some cases to send a cancellation
>message, as described in Section ‎3.2. On the other hand, the 15 second
>duration yields a very low probability of a reboot or a permission change.
>I still think, especially while this as at experimental, you should scope this with
>an absolute max. But I'm just one reviewer. Work it out with your AD.
>
>>
>>
>> NETCONF (RFC 6241) does not explicitly define an authorization scheme, and
>it is probably not within the scope of the current draft to define such a
>scheme either.
>> Quoting RFC 6241:
>>

---

From: Andy Bierman <[email-masked]>
Date: Tue, 4 Aug 2015 18:02:52 -0700

Hi,

The draft adds the invoke-at-time capability to a small set
of NETCONF operations (via augment-stmt).
The mechanism cannot be used for any other operations.
It appears this is the entire list of operations supported:

   - get-config
   - get
   - copy-config
   - edit-config
   - delete-config
   - lock
   - unlock
   - close-session
   - kill-session
   - commit

Why was this subset of all operations selected?

I cannot find any text in the draft that says what happens
if the client session terminates for any reason.  There are
commands that support the 'execution-time' parameter
like <lock> that explicitly require a session to be maintained.
Not sure a delayed <close-session> even makes sense.

If the session is gone when the scheduled operation is about
to be executed, does the server cancel it or attempt it?
Without a session, the server cannot send an <rpc-reply>,
so it should not attempt the command.

What if commands are scheduled at the same time?
Is the server expected to serialize these commands or
invoke them in parallel?  Note that operations within
a single session MUST be invoked in order, but this only
seems to apply to the original <rpc> to schedule the delayed
operation.

Andy

On Tue, Aug 4, 2015 at 4:41 PM, Tal Mizrahi <[email-masked]> wrote:

> Well, the thing is that 15 seconds (or 'a few seconds' for that matter) is
> a long enough time to send thousands (or more) of scheduled RPCs, so I am
> not sure the sched-max-future mitigates the buffer overflow threat.
> Generally speaking, Section 3.6 discusses erroneous scenarios, and not
> security threats.
>
> I would suggest to add some text to the security considerations section,
> which discusses the overflow attack you mentioned here. Would this address
> your concern?
>
>
> I can't comment about what is deployed in production today, although I am
> sure there are operators out there who can comment about that. RFC 6536,
> which defines a NETCONF access control model, is cited by 6 other RFCs, so
> I do not think access control has been overlooked by the community.
> Nevertheless, I believe that (much like RFC 6241) the access control
> specifics are not within the scope of the current draft.
>
>
> Thanks,
> Tal.
>
>
> >> Here is the new text we suggest. Please let us know if this addresses
> your
> >comment:
> >>
> >>
> client,
> cancel
> >> Indeed, there is a need for an authorization scheme, which is able to
> set and
> >verify the authorization level.

---

From: Andy Bierman <[email-masked]>
Date: Tue, 4 Aug 2015 20:45:36 -0700

Hi,

I think this draft is severely constrained by RFC 6241, sec. 4.5.

4.5.  Pipelining

   NETCONF <rpc> requests MUST be processed serially by the managed
   device.  Additional <rpc> requests MAY be sent before previous ones
   have been completed.  The managed device MUST send responses only in
   the order the requests were received.

There are no existing NETCONF servers that will start processing rpc(N+1)
while rpc(N) is in progress for a session.  It might violate the MUST
in sec. 4.5.

A different NETCONF session is required for each scheduled operation.
Only 1 scheduled operation can be pending on each session.
A different session is also required to receive the
<netconf-scheduled-message>
since the session sending the scheduled operation is busy waiting for the
<rpc-reply>.  The <cancel-schedule> RPC also must be on another session,
for the same reason.

Andy

---

From: Tal Mizrahi <[email-masked]>
Date: Thu, 6 Aug 2015 05:46:58 +0000

Hi Andy,

We appreciate the detailed review.

>Why was this subset of all operations selected?

This list contains most of the RPCs defined in RFC6241, with the exception of three RPCs: cancel-commit, validate, and discard-changes. It did not make sense to us to support the <scheduled-time> element for these RPCs.
The intended status of this document is experimental. For experimental purposes, it appears that this list of RPCs is sufficient. However, if you feel that there is an RPC that needs to be added to the list please let us know.

>if the client session terminates for any reason.  

(*) Agreed. We will add text that says that when a session terminates, the server cancels all scheduled RPCs that were received in that session.

>There are

Agreed. We will remove the <close-session> RPC from the YANG module.

>so it should not attempt the command.

Agreed. See the response to (*) above.

>seems to apply to the original <rpc> to schedule the delayed

Good point. Scheduled RPCs should be processed serially, as is currently defined in RFC 6241. If two or more scheduled RPCs have the same <scheduled-time> the server can process them in an arbitrary order.
We will add text that clarifies this in the draft.

>   the order the requests were received.

Yes, the PURPOSE of the :time capability is to allow RPCs to be executed according to their scheduled time, and not according to their reception order. 
I suggest that we add some text that clarifies that replies to scheduled RPCs are sent in the order of their schedule, while replies of unscheduled RPCs (as defined in RFC 6241) are sent by the reception order of the corresponding RPCs.

>while rpc(N) is in progress for a session.  It might violate the MUST

Indeed, that is the behavior of *existing* servers, but this does not impose an interoperability problem with the :time capability. As all capability-based extension, interoperability with *existing* servers is guaranteed by the capability exchange: a server that does not support the :time capability will continue to send RPC replies according to their arrival order. A server that supports the :time capability will perform (and reply to) scheduled RPCs according to their schedule, and unscheduled RPCs according to their reception order.  

The :time capability must be:
1. Interoperable with existing servers by means of the capability exchange (we believe this is currently satisfied).
2. Well defined.
To guarantee (2), I suggest that in the capability definition (section 4) we will clarify how the :time capability affects the order of RPC replies of scheduled RPCs.

>A different session is also required to receive the <netconf-scheduled-message>

If I understand correctly, you are suggesting a workaround that allows the :time capability without violating the reply-in-reception-order requirement. Right?

Thanks,
Tal.

---

From: Andy Bierman <[email-masked]>
Date: Thu, 6 Aug 2015 09:42:26 -0700

On Wed, Aug 5, 2015 at 10:46 PM, Tal Mizrahi <[email-masked]> wrote:

> This list contains most of the RPCs defined in RFC6241, with the exception
> of three RPCs: cancel-commit, validate, and discard-changes. It did not
> make sense to us to support the <scheduled-time> element for these RPCs.
> The intended status of this document is experimental. For experimental
> purposes, it appears that this list of RPCs is sufficient. However, if you
> feel that there is an RPC that needs to be added to the list please let us
> know.
>
>
IMO, the only operation that has a reasonable use-case is <get>

> (*) Agreed. We will add text that says that when a session terminates, the
> server cancels all scheduled RPCs that were received in that session.
>
> >There are
> Good point. Scheduled RPCs should be processed serially, as is currently
> defined in RFC 6241. If two or more scheduled RPCs have the same
> <scheduled-time> the server can process them in an arbitrary order.
> Yes, the PURPOSE of the :time capability is to allow RPCs to be executed
> according to their scheduled time, and not according to their reception
> order.
> I suggest that we add some text that clarifies that replies to scheduled
> RPCs are sent in the order of their schedule, while replies of unscheduled
> RPCs (as defined in RFC 6241) are sent by the reception order of the
> corresponding RPCs.
>
>

NETCONF capabilities are not allowed to undo requirements of
other capabilities.  In order to undo the requirements of sec. 4.5,
the server cannot advertise NETCONF base 1.0 or 1.1, and also
advertise the 'time' capability.

> Indeed, that is the behavior of *existing* servers, but this does not
> impose an interoperability problem with the :time capability. As all
> capability-based extension, interoperability with *existing* servers is
> guaranteed by the capability exchange: a server that does not support the
> :time capability will continue to send RPC replies according to their
> arrival order. A server that supports the :time capability will perform
> (and reply to) scheduled RPCs according to their schedule, and unscheduled
> RPCs according to their reception order.
>
>
IMO you need a new revision of the protocol that removes the pipelining
requirements in order to pursue this solution.

> 1. Interoperable with existing servers by means of the capability exchange
> (we believe this is currently satisfied).
> 2. Well defined.
> To guarantee (2), I suggest that in the capability definition (section 4)
> we will clarify how the :time capability affects the order of RPC replies
> of scheduled RPCs.
>
> If I understand correctly, you are suggesting a workaround that allows the
> :time capability without violating the reply-in-reception-order
> requirement. Right?
>
>
No, I think RFC 6241 is quite clear how it will handle new <rpc> requests on
the same session as an operation already in progress.  It will buffer the
requests and process them in the order received.

Other solutions such as multiple <rpc-reply> for a single request are
designed to work within the existing NETCONF protocol rules.
IMO that solution approach would be better for this problem.
(i.e. get rid of the notification and return <ok/> to the schedule request.
Then send another <rpc-reply> when the executed task is invoked
or cancelled.)

Thanks,

Andy

---

From: Tal Mizrahi <[email-masked]>
Date: Sun, 16 Aug 2015 08:02:11 +0000

Hi Robert,

Thanks for the detailed comments.

We have posted an updated draft. 
https://datatracker.ietf.org/doc/draft-mm-netconf-time-capability/

We believe this draft addresses all the issues you raised.

Please let us know if you have further comments about the current draft.

>>an absolute max. But I'm just one reviewer. Work it out with your AD.

Regarding an absolute max of sched-max-future: as suggested, we consulted our AD about this issue. Since the intended status is experimental, we prefer not to define an upper bound, so as not to limit the experimental scope of the time capability. 
However, we added a new subsection "The Tradeoff in Setting the sched-max-future Value", and we believe the tradeoff is currently clear to the reader.

>>to happen at once). I suspect there are other things that pressured adding
>the "near future"
>Well, the thing is that 15 seconds (or 'a few seconds' for that matter) is a long
>enough time to send thousands (or more) of scheduled RPCs, so I am not sure
>the sched-max-future mitigates the buffer overflow threat. Generally
>speaking, Section 3.6 discusses erroneous scenarios, and not security threats.
>
>I would suggest to add some text to the security considerations section, which
>discusses the overflow attack you mentioned here. Would this address your
>concern?

We updated the security considerations section, and it currently includes the buffer overflow, and multiple-RPCs-at-the-same-time attacks.

Thanks,
Tal.

>From: Tal Mizrahi
>Sent: Wednesday, August 05, 2015 2:41 AM
>To: 'Robert Sparks'
>Subject: RE: Gen-art LC review: draft-mm-netconf-time-capability-05
>
>Hi Robert,
>
>>to happen at once). I suspect there are other things that pressured adding
>the "near future"
>Well, the thing is that 15 seconds (or 'a few seconds' for that matter) is a long
>enough time to send thousands (or more) of scheduled RPCs, so I am not sure
>the sched-max-future mitigates the buffer overflow threat. Generally
>speaking, Section 3.6 discusses erroneous scenarios, and not security threats.
>
>I would suggest to add some text to the security considerations section, which
>discusses the overflow attack you mentioned here. Would this address your
>concern?
>
>
>>authorization policy is "the peer was able to send me a packet". Is
>>that wrong?
>
>which defines a NETCONF access control model, is cited by 6 other RFCs, so I
>do not think access control has been overlooked by the community.
>Nevertheless, I believe that (much like RFC 6241) the access control specifics
>are not within the scope of the current draft.
>
>
>Thanks,
>Tal.
>
>
>>>>> commit; a client needs to trigger a commit at n servers, so that
>>>>> the n servers perform the commit as close as possible to simultaneously.
>>>>> Without the time capability, the client sends a sequence of n
>>>>> commit messages, and thus each server performs the commit at a
>>>>> different time. By using the time capability, the client can send
>>>>> commit messages that are scheduled to take place at time Ts, which
>>>>> is 5 seconds in the future, causing the servers to invoke the
>>>>> commit as close
>>as possible to time Ts.
>>>>> by the client, potentially to multiple servers, and in some cases
>>>>> to send a cancellation message, as described in Section ‎3.2. On
>>>>> the other hand, the 15 second duration yields a very low
>>>>> probability of a
>>>> least _some_ explanation - thanks for that.  I'll leave it to the
>>>> ADs and other reviewers in the field to see if it's sufficient for
>>>> an experimental protocol.
>>>>> since one of the goals was to define a generic tool that can be
>>>>> used for various different environments. The draft clearly states
>>>>> the intention of using near-future-scheduling, but the requirements
>>>>> and constraints of different environments may require the
>>the draft 06 there is a more detailed discussion about the issues we
>>are trying to prevent by using near-future scheduling (Section 3.6).
>>previous point (*), is that we add more text explaining the factors
>>that affect sched-max-future to Section 3.6 .
>>>
>>the time at which the RPC is sent and the time at which it is scheduled
>>to be executed the following erroneous events may occur:
>>to happen at once). I suspect there are other things that pressured adding
>the "near future"
>>perform an action that is inconsistent with the current network policy,
>>or inconsistent with the currently active clients.
>>>
>>examples above have a low probability of occurring during the
>>sched-max- future period, and even when they do, the period of
>>inconsistency is limited to sched-max-future, which is a short period of time.
>>>
>>low enough to allow a low probability of the erroneous events above,
>>typically on the order of a few seconds. Note that even if
>>sched-max-future is configured to a low value, it is still possible
>>(with a low probability) that an erroneous event will occur. However,
>>this short potentially hazardous period is not significantly worse than
>>in conventional (unscheduled) RPCs, as even a conventional RPC may in
>>some cases be executed a few seconds after it was sent by the client.
>>>
>>> This
>>cancellation message, as described in Section ‎3.2. On the other hand,
>>the 15 second duration yields a very low probability of a reboot or a
>permission change.
>>scope this with an absolute max. But I'm just one reviewer. Work it out with
>your AD.
>>
>>>
>>>
>>>>> considered sensitive or vulnerable in some network environments.
>>>>> Since the value of the <schedule-id> is known to all the clients
>>>>> that are subscribed to notifications from the server, the
>>>>> <cancel-schedule> RPC may be used maliciously to attack servers by
>>>>> canceling their pending
>>RPCs.
>>>>> This attack is addressed in two layers: (i) security at the
>>>>> transport layer, limiting the attack only to clients that have
>>>>> successfully initiated a secure session with the server, and (ii)
>>>>> the authorization level required to cancel an RPC should be the same as
>the level required to schedule it.
>>>> To help me along, point me to the specifics of what you use to set
>>>> and verify such an authorization level?
>>it is probably not within the scope of the current draft to define such
>>a scheme either.
>>> Quoting RFC 6241:
>>>
>>authorization policy is "the peer was able to send me a packet". Is
>>that wrong?
>>>
>>>
>>>
>>> Please let us know if you have further comments or concerns about any
>>> of
>>the issues above.
>>>
>>> Thanks,
>>> Tal.

---

From: Tal Mizrahi <[email-masked]>
Date: Sun, 16 Aug 2015 07:59:42 +0000

Hi Andy,

We want to thank you again for the thorough review, and for many helpful comments you sent us, both on the mailing list and offline.

We carefully considered each and every comment you sent. In most cases we updated the document to resolve the issues you raised. In other cases we sent a response that explains the rationale in not updating.

We have posted an updated draft:
https://datatracker.ietf.org/doc/draft-mm-netconf-time-capability/

>   the order the requests were received.

Indeed, we have added a new subsection, “4.5.2. Processing Scheduled Operations”, which discusses this issue, and explains the order of scheduled RPCs, and how the time capability interoperates with existing implementations.

>other capabilities.

That is correct. However, according to RFC 6241, the definition of a new capability should include the section “Modifications to Existing Operations”.
Hence, clearly a new capability can modify the behavior of existing operations.
That is exactly what we are doing in the current draft.
We are not suggesting to undo the “MUST” you quoted. We are only defining “modifications” that are applicable to specific operations: the operations listed in Section “4.5.1. Affected Operations”.
Specifically, these “modifications” are only limited to scheduled RPCs (as discussed in section 4.5.2), and thus do not affect the behavior of existing NETCONF servers, and do not compromise interoperability.

>while rpc(N) is in progress for a session.  It might violate the MUST

Right. Let’s consider two cases:

(1)    *Existing* RFC6241 server implementations do not support the time capability. So there is no problem there, because based on the capability exchange, the client knows these servers do not support the time capability, and the rpc-reply order is exactly as defined in RFC6241.

(2)    *New* NETCONF implementations can support the time capability.
For *unscheduled* RPCs, the order will be exactly as defined in RFC6241.
For *scheduled* RPCs, the order will be as defined in our draft – this complies to the “modifications to existing operations”.

Based on (1)+(2), there is no conflict between the time capability and *existing* or *new* implementations.

Thanks,
Tal.

From: Andy Bierman [mailto:[email-masked]]
Sent: Thursday, August 06, 2015 7:42 PM
To: Tal Mizrahi
Cc: Robert Sparks; General Area Review Team; [email-masked]; [email-masked]
Subject: Re: Gen-art LC review: draft-mm-netconf-time-capability-05

On Wed, Aug 5, 2015 at 10:46 PM, Tal Mizrahi <[email-masked]<mailto:[email-masked]>> wrote:
Hi Andy,

Thanks,
Tal.

Andy