Subject: Genart last call review of draft-ietf-rmcat-eval-test-08

From: Stewart Bryant <[email-masked]>
Date: Mon, 04 Feb 2019 11:28:41 -0800

Reviewer: Stewart Bryant
Review result: Almost Ready

I am the assigned Gen-ART reviewer for this draft. The General Area
Review Team (Gen-ART) reviews all IETF documents being processed
by the IESG for the IETF Chair.  Please treat these comments just
like any other last call comments.

For more information, please see the FAQ at

<https://trac.ietf.org/trac/gen/wiki/GenArtfaq>.

Document: draft-ietf-rmcat-eval-test-08
Reviewer: Stewart Bryant
Review Date: 2019-02-04
IETF LC End Date: 2019-02-11
IESG Telechat date: Not scheduled for a telechat

Summary:

A well written documents an close to being ready for publication.

I am concerned that the Security section is weak on use outside a
controlled environment.

There are a fair number of minor issues and nits that need attention,
but most of them are simple to fix.

One concern that I have that I doubt is readily fixable is that long
multi-nested lists do not work well in paginated ASCII with line spaces
and sometimes it is difficult to be sure of the context of a test element note.

Major issues:

8.  Security Considerations

   The security considerations in [I-D.ietf-rmcat-eval-criteria] and the
   relevant congestion control algorithms apply.  The principles for
   congestion control are described in [RFC2914], and in particular any
   new method MUST implement safeguards to avoid congestion collapse of
   the Internet.

   The evaluation of the test cases are intended to be run in a
   controlled lab environment.

SB> I wonder if there shouldn't me a MUST in that sentence?
SB> There have been issues on SP networks with users running unsuitable
SB> performance benchmarks on live networks, including complaints to the
SB> operators concerning the results achieved.

   Hence, the applications, simulators and
   network nodes ought to be well-behaved and should not impact the
   desired results.  It is important to take appropriate caution to
   avoid leaking non-responsive traffic from unproven congestion
   avoidance techniques onto the open Internet.

SB> Again I am surprised this is not much stronger in prohibiting
SB> use on the Internet.

========

Minor issues:

   This memo describes a set of test cases for evaluating congestion
   control algorithm proposals for real-time interactive media.

SB> It would be useful to add here the statement in the abstract that
SB> these tests should be done in a controlled environment.

===========

   Expected behavior: depending on the convergence observed in test case
   5.1 and 5.2, the candidate algorithm may be able to avoid congestion
   collapse.  In the worst case, the media stream will fall to the
   minimum media bit rate.

SB> Do you need to specify the variant of TCP? You do state it later, but some
comment here would be useful. SB> What behaviour do you expect the TCP to show.
It would be bad if SB> an aggressive media application kill the TCP completely.

============
   the first flow
   (S1) MUST arrive at a steady-state rate approximately twice of that
   of the other two flows (S2 and S3).

SB> I am not sure what you mean by priority I assume that you mean
SB> QoS ranking in the routing system. In which case I don't see
SB> how you can expect the result you specify.

============

   Expected behavior: the candidate algorithm is expected to achieve
   full utilization at both bottleneck links without starving any of the
   three congestion controlled media flows.

SB> I am not sure what you mean by this. Do you mean that the bottlenecks
SB> will saturate, but make no comment about how much of the bottleneck
SB> capacity each flow gets for itself?

============

Nits/editorial comments:

 Checking nits according to https://www.ietf.org/id-info/checklist :
  ----------------------------------------------------------------------------

  ** There are 9 instances of too long lines in the document, the longest one
     being 4 characters in excess of 72.

  == Outdated reference: A later version (-06) exists of
     draft-ietf-rmcat-wireless-tests-05

===========

3.  Structure of Test cases

SB> In the text below it was sometimes hard to get the context right in the
SB> triple (or more) nested list. Please consider using subsections or some
other SB> demarcation.

===========

         +  Bottleneck queue type: for example, Droptail, FQ-CoDel, or
            PIE.

SB> There need references, and by convention expansion on first use.

==========

         +  Path loss ratio: characterizes the non-congested, additive,
            losses to be generated on the end-to-end path.  MUST
SB> s/MUST/This MUST/ ?

==========

       B.  Variation in sending bit rate and goodput.  Mainly observing
           the frequency and magnitude of oscillations.

SB> goodput needs a reference or a definition. I don't think it is a
universally known term.

===========

   Expected behavior: the candidate algorithm is expected to detect the
   path capacity constraint, converges to the bottleneck link's capacity
SB> s/converges/converge/

===========

Due to asymmetric nature of the link

SB> s/Due to/Due to the/

===========

SB> Is there a diagram error in the figure above?

     Figure 6: Testbed Topology for TCP vs congestion controlled media
                                   Flows

===========

   have the same bandwidth share on the link.  It has to make it's way
SB>s/it's/its/

===========

The candidate algorithm MUST reflect the relative priorities
   assigned to each media flow.  In the previous example,
SB> An explicit reference to the test would help the reader

==========

---

From: Zaheduzzaman Sarker <[email-masked]>
Date: Thu, 7 Feb 2019 13:27:42 +0000

Hi Stewart,

Thanks for a good review. 

For the security consideration section, we can use stronger words if that is required. This document merely specifies test cases when people are testing their algorithm in a controlled environment and does not specify protocol usage. I was wondering if using normative language is an overkill here. For those reasons we are actually thinking of taking out 2119 usage completely. I have a modified text proposal below. 

Please see inline below for more.

BR
Zahed

On 2019-02-04, 20:29, "Stewart Bryant" <[email-masked]> wrote:

    Reviewer: Stewart Bryant

    

    Summary:

    

[ZS] I share the concern here, but I don’t think I have a better alternative now.
    
    Major issues:

    

    

[ZS] what about :

" The evaluation of the test cases are intended to be run in a
   controlled lab environment.  Hence, the applications, simulators and
   network nodes must be well-behaved and should not impact the
   desired results.  Moreover, proper measures must be taken to
   avoid leaking non-responsive traffic from unproven congestion
   avoidance techniques onto the open Internet. "
    
    ========
    
    Minor issues:

    SB> It would be useful to add here the statement in the abstract that
    SB> these tests should be done in a controlled environment.
[ZS] The abstract mentions the this "This document describes
   the test cases to be used in the performance evaluation of such
   congestion control algorithms in a controlled environment."

We can repeat that in the intro as well.
    
    ===========

    SB> Do you need to specify the variant of TCP? You do state it later, but some
    comment here would be useful. 

[ZS] Not sure what would be useful to describe here more, the expected behaviour is not really coupled to what TCP congestion control is used, the general demand is to avoid congestion collapse here.
 
SB> What behaviour do you expect the TCP to show.
    It would be bad if SB> an aggressive media application kill the TCP completely.

[ZS] I don’t think we should say anything about TCP behaviour here. The idea is to test the new congestion control behaviour with available TCP behaviours not vice versa. But TCP can certainly improve its performance from the test results (.
    
    ============
       the first flow

[ZS] no this is not about routing priority on the network nodes, this is at the media sender. When a media sender has multiple flows that shares the same bottleneck then the media sender can use techniques to distribute the available bandwidth to the multiple flows that it is sending. The point here is the media flows should get their share of the available bandwidth as per the priority set by the application.
    
    ============

[ZS] bottlenecks will saturate -- yes. The success criteria --- the existence of multiple bottleneck should not result in flow starvation to any flows that is sharing those bottlenecks. Yes, there is no comment on fairness here explicitly. Not sure we need one here, do we?
    
    ============
    
    Nits/editorial comments:
[ZS] thanks for those nits. will take care of them.

    
    
    ===========

    
    ===========

    
    ==========

    ==========

    
    ===========

    ===========

    SB> s/Due to/Due to the/
    
    ===========

    
    ===========

    ===========

    ==========

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 8 Feb 2019 13:24:41 +0000

I commented at the WG meeting, and would like to add a few comments here:

>      SB> operators concerning the results achieved.

I'm not sure I agree:

To me, the MUST is appropriate for an IETF-defined method that

could be deployed on the general Internet - by design, or by generating

traffic flowing over a virtual network interface. I'd be really concerned

if an IETF-defined mechanism didn't protect from congestion collapse.

Whether that could mean implementing a dynamic CC algorithm, or a

circuit-breaker like function.

If the work targets use in constrained environment, as it might in

this case then other mechanisms are needed to prevent this

traffic from appearing in the Internet.

>     avoidance techniques onto the open Internet. "
I'm not sure that's quite there - see above.
> We can repeat that in the intro as well.
maybe reference to RFC2914 would help?
Perhaps: /bandwidth/capacity/ - or at least be very careful in cases 
where bandwidth is used.

---

From: Stewart Bryant <[email-masked]>
Date: Fri, 15 Feb 2019 11:45:45 +0000

Hi Zahed

The work that made me particularly conscious of the security issue in 
this draft was RFC6815, and although RFC6815 applies to RFC2544, I would 
imagine that the same considerations apply here. Perhaps a reference to 
RFC6815 would help the reader of this draft better appreciate the danger 
of running these tests outside the lab.

I note that a statement on the need to run these tests in a controlled 
environment was not added to the Abstract, something which might be 
useful in highlighting the danger to someone supervising a lab, but not 
involved in the detail.

It is really for the security directorate to decide their position, but 
I would have thought that it was better to emphasis (with RFC2119 
language) the real danger to production networks (of all flavours, not 
just the Internet) of this test traffic leaking and causing disruption.

I will leave this issue with the security reviewer from here on and take 
a look at the other text changes.

- Stewart