Hello, I have been selected as the Routing Directorate reviewer for this draft. The Routing Directorate seeks to review all routing or routing-related drafts as they pass through IETF last call and IESG review, and sometimes on special request. The purpose of the review is to provide assistance to the Routing ADs. For more information about the Routing Directorate, please see ​ http://trac.tools.ietf.org/area/rtg/trac/wiki/RtgDir Although these comments are primarily for the use of the Routing ADs, it would be helpful if you could consider them along with any other IETF Last Call comments that you receive, and strive to resolve them through discussion or by updating the draft. Document: draft-ietf-idr-route-oscillation-stop-00 Reviewer: Tony Przygienda Review Date: 5/26/15 Intended Status: Standards Summary: I have some minor concerns about this document that I think should be resolved before publication. In short: very good BCP draft albeit terse unless very skilled in the art, too loose for a standards track (at least in the current form) IMO. Comments below.     Network Working Group                                          D. Walton Internet-Draft                                          Cumulus Networks Intended status: Standards Track                               A. Retana   PRZ> I would like to question whether this is a standards Track document ? PRZ> It looks to me more BCP than Standards track. It relies on peers PRZ> supporting on optional capability and then it only SHOULDs the PRZ> intended behavior. In fact there is not a single normative MUST in PRZ> the whole document.   Expires: August 6, 2015                                          E. Chen                                                      Cisco Systems, Inc.                                                               J. Scudder                                                         Juniper Networks                                                         February 2, 2015                    BGP Persistent Route Oscillation Solutions                 draft-ietf-idr-route-oscillation-stop-00   Abstract      In this document we present two sets of paths for an address prefix    that can be advertised by a BGP route reflector or confederation ASBR    to eliminate the MED-induced route oscillations in a network.  The    first set involves all the available paths, and would achieve the    same routing consistency as the full IBGP mesh.  The second set,    which is a subset of the first one, involves the neighbor-AS based    Group Best Paths, and would be sufficient to eliminate the MED-    induced route oscillations (subject to certain commonly adopted    topological constrains).   Status of This Memo      This Internet-Draft is submitted in full conformance with the    provisions of BCP 78 and BCP 79.      Internet-Drafts are working documents of the Internet Engineering    Task Force (IETF).  Note that other groups may also distribute    working documents as Internet-Drafts.  The list of current Internet-    Drafts is at http://datatracker.ietf.org/drafts/current/ .      Internet-Drafts are draft documents valid for a maximum of six months    and may be updated, replaced, or obsoleted by other documents at any    time.  It is inappropriate to use Internet-Drafts as reference    material or to cite them other than as "work in progress."      This Internet-Draft will expire on August 6, 2015.   Copyright Notice      Copyright (c) 2015 IETF Trust and the persons identified as the    document authors.  All rights reserved.           Walton, et al.           Expires August 6, 2015                 [Page 1] Àpar Internet-Draft          BGP Oscillation Solutions          February 2015        This document is subject to BCP 78 and the IETF Trust's Legal    Provisions Relating to IETF Documents    ( http://trustee.ietf.org/license-info ) in effect on the date of    publication of this document.  Please review these documents    carefully, as they describe your rights and restrictions with respect    to this document.  Code Components extracted from this document must    include Simplified BSD License text as described in Section 4.e of    the Trust Legal Provisions and are provided without warranty as    described in the Simplified BSD License.   Table of Contents      1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2    2.  Requirements Language . . . . . . . . . . . . . . . . . . . .   3    3.  Advertise the Available Paths . . . . . . . . . . . . . . . .   3    4.  Advertise the Group Best Paths  . . . . . . . . . . . . . . .   3    5.  Route Reflection and Confederation  . . . . . . . . . . . . .   4      5.1.  Route Reflection  . . . . . . . . . . . . . . . . . . . .   5      5.2.  Confederation . . . . . . . . . . . . . . . . . . . . . .   5    6.  Deployment Considerations . . . . . . . . . . . . . . . . . .   5    7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   6   8.  Security Considerations . . . . . . . . . . . . . . . . . . .   6    9.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .   7    10. References  . . . . . . . . . . . . . . . . . . . . . . . . .   7      10.1.  Normative References . . . . . . . . . . . . . . . . . .   7      10.2.  Informative References . . . . . . . . . . . . . . . . .   7    Appendix A.  Why the Group Best Paths Are Adequate? . . . . . . .   7    Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   8   1.  Introduction      As documented in [RFC3345], the routing information reduction by BGP    Route Reflection [RFC4456] or BGP Confederation [RFC5065] can result    in persistent IBGP route oscillations with certain routing setup and    network topologies.  Except for a couple artificially engineered    network topologies, the MED attribute [RFC4271] has played a pivotal    role in virtually all of the known persistent IBGP route    oscillations.  For the sake of brevity, we use the term "MED-induced    route oscillation" hereafter to refer to a persistent IBGP route    oscillation in which the MED plays a role.      In order to eliminate the MED-induced route oscillations and to    achieve consistent routing in a network, clearly a route reflector or    a confederation ASBR needs to advertise more than just the best path    for an address prefix.  Our goal is to identify the "right" set of    paths for an address prefix that needs to be advertised by a route    reflector or a confederation ASBR.         Walton, et al.           Expires August 6, 2015                 [Page 2] Àpar Internet-Draft          BGP Oscillation Solutions          February 2015        In this document we present two sets of paths for an address prefix    that can be advertised by a BGP route reflector or confederation ASBR    to eliminate the MED-induced route oscillations in a network.  The    first set involves all the available paths, and would achieve the    same routing consistency as the full IBGP mesh.  The second set,    which is a subset of the first one, involves the neighbor-AS based    Group Best Paths, and would be sufficient to eliminate the MED-    induced route oscillations (subject to certain commonly adopted    topological constrains).      These paths can be advertised using the mechanism described in ADD-    PATH [I-D.ietf-idr-add-paths] for advertising multiple paths.   PRZ> I suggest to indicate in the document that all routers in AD MUST be PRZ> configured to behave consistently when comparing MEDs PRZ> (i.e. always-compare-med, missing-as-worst and so on need to be PRZ> consistent). PRZ> There seems to me a hidden assumption PRZ> in the document PRZ> albeit likely obvious for the skilled in the art that PRZ> always-compare-med is used here (and in fact the overall behavior PRZ> simulates deterministic-MED?) but IMO it must be mentioned what the PRZ> assumptions are. Especially for routers that want to the RRC but PRZ> do NOT support add-path e.g. PRZ> that they are sometimes employed to fix some of PRZ> those issues and need to be considered.   2.  Requirements Language      The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",    "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this    document are to be interpreted as described in [RFC2119].   3.  Advertise the Available Paths      Observe that in a network that maintains a full IBGP mesh all the BGP    speakers have consistent and equivalent routing information.  Such a    network is thus free of the MED-induced route oscillations and other    routing inconsistencies such as forwarding loops.      Therefore one approach is to allow a route reflector or a    confederation ASBR to advertise all the available paths for an    address prefix.  Clearly this approach would yield the same amount of    routing information and achieve the same routing consistency as the    full IBGP mesh in a network.      This approach can be implemented using the mechanism described in    ADD-PATH [I-D.ietf-idr-add-paths] for advertising multiple paths for    certain prefixes.      For the sake of scalability the advertisement of multiple paths    should be limited to those prefixes which are affected by MED-induced    route oscillation in a network carrying a large number of alternate    paths.    PRZ> Suggest to specify the precise criteria since those are fairly PRZ> complicated as far I see or at least refer as example PRZ> to e.g. 4456 section 11 PRZ> rather than the vague PRZ> English relative clause encountered here during initial reading.  PRZ> Or pull up the 4456 reference from section 4 PRZ> and refer from section 4 again in a short form. PRZ> This will improve logical flow of the document. PRZ>  PRZ> Moreover, other mechanisms than avoiding well-known criteria can be PRZ> imagined, e.g. something that has a hysterisis on # of flaps in PRZ> a prefix and based on that qualifying a NLRI as a 'MED-induced PRZ> oscillation affected prefix' to apply the techniques here.    4.  Advertise the Group Best Paths      The term neighbor-AS for a route refers to the neighboring AS from    which the route was received.  The calculation of the neighbor-AS is    specified in Sect. 9.1.2.2 of [RFC4271], and Section 7.2 of    [RFC5065].  By definition the MED is comparable only among routes    with the same neighbor-AS.     PRZ> We all know this can be violated by vendor specific configs and PRZ> merits mentioning as reference to 4456 again since this is a very PRZ> 'practical deployment' PRZ> targeted draft. In fact this draft should reference how two best PRZ> routes to same prefix via two different ASes (two group best paths PRZ> to same prefix) should be resolved or ECMP’ed.               Thus the route selection procedures       Walton, et al.           Expires August 6, 2015                 [Page 3] Àpar Internet-Draft          BGP Oscillation Solutions          February 2015        specified in [RFC4271] would conceptually involve two steps: first    organize the paths for an address prefix into groups according to    their respective neighbor-AS's, and calculate the most preferred one    (termed "Group Best Path") for each of the groups; Then calculate the    overall best path among all the Group Best Paths.   PRZ> Per above, pls refer to the resolution and whether it must be PRZ> uniform across all participating RRCs and iBGP peers involved.      As a generally recommended ([RFC4456], [RFC5065]) and widely adopted    practice, a route reflection cluster or a confederation sub-AS should    be designed such that the IGP metrics for links within a cluster (or    confederation sub-AS) are much smaller than the IGP metrics for the    links between the clusters (or confederation sub-AS).  This practice    helps achieve consistent routing within a route reflection cluster or    a confederation sub-AS.      When the aforementioned practice for devising a route reflection    cluster or confederation sub-AS is followed in a network, we claim    that the advertisement of all the Group Best Paths by a route    reflector or a confederation ASBR is sufficient to eliminate the MED-    induced route oscillations in the network.  This claim is validated    in Appendix A.      Note that a Group Best Path for an address prefix can be identified    by the combination of the address prefix and the neighbor-AS.  Thus    this approach can be implemented using the mechanism described in    ADD-PATH [I-D.ietf-idr-add-paths] for advertising multiple paths, and    in this case the neighbor-AS of a path may be used as the path    identifier of the path.      It should be noted that the approach of advertising the Group Best    Paths requires certain topological constrains to be satisfied in    order to eliminate the MED-induced route oscillation.      PRZ> Please give at least ONE example of what those 'certain topological PRZ> constraints' are if there are more than the 'short IGP distance inside'       In addition,    the BGP speakers still depend on the route selection by the route    reflector or the confederation ASBR.  As the route selection involves    the comparison of the nexthop's IGP metrics which are specific to a    particular BGP speaker, the routing information advertised by a route    reflector or a confederation ASBR may still be inadequate to avoid    other routing inconsistencies such as forwarding loops in certain    networks.    PRZ> This is simply too vague, especially for a standards track. Please PRZ> give at least one example of such looping.   5.  Route Reflection and Confederation      To allow a route reflector or a confederation ASBR to advertise    either the Available Paths or Group Best Paths using the mechanism    described in ADD-PATH [I-D.ietf-idr-add-paths], the following    revisions are proposed for BGP route reflection and BGP    Confederation.           Walton, et al.           Expires August 6, 2015                 [Page 4] Àpar Internet-Draft          BGP Oscillation Solutions          February 2015     5.1.  Route Reflection      Depending on the configuration, for a particular a route    reflector SHOULD include the with the "Send/Receive"    field set to 2 or 3 in the ADD-PATH Capability    [I-D.ietf-idr-add-paths] advertised to an IBGP peer.  When the ADD-    PATH Capability is also received from the IBGP peer with the "Send/    Receive" field set to 1 or 3 for the same , then the    following procedures shall be followed:      If the peer is a route reflection client, the route reflector SHOULD    advertise to the peer the Group Best Paths (or the Available Paths)    received from its non-client IBGP peers.  Depending on the    configuration, the route reflector MAY also advertise to the peer the    Group Best Paths (or the Available Paths) received from its clients.      If the peer is a non-client, the route reflector SHOULD advertise to    the peer the Group Best Paths (or the Available Paths) received from    its clients.   5.2.  Confederation     Depending on the configuration, for a particular a    confederation ASBR SHOULD include the with the "Send/    Receive" field set to 2 or 3 in the ADD-PATH Capability    [I-D.ietf-idr-add-paths] advertised to an IBGP peer, and to a    confederation external peer.  When the ADD-PATH Capability is also    received from the IBGP peer or the confederation external peer with    the "Send/Receive" field set to 1 or 3 for the same , then    the following procedures shall be followed:      If the peer is internal, the confederation ASBR SHOULD advertise to    the peer the Group Best Paths (or the Available Paths) received from    its confederation external peers.      If the peer is confederation external, the confederation ASBR SHOULD    advertise to the peer the Group Best Paths (or the Available Paths)    received from its IBGP peers.   PRZ> This needs specification WHAT is advertised to the peers not supporting PRZ> add-path? if we follow today's behavior, it would be any PRZ> of the best-group paths broken on IGP metric. PRZ> I think it is worth mentioning that if ANY of the involved RRs does PRZ> not support add-path, the solution COULD loop again using IGP as tie-breaker PRZ> for routes from different ASes. PRZ> PRZ> Or is one of the restrictions this draft suggests PRZ> that all RRs MUST be PRZ> add-path capable ?   6.  Deployment Considerations      Some route oscillations, once detected, can be eliminated by simple    configuration workarounds.  As carrying additional paths impacts the    memory usage and routing convergence in a network, it is recommended    that the impact be evaluated and the approach of using a    configuration workaround be considered in deciding whether to deploy    the proposed mechanism in a network.  In addition, the advertisement         Walton, et al.           Expires August 6, 2015                 [Page 5] Àpar Internet-Draft          BGP Oscillation Solutions          February 2015        of multiple paths should be limited to those prefixes which are    affected by MED-induced route oscillation.      While the route reflectors or confederation ASBRs in a network need    to advertise the Group Best Paths or Available Paths, the vast    majority of the BGP speakers in the network only need to receive the    Group Best Paths or Available Paths, which would involve only minor    software changes.      It should be emphasized that in order to eliminate the MED-induced    route oscillations in a network using the approach of advertising the    Group Best Paths, the recommended practice for devising a route    reflection cluster or confederation sub-AS with respect to the IGP    metrics ([RFC4456], [RFC5065]) should be followed.      It is expected that the approach of advertising the Group Best Paths    would be adequate to achieve consistent routing for the vast majority    of the networks.  For a network that has large number of alternate    paths, the approach should be a good choice as the number of paths    advertised by a reflector or a confederation ASBR is bounded by the    number of the neighbor-AS's for a particular address prefix.  The    additional states for an address prefix would also be per neighbor-AS    based rather than per path based.  The number of the neighbor-AS's    for a particular address prefix is typically small because of the    limited number of upstream providers for a customer and the nature of    advertising only customer routes at the inter-exchange points.      The approach of advertising the Group Best Paths, however, may still    be inadequate for certain networks to avoid other routing    inconsistencies such as forwarding loops.  The required topological    constrains could also be operationally challenging.  In these cases    the approach of advertising the Available Paths may be used, but    should be limited to those prefixes which are affected by MED-induced    route oscillation in a network carrying a large number of alternate    paths.  Note that the number of paths that need to be maintained and    advertised can be greatly reduced by accepting the IGP metric based    MEDs from other peering networks.    PRZ> does 'IGP metric based MED' mean 'copy IGP into MED & advertise?'. PRZ> If so, please define the term clearly.   7.  IANA Considerations      This memo includes no request to IANA.   8.  Security Considerations      This extension to BGP does not change the underlying security issues    inherent in the existing BGP [RFC4271].           Walton, et al.           Expires August 6, 2015                 [Page 6] Àpar Internet-Draft          BGP Oscillation Solutions          February 2015     9.  Acknowledgements      We would like to thank David Cook and Naiming Shen for their    contributions to the design and development of the solutions.   10.  References   10.1.  Normative References      [I-D.ietf-idr-add-paths]               Walton, D., Retana, A., Chen, E., and J. Scudder,               "Advertisement of Multiple Paths in BGP", draft-ietf-idr-               add-paths-10 (work in progress), October 2014.      [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate               Requirement Levels", BCP 14, RFC 2119, March 1997.      [RFC4271]  Rekhter, Y., Li, T., and S. Hares, "A Border Gateway               Protocol 4 (BGP-4)", RFC 4271, January 2006.      [RFC4456]  Bates, T., Chen, E., and R. Chandra, "BGP Route               Reflection: An Alternative to Full Mesh Internal BGP               (IBGP)", RFC 4456, April 2006.      [RFC5065]  Traina, P., McPherson, D., and J. Scudder, "Autonomous               System Confederations for BGP", RFC 5065, August 2007.   10.2.  Informative References      [RFC3345]  McPherson, D., Gill, V., Walton, D., and A. Retana,               "Border Gateway Protocol (BGP) Persistent Route               Oscillation Condition", RFC 3345, August 2002.   Appendix A.  Why the Group Best Paths Are Adequate?   PRZ> It seems to me there is another, more easily understood PRZ> ‘proof’ by basically saying that PRZ> IGP metric is comparable now only within each AS’s paths (MED), i.e. PRZ> an ‘best-path’ PRZ> cannot change MEDs based on comparing IGP metrics of two different PRZ> ASs (= there is now a ‘best-path’ per AS). This is normally the source of PRZ> ‘MED loops’ I saw, i.e. a RR PRZ> changes opinion and advertises different MEDs on the route because PRZ> the IGP metric comparison between routes with 2 different MEDs PRZ> triggers the ‘change of heart’.      It is assumed that the following common practice is followed.  A    route reflection cluster or a confederation sub-AS should be designed    such that the IGP metrics for links within a cluster (or    confederation sub-AS) are much smaller than the IGP metrics for the    links between the clusters (or confederation sub-AS).  This practice    helps achieve consistent routing within a route reflection cluster or    a confederation sub-AS.      Observe that in a network that maintains full IBGP mesh only the    paths that survive the (Local_Pref, AS-PATH Length, Origin, MED)    comparisons [RFC4271] would contribute to the route selection in the    network.         Walton, et al.           Expires August 6, 2015                 [Page 7] Àpar Internet-Draft          BGP Oscillation Solutions          February 2015        Consider a route reflection cluster that sources one or more paths    that would survive the (Local_Pref, AS-PATH Length, Origin, MED)    comparisons among all the paths in the network.  One of these    surviving paths would be selected as the Group Best Path by the route    reflector in the cluster.  Due to the constrain on the IGP metrics as    described previously, this path would remain as the Group Best Path    and would be advertised to all other clusters even after a path is    received from another cluster.      On the other hand, when no path in a route reflection cluster would    survive the (Local_Pref, AS-PATH Length, Origin, MED) comparisons    among all the paths in the network, the Group Best Path (when exists)    for a route reflector would be from another cluster.  Clearly the    advertise of the Group Best Path by the route reflector to the    clients only depends on the paths received from other clusters.      Therefore there is no MED-induced route oscillation in the network as    the advertisement of a Group Best Path to a peer does not depend on    the paths received from that peer.      The claim for the confederation can be validated similarly.   Authors' Addresses      Daniel Walton    Cumulus Networks    140C S. Whisman Rd.    Mountain View, CA  94041    USA      Email: dwalton at cumulusnetworks.com        Alvaro Retana    Cisco Systems, Inc.    7025 Kit Creek Rd.    Research Triangle Park, NC  27709    USA      Email: aretana at cisco.com                       Walton, et al.           Expires August 6, 2015                 [Page 8] Àpar Internet-Draft          BGP Oscillation Solutions          February 2015        Enke Chen    Cisco Systems, Inc.    170 W. Tasman Dr.    San Jose, CA  95134    USA      Email: enkechen at cisco.com        John Scudder    Juniper Networks    1194 N. Mathilda Ave    Sunnyvale, CA  94089    USA      Email: jgs at juniper.net                                                                       Walton, et al.           Expires August 6, 2015                 [Page 9]