Subject: RE: Last Call: <draft-ietf-sidr-origin-ops-21.txt> (RPKI-Based Origin Validation Operation) to Best Current Practice

From: "George, Wes" <[email-masked]>
Date: Mon, 23 Sep 2013 08:58:01 -0400

I've reviewed multiple iterations of this draft, and I believe it is mostly ready to go.

However, the concerns I raised during WGLC in http://www.ietf.org/mail-archive/web/sidr/current/msg05010.html regarding the ambiguity of some of the guidance regarding location of RPKI caches ("close") in section 3 still have not been addressed. IMO if it is important enough to discuss proximity, it is important enough to devote some words to explaining the rationale for that recommendation so that operators can make an informed design decision.

Thanks,

Wes George

> -----Original Message-----
> From: [email-masked] [mailto:ietf-announce-
> [email-masked]] On Behalf Of The IESG
> Sent: Tuesday, September 17, 2013 10:52 AM
> To: IETF-Announce
> Cc: [email-masked]
> Subject: Last Call: <draft-ietf-sidr-origin-ops-21.txt> (RPKI-Based
> Origin Validation Operation) to Best Current Practice
>
>
> The IESG has received a request from the Secure Inter-Domain Routing WG
> (sidr) to consider the following document:
> - 'RPKI-Based Origin Validation Operation'
>   <draft-ietf-sidr-origin-ops-21.txt> as Best Current Practice
>
> The IESG plans to make a decision in the next few weeks, and solicits
> final comments on this action. Please send substantive comments to the
> [email-masked] mailing lists by 2013-10-01. Exceptionally, comments may
> be
> sent to [email-masked] instead. In either case, please retain the
> beginning of the Subject line to allow automated sorting.
>
> Abstract
>
>
>    Deployment of RPKI-based BGP origin validation has many operational
>    considerations.  This document attempts to collect and present those
>    which are most critical.  It is expected to evolve as RPKI-based
>    origin validation continues to be deployed and the dynamics are
>    better understood.
>
>
>
>
>
> The file can be obtained via
> http://datatracker.ietf.org/doc/draft-ietf-sidr-origin-ops/
>
> IESG discussion can be tracked via
> http://datatracker.ietf.org/doc/draft-ietf-sidr-origin-ops/ballot/
>
>
> No IPR declarations have been submitted directly on this I-D.
>

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: Randy Bush <[email-masked]>
Date: Mon, 23 Sep 2013 08:51:59 -1000

> However, the concerns I raised during WGLC in
> http://www.ietf.org/mail-archive/web/sidr/current/msg05010.html
> regarding the ambiguity of some of the guidance regarding location of
> RPKI caches ("close") in section 3 still have not been addressed. IMO
> if it is important enough to discuss proximity, it is important enough
> to devote some words to explaining the rationale for that
> recommendation so that operators can make an informed design decision.

hey, thanks for caring and reviewing

i think the two paragraphs you would like to see improved are

   As RPKI-based origin validation relies on the availability of RPKI
   data, operators SHOULD locate caches close to routers that require
   these data and services.  'Close' is, of course, complex.  One should
   consider trust boundaries, routing bootstrap reachability, latency,
   etc.

   If insecure transports are used between an operator's cache and their
   router(s), the Transport Security recommendations in [RFC6810] SHOULD
   be followed.  In particular, operators MUST NOT use insecure
   transports between their routers and RPKI caches located in other
   Autonomous Systems.

i am not against further explanation, send text. but short text.  :)

experiments have shown that latency between cache and router, and
between caches in a cache dag, is not an appreciable issue.  as the
reports of these experiments are not peer reviewed, it is not clear that
they are good to reference.  one has been accepted at conext, but page
count limit prevented the latency issue from being included :(

i thought bootstrap reachability would be fairly obvious to an operator.
but if simple routing and no dns dependency were not obvious to you,
then a few words are indeed needed.  or am i missing your point here?

as the rpki-rtr protocol is beyond the border of object-based security,
the operator only has transport security between the cache(s) and the
router.  it would probably be good to add this as it motivates the
"close" and "trust boundary" cautions.

what is missing from the second paragraph?  section 7 of rfc6810, as
referenced, was very heavily reviewed and pretty much bludgeons this one
to death.  of course if you have specific ideas for improvement, that'd
be great.

> "ok, we're going to stick it in a VM on our two national data center
> compute infrastructures like the rest of our servers, we can spin up
> more instances if you need to scale it" 
> "but RFC mumble mumble says that we shouldn't do that..."
> "ok, why? And where do you want to put it?"
> "ummm... 'close' to the routers? Because...reasons"

i am not sure it would be useful to go into the general issue of why
caches should be proximal to the consumer as it is a rather well-
explored area, from akamia and limelight to dns.  but if you have a
sentence or two that communicates this, send it over.

randy

---

From: Randy Bush <[email-masked]>
Date: Mon, 23 Sep 2013 10:46:54 -1000

take two paragraphs and call back in the morning if you are still in
pain :)

randy

   In order that routers need not perform certificate validation,
   cryptographic operations, etc., the RPKI-Router protocol, [RFC6810],
   does not provide object-based security to the router.  I.e. the
   router may not validate the data cryptographically from well-known
   trust anchor.  The router trusts the cache to provide correct data
   and relies on transport based security for the data received from the
   cache.  Therefore the authenticity and integrity of the data from the
   cache should be well protected, see Section 7 of [RFC6810].

   As RPKI-based origin validation relies on the availability of RPKI
   data, operators SHOULD locate caches close to routers that require
   these data and services.  'Close' is, of course, complex.  One should
   consider trust boundaries, routing bootstrap reachability, latency,
   etc.  E.g. as the router can not validate the received data using a
   trust anchor, it should only accept data from caches it strongly
   trusts to provide valid data.  And a router should bootstrap from a
   chache which is reachable without relying on other infrastructure
   such as DNS or routing protocols.

---

From: "George, Wes" <[email-masked]>
Date: Tue, 24 Sep 2013 12:26:38 -0400

> From: Randy Bush [mailto:[email-masked]]
[snip]
> i am not against further explanation, send text. but short text.  :)
[WEG] just the first paragraph really, and as I'll note below - I'd love to send text, but I don't understand one of the recommendations
> between caches in a cache dag, is not an appreciable issue.
[WEG] ok, so why is "close" important then?

> then a few words are indeed needed.  or am i missing your point here?
[WEG] yes, completely obvious. Though the last two sentences of your suggested text in the other email is a useful addition
>
>
> what is missing from the second paragraph?
[WEG] I'm actually happy with second para
> sentence or two that communicates this, send it over.
[WEG] Generally, I understand why closer is better for content caches (Akamai/llnw) and DNS, but not for RPKI caches. You're making a link between the two that I'm not following. Both of the former benefit from proximity because it reduces latency and reduces unnecessary backhaul and potentially allows for a geographically customized service, resulting in improved user experience. If latency isn't a factor (at least in the average-sized propagation domain), and RPKI caches aren't particularly bandwidth intensive, why does proximity matter? Is this just an extension of the trust domain and limited dependence on routing protocols? If so, I'd dispense with recommending "close" because it confuses the issue and just keep the discussion about secondary dependencies and trust domains.

Thanks
Wes George

---

From: Randy Bush <[email-masked]>
Date: Tue, 24 Sep 2013 08:53:13 -1000

hi wes,

> why does proximity matter? Is this just an extension of the trust
> domain and limited dependence on routing protocols? If so, I'd
> dispense with recommending "close" because it confuses the issue and
> just keep the discussion about secondary dependencies and trust
> domains.

are you really saying that i should be comfortable configuring a seattle
router to use a cache in tokyo even though both are in my network and
there is a pretty direct hop?

 1  r0.sea.rg.net (147.28.0.4)  0.189 ms  0.306 ms  0.230 ms
 2  r1.sea.rg.net (147.28.0.5)  0.307 ms  0.259 ms  0.221 ms
 3  sea001bf00.iij.net (206.81.80.237)  0.336 ms  0.383 ms  0.348 ms
 4  tky008bf00.IIJ.net (206.132.169.110)  88.723 ms  94.377 ms  124.044 ms
 5  tky001bb10.IIJ.Net (58.138.80.18)  83.639 ms  83.701 ms  89.133 ms

i am willing to hack the text.  but i am just not sure that proximity is
not a good attribute.  the longer the wire the greater the likelihood of
it being cut.

randy

---

From: Randy Bush <[email-masked]>
Date: Tue, 24 Sep 2013 09:29:03 -1000

> Following this line of reasoning (which is not unreasonable); if the
> router requires the cache to arrive at correctness, maybe the cache
> should be _inside_ the router.

yep.  but no chance of it fitting in existing routers, and routers today
don't have the crypto oomph to validate (frequently).  hence rpki-rtr
protocol, rfc6810.

randy

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 24 Sep 2013 20:38:56 -0400

On Tue, Sep 24, 2013 at 12:26 PM, George, Wes <[email-masked]> wrote:
> [WEG] Generally, I understand why closer is better for content caches (Akamai/llnw) and DNS, but not for RPKI caches. You're making a link between the two that I'm not following. Both of the

[CLM]
this is about the consumer of the data, right? (and convenience to the
operator that hosts the cache).

In the akamai/cdn example 'consumer' is 'cable/dsl/etc' user. "Close"
is convenient-to-the-operator close to the 'cable/dsl/etc' user.
Perhaps 'metro' is close enough for CDNs, balancing latency and
backhaul requirements.

In the RPKIcache example, 'consumer' is 'routers in your network'.
'Close' is 'close enough that bootstrapping isn't a problem', balanced
with 'gosh, maybe I don't want to put one on top of each router! plus
associated management headaches to deal with these new
systems/appliances'.

former benefit from proximity because it reduces latency and reduces
unnecessary backhaul and potentially allows for a geographically
customized service, resulting in improved user experience. If latency
isn't a factor (at least in the average-sized propagation domain), and
RPKI caches aren't particularly bandwidth intensive, why does
proximity matter? Is this just an extension of the trust domain and
limited dependence on routing protocols? If so, I'd dispense with
recommending "close" because it confuses the issue and just keep the
discussion about secondary dependencies and trust domains.

[CLM]
I guess one way is to say: "People should understand the dependencies
and engineer appropriately" ... which you kind of asked to not say in
the original comment. (or is the issue that the dependencies aren't
clear?)

> _______________________________________________
> sidr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidr

---

From: Tony Tauber <[email-masked]>
Date: Tue, 24 Sep 2013 15:14:09 -0400

> there is a pretty direct hop?

Why shouldn't it just be "in the cloud" and you not care about where?
Oh, because RPKI-RTR isn't an application, it's infrastructure!

> not a good attribute.  the longer the wire the greater the likelihood of

Following this line of reasoning (which is not unreasonable); if the router
requires the cache to arrive at correctness, maybe the cache should be
_inside_ the router.

I'm not trying to be snarky, I know all the bootstrapping concerns have
been hashed and re-hashed.
To me, this funkiness seems like a holdover from that.

Tony

---

From: "George, Wes" <[email-masked]>
Date: Wed, 25 Sep 2013 12:35:39 -0400

> there is a pretty direct hop?
[WEG] not necessarily. But I'm also not saying that there would *never* be a scenario where that makes sense.
> not a good attribute.  the longer the wire the greater the likelihood of
[WEG] Proximity is generally a good attribute, but it's not free, so help the reader understand the tradeoffs so that they can justify spending money to get better proximity. The draft hasn't provided a good explanation of how proximity improves RPKI (or what problems a lack of it creates), nor how to determine whether the level of proximity in a given design is "good enough" to provide the perceived benefit.

Your statement about long wires being cut is an argument for redundancy and geographic diversity, not proximity. Of course an operator needs to take into account the topology of their network when considering geographic diversity, but that's not necessarily going to translate to a need for proximity.

The draft says one should consider latency, but never explains how increased latency impacts RPKI, nor gives any guidance on how much might be too much, especially when one considers that RPKI is an asymmetric system that changes on mostly human time-scales (order of hours or days). Most places where latency matters, it's a function of what RTT does to throughput or the perception of speed for interaction (how long it takes between when I click and when something happens). This isn't an interactive system. What is latency-sensitive in this system on the subsecond scale such that it can be affected by moving the cache closer? Even on systems configured for very frequent synchronization, are we expecting anything to change on that timescale?

See my other response to CMorrow for questions around bootstrapping.

Wes

---

From: "George, Wes" <[email-masked]>
Date: Wed, 25 Sep 2013 12:38:45 -0400

> From: [email-masked] [mailto:[email-masked]]
>
> [CLM]
[WEG] that's part of my issue - the only way that you get "close enough that bootstrapping isn't a problem" is when the cache and router are directly connected. Otherwise there *is* going to be some amount of time while the router is coming up that it can't talk to its configured caches e.g. while it learns the route(s) to the cache(s). I think that supports a recommendation to put the caches in your IGP instead of BGP, so that you get faster convergence of those routes and therefore have access to the cache when BGP comes up and starts converging, rather than once BGP is partially converged. But the draft doesn't say that.
The question is, does the propagation/convergence delay for an IGP in an average network (let's call it somewhere between subsecond and 5 seconds) make a non-trival difference in RPKI's bootstrap behavior, especially since BGP convergence is also dependent on IGP convergence? Can we make a clearer recommendation of the performance envelope we're shooting for so that people can design accordingly? I'm not sure I buy a general "faster(or closer) is always better" recommendation - at some point, we hit diminishing returns, given that this is mostly a human time-scale system. The document doesn't provide clear guidance on how to balance that tradeoff.
> the original comment. (or is the issue that the dependencies aren't

[WEG] The issue is that the dependencies aren't clear. I'm not expecting the text to be too prescriptive here, because all networks are different, but I need enough technical discussion to properly "understand the dependencies and engineer accordingly". This is an operational considerations document, so it needs to tell operators what breaks if they don't do it as recommended. If this is about bootstrapping, then we need to be clearer about the relationship between bootstrapping and network convergence (since recommending that the cache is directly connected to the router is impractical) and how it impacts RPKI cache-router communication and performance. If it's about reducing latency via proximity, then we need to explain how much latency is too much latency and why. If it's about proper geographic diversity within a network's topology, then we need to say that. If we don't actually know if it makes a difference, and so are defaulting to recommendations that most folks agree are generally a good idea, we should say that. But right now we're assuming too much, IMO.

Wes

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 25 Sep 2013 22:22:52 -0400

On Wed, Sep 25, 2013 at 12:38 PM, George, Wes <[email-masked]> wrote:
> [WEG] that's part of my issue - the only way that you get "close enough that
> bootstrapping isn't a problem" is when the cache and router are directly

there's some baseline that's acceptable, you intimate that IGP comes
up before EGP below. that makes some sense, and thus maybe the target
is 'in your igp, close enough that fiber failures won't be a problem'
then?

> connected. Otherwise there *is* going to be some amount of time while
> the router is coming up that it can't talk to its configured caches e.g. while

but the data in the cache only REALLY matters for bgp validation... so
your IGP clue below isn't unreasonable.

> it learns the route(s) to the cache(s). I think that supports a recommendation
> to put the caches in your IGP instead of BGP, so that you get faster

I actually didn't note a [ie]GP recommendation in the doc.

> convergence of those routes and therefore have access to the cache
> when BGP comes up and starts converging, rather than once BGP is
> partially converged. But the draft doesn't say that.

ok

> The question is, does the propagation/convergence delay for an IGP in an
> average network (let's call it somewhere between subsecond and 5 seconds)
> make a non-trival difference in RPKI's bootstrap behavior, especially since
> BGP convergence is also dependent on IGP convergence? Can we make a
> clearer recommendation of the performance envelope we're shooting for so
> that people can design accordingly? I'm not sure I buy a general "faster(or
> closer) is always better" recommendation - at some point, we hit diminishing
> returns, given that this is mostly a human time-scale system. The document
> doesn't provide clear guidance on how to balance that tradeoff.

i think a bunch of this really also depends on the operator deploying
though... 'its hard to get server people to do X for me' or 'gosh,
these appliances can be managed by network-operations! and they are
cheap-ish' or 'gosh, we don't have 1gbps ports anymore in general,
crap...'

I do think the original intent was to not dictate: "Must be 5ms from
the router, or else!!" and rely upon the operator to do the tradeoff
you just made above. Each network is different in it's expectations
from the infra, and each has different igp/egp designs as well as
fiber plant restrictions. I think it's going to be rough going making
a recommendation much more than:
  1) make sure the cache is available before BGP starts to converge for a device

and I actually can't come up with something else that's super helpful
:( even the above might be 'too much advice', if your plan is to
accept all routes and simply de-pref until validation might happen
then re-evaluate as you can.

> [WEG] The issue is that the dependencies aren't clear. I'm not expecting the
> text to be too prescriptive here, because all networks are different, but I need
> enough technical discussion to properly "understand the dependencies and
> engineer accordingly". This is an operational considerations document, so it
> needs to tell operators what breaks if they don't do it as recommended. If this

ok...

> is about bootstrapping, then we need to be clearer about the relationship
> between bootstrapping and network convergence (since recommending
> that the cache is directly connected to the router is impractical) and how
> it impacts RPKI cache-router communication and performance. If it's about
> reducing latency via proximity, then we need to explain how much latency is
> too much latency and why. If it's about proper geographic diversity within a
> network's topology, then we need to say that. If we don't actually know if it
> makes a difference, and so are defaulting to recommendations that most folks
> agree are generally a good idea, we should say that. But right now we're
> assuming too much, IMO.

ok, the current text is:
"   As RPKI-based origin validation relies on the availability of RPKI
   data, operators SHOULD locate caches close to routers that require
   these data and services.  'Close' is, of course, complex.  One should
   consider trust boundaries, routing bootstrap reachability, latency,
   etc"

Maybe something like:
"   As RPKI-based origin validation relies on the availability of RPKI
   data, operators SHOULD locate caches close enough to routers that
   require these data and services such that failures in local device
routing domain
   do not impact cache availability. One should consider trust
boundaries, routing
   bootstrap reachability, latency, etc"

-chris

(content warning removed.. since it didn't come from TWC, and my words
are not as restricted)

---

From: Randy Bush <[email-masked]>
Date: Wed, 25 Sep 2013 16:45:17 -1000

>> [WEG] that's part of my issue - the only way that you get "close
>> enough that bootstrapping isn't a problem" is when the cache and
>> router are directly

but do we want to go to that level of detail?  it's just nit-pick fodder

> your IGP clue below isn't unreasonable.

and the folk who smoked the "use ibgp for your igp" dope at&t handed out
to customers?

i really don't think we want to get into the nitty gritty of how folk
should configure their networks.  way to many styles, variables, corner
cases, ...

explain the underlying physics and what's prudent and let the network's
engineers design their network.

but yes, 'close' may be a bit so loose, and seems to be major
nit-picking fodder.

>> it learns the route(s) to the cache(s). I think that supports a
>> recommendation to put the caches in your IGP instead of BGP, so that
>> you get faster

not because i ran out of ink (credit to nw)

>> partially converged. But the draft doesn't say that.

"but i only have that one router.  and if bgp does not come up, then the
cache can not load up."

do you *really* want to go down these myriad twisty passages?

> cheap-ish' or 'gosh, we don't have 1gbps ports anymore in general,

yes, makes one want to go back to running a small network.

> a recommendation much more than:

i agree up to here

>   1) make sure the cache is available before BGP starts to converge

you can't.  cache may need routing to fill its little tummy.

> and I actually can't come up with something else that's super helpful

which is why i stopped where i did.  though i think one could amplify a
bit on 'close'.  your paragraph is not bad

> "As RPKI-based origin validation relies on the availability of RPKI
>  routing domain do not impact cache availability. One should consider
>  trust boundaries, routing bootstrap reachability, latency, etc"

i would modify slightly (e.g. s/do not impact/are unlikely to impact/)
but can stitch it in.  and latency may be a red herring.  but i eat
herring, so wth.

randy

---

From: Randy Bush <[email-masked]>
Date: Wed, 25 Sep 2013 17:29:33 -1000

how about

   To relieve routers of the load of performing certificate validation,
   cryptographic operations, etc., the RPKI-Router protocol, [RFC6810],
   does not provide object-based security to the router.  I.e. the
   router may not validate the data cryptographically from a well-known
   trust anchor.  The router trusts the cache to provide correct data
   and relies on transport based security for the data received from the
   cache.  Therefore the authenticity and integrity of the data from the
   cache should be well protected, see Section 7 of [RFC6810].

   As RPKI-based origin validation relies on the availability of RPKI
   data, operators SHOULD locate RPKI caches close to routers that
   require these data and services in order to minimize the impact of
   likely failures in local routing, intermediate devices, long
   circuits, etc.  One also should consider trust boundaries, routing
   bootstrap reachability, etc.  E.g. a router should bootstrap from a
   chache which is reachable with minimal reliance on other
   infrastructure such as DNS or routing protocols.

randy

---

From: "George, Wes" <[email-masked]>
Date: Thu, 26 Sep 2013 09:43:31 -0400

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Randy Bush
>
> how about
>
[WEG] fine, though it's unclear if "may" in the above is intended to be normative. I think it's not, but just pointing it out.

>    infrastructure such as DNS or routing protocols.
[WEG] this is better, but I still maintain that in the first sentence, "close" isn't actually the goal we're trying for.

How about:

...operators SHOULD consider the relationship between the routers that require these data and services and the location of the RPKI caches in the network's topology. Caches SHOULD be located so that they can take advantage of geographic redundancy and minimize the impact of likely failures in local routing, ....

And add this at the end to explain why reliance on routing protocols @ bootstrap is risky:

...or routing protocols. Reliance on routing protocol convergence to reach a cache at bootstrap time can result in significant increases in total convergence time as the router converges partially, synchronizes with the RPKI cache, and then must re-converge based on the data from the cache.

Thanks

Wes

---

From: Randy Bush <[email-masked]>
Date: Thu, 26 Sep 2013 08:28:40 -1000

> [WEG] fine, though it's unclear if "may" in the above is intended to
>    be normative. I think it's not, but just pointing it out. 

actually, it was a grammar error which i caught right after posting.  in
my edit buffer, it is 'can' as in ablility not 'may' as in permission.

> [WEG] this is better, but I still maintain that in the first sentence,
> "close" isn't actually the goal we're trying for.

for some definitions of 'we' :)

> How about:
> ...operators SHOULD consider the relationship between the routers that
> require these data and services and the location of the RPKI caches in
> the network's topology. Caches SHOULD be located so that they can take
> advantage of geographic redundancy and minimize the impact of likely
> failures in local routing, ....

i don't even know what geographic redundancy is, alternate earths?  if
you mean redundant network topology, i think it is more than that.  i
really think physical line length matters.  as i said, the longer the
circuit the more likely a boat anchor will whack it.  hence close.

> ...or routing protocols. Reliance on routing protocol convergence to
> reach a cache at bootstrap time can result in significant increases in
> total convergence time as the router converges partially, synchronizes
> with the RPKI cache, and then must re-converge based on the data from
> the cache.

i think i understand what you want made more explicit.  today's try

   router can not validate the data cryptographically from a well-known

   circuits, etc.  One should also consider trust boundaries, routing

   For example, if a router needs its BGP and/or IGP to converge for the
   router to reach a cache, once a cache is reachable, the router will
   then have to reevaluate prefixes already learned via BGP.  Such
   configurations should be avoided if possible.

randy

---

From: "George, Wes" <[email-masked]>
Date: Thu, 26 Sep 2013 17:19:39 -0400

> i don't even know what geographic redundancy is, alternate earths?
[WEG] nah, the latency is too high until we sort out IP over Quantum Entanglement. ;-) Geographic redundancy in the context of things that live on servers is that it exists on servers in more than one physical location (e.g. Datacenter East and Datacenter West) so that there isn't a single point of failure. In this case, that'd probably be in the form of two caches backing each other up (router configured to use both).

 if
> circuit the more likely a boat anchor will whack it.  hence close.
[WEG] yes, but ultimately that's dependent on your network topology. If "close" is the only way to mitigate that (vs having a truly redundant backup that doesn't share any topology in its path), then sure, that's what you do. But I don't think it directly translates to "should be close". We clearly disagree, but I'm not going to belabor the point any further.
>    configurations should be avoided if possible.

[WEG] close enough, ship it.

Wes

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 27 Sep 2013 09:16:40 -0400

> [WEG] close enough, ship it.

hurray! :) (I'm also ok with the last edit buffer fun)

thank wes and randy for a fun discussion.

-chris