Subject: motivations (was: Re: draft-housley-two-maturity-levels-00)

From: RJ Atkinson <[email-masked]>
Date: Tue, 22 Jun 2010 06:43:58 -0400

On 22nd June 2010, at 10:12:13 CET, Eliot Lear wrote:
> This then leads to a question of motivations.  What are the motivations
> for the IESG, the IETF, and for individual implementers?  Traditionally
> for the IETF and IESG, the motivation was meant to be a signal to the
> market that a standard won't change out from underneath the developer.

The above seems fairly muddled as written.

Traditionally, "the market" refers to consumers, users, 
and operators, rather than "implementers" or "developers" 
of products.

Indeed, moving beyond Proposed Standard has long been a signal 
to users, consumers, and operators that a technology now has
demonstrated multi-vendor interoperability.  

Further, by moving technology items that lacked multi-vendor 
interoperability into optional Appendices, or downgrading
them to "MAY implement" items, that process also makes clear
which parts of the technology really were readily available, 
as different from (for example) an essentially proprietary 
feature unique to one implementation.

In turn, that tends (even now) to increase the frequency that 
a particular IETF-standardised technology appears in RFPs 
(or Tender Announcements).  In turn, that enhanced the business 
case for vendors to implement the interoperable standards.

Standards are useful both for vendors/implementers and also
for consumers/users/operators.  However, standards are useful
to those 2 different communities in different ways.  

The IETF already has a tendency to be very vendor-focused &
vendor-driven.  It is best, however, if the IETF keeps the 
interests of both communities balanced (rather than tilting 
towards commercial vendors).

> Question #1: Is such a signal needed today?  

Yes.  Users/operators/consumers actively want and need
independent validation that a standard is both interoperable
and reasonably stable.

> If we look at the 1694
> Proposed Standards, are we seeing a lack of implementation due to lack
> of stability?  I would claim that there are quite a number of examples
> to the contrary (but see below).

Wrong question.  How clever to knock down the wrong strawman.

The right questions are:
	A) whether that signal is useful to consumers/users/operators 

	The answer to this is clearly YES, as technologies that
	have advanced beyond Proposed Standard (PS) have a higher
	probability of showing up in RFPs and Tender Requirements.

	As examples, the JITC and TIC requirements pay a great
	deal of attention to whether some technology is past PS.
	Various IPv6 Profile documents around the world also pay
	much attention to whether a particular specification is
	past PS.

	B) whether that signal has a feedback loop to implementers/
	   vendors that still works.

	The answer to this is also clearly YES.  Technologies that
	appear in RFPs or Tender Requirements have a stronger
	business case for vendors/implementers, hence are more
	likely to be widely implemented.

	Items that appear in the TIC or JITC requirements are very
	very likely to be broadly implemented by many network
	equipment vendors.  The same is true for technologies 
	in various IPv6 Profiles around the world.

> Question #2: Is the signal actually accurate?  

Yes.

> Is there any reason for a developer to believe that the day after
> a "mature" standard is announced, a new Internet Draft won't
> in some way obsolete that work? 

Again, the wrong question, and an absurdly short measurement
time of 1 day.  Reductio ad absurdum is an often used technique
to divert attention when one lacks a persuasive substantial
argument for one's position.

By definition, Internet-Drafts cannot obsolete any 
standards-track document while they remain Internet-Drafts.  

Only an IESG Standards Action can obsolete some mature standard, 
and that kind of change happens slowly, relatively infrequently, 
and with long highly-visible lead times.

> What does history say about this effort? 

History says that 2-track has NOT happened several times already
because people (e.g. Eliot Lear) quibble over the details,
rather than understand that moving to 2-track is an improvement
and that "optimum" is the enemy of "better" in this situation.

> Question #3: What does such a signal say to the IETF?  

It is a positive feedback loop, indicating that work is
stable and interoperable.  It also says that gratuitous
changes are very unlikely to happen.  By contrast, 
technologies at Proposed Standard very frequently have 
substantial changes, often re-cycling back to PS with
those major changes.

Further, the new approach will have the effect of making
it easier to publish technologies at Proposed Standard,
which would be good all around.

> I know of at least one case where work was not permitted
> in the IETF precisely because a FULL STANDARD was said
> to need soak time.  It was SNMP, and the work that was
> not permitted at the time was what would later become
> ISMS.

That is history from long ago, under very different
process rules from now, so is totally irrelevant.  
It isn't even a good example of how things generally 
worked at that time.

> Question #4:  Is there a market advantage gained by an implementer
> working to advance a specification's maturity?

Again, wrong question.

That noted, the answer is clearly Yes.  Early implementers who show
interoperability are well positioned to win RFPs that require a
technology that has moved beyond Proposed Standard, while trailing
implementers often end up unqualified to bid/tender due to the
absence of such a feature.

> If there *is* a market advantage, is that something a standards
> organization wants?  

Yes, because it encourages broad implementation, broad interoperability,
and broad adoption of openly specified standards.

> Might ossification of a standard retard innovation
> by discouraging extensions or changes?

This is wildly less likely under 2-track than it already
is today, partly because it will be much easier for a
sensible revision to move to Proposed Standard.

> Question #5:  Are these the correct questions, and are there others that
> should be asked?

Largely not.  

You've clearly got a lot of vendor-bias in your writing above.

Your note cearly neglected the user/operator/consumer community 
(who benefit most from having a 2-track system, although 
vendors/implementers also benefit).

Yours,

Ran

---

From: Eliot Lear <[email-masked]>
Date: Wed, 23 Jun 2010 14:45:59 +0200

Hi Ran, and thanks for your reply.

There are two separate issues that we need to distill.  First, what to
do about draft-housley-two-maturity-levels-00, and second, how do take
input to improve the overall process?

I have not really come down on one side or the other on this draft
(yet).  To be sure, two maturity levels seem better than three, and as
you know, I've proposed a single maturity level in the past, so to me,
the draft goes in the right direction.  However, I do not know how many
times we get to change this sort of procedure, and I believe the
community and IESG choice could be better informed than it is today. 
Having been involved in NEWTRK, and having produced what I think was the
only output from that group, in terms of RFCs or processes, I think I
know about which I write, when I say this community can become a bit of
an echo chamber, and could use a bit of formal academic input. 
Conveniently, there are researchers in this area.  This is an even
stronger reason for for me to not state an opinion about whether to
advance the draft.

As to the questions I asked, you and I obviously hold very different
views.  In the end, it is of course for researchers (who are the real
target of my questions) to ask what questions that they think might be
telling about our process.  I hope this discussion informs them, if and
when they review it.

You claim I have a vendor bias.  Guilty, but I am also concerned that we
do the right things for the right reasons, and that motivations are
reasonably aligned so that we have some reason to believe that what is
proposed will work and not have perverse impact.  Absent some serious
analysis, we also are making the assumption that the logic of decisions
of over twenty years ago holds today, when in fact we don't really even
know if it held then.

And now as to your specifics, you have placed a lot of weight on one
example, seemingly extrapolating from it, the Joint Interoperability
Test Command.  I have no experience working with them, and defer to
yours.  However, when you say,

> 	much attention to whether a particular specification is

It leads to the following questions:

    * Would the vendors have implemented the functionality ANYWAY? 
      Specifically, would other RFPs have already driven vendors in this
      direction?  Can you cite a counter example, where that was not the
      case?
    * Is the defense industry at all representative of the broader
      market?  My own experience leads to an answer of, “barely at all”,
      and this has been assuredly the case with the Internet where a
      huge portion has run on on PS, Internet-Drafts, and proprietary
      standards, and not waited for advancement.  Examples have included
      BGP, MPLS-VPNs, HTTP, SSL, and Netflow, just to name a few. 

But again, I would like to see a rigorous analysis, rather than simply
rely on either of our personal experiences.

> towards commercial vendors).
While this is a perhaps laudable idea, someone has to do the work to get
specifications to the next standards level.  The whole point of my
questions is to determine what motivations that someone might have for
actually performing that work.

> Wrong question.  How clever to knock down the wrong strawman.

There's no need to be rude or snarky with me, even if you disagree.  You
are looking at this from the angle of the customers, and that's
perfectly reasonable.  I'm looking at it from the developers' point of
view, and from the supply side of your equation.  Both seem reasonably
valid, and so I have no qualms with the question part of your (A),
although as I mentioned above, I question your answer.

> 	likely to be widely implemented.

Certainly so, but I don't understand how you made the leap of logic from
your question to your answer.  Do we have situations, for instance,
where a proposed standard is compared to a draft standard, or a draft
standard is compared to a full standard, and one is chosen over the
other?  If so, are they the norm, and are they likely to drive
implementation?  Also, if all this gets you is interoperability, but
doesn't actually solve your problem, is THAT a good thing for the
customer?  IMHO this was precisely the case for SNMPv3.

> standards-track document while they remain Internet-Drafts

I think you misunderstand what I mean by "obsolete".  I don't mean
"obsolete" in the sense that you see it in the RFC directory, but
whether a new idea will overtake what has just been standardized.  Put
another way, if a developer is going to spend money ossifying work, what
is the risk they will lose their investment to something new?  I'll
grant you it's almost impossible to calculate, but perhaps history can
be of some use.  That analysis should be done.

> substantial changes, often re-cycling back to PS with

Where do we see cases where gratuitous changes take place at Proposed?

> which would be good all around.

Why do you come to this conclusion, given that PS isn't changing?

> It isn't even a good example of how things generally 

I'll clarify.  This was with SNMPv3 – and while "long ago" is a matter
of perspective, I am unaware of any rule changes that would have
impacted that case.  And since there are few cases of full standards in
any event, the sample size is a problem ;-)

>> working to advance a specification's maturity?

Why?  Keeping in mind we're  looking at reasons for people to do the work.

> absence of such a feature.

Again, there seems to be a leap of logic here that somehow because a
specification has attained draft or full standard will cause work to be
done that wasn't done before.  Those are facts not in evidence.

> sensible revision to move to Proposed Standard.
 
I don't understand what you're saying.  Are you arguing that recycling
back down to proposed will be easier?

Regards,

Eliot

---

From: RJ Atkinson <[email-masked]>
Date: Wed, 23 Jun 2010 09:31:39 -0400

> yours.  However, when you say,

Again, you setup an incorrect strawman, and then knock it down.

In the quote (below), I also mentioned the "various IPv6
Profile documents around the world", which you ignore,
apparently in order to incorrectly characterise my note 
as using a single example.  There are a number of cases
where a large customer's requirements (in RFPs or Tender
opportunities) have driven feature priorities.  The TIC
and JITC are merely examples.  Numerous other examples
exist, including a large bank in central Europe and 
several ISPs.

Yes.

There are certainly numerous cases where vendor implementation 
timing and new feature prioritisation were directly impacted 
by a profile document cited in some RFP, and where that profile
document's contents were directly impacted by whether a
particular technology was at Proposed Standard or some more
advanced stage in the IETF processes.

The most obvious examples come from the various IPv6 Profiles
around the world.  There are some number of these in Japan,
in Europe, in the USA, and in other countries.

Various examples also exist outside the IPv6 Profile universe,
including but not limited to large customers (e.g. the JITC and TIC).

>      BGP, MPLS-VPNs, HTTP, SSL, and Netflow, just to name a few. 

I provided non-defense examples in both my original note
(which examples you have ignored for some reason) and also
in my response above.

> actually performing that work.

I was quite detailed on that front, although you seem to have
selectively ignored that part of my note.

> There's no need to be rude or snarky with me, even if you disagree.  

I wasn't rude, and can't find "snarky" in the OED.

> You are looking at this from the angle of the customers, and that's
> view, and from the supply side of your equation. 

I've been both customer/user/operator and vendor/implementer
at various points in time.  So I look at it from both points
of view, and my earlier note included discussion of both
vendor advantages and user/operator/customer advantages.

It seems quite odd that you seem to have ignored my note 
so selectively.

> other?  

Yes, we do.

> If so, are they the norm, and are they likely to drive
> implementation?  

Such decisions in various IPv6 Profiles around the world,
in large customer requirements documents around the world
(e.g. JITC, TIC) regularly have driven implementation priorities 
and new feature timetables in the past.  

Folks at many vendors have experienced this.  I witnessed
it at every vendor I've ever worked for.  It isn't a surprise 
that a business case would drive these things NOR is it 
a surprise that standards status would drive an RFP 
(and hence drive the business case).

> Also, if all this gets you is interoperability, but

We disagree about SNMPv3 as an example.

By its very definition, "interoperability" is always solving 
a problem.  In my experience, having been both implementer 
and consumer/operator/user, it solves a problem both for
implementers/vendors and also for users/operators/customers.

> whether a new idea will overtake what has just been standardized.  

As time goes to infinity, the entire Internet is likely to be
replaced.  So that's the wrong question (again).  

The right question is about speed that a new technology will 
supercede an older one.  The IETF has extensive experience 
that it takes a substantial time for nearly anything to move 
to even Proposed Standard.  

While we believe that the time required between innovation and 
a suitable Proposed Standard will diminish with the new 2-track 
scheme [A], the long-standing requirements for openness and 
due process in the IETF standards process combine with more 
recent practices (e.g. Area reviews) to ensure that a technology 
won't be replaced instantly.

> I'll grant you it's almost impossible to calculate,
> but perhaps history can be of some use.  That analysis should be done.

To be frank, those sentences appear to just be trying to stall 
and divert discussion.

> Where do we see cases where gratuitous changes take place at Proposed?

Those happen pretty frequently.  You yourself have complained
about several changes, from time to time, in the past.

> Why do you come to this conclusion, given that PS isn't changing?

[A]

The public statement by the IETF Chair to the ietf discussion list,
here (1st paragraph from Russ):
	<http://www.ietf.org/mail-archive/web/ietf/current/msg62019.html>

and also text within draft-housley-two-maturity-levels.

> for people to do the work.

Its the wrong question because it ignores the presence
of users/customers/operators.

> done that wasn't done before.  Those are facts not in evidence.

Actually they are quite evident to folks within the product
development/feature prioritisation functions at most equipment
vendors.  I've outlined them before, but I'll try repeating
a currently common sequence of events here:

	- Technology shows interoperability and moves beyond 
	  Proposed Standard RFC.
	- RFC beyond Proposed Standard increases user/customer/operator
	  confidence that a technology is interoperable, 
	  stable, and does not lock them into a single vendor.
	- Operator/user/customer becomes MUCH more likely 
	  to include that technology as a "MUST implement" 
	  in their RFPs or Tender opportunities.
	- RFP requirements increase the business case 
	  for vendors to implement that particular feature
	- Business case pressures drive feature prioritisation
	  and implementation resourcing at vendors
	- Result is that technologies specified in RFCs
	  beyond Proposed Standard have more value to both
	  vendors and users/operators/customers, and so are
	  both more likely to be implemented and also more
	  likely to be widely deployed.

> I don't understand what you're saying.  
> Are you arguing that recycling back down to proposed will be easier?

It relates to the reference [A] above.

The current challenges of publishing any innovation as
a Proposed Standard RFC should diminish with Russ's proposal.

Yours,

Ran

---

From: Eliot Lear <[email-masked]>
Date: Wed, 23 Jun 2010 20:42:45 +0200

On 6/23/10 3:31 PM, RJ Atkinson wrote:
> as using a single example.  

I did not do this, and you have ignored the language that I used, and
ascribed motives to me in the process of doing so, the very thing you
accuse me of.

Eliot