Subject: Re: draft-housley-two-maturity-levels-00

From: "Bernard Aboba" <[email-masked]>
Date: Mon, 21 Jun 2010 11:09:14 -0700

Russ,

I'd also like to think you for revisiting this topic. 

I support the recommendation to eliminate the "Standard" maturity level, and
also agree with your recommendation on Maturity Level 2 (similar to Draft
Standard). 

We need more thought on what to do with the other levels though. 

In practice, we often see a document initial go to Proposed Standard, then
go through a "bis" to enable clarifications and interop improvements. 

Often these changes are too substantial to enable advancement to Draft, but
they nevertheless represent an important advancement in status.   

I'd like to see some way that this advancement can be recognized formally.  

Also, in some areas (e.g. Transport) the first stage is publication of an
Experimental RFC.  These documents are published with the understanding that
implementation experience will be incorporated into a future revision. 

So perhaps the hierarchy should be:

a.       Experimental.  

b.      Proposed Standard (e.g. a "bis").

c.       Interoperable Standard/Draft Standard.

---

From: Yoav Nir <[email-masked]>
Date: Tue, 22 Jun 2010 11:00:03 +0300

I like this proposal, but there should be a (relatively) easy process to advance from Experimental to Proposed, especially if implementation experience shows no need for bits-on-the-wire changes.

We should be able to say that for a particular experimental RFC there have been this many independent implementation, and they interoperate OK, and only so-and-so clarifications need to be added, and the document is ready for "Proposed".

On Jun 21, 2010, at 9:09 PM, Bernard Aboba wrote:

Russ,

I’d also like to think you for revisiting this topic.

I support the recommendation to eliminate the “Standard” maturity level, and also agree with your recommendation on Maturity Level 2 (similar to Draft Standard).

In practice, we often see a document initial go to Proposed Standard, then go through a “bis” to enable clarifications and interop improvements.

Often these changes are too substantial to enable advancement to Draft, but they nevertheless represent an important advancement in status.

I’d like to see some way that this advancement can be recognized formally.

Also, in some areas (e.g. Transport) the first stage is publication of an Experimental RFC.  These documents are published with the understanding that implementation experience will be incorporated into a future revision.

a.       Experimental.
b.      Proposed Standard (e.g. a “bis”).
c.       Interoperable Standard/Draft Standard.
<ATT00001..txt>

---

From: "Bernard Aboba" <[email-masked]>
Date: Tue, 22 Jun 2010 08:32:08 -0700

Overall, I'd suggest that the goal should be to merely recognize and
document the maturity levels that already exist in practice, not to change
them.  

My understanding is that the  process for advancing from Experimental to
Proposed today largely involves review of implementation experience (e.g.
the results of the "experiment"), and in Transport, a demonstration that the
proposal is not catastrophic to the Internet.  Sometimes the changes
required can be substantial (e.g. changes made in EAP going from RFC 2284 to
3748, and in SIP going from 2543 to 3261),  but I don't think this should
hinder advancement to Proposed.   Problems in interoperability are often
addressed in "bis" documents, so we shouldn't require a detailed interop
assessment either (that's for Draft level).   I can imagine blocking
advancement of a "bis" to Proposed in only a limited number of situations,
such as where there was no implementation experience.  Today "recycling at
Experimental" is pretty rare -- if there is motivation for a "bis" typically
this implies that there was interest/usefulness.   

From: Yoav Nir [mailto:[email-masked]] 
Sent: Tuesday, June 22, 2010 1:00 AM
To: Bernard Aboba
Cc: [email-masked]
Subject: Re: draft-housley-two-maturity-levels-00

I like this proposal, but there should be a (relatively) easy process to
advance from Experimental to Proposed, especially if implementation
experience shows no need for bits-on-the-wire changes.

We should be able to say that for a particular experimental RFC there have
been this many independent implementation, and they interoperate OK, and
only so-and-so clarifications need to be added, and the document is ready
for "Proposed".

Russ,

a.       Experimental. 

<ATT00001..txt>

---

From: Russ Housley <[email-masked]>
Date: Wed, 23 Jun 2010 00:58:15 -0400

Bernard:

> In practice, we often see a document initial go to Proposed Standard,
> then go through a “bis” to enable clarifications and interop improvements.
> 
> Often these changes are too substantial to enable advancement to Draft,
> but they nevertheless represent an important advancement in status.   
> 

I do not see how the document we are discussing encourages or
discourages recycling at Proposed Standard.  This is common with "bis"
documents today.  If there is an interoperability report, this can
happen at the proposed Interoperability Standard maturity level too,
which is not possible under the current set of rules.

Russ

---

From: Jari Arkko <[email-masked]>
Date: Wed, 23 Jun 2010 16:30:51 +0300

Yoav,
> I like this proposal, but there should be a (relatively) easy process 
> to advance from Experimental to Proposed, especially if implementation 
> We should be able to say that for a particular experimental RFC there 
> have been this many independent implementation, and they interoperate 
> OK, and only so-and-so clarifications need to be added, and the 
> document is ready for "Proposed".

I think we already have that. There is really no requirement to produce 
a new draft, for instance. You can reclassify an RFC to a different 
status (and we do it sometimes). Additional knowledge outside the 
document about implementations, market acceptance, and lack of problems 
would easily convince me at least that the document is worthy of PS status.

(Of course, you might end up making an edit in the document anyway if it 
talked about experiments, and it is very likely most documents would 
have at least some details to correct.)

Jari

---

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 23 Jun 2010 09:10:45 -0500

Hi, Jari,

>> have been this many independent implementation, and they interoperate OK, 
>> and only so-and-so clarifications need to be added, and the document is 
>> ready for "Proposed".
>
> I think we already have that. There is really no requirement to produce a 
> new draft, for instance. You can reclassify an RFC to a different status 
> (and we do it sometimes). Additional knowledge outside the document about 
> implementations, market acceptance, and lack of problems would easily 
> convince me at least that the document is worthy of PS status.

I think a key question in choosing between a multi-level standards track and 
a single-level standards track is what the first publication step looks 
like, so I've been interested in recent proposals to use Experimental as a 
first publication step. Having said that ...

As I read http://www.rfc-editor.org/rfc/rfc5741.txt, Experimental RFCs would 
be Category: Experimental on the first page, and I'd expect them to be 
revised when they are reclassified, if only to make this say Category: 
Standards Track. So that's at least a small barrier to reclassification in 
place.

Thanks,

Spencer

---

From: Jari Arkko <[email-masked]>
Date: Wed, 23 Jun 2010 17:17:59 +0300

Spencer,

> As I read http://www.rfc-editor.org/rfc/rfc5741.txt, Experimental RFCs 
> would be Category: Experimental on the first page, and I'd expect them 
> to be revised when they are reclassified, if only to make this say 
> Category: Standards Track. So that's at least a small barrier to 
> reclassification in place.

Yes, though that is something which could be handled by setting the 
tracker intended status and perhaps an RFC Editor note correctly to PS. 
The last call notice would say the correct thing after this, for 
instance. Its really very similar to what we've been doing for some 
documents already. "RFC such and such for Draft" and at that time such 
and such is still a PS, and a new RFC # will be allocated for the DS RFC.

Jari

---

From: Russ Housley <[email-masked]>
Date: Wed, 23 Jun 2010 10:37:58 -0400

Dave:

This observation was based on many hallway discussions with many people
over many years.  You are correct to observe that there are many factors
at play, and many of them were discussed at the mic in plenary.  The
draft changes process in many different places.  I've attempted to
balance many differnt things in this proposal.  Only community
discussion will determine if the balance is acceptable.

There is always risk in process changes.  The goal is to make changes
where the benefits outweigh the risk.

Russ

On 6/23/2010 5:25 AM, Dave CROCKER wrote:
> 
> Russ,
> 
> In reading the latest version of your proposal, I finally realized that
> a motivating premise:
> 
>>     o Since many documents are published as Proposed Standard and never
>>       advances to a higher maturity level, the initial publication
>>       receives much more scrutiny that is call for by RFC 2026 [1].
> 
> is well-intentioned, sounds reasonable, but is actually without any
> practical foundation.  In other words, we do not know that the premise
> is valid.
> 
> There are many likely reasons that the process of getting through the
> IESG, for Proposed, has become such a high burden.  (Perhaps you did not
> mean to limit the reference to mean only IESG scrutiny, but in formal
> terms, it really is the only one that matters, in terms of 'scrutiny'.)
> 
> I think that a detached and thorough exploration of those possible
> reasons is likely to show that the problem pre-dates the move towards
> leaving documents at Proposed and, in fact, well might have contributed
> to it.  By virtue of having Proposed be so difficult to attain, there is
> a disincentive for going through the process again, for the same protocol.
> 
> A tendency for all of us in this kind of topic is to make simple
> assertions of cause or of likely behavioral change that sound reasonable
> but have little empirical basis.
> 
> We need to be careful to avoid falling into that trap, because the
> changes being discussed are strategic, and could well be impossible to
> reverse completely...
> 
> d/
>

---

From: Dave CROCKER <[email-masked]>
Date: Wed, 23 Jun 2010 11:25:07 +0200

Russ,

In reading the latest version of your proposal, I finally realized that a 

>       receives much more scrutiny that is call for by RFC 2026 [1].

is well-intentioned, sounds reasonable, but is actually without any practical 
foundation.  In other words, we do not know that the premise is valid.

There are many likely reasons that the process of getting through the IESG, for 
Proposed, has become such a high burden.  (Perhaps you did not mean to limit the 
reference to mean only IESG scrutiny, but in formal terms, it really is the only 
one that matters, in terms of 'scrutiny'.)

I think that a detached and thorough exploration of those possible reasons is 
likely to show that the problem pre-dates the move towards leaving documents at 
Proposed and, in fact, well might have contributed to it.  By virtue of having 
Proposed be so difficult to attain, there is a disincentive for going through 
the process again, for the same protocol.

A tendency for all of us in this kind of topic is to make simple assertions of 
cause or of likely behavioral change that sound reasonable but have little 
empirical basis.

We need to be careful to avoid falling into that trap, because the changes being 
discussed are strategic, and could well be impossible to reverse completely...

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Russ Housley <[email-masked]>
Date: Thu, 24 Jun 2010 15:10:17 -0400

I strongly disagree with this characterization.  In my view, too many
things got bundled together, and the thing that was unacceptable too the
whole bundle down.

Russ

On 6/24/2010 2:52 PM, Phillip Hallam-Baker wrote:
> Last time the reforms were blocked without the IETF at large even
> knowing who was responsible. It was a decision the IESG took in private
> as if it only affected them and they were the only people who should
> have a say. So much for bottom up organization.

---

From: Russ Housley <[email-masked]>
Date: Fri, 25 Jun 2010 10:49:43 -0400

Phillip:

Obviously, I was not General AD when this happened.  However, I was
Security AD at the time, so I was involved in the discussions that
included the whole IESG.

I made my reply to your posting because I want people to realize that
there is another side to the story.  We need to learn from the history,
but we need to act toward improving the future.

The IESG spent a huge amount of time on the NEWTRK documents in retreat.
 The ISD proposal hit the IESG in a very bad way.  The ISD proposal
required the IESG spend a lot of time that the individuals simply did
not have.  Further, this came at a very, very bad time.  Admin-Rest had
consumed way to many cycles.  Perhaps the 1-step or 2-step proposals
could have been separated from ISDs, but that was not the path that was
taken.  I do not know the reasons.

Russ

On 6/24/2010 6:11 PM, Phillip Hallam-Baker wrote:
> My point is that I am unable to have any characterization whatsoever
> since nobody has ever told me the reason that the changes did not go ahead. 
> 
> And since I have asked for reasons in a plenary and never got any
> statement that was not phrased in the passive voice, I don't think it is
> unfair to describe the decision as having been made in private.
> 
> If the history is not confidential then I want to know what it was.
> Otherwise I don't see why it is inaccurate to describe the process as
> top down.
> 
> If the process is going to be described as consensus based and bottom up
> then at a minimum the people who take the decision have to be prepared
> to state their reasons.
> 
> 
> 
> On Thu, Jun 24, 2010 at 3:10 PM, Russ Housley <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
> Website: http://hallambaker.com/
>

---

From: "Spencer Dawkins" <[email-masked]>
Date: Fri, 25 Jun 2010 10:31:53 -0500

I've mentioned this to Russ privately, but it's worth saying it out loud ...

> taken.  I do not know the reasons.

IESG discussions are now a lot more visible to anyone from the community who 
cares to look, than they were during any part of NEWTRK. Beginning in late 
2005, the IESG has published narrative minutes from every official telechat 
(details at http://www.ietf.org/iesg/minutes/2010/ for this year).

When I served as IESG scribe, there were VERY few unminuted discussions, and 
they were identified as "executive session" in the minutes.

In addition, I THOUGHT Scott Bradner had requested publication of 
https://datatracker.ietf.org/doc/draft-ietf-newtrk-repurposing-isd/, and 
that was the straw that broke the camel's back, but I didn't see any IESG 
evaluation records for any of the NEWTRK documents, except for the DECRUFT 
draft 
(https://datatracker.ietf.org/doc/draft-ietf-newtrk-decruft-experiment/). 
Speaking only for myself, I didn't know much about what the IESG thought 
before the NEWTRK session at IETF 63 in Paris, even though I was note-taker 
for every NEWTRK meeting, and I was an active working group participant 
(perhaps too active :-).

I did not have the understanding about the IESG reaction to NEWTRK that Russ 
described in his note - and I don't think Russ is mistaken about that 
reaction.

We have significantly improved the transparency of the IESG to the community 
since NEWTRK was on the docket. I can't tell you that Russ's current 
proposal will be accepted, but I think I can tell you that members of the 
community will better understand what the IESG's concerns are, and members 
of the community will understand which ADs have those concerns, so we can 
work on resolving them.

This isn't a knock at anyone serving on the IESG during the NEWTRK era, all 
of whom except for Russ have moved on from their honorable service, and 
certainly not at Brian, who supported IESG narrative minutes as General Area 
Director when that was still controversial. It was just a different time.

We still need to improve, of course, but it's fair to recognize the progress 
we've made.

Thanks,

Spencer

---

From: [email-masked] (Scott O. Bradner)
Date: Fri, 25 Jun 2010 12:53:38 -0400 (EDT)

> The ISD proposal
> not have.

so the IESG insisted - that was not the opinion of the newtrk chair
(who thought that ISDs would likely reduce the load on ADs

> Further, this came at a very, very bad time

and that, apparently, kept (at least some) members of the IESG from seriously
considering what was being proposed

this was not the IESG's finest hour - lets leave it at that

Scott
(ex) newtrk chair

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 24 Jun 2010 14:52:33 -0400

It seems to me that we hit a vicious circle in which the lack of standards
progressing beyond proposed has meant that proposed standard has been
considered sufficiently advanced for adoption leading to more people
worrying about rough edges in proposed standards and demanding more complete
documents further erasing the practical distinction between proposed and
draft, thus further reducing the incentive to progress the drafts and so on.

The result is that the criteria for proposed today are more than sufficient
for a draft standard.

As many people have pointed out, the use Informational and Experimental
means that we will still have scope for a three step process in cases where
it is relevant.

I completely disagree with Martin Rex on this one. We have tried to make a
broken system work for twenty years. The issues Russ is raising now have
been raised before. The people who argued to give the system as written one
last chance got their way last time. THEY MUST NOT GET THEIR WAY AGAIN.

We have been there and done that. Trying to make the broken system work
didn't work last time. There is no reason to expect a different outcome if
we try the same thing the same way yet another time. That is the definition
of insanity according to someone famous.

Last time the reforms were blocked without the IETF at large even knowing
who was responsible. It was a decision the IESG took in private as if it
only affected them and they were the only people who should have a say. So
much for bottom up organization.

I think that we need to be assured that this time the ground rules for
debate will be different and that either the IESG is going to accept the
need for change or at the very least reveal who objected and what their
reasons were.

On Wed, Jun 23, 2010 at 10:37 AM, Russ Housley <[email-masked]> wrote:

> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf
>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 24 Jun 2010 18:11:53 -0400

My point is that I am unable to have any characterization whatsoever since
nobody has ever told me the reason that the changes did not go ahead.

And since I have asked for reasons in a plenary and never got any statement
that was not phrased in the passive voice, I don't think it is unfair to
describe the decision as having been made in private.

If the history is not confidential then I want to know what it was.
Otherwise I don't see why it is inaccurate to describe the process as top
down.

If the process is going to be described as consensus based and bottom up
then at a minimum the people who take the decision have to be prepared to
state their reasons.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 25 Jun 2010 13:02:13 -0400

Yes, I agree that the IETF has become a lot more open than the
self-perpetuating cabal that ran it during the mid 90s.

We no have de-facto term limits for ADs and it is no longer considered
acceptable for an AD to chair a working group (excepting process related
groups chaired by the IETF chair).

Gradually the IETF is shedding the constitutional innovations that were
meant to be better than democracy in favor of processes and procedures that
have been observed to work.

No, I do not expect a repeat of last time when the NEWTRK proposals were
killed in private and everyone sat in silence like a row of puddings when I
asked them to explain their reasons in the plenary.

At this point we do not have an assumption that the IESG operates by
collective responsibility makes decisions for the whole IETF in private and
then refuses to give reasons. That assumption is now dead, deceased, ...
snuffed it, that is an ex-assumption.

Just thought I would take the opportunity to point it out.

On Fri, Jun 25, 2010 at 11:31 AM, Spencer Dawkins <[email-masked]
> wrote:

> IESG discussions are now a lot more visible to anyone from the community
> who cares to look, than they were during any part of NEWTRK. Beginning in
> late 2005, the IESG has published narrative minutes from every official
> telechat (details at http://www.ietf.org/iesg/minutes/2010/ for this
> year).
>
> When I served as IESG scribe, there were VERY few unminuted discussions,
> and they were identified as "executive session" in the minutes.
>
> draft (
> https://datatracker.ietf.org/doc/draft-ietf-newtrk-decruft-experiment/).
> I did not have the understanding about the IESG reaction to NEWTRK that
> Russ described in his note - and I don't think Russ is mistaken about that
> reaction.
>
> We have significantly improved the transparency of the IESG to the
> community since NEWTRK was on the docket. I can't tell you that Russ's
> current proposal will be accepted, but I think I can tell you that members
> of the community will better understand what the IESG's concerns are, and
> members of the community will understand which ADs have those concerns, so
> we can work on resolving them.
>
> We still need to improve, of course, but it's fair to recognize the
> progress we've made.
>
> Thanks,
>
> Spencer
>

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 26 Jun 2010 19:24:48 +1200

> taken.  I do not know the reasons.

I was General AD at the time. There was certainly no meeting of the minds
between NEWTRK's rough consensus for the ISD proposal and the IESG's
understanding of what ISDs would mean in practice. Also there was no
sign of consensus in NEWTRK for moving to a 1-step or 2-step standards
process as a first step, rather than ISDs as the first step. So basically
we got collectively stuck. I tried setting up a special design team
to unstick us and that didn't work either.

Which is why, basically, I support the latest 2-step proposal, as a way
to unstick this discussion and move in the direction of simplicity.

    Brian

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sat, 26 Jun 2010 12:05:29 -0400

One approach that may help avoid that blockage is to use a Quaker poll.

[Yes, Yes, consensus, blah, blah]

At the moment the mode of discourse is that everyone proposes their
preferred solution. So the clear consensus that the three step process is
not being applied is lost because everyone is encouraged to propose
alternatives.

I prefer a two step process, but I can live with a one step process.

I do not see any value to a three step process and even if I did, we have
fifteen years experience of that scheme not having worked.

On the downrefs thing. I do not see a reason for a downref to block progress
on standards track provided that the specification being referenced is
stable (i.e. an RFC, not an Internet draft).

I still think the ISDs are a good idea. There is a big problem with
standards being fragmented across a series of documents without any document
that ties them together. And there are currently overview documents that
perform that role. But clearly the first priority has to be to fix the
broken three step process.