Subject: Re: No single problem... (was Re: what is the problem bis)

From: Ted Hardie <[email-masked]>
Date: Mon, 1 Nov 2010 11:29:32 -0700

On Sat, Oct 30, 2010 at 1:17 AM, John C Klensin <[email-masked]> wrote:
<snip>
> However, a change to the handling of documents that are
> candidates for Proposed Standard is ultimately in the hands of
> the IESG.  In principle, they could announce tomorrow that any
> document submitted for processing after IETF 79 would be
> evaluated against the criteria in 2026 and no others other than
> reasonable document clarity.  If the IESG has the will --and
> whatever community backing is needed-- to do that, then the
> "two-step" document is not needed.  If the IESG does not have
> that will and backing, then community of the "two-step" document
> would merely leave us --as far as this issue/problem is
> concerned-- with one more set of rules we aren't following.
>

John,

Thanks for your thoughts on this.  While I agree that the IESG could
theoretically make the change you describe above, I don't think
they can practically do so.  Along with the increasing IESG review,
there has been an increasing sense that Proposed Standards
may be winnowed but likely will not be changed in core mechanism
unless there is significant breakage.  Given the weight of the "Installed base"
argument, however wrong it may be, I doubt the IESG would
be comfortable reverting to 2026 without some label changes
either in Proposed or the follow-on levels.  They need something
beyond an IESG note to hang their collective hats on.  A lot of the
education on the point "RFC != IETF Standards track document"
may actual tie our hands here, because its the first level at which
we've told folks in external SDOs that they can count on.

The current functional case is, in my opinion:

Subsets of the drafts adopted by working group (those that are
functionally complete, bascially)
are the new Proposed Standards

Proposed Standards are now close to what Draft once was, with
significant comfort
that this is what will be there for a long time.

Draft is used only when Proposed Standards include something that needs cutting
or an external driver requires advancement

Standard is used only when an external driver requires advancement.

An issue (again, not the *the* issue) is that only those deeply involved in
a WG can tell which drafts have gone into the subset.  That may hinder
cross-area review (since potential reviewers can't tell which ones are
important enough to do).

When I re-write the advance mechanics draft, I will propose something
along the following lines:

1) A WG snapshot-like status achieved after agreement by the working group
and a posting by the WG chair to IETF-announce notifying the wider community
and inviting review (presumably by review teams).  Any document may
reference this level for any level of maturity; it is not just
functionally archival, but a
recognized state.

2) A status called "IETF Standard" that is reached after the current (real)
procedures for Proposed.

3) A status called "Internet Standard" reached when an "IETF Standard" has
spent at least 3 years as an "IETF Standard" without any errata, objections,
or the creation of a -bis or -ter effort.  A new document may be
issued to correct
errata without requiring re-spinning in "IETF Standard" grade.

This is either a 3-stage document model or a 1-stage, depending on
whether you are counting labels or trips through the IESG.

This does not solve all the problems I put forward; it does not magically
breathe energy into any working group, for example, nor does it
handle the pulse-of-activity timing.  But it does solve the marketing
problem and recognize the role of the subset of agreed I-Ds in the real
world.  For one document, that's probably the best we can do.

For what it is worth, I don't think this is very different from what is Russ's
document, other than my being more willing to fiddle with status
names and him being less willing to do so.  Possibly this is because
I really do read his statement of restoring "Proposed" to the core
2026 values at face value.  Were that in place, we could realistically
expect WGs to throw documents over the IESG wall periodically
as "snapshots" and get much the same place.  I personally suspect,
however, that we need *some* rejiggering of the labels  at this point,
just to highlight the rules in place at the time something was approved.

regards,

Ted

PS.  On re-labeling things en-masse to make them fit the new
scheme, I don't really want to, because I can't see how to make
it work without a massive effort by someone.  Changing the
names in actual use may help that, as an SDO can say
"Proposed Standard or IETF Standard" for its reference
guidelines.

> So, if we are serious about changing (or reverting) those
> criteria, then let the IESG issue a statement about the new
> rules, schedule, and any transition plan.  Let's see if such a
> statement is successfully appealed by someone (I'd hope, given
> the concerns on this list about the problem, that wouldn't
> happen).  And then let's see if we can actually do it.
>
> There is lots of time to change the written procedures if such
> an effort works, even experimentally.  After all, we've been out
> of synch with 2026 for 14 years now and it hasn't shut us down.
>
> best,
>   john
>
> p.s. I also believe that, if part of the intent of the
> "two-step" document is to restore that bar, it is _very_ hard to
> deduce that from the document itself.  I'd feel better about the
> document if that were more clear.  But the document is really
> not the issue, the strategy is.  At least IMO.
>
>

---

From: Ted Hardie <[email-masked]>
Date: Mon, 1 Nov 2010 11:56:37 -0700

On Sat, Oct 30, 2010 at 8:16 AM, Hadriel Kaplan <[email-masked]> wrote:

> So is your expectation that if Russ's draft gets published, the bar for PS will suddenly drop?
>
> If so, why do we need Russ's draft to begin with?  We already have rfc2026.  Why would a new RFC which says "follow this other RFC" be needed??

I think it could, but that would require feedback from the community
on Russ's draft,
pushing that way of going forward.  As you'll see in my response to
John, I think there
are problems there and other ways forward.

>
> Later you say:
>> if the community
>> accepts that this restores the 2026 bar for the first rung *and holds the IESG
>> to it*
>
>
> How do we do that?  Why aren't we doing it now, since rfc2026 already exists?  In what way does Russ's draft make it any easier (or possible) to hold them to it?
>

Feedback to the IESG that the community wants this aspect of 2026
restored could take a
couple of forms:  messages (public or private) to that effect,
appeals, feedback to NomCom,
and recalls.  I'd start with the first, honestly, because this is
really an issue between
the IESG and the community.  Everyone is looking for a way forward
that meets the
community's needs.

Just my two cents,

regards,

Ted

> -hadriel
>
>
> On Oct 29, 2010, at 7:15 PM, Ted Hardie wrote:
>
>> As is moderately obvious from the stream of commentary on this
>> thread and there companions, there is no *one* problem at
>> the root of all this.  One way to draw this is:
>>
>> Issue:  Documents are too slow in achieving the first rung of the
>> standards process
>>
>> Contributing issues:
>>
>> --------->WG formation is slow, as there are now often 2 BoFs before work begins
>> --------->Working group activity is slow, as it pulses to physical meetings
>> --------->Late surprises arrive from late cross-area review (often
>> from teams) or the IESG
>> --------------->Because there is little early cross-area review after a BoF
>>
>> Resulting issues:
>>
>> ------->Little energy remains in working groups to advance documents
>> once they do complete
>> -------->The IESG sees that few documents get early re-review as part
>> of advancement, and
>>             raises the document quality requirement for the first
>> rung to prevent impact on the
>>             rest of the ecosystem
>>
>> Results of the results:
>>
>> ------->Things get slower
>> ------->More work is done outside the IETF and brought in only to be blessed
>> ------->More of the internet-runs on Internet Drafts.
>>
>> Results of the results of the results:
>>
>> --------->It's harder to tell which documents are actually the ones
>> you need, both because
>>             some actual Standards documents are obsoleted by drafts
>> and because some sets
>>             of drafts have functional consensus and some don't.
>>
>> Results of the results of the results:
>>
>> ---------->ADs and others want more tutorial data added to the RFCs,
>> which makes
>>           producing them slower.
>>
>>
>> Try to find one place to tug on this and the actual results of your
>> tugging won't
>> really be seen until a full document cycle, and there will be odd
>> states in between.
>> That causes debate and discussion, and worry with all the nice people
>> we've tasked
>> to worry about these things (and many others besides).  That burns energy that
>> could be going into working groups and, well, you get the picture
>> (things get slower).
>>
>> Should we do nothing?  No.  But we need to accept that no single thing we do
>> is going to solve all of the problems.  Changing the document labeling
>> will not increase
>> early cross-area review.  But if the top-line issue is "Documents achieving the
>> first rung of the standards track do so too slowly" we may have to tackle
>> it, the WG creation problem, and the WG "pulsed activity" problem at once
>> to make real progress.
>>
>> If the problem we want to tackle is "The first rung is set too high", then there
>> are other possibilities (including simply recognizing that the first rung is
>> really WG draft, marked or unmarked as it may be).   I personally don't
>> think that's quite enough, as the value of the IETF (as opposed to its
>> working groups) is that it can and does cross-WG and area review.  But I
>> see the attraction--if the first rung goes lower, the documents may be produced
>> faster, which can mean that there is enough energy to go up the track plus
>> the cross-area review is functionally earlier. My worry (yes, I worry)
>> is that if we
>> re-use a label for the first rung after lowering its bar, we create a
>> confusion that we
>> can't easily solve *especially if the energy does not magically appear*.
>>
>> As we stare down this rathole one more time, let's at least be certain
>> that there is more than one rat down there, and be realistic about the
>> energy we have on how many we can tackle.  Russ's draft tries to
>> do two things:
>>
>> Restore the 2026 rules for Proposed as the functionally in-use bar for the
>> first rung.
>>
>> Reduce the bar for Standard to the old bar for Draft.
>>
>> Listening to the discussion, I think we have focused a great
>> deal on point two, but have either not really noticed point
>> one or didn't believe it.    I think this addresses a marketing problem
>> (long an issue, though now commonly explained away) and it
>> focuses on the first two "resulting issues" in the quasi-chart above, and
>> thus may have some cascade effects on the other two.  It doesn't
>> tackle any of the contributing issues, but this is not really a defect
>> in my eyes, as those can't really be addressed by document issues.
>>
>> Are there other ways to tackle this?  Sure.  But if the community
>> accepts that this restores the 2026 bar for the first rung *and holds the IESG
>> to it*, then I think this is one useful place to tug on the tangle of issues.
>>
>> Just my two cents,
>>
>> regards,
>>
>> Ted
>> _______________________________________________
>> Ietf mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/ietf
>
>

---

From: John Leslie <[email-masked]>
Date: Mon, 1 Nov 2010 15:17:30 -0400

Ted Hardie <[email-masked]> wrote:
> 1) A WG snapshot-like status achieved after agreement by the working
>    group and a posting by the WG chair to IETF-announce notifying the
>    wider community and inviting review (presumably by review teams).
>    Any document may reference this level for any level of maturity;
>    it is not just functionally archival, but a recognized state.
> 
> 2) A status called "IETF Standard" that is reached after the current
>    (real) procedures for Proposed.
> 
> 3) A status called "Internet Standard" reached when an "IETF Standard"
>    has spent at least 3 years as an "IETF Standard" without any errata,
>    objections, or the creation of a -bis or -ter effort. A new document
>    may be issued to correct errata without requiring re-spinning in
>    "IETF Standard" grade.
> 

   I think this deserves some discussion. One trip through the IESG has
definite merit...

   In practice, I-Ds have become archival already. I think a WG deserves
to be empowerd to label one or more I-Ds as universally citable.

   Formalized, hopefully earlier, review seems like a good thing.

   WG Snapshot seems like an OK name...

   One trip through the IESG to get Proposed Standard status has been
honed to about as efficient a process as we can hope for. The glitches,
IMHO, are heavily correlated with deficiencies in cross-area review.

   I see little reason, though, to change the name from "Proposed Standard."
And, IMHO, the RFC 2026 specifications for Proposed Standard are pretty
good. I'd ask for pretty strong justification for changing either the
name or the RFC 2026 definition.

   As to what follows Ted's Step Two, I think that needs work; but the
idea of formalizing something which doesn't require another trip through
the IESG sounds promising.

--
John Leslie <[email-masked]>

---

From: Bill McQuillan <[email-masked]>
Date: Tue, 2 Nov 2010 01:04:29 -0700

>>    it is not just functionally archival, but a recognized state.

<......>

>    WG Snapshot seems like an OK name...

How about "First Review" as a status? It says what it is without using the
word "Standard" and implies the tentative nature of its maturity.

-- 
Bill McQuillan <[email-masked]>

---

From: Ross Callon <[email-masked]>
Date: Thu, 4 Nov 2010 05:50:14 -0400

Commenting on one issue from John's email from Sat 10/30/2010 4:18am
(and ignoring the issue of what John was doing up at 4am):
 
> "two-step" document is not needed. ...

I don't quite agree with this. I think that if the IESG wanted to step 
back to a "closer to the wording of 2026" process for proposed standard
documents, then we *do* need to also move to a two step process (rather 
than a three step process). The reason is that moving from proposed 
standard to draft standard is a step that isn't worth the trouble. This 
means that most RFCs can't ever get to full standard. If we want the 
first step to really *only be the first step*, we need a second step 
that will actually be worth enough that someone will take the effort 
to follow it for the vast majority of useful standards. 

I won't claim that the two-step document is actually going to cause the
IESG to make the first step easier. However, the IESG has noticed the 
message from the community that we don't want many silly discuss votes 
to drag out document approval unnecessarily, and has done some serious 
navel gazing on the subject. To me it appears that there has been some 
improvement in this area over the past five or six years (and I can 
recall some rather frustrating examples five or six years ago that I 
would rather not dissect in public -- or anywhere else without a beer 
in front of me). 

Ross

---

From: John C Klensin <[email-masked]>
Date: Thu, 04 Nov 2010 20:58:04 -0400

--On Thursday, 04 November, 2010 05:50 -0400 Ross Callon
<[email-masked]> wrote:

> Commenting on one issue from John's email from Sat 10/30/2010
> 4:18am (and ignoring the issue of what John was doing up at
> 4am):  

:-)

>> evaluated against the criteria in 2026 and no others other
>> than reasonable document clarity.  If the IESG has the will
>> --and whatever community backing is needed-- to do that, then
>> the "two-step" document is not needed. ...
> 
> I don't quite agree with this. I think that if the IESG wanted
> to step  back to a "closer to the wording of 2026" process for
> proposed standard documents, then we *do* need to also move to
> a two step process (rather  than a three step process). The
> reason is that moving from proposed  standard to draft
> standard is a step that isn't worth the trouble. This  means
> that most RFCs can't ever get to full standard. If we want the 
> first step to really *only be the first step*, we need a
> second step  that will actually be worth enough that someone
> will take the effort  to follow it for the vast majority of
> useful standards. 
> 
> I won't claim that the two-step document is actually going to
> cause the IESG to make the first step easier. However, the
> IESG has noticed the  message from the community that we don't
> want many silly discuss votes  to drag out document approval
> unnecessarily, and has done some serious  navel gazing on the
> subject. 

Ross,

We (the community) are having a serious communication problem
here -- different clusters of us are just not communicating.

In the hope of explaining at least part of the problem, let me
respond to what you said rather than what I'm quite sure you
meant.

I don't see proceeding by small, incremental changes to be a
problem. Indeed, I usually consider it an advantage as long as
there is reasonable confidence that the changes that are made
won't foreclose real solutions later.  That risk can never be
completely eliminated, at least without a comprehensive
long-term plan, but I'm asking only for reasonable confidence.
Such confidence would arise, for example, from a clear statement
of a particular, appropriately-narrow, problem and a logical
explanation as to why a proposed solution will focus narrowly on
it and fix it.   Again, that explanation doesn't need to be
proof to mathematical certainty, just something that is able to
be examined logically and that seems to have a rational
cause-and-effect relationship to the problem it is intended to
solve.

I also don't think there is anyone in the community who believes
there is no problem with the way maturity levels are now handled
and used, at least no one who has been awake during the last
many years.

But what we have been given isn't the kind of "stated problem
and plausible solution to it" analysis suggested above.  What we
have instead been given is fairly close to a statement that the
IESG gazed upon its collective navel and, in the depths of said
navel, found a revelation that eliminating full standard and
renaming Draft would cause a miracle to happen, where that
miracle is that various fairly-unspecified problems will be
solved.

If the critical-path problem is that it is too hard to get to
Proposed, then let's address that.

The problem isn't that nothing moves to Draft or Full Standard
under the current rules because some things do.  Perhaps there
is an issue with the characteristics of those that do or don't
and perhaps understanding that would be a useful exercise.  From
my own observations, it differs somewhat by topic area (which
may or may not align with IETF areas).  If that is the case,
shouldn't we be looking both at the differences and at whether
some area-specific rules would be in order?  (Note, fwiw, that
draft-klensin-isdbis explores just that option, rather than
taking the more global approach of its predecessor.)

If we are really going to argue that the number of documents
that move to Draft is so miniscule that whether or not they
advance to Full is irrelevant, then let's either address why
there are so few advancements to Draft or stop talking about
maturity levels entirely, moving either to one-step or to a
different model.

Rephrased into your language, "we need a second step  that will
actually be worth enough that someone will take the effort  to
follow it for the vast majority of useful standards."  I think
that assumes that changing the name of "Draft" to "Full" will
provide that value.  I see no evidence of that whatsoever
--other than wishful thinking-- largely because I think that the
main reason things don't advance to Draft today is that there is
no real incentive to reopen documents after Proposed, especially
if the protocols are already deployed and the WG (or whatever)
process was exhausted of all energy in getting _to_ Proposed.
No evidence has been offered that eliminating Full Standard will
change that.

But it has seemed to me that what the current two-step draft
says is "well, too few documents move to Draft, so let's
increase the incentives by eliminating Full"... with absolutely
no evidence that eliminating Full would change the incentives to
move to Draft, or the number of documents that are advanced, by
one iota.

So, let's stop the navel-gazing and move to a realistic and
focused analysis and explanation of what problem we are trying
to solve and why any particular proposed measure will solve it.  

best,
    john

---

From: Ross Callon <[email-masked]>
Date: Thu, 4 Nov 2010 23:52:49 -0400

> won't foreclose real solutions later...

This is my understanding of what is proposed. 

>  ...That risk can never be
> it and fix it.

The document currently states: 

   These changes are designed to
   simplify the IETF Standards Process and reduce impediments to
   standards progression while preserving the benefits of the IETF
   engineering approach.

Would you be happier adding some sort of statement along the lines of:

   The changes proposed in this document are focused on reducing the
   number of steps in the progression of standards track documents, 
   and to the reduction of limitations based on downrefs. This 
   document does not consider other changes, and is not intended to 
   discourage nor to prevent any additional process changes which 
   may be proposed and progressed independently. 

I think that something along this lines might help to clarify the 
intent of the document (assuming that I have correctly understood
the intent). 

>   ...Again, that explanation doesn't need to be

Mathematical certainty is not going to happen in this area. 

> and used, at least no one who has been awake during the last

I am concerned that there is a clear problem with the three step
process that might not get fixed because people are arguing
issues which are large orthogonal to whether we would be better 
off with two steps or with three steps. 

> miracle is that various fairly-unspecified problems will be

I haven't noticed any statement to the effect that a miracle is 
going to happen, nor that all unspecified problems will be solved
by this particular document. Nor have I noticed any statement 
precluding additional orthogonal changes being proposed by others. 

> Proposed, then let's address that.

I think that moving to a two-step process, rather than a three-
step process is *necessary* to simplify the effort to make it 
easier to get to proposed. I don't think that it is *sufficient.* 
If people want to propose other changes to make it easier to get
to proposed then please write up the other changes in another 
document.  

> taking the more global approach of its predecessor.)

Feel free to start this exercise. I suspect that you are probably 
correct that there is some variation by area (for some definition 
of "area").

> No evidence has been offered that eliminating Full Standard will

I agree that if we move to a two step process the incentive to take 
the second step will still be small. I don't think that there is 
any way to accurately predict whether this small incentive will be 
sufficient to cause the second step to actually happen, unless we 
try the process change and see what happens. It is pretty clear 
that with a three step process the incentive to take the second 
step is insufficient. 

> ...

I think that you are expecting Russ' document to solve more problems 
than it is intended to solve. It solves two problems: The three step 
process is obviously at least one step too many (thus two is better 
than three, which does not address the issue of whether one step is 
better or worse than two steps). The downref rules discourage document 
advancement. 

That is all that Russ' document solves. If you want to solve other 
problems, you are free to propose fixes, but don't expect this one 
modest document to solve a wider range of problems. 

I can see that if we were to agree and put two-step into place, then 
it would likely take at least a couple of years of trying it out 
before we are likely to be willing to consider going from a two-step 
process to a one-step process. Personally I haven't seen any consensus
that we should go directly to one-step. 

Other changes to the process can be considered, should be considered
independently and we should not expect Russ's document to either 
propose nor to preclude any other changes that people want to consider. 

It makes sense to me to explicitly state that this document does not 
preclude other orthogonal changes to the process.

Ross