Subject: Review of: Characterization of Proposed Standards

From: Dave Crocker <[email-masked]>
Date: Thu, 24 Oct 2013 09:29:03 -0400

Review of:    Characterization of Proposed Standards
I-D:          draft-kolkman-proposed-standards-clarified-04

Reviewed by:  D. Crocker
Date:         24 October 2013

Summary:

  This draft offers revised normative text for the criteria used to
qualify IETF drafts as entry-level Proposed Standards status. The
draft's background discussion makes a number of assertions of fact
without substantiation, at least two of which are probably not correct.
The revised normative text includes two criteria that also are likely
inaccurate.

    The draft is not ready for approval.

Detail:

> Abstract
>
> RFC 2026 describes the review performed by the IESG on IETF Proposed
> Standard RFCs and states the maturity level of those documents.
> This document clarifies those descriptions and updates RFC 2026 by
> providing a new characterization of Proposed Standards.

"those" descriptions?  As the Introduction confirms, the draft covers
only one of them.

> 2. IETF Review of Proposed Standards
>
>
> The entry-level maturity for the standards track is "Proposed
> Standard".  A specific action by the IESG is required to move a
> specification onto the standards track at the "Proposed Standard"
> level. Initially it was assumed that most IETF technical
> specifications would progress through a series of maturity stages
> starting with Proposed Standard, then progressing to Draft Standard
> then, finally, to Internet Standard (see RFC 2026 section 6). Over
> time, for a number of reasons, this progression became less common.

It was /always/ uncommon.  There was never a time when progression
through the full sequence typically happened.

> In response, the IETF strengthened its review of Proposed Standards,
> basically

It was not 'in response'.  There was never a process of discussing the
issue and changing a policy formally and with clear IETF rough consensus.

The higher bar for Proposed happened organically and mostly through many
independent decisions of those running the standards process over the
years.

Even better, it was always controversial, when discussed in public.  Or
rather, the question of how high the bar should be was always
controversial and was never subject to an explicit rough consensus process.

> operating as if the Proposed Standard was the last chance for the
> IETF to ensure the quality of the technology and the clarity of the
> Standard Track document.  The result was that IETF Proposed
> Standards approved over the last decade or more have had extensive
> reviews.

Many have.  Many have not.  It has depended upon the diligence chosen by
everyone working on the draft or doing reviews.

I'm pressing this point because the language being used for this draft
is leaning towards mixing formal process measures to raise the /effort/
needed to get approved, with actual quality assurance /results/.  In
fact we often publish drafts that have actually have very few eyes doing
deep and thoughtful review and we often publish drafts that don't work
very well.  That is, the much higher formal bar still allows quite a bit
of cruft to jump over...

> Because of this change in review assumptions, IETF Proposed
> Standards should be considered to be at least as mature as final
> standards from other standards development organizations.

An assertion like this requires some discussion detail that compares the
IETF process to specific, other SDO processes, both formally and
effectively.  I'm not disagreeing with the conclusion, but with its
being made rather casually and firmly, but without substantiation.
(Part of the real challenge in doing the comparison is looking for
alternative forms of quality assurance that other SDOs might do, other
than the explicit reviews the IETF does.)

By the way, such a comparison discussion will probably serve to greatly
strengthen the ability to explain/argue for the comparative maturity of
Proposed...

> The IETF review is possibly more extensive than that done in most
> other SDOs owing to the cross-area technical review performed by the
> IETF, exemplified by technical review by the full IESG at the last
> stage of specification development.  That position is further
> strengthened by the common presence of interoperable running code and
> implementation before publication as a Proposed Standard.

There's actually a reasonable possibility that we do /less/ pre-Proposed
interoperable implementation now than we did 20 years ago.

Consequently an assertion that implementation is 'common' needs some
objective substantiation in comparison to the time we defined the
standards labels.

> 3. Characterization of Specification
>
>
> Section 3.1 of this document replaces RFC 2026 Section 4.1.1.
> Section 3.2 is a verbatim copy of the characterization of Internet
> Standards from RFC 2026 Section 4.1.3 and is provided for convenient
> reference.
>
> 3.1. Characterization of IETF Proposed Standard Specifications
>
>
> The entry-level maturity for the standards track is "Proposed
> Standard".  A specific action by the IESG is required to move a
> specification onto the standards track at the "Proposed Standard"
> level.
>
> A Proposed Standard specification is stable, has resolved known
> design choices, is well-understood, has received significant
> community review, and appears to enjoy enough community interest to
> be considered valuable.

The text has two major flaws that serve to assert a much higher
practical maturity for the Proposed Standard documents we publish than
is very often observably true:

Well understood:

    For a specification with any interesting level of innovation or
complexity, it is almost never 'well understood' at the time it is
approved by the IETF.  That requires significant implementation,
deployment and use experience.

    The fact that many eyes might have done a static review or even that
a couple of folk have written some code or even that those folk
interoperated does not come close to qualifying the specification as
"well understood".

Enough community interest:

    We often publish specifications that actually have very little
community support.  This is evidenced by a) very low working group
participation levels, and b) failure to successfully deploy and get used.

>
> Usually, neither implementation nor operational experience is
> required for the designation of a specification as a Proposed
> Standard.  However, such experience is highly desirable, and will
> usually represent a strong argument in favor of a Proposed Standard
> designation.
>
> The IESG may require implementation and/or operational experience
> prior to granting Proposed Standard status to a specification that
> materially affects the core Internet protocols or that specifies
> behavior that may have significant operational impact on the
> Internet.
>
> A Proposed Standard will have no known technical omissions with
> respect to the requirements placed upon it.  Proposed Standards are
> of such quality that implementations can be deployed in the
> Internet.

And yet we know that this often is not true.  However, language like
this is going to cause the IESG to naturally raise the bar even higher,
for fear that a spec still is not good enough.

The real goal of the language change, here, is to give assurances that
the specification is "sufficiently" mature, in terms of concern for
market deployment.

The part that the document can't say, but has as an implicit point since
it is motivating the draft, is that the maturity is at least as good as
that produced by other SDOs.

Perhaps:

     Proposed Standards are of such quality that they are ready for the
usual market-based product development and deployment efforts into the
Internet.

> However, as with all technical specifications, Proposed Standards
> may

Remove "However".

> be revised if problems are found or better solutions are identified,
> when experiences with deploying implementations of such technologies
> at scale is gathered.

> 4. Further Considerations
>
>
> While less mature specifications will usually be published as
> Informational or Experimental RFCs, the IETF may, on occasion,

On occasion?  In fact it's quite common and possibly the norm.  Let's
not pretend otherwise.

> publish a specification that still contains areas for improvement or
> certain uncertainties about whether the best engineering choices are
> made.  In those cases that fact will be clearly and prominently
> communicated in the document e.g.  in the abstract, the
> introduction, or a separate section or statement.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Abdussalam Baryun <[email-masked]>
Date: Thu, 24 Oct 2013 17:03:39 +0100

I don't agree with comparing our standards with other SDO standards within
our work, or even making our work process depend on other SDO products.
IETF RFC should try its best to have normative references that are RFCs not
dependent/government oriented. We may end up with a new name for our IETF,
as dependent IETF (DIETF).

AB

---

From: "Moriarty, Kathleen" <[email-masked]>
Date: Thu, 24 Oct 2013 13:53:51 -0400

I don't see how that is possible.  Different SDOs have different focus areas with clear interdependencies between the work.  We don't need to try to replicate the work happening elsewhere, but rather should continue to play nice with other SDOs.  It would be really hard to get all of the various experts needed attend multiple forums because one SDO didn't want to reference the work they did in another SDO.

Regards,
Kathleen

From: [email-masked] [mailto:[email-masked]] On Behalf Of Abdussalam Baryun
Sent: Thursday, October 24, 2013 12:04 PM
To: ietf
Cc: [email-masked]
Subject: Review of: Characterization of Proposed Standards

I don't agree with comparing our standards with other SDO standards within our work, or even making our work process depend on other SDO products. IETF RFC should try its best to have normative references that are RFCs not dependent/government oriented. We may end up with a new name for our IETF, as dependent IETF (DIETF).

AB

---

From: Dave Crocker <[email-masked]>
Date: Thu, 24 Oct 2013 14:00:40 -0400

On 10/24/2013 1:53 PM, Moriarty, Kathleen wrote:
> I don’t see how that is possible.  Different SDOs have different focus
> areas with clear interdependencies between the work.  We don’t need to
> try to replicate the work happening elsewhere, but rather should
> continue to play nice with other SDOs.  It would be really hard to get
> all of the various experts needed attend multiple forums because one SDO
> didn’t want to reference the work they did in another SDO.

For reference, the comment about other SDOs in my review note was 
specific to the existing draft text that compared an aspect of IETF 
documents to documents from other SDOs and my suggestion was merely to 
substantiate the claim(s) being made.

d/

---

From: Abdussalam Baryun <[email-masked]>
Date: Thu, 31 Oct 2013 01:49:26 +0100

Hi Kathleen

As my understanding, this procedure update document under review allows
comparing SDO work within our IETF work, so as IETF is already open to
allow all participation of businesses and SDOs into IETF, this issue of
comparing discussions while our standards draft developing will create
longer discussions and even disagreements between businesses/companies.

Any cross-organisation standard work issues with external bodies we need an
agreement policy like our one with ITU. I don't think it is good to discuss
comparisons in IETF because the IETF is open (other May not be), and we
should focus on the development of our standards following IETF aim. The
document does not mention any agreement requirement between IETF and
other SDOs, if we want to allow such process.

IMO the IETF has special aim and different standard vision than other SDOs
, so IETF standards need to be interoperable, competitive, and not
dependant. I suggest to add a requirement of existing agreement policy when
doing work that involve other SDO.

AB

On Thursday, October 24, 2013, Moriarty, Kathleen wrote:

> areas with clear interdependencies between the work.
>
> I suggest
Interoperable but not interdependent

> We don’t need to try to replicate the work happening elsewhere, but rather
> should continue to play nice with other SDOs.
>
> We can do better work than others so why replicate, but it is ok to build
on others sub-work.

> It would be really hard to get all of the various experts needed attend
> multiple forums because one SDO didn’t want to reference the work they did
> in another SDO.  ****
>
> Referencing is always good but following without progress is not good .

> ** **
>
> Regards,****
>
> Kathleen****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Abdussalam Baryun
> *Sent:* Thursday, October 24, 2013 12:04 PM
> *To:* ietf
> *Cc:* [email-masked]
> *Subject:* Review of: Characterization of Proposed Standards****
>
> ** **
>
> as dependent IETF (DIETF).****
>
> ** **
>
> AB****
>

---

From: Olaf Kolkman <[email-masked]>
Date: Thu, 31 Oct 2013 09:44:53 +0100

On 31 okt. 2013, at 01:49, Abdussalam Baryun <[email-masked]> wrote:

> Hi Kathleen
> 
> As my understanding, this procedure update document under review allows comparing SDO work within our IETF work, so as IETF is already open to allow all participation of businesses and SDOs into IETF, this issue of comparing discussions while our standards draft developing will create longer discussions and even disagreements between businesses/companies. 
> 
> Any cross-organisation standard work issues with external bodies we need an agreement policy like our one with ITU. I don't think it is good to discuss comparisons in IETF because the IETF is open (other May not be), and we should focus on the development of our standards following IETF aim. The document does not mention any agreement requirement between IETF and other SDOs, if we want to allow such process.
> 
> IMO the IETF has special aim and different standard vision than other SDOs , so IETF standards need to be interoperable, competitive, and not dependant. I suggest to add a requirement of existing agreement policy when doing work that involve other SDO.
> 

Dear Abdussalam,

Just to clarify what the draft is about: It is about documenting our processes and giving a characterization of Proposed Standards that match what amount of review we have performed and what the resulting quality is. 

There has been quite some pushback on the language that compares the quality of our output with the quality of other SDOs output. A version 05 will shortly be available (and I will post a message to the IETF list when it is) that is far more neutral (less condescending) than the previous versions of the document.

Cooperation with other SDOs, in the form of e.g. liaison relations, is not the subject of this document. The document also doesn’t talk about various types of normative relations that IETF standards may have with other documents.  I think that this document will not cause ‘longer discussions’ by itself.

I hope this clarifies, if not feel free to contact me off-list.

Thanks for engaging and kind regards,

—Olaf

---

From: Olaf Kolkman <[email-masked]>
Date: Thu, 31 Oct 2013 17:34:17 +0100

>   The draft is not ready for approval.

Based on these comments and the comments made during the IESG ballot I’ve made changes to the document.
Just posted as:
http://tools.ietf.org/html/draft-kolkman-proposed-standards-clarified-05

See Appendix B.6 for a somewhat detailed in-document log of the changes or the excellent diff tool:
http://tools.ietf.org/rfcdiff?url2=draft-kolkman-proposed-standards-clarified-05.txt

There was a suggestion to remove the characterization of Internet Standard (for completeness) I think that is a matter of taste and not of content.

More a matter of content is Dave’s (off-list) suggestion to remove section 4: That section is result of a discussion on the IETF list between Sep 3 and 13 and addresses the worries about setting the bar to high for publication.

I believe that with this version all is addressed and we’ve arrived at if not consensus, then only ripply consensus, certainly not rough…. but off course Jari/IESG is the judge of that.

Over (and out?)

—Olaf

---

From: Dave Crocker <[email-masked]>
Date: Thu, 31 Oct 2013 11:06:21 -0700

Folks,

Thanks to Olaf, et al for putting up with my after-the-deadline 
feedback.  However this document goes to the core of the IETF and it's 
likely to stand for some decades.  So I think we all need to be 
particularly diligent at considering its uses and the opportunities for 
problems with it.  In other words, this is a good document for being 
fussy about fine points.

Note that the motivation for Olaf's draft is the misunderstanding by 
others of the nature and worth of Proposed Standards documents.  We do 
not have complaints about the actual /quality/ of the specifications, 
but of the language describing their formal status.

In effect, such folk are distracted by some of the words in RFC 2026. 
We can't fix their style of reading or limitations in how they integrate 
information and also we cannot ignore such folk.

What we /can/ do is try to have the draft anticipate problems with how 
such folk will read the text.  This requires a very different style of 
draft analysis than we typically expect.  Rather than looking for 
clarity, we need to look for opportunities to misunderstand.  (I call 
that necessary kind of language 'political' speech, since the author 
must worry about active misinterpretation and misrepresentation...)

That's what has motivated much of my feedback, including the additional 
points below...

And by the way...

      1.  For a document that goes to the very heart of the IETF's 
reason for being and especially given how long-lasting its effect is 
likely to be, it has received astonishingly little active, public 
review, based on the public record.

      2.  I also wonder whether we shouldn't circulate the draft more 
broadly, outside of the IETF, to request the comments.  Will it 
accomplish what we want it to accomplish?

Into some details...

On 10/31/2013 9:34 AM, Olaf Kolkman wrote:
 >  There was a suggestion to remove the characterization of Internet
 > Standard (for completeness) I think that is a matter of taste and not
 > of content.

When there is more than one source for something as basic as 
"characterization of Internet Standard" the different copies will, over 
time, diverge and create an opportunity for confusion.  This draft seeks 
to be a normative document.  It needs to stay within its scope.  Its 
scope does not include characterization of Internet Standard.  It 
should, therefore, merely cite the proper text contained elsewhere.

By way of making offering a typical example:  Keep this redundant text 
in the current draft.  Years from now, change the related text in RFC 
2026, in ways that are significantly different from what is in this 
current draft.  There will be no synchronization effort to ensure that 
this draft is fixed to match the new, official text, because there is no 
formal linking of the text.  Anyone reading the RFC version of this 
current draft will therefore be reading an incorrect characterization.

 > More a matter of content is Dave’s (off-list) suggestion to remove
 > section 4: That section is result of a discussion on the IETF list
 > between Sep 3 and 13 and addresses the worries about setting the bar
 > to high for publication.

My concern about section 4 is that it only serves to invite criticism of 
IETF documents.  The statements in Section 4 really apply to all 
technical specifications and all organizations that publish them.  We 
never achieve perfection and we always reserve the right to produce a 
better version.

So including text that says nothing distinctive but has language about 
likely flaws invites criticism; it mostly serves as material for those 
looking for the opportunity.

Again, my apologies for the added round of late-stage effort I've 
triggered and many thanks to Olaf et al for responding so helpfully.

d/

---

From: Barry Leiba <[email-masked]>
Date: Thu, 31 Oct 2013 16:53:04 -0400

So far, I've just held the DISCUSS ballot that's been the formal
process to keep this discussion open.  It's time to start paddling the
oar that I've stuck in.

It seems that there are still three points that Dave has:

1. A general point: the document doesn't seem to have had the review
and scrutiny that one would expect for a significant process change.

2. Simple duplication of text from 2026 (in Section 3.2) is not a good
idea, as it invites future divergence.

3. Section 4 should be removed, as it weakens the statements made
elsewhere in the document and invites criticism, and is in effect self
evident anyway.

To point 1, I have a couple of notes in response.  I agree that this
is a formal process change -- some say it's not, but is only
documenting what we already do, and I think that's ignoring the fact
that, notwithstanding that, this document *does* change the formal
definition of what Proposed Standard means.  It doesn't matter that
it's "merely reflecting reality"; it's still a formal change.  That
said, I believe that the light volume of comments is because (1) it
*is* reflecting reality, and that's mostly not controversial and (2) a
lot of this was discussed when RFC 6410 was being developed, and these
changes are really artifacts of that discussion (which did not lack in
participation).  Further, the document has been out here for
discussion; the opportunity has been here, and IETF folks aren't
normally shy if they have things to say.

So, while I understand Dave's concern, I don't share it.  But Dave goes on:

> broadly, outside of the IETF, to request the comments.  Will it accomplish
> what we want it to accomplish?

We could post it to the new-work mailing list, or explicitly send it
out through our liaisons.  It's likely, I think, that it would get
little attention that way, though we won't know unless we do it.  On
the other hand, we don't normally send our process documents out that
way -- we didn't do that with the draft that became 6410, for example.
 I have no objection to soliciting external reviews and comments, but
I'm doubtful that it'll be useful.

>> Standard (for completeness) I think that is a matter of taste and not

I think that's not really addressing the core of Dave's concern, which
isn't that the repetition is unnecessary (which would be a matter of
taste), but that it's actually harmful, in that future changes could
leave us with two divergent definitions of "Internet Standard".

On this, I'll note that we're already somewhat along that path, as
6410 quoted text from 2026 in its definition of "Internet Standard".
And I'll also note to things along that line:

a. 6410 tweaked the definition of Internet Standard, in order to merge
in aspects of Draft Standard.  That this document quotes 2026 without
also bringing in the changes from 6410 somewhat misstates things (in a
small way, but nonetheless...).

b. 6410 had the opportunity to do the same thing, copying the
definition of Proposed Standard (which it did not change) from 2026,
and it didn't do it.  Instead, it said this:

   The stated requirements for Proposed Standard are not changed; they
   remain exactly as specified in RFC 2026 [1].  No new requirements are
   introduced; no existing published requirements are relaxed.

I suggest that that's a better approach, and would suggest that
Section 3.2 would be better done this way:

   The stated requirements for Internet Standard are not changed; they
   remain exactly as specified in Section 2.2 of [RFC6410], as it updates
   Section 4.1.3 of [RFC2026].

On point 3, I'll note that Section 4 is in this document exactly in
response to my comments.  I was concerned that as this document
tightens the formal criteria for PS, it's likely to cause even more
strictness in IESG approval.  We've always had the option (and have
not infrequently exercised it) to approve a document at PS that *does*
have known limitations.  This is from Section 4.1.1 of RFC 2026:

   A Proposed Standard should have no known technical omissions with
   respect to the requirements placed upon it.  However, the IESG may
   waive this requirement in order to allow a specification to advance
   to the Proposed Standard state when it is considered to be useful and
   necessary (and timely) even with known technical omissions.

Without Section 4, I believe we are removing the option that the
second sentence gives the IESG, and I want to be absolutely sure we
want to do that, before we do.  It's my opinion that we don't want to
remove it, and that's the purpose of Section 4.

> IETF documents.  The statements in Section 4 really apply to all technical
> specifications and all organizations that publish them.  We never achieve
> perfection and we always reserve the right to produce a better version.

I believe that Section 4 is saying more than the things that apply to
all technical specifications: it's not just saying that they're not
perfect and that we might find problems later.  It's meant to say that
we might, sometimes, approve specifications at Proposed Standard that
we *know* to have imperfections and limitations, and we're approving
them anyway.  And what it's adding to what 2026 says about that is
that when we do that, we'll explicitly say so in the document.

Perhaps you have a suggestion for different wording, Dave, that will
address your concern while still addressing mine?

Barry

---

From: SM <[email-masked]>
Date: Thu, 31 Oct 2013 17:00:37 -0700

Hi Dave,
At 11:06 31-10-2013, Dave Crocker wrote:
>others of the nature and worth of Proposed Standards documents.  We 
>do not have complaints about the actual /quality/ of the 
>specifications, but of the language describing their formal status.

Yes.

>In effect, such folk are distracted by some of the words in RFC 
>2026. We can't fix their style of reading or limitations in how they 
>integrate information and also we cannot ignore such folk.

The IAB sent a comment about ICT standardization to a directorate of 
the European Commission in 2009.  My interpretation of the intent is 
that it was so that the IETF is recognized more broadly among 
standardization organizations.  The comment stated that the IETF and 
IAB exert no control over the nature or identity of participants in IETF
activities.  Some parts of the comment was about standards 
making.  Most of the comments are usually about the process instead 
of the quality of the specifications.  If IETF  ignores such folks 
(see above) it can affect where the RFCs can be used.

In Section 2:

   "Hence IETF Proposed Standards are of such quality that they
    are ready for the usual market-based product development and
    deployment efforts into the Internet."

I am not comfortable with having that text in BCP 9.  The argument up 
to now has been running code and that that is the line which has been 
used for test of quality.  A better test of quality might be someone 
who has not followed the working group and who can implement the 
specification.  There is also the IPR test.  That is also one of the 
issues mentioned by the audience which the document targets.

Regards,
-sm