Subject: Re: Alternate entry document model (was: Re: IETF processes (wasRe: draft-housley-two-maturity-levels))

From: "t.petch" <[email-masked]>
Date: Mon, 1 Nov 2010 11:24:32 +0100

---- Original Message -----
From: "Andrew Sullivan" <[email-masked]>
To: <[email-masked]>
Sent: Friday, October 29, 2010 9:39 PM
> On Fri, Oct 29, 2010 at 01:20:23PM -0700, SM wrote:
> > It would be difficult to get buy-in if the document is not published as a
> > RFC.
>
> Supppse we actually have the following problems:
>
>     1.  People think that it's too hard to get to PS.  (Never mind the
>     competing anecdotes.  Let's just suppose this is true.)
>
>     2.  People think that PS actually ought to mean "Proposed" and not
>     "Permanent".  (i.e. people want a sort of immature-ish level for
>     standards so that it's possible to build and deploy something
>     interoperable without first proving that it will never need to
>     change.)
>
>     3.  We want things to move along and be Internet STANDARDs.
>
>     4.  Most of the world thinks "RFC" == "Internet Standard".

I think that this point is crucial and much underrated.  I would express
slightly differently,
that, for most of the world, an RFC is a Standard produced by the IETF, and
that the number of organisations that know differently are so few in number,
even if
some are politically significant, that they can be ignored.

And that this is something outside our control and that we are powerless to
change.

So whether we have XStandard, YStandard or ZStandard and how we move
between them is irrelevant (to most of the world).

Hence my focus is on how we can get an RFC published in the first place, in a
more
timely manner with, ideally, an improvement in the quality.

Tom Petch

> If all of those things are right and we're actually trying to solve
> them all, then it seems to me that the answer is indeed to move to _n_
> maturity levels of RFC, where _n_ < 3 (I propose 1), but that we
> introduce some new document series (call them TRFC, for "Tentative
> Request For Comment", or whatever) that is the first step.  Then we
> get past the thing that people are optimizing for ("everything stays
> as Proposed Standard once it gets published") by simply eliminating
> that issue permanently.
>
> Ah, you say, but now things will stick at TRFC.  Maybe.  But we could
> on purpose make it easier to get TRFC than it is now to get PS (say,
> by adopting John's limited DISCUSS community for TRFC, or one of the
> other things discussed in this thread).  Also, the argument about
> everyone thinking that RFCs are "standard", and the resulting pressure
> to make them perfect and permanent, would be explicitly relieved (at
> least for a while), because nobody thinks that TRFCs are standards.
>
> Note that this is not to denigrate SM's suggestion, which also doesn't
> seem wrong to me.  But since one of the issues appears to be that
> anything called "RFC" is set in stone, then if we just stop calling
> the early-publication documents "RFC" that and introduce something
> after I-D (which is formally only on the _way_ to some consensus, and
> not actually the product of it), the blockage might be removed.
>
> A
> --
> Andrew Sullivan
> [email-masked]
> Shinkuro, Inc.

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 1 Nov 2010 11:40:02 -0700

Hi -

> From: "t.petch" <[email-masked]>
> To: "Andrew Sullivan" <[email-masked]>; <[email-masked]>
> Sent: Monday, November 01, 2010 3:24 AM
> Subject: Re: Alternate entry document model (was: Re: IETF processes (wasRe:draft-housley-two-maturity-levels))
...
> timely manner with, ideally, an improvement in the quality.
...

Ironically, the more we emphasize improving the "quality" of RFCs, the more
we reinforce the myth that all RFCs are standards.  I higher percentage
of obviously immature, speculative, or even outright garbage documents
might help dispel the myth.  :-) * 0.5

Randy

---

From: Ted Hardie <[email-masked]>
Date: Mon, 1 Nov 2010 11:57:30 -0700

> might help dispel the myth.  :-) * 0.5

Maybe should make April 1 happen quarterly.....

Ted

---

From: Martin Rex <[email-masked]>
Date: Tue, 2 Nov 2010 16:09:53 +0100 (MET)

t.petch wrote:
> slightly differently,  That, for most of the world, an RFC is a Standard
> produced by the IETF, and that the number of organisations that know
> differently are so few in number, even if some are politically
> significant, that they can be ignored.

The underlying question is acutally more fundamental:
do we want to dillute specification so much that there will be
multiple incompatible / non-interoperable versions of a specification
for the sake of having a document earlier that looks like the
real thing?

There have been incompatible versions of C++ drafts (and compilers
implementing it) over many years.  HD television went through
incompatible standards.  WiFi 802.11 saw numerous of "draft-N"
and "802.11g+" products.  ASN.1 went through backwards incompatible
revisions.  Java/J2EE went through backwards-incompatible revisions.

Publishing a specification earlier, with the provision "subject to change"
appears somewhat unrealistic and counterproductive to me.  It happens
regularly that some vendor(s) create an installed base that is simply
to large to ignore, based on early proposals of a spec, and not
necessarily a correct implementation of the early spec -- which is
realized after having created an installed base of several millions
or more...

Would the world be better off if the IETF had more variants of
IP-Protocols (IPv7, IPv8, IPv9 besides IPv4 and IPv6)? Or if
we had SNMP v4+v5+v6 in addition to v3 (and historic v2)?
Or if we had HTTP v1.2 + v1.3 + v1.4 in addition to HTTPv1.0 & v1.1?

I do not believe that having more incompatible variants of a protocol
is going to improve the situation in the long run, and neither do 
I believe in getting entirely rid of cross-pollination by issuing
WG-only documents as "proposed standards".

What other motivation could there be to publishing documents earlier
than vendors implementing and shipping it earlier?  And if they do
that, there is hardly any room for any substantial or backwards-
incompatible changes.  And the size of the installed base created
by the early adopters significantly limits usefulness of any features
or backwards-compatible changes that are incorporated into later
revisions of the document.

-Martin

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 2 Nov 2010 11:45:57 -0400

> Or if we had HTTP v1.2 + v1.3 + v1.4 in addition to HTTPv1.0 & v1.1?

Maybe.  Arguing about counterfactuals is always fun, though a little
difficult.  But in general, there is more than one way in which a
plethora of different standards can affect things.

One way -- the one you seem to be (quite reasonably) worried about --
is that you get a bunch of non-interoperating, half-baked things that
are later rendered broken by updates to the specification.  Note that
this method is actually the one we claim potentially to have; the
two-maturity-levels draft does not change that.  The idea is that you
are supposed to try things out as early as reasonable, and if there
are significant changes, they'll turn up when an effort is made to
move things along the maturity track.  

Some people have argued in this thread (and the other related ones)
that there is a problem from the IETF attempting to prevent the
problem you're talking about.  That attempt, which is not documented
anywhere, amounts to a high bar of difficulty in getting an RFC
published at all.  I'm not actually sure I have the empirical data to
support a claim that it really is harder to get that initial-version
RFC published; but people seem to think that it is, anyway.
Suppositing that it really is harder, then the current documented
standards track, and the two-maturity-levels document, will both be
wrong.  There will be one effective maturity level.

The argument in favour of publish-early, revise-often approaches is
that iterations will, or ought to, improve things.  Imagine: in some
other possible world, they're up to IPv10 now, but it took those
intervening versions to discover that you really really needed some
clean interoperation layer with the "legacy" IPv4 networks.  In that
other possible world, the early deployment failures of
quickly-produced IPv6 specifications were taken to mean that
deployment was too hard, and so a _more_ interoperable system was
crafted.  (Having come from a community where people talked about "the
possible world where I am a mud puddle", the possible world where
deployment failure leads protocol designers to think harder about real
deployment doesn't seem unrealistic to me.)

I should say that personally, I'm not sure where I land on the
early-often/forward-compatible continuum; I suspect it differs
depending on the protocol.  But as others have been pointing out, it
sure looks like there's more than one problem here, and we need to be
clear on what we think we're solving before we jump in and start doing
it.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: "Spencer Dawkins" <[email-masked]>
Date: Tue, 2 Nov 2010 10:56:19 -0500

You could call me a blue-eyed optimist, but I have brown eyes ...

> revisions of the document.

I think you're conflating two steps here:

- implement and test, including interop testing

- shipping in a product

There is a reason to implement and test, including Interop testing. Robert 
Sparks can speak (orders of magnitude :-) more authoritatively than I can, 
but my impression from looking at SIPit reports, is that people in the RAI 
community really do have implentations, including implementations used for 
interop testing, that are NOT present in their products, and they do expect 
to feed back based on interop testing and see changes to the protocol that 
might require implementation changes.

But beyond this ... you can implement any draft - you just have to think 
that's a good idea, and right now, people are making the decision to 
implement any particular draft with no guidance.

In a previous life, I worked for a company making network monitor equipment. 
We hade requests from large carriers to support monitoring one of the 
SIGTRAN specs - I'm thinking it was 
https://datatracker.ietf.org/doc/draft-ietf-sigtran-sua/ - in about six 
different flavors of the draft (out of the 16 versions published as working 
group documents).

- All had been implemented.

- All had been deployed.

- No two were 100-percent compatible.

- None carried an identifier that said "this is -09" or whatever, so we used 
heuristics to guess which version of the protocol we were looking at.

- Some carriers had more than one version of the protocol running in the 
networks (-07 between these two boxes, -09 between these other two boxes).

And none of these large carriers had waited for a proposed standard (it 
still hadn't been approved for publication when they were asking us to 
support six flavors of the draft).

If having a working group say "we think this version of the draft is stable 
enough to code to" is going to make things worse, I wouldn't mind 
understanding why that's true.

I will agree that I hear people say "we can't change the spec from v03 of 
this because people have implemented it", but looking back, they were 
pointing to TINY communities, compared to real deployments that happened 
later. In the case I was talking about with Eric Berger yesterday, the 
"can't change the spec from -03 because people have implemented it" draft 
was finally published at -08. -03 DID change, several times.

If someone manages to make a convincing argument to protect an early 
deployment to a tiny user community, that prevents us from solving problems 
with a protocol that would allow wider deployment, that's our fault.

Spencer

---

From: Yoav Nir <[email-masked]>
Date: Tue, 2 Nov 2010 18:08:40 +0200

Strange. I look at the same facts, and reach the opposite conclusions.

The fact that there were many implementations based on drafts of standards shows that industry (not just us, but others as well) does not wait for SDOs to be "quite done".  They are going to implement something even we label them "danger - still a draft pretty please don't implement"

Everybody in our industry has heard of Internet Drafts. They know that these are the things that end up being RFCs, which are, as others have said, synonymous with standards. If we don't get the drafts reviewed well enough to be considered "good enough to implement" fast enough, industry is just going to ignore us and implement the draft.

My conclusion is that we can't just ignore industry and keep polishing away, but that we have to do things in a timely manner.  One thing we've learned from the TLS renegotiation thing was that it is possible to get a document from concept to RFC in 3 months. Yes, you need commitment from ADs and IETFers in general (IIRC you and I were among those pushing to delay a little), but it can be done.

It's a shame that we can't summon that energy to regular documents, and that's how we get the SCEP draft which has been "in process" for nearly 11 years, and it's still changing. But that is partially because we (IETFers) all have day jobs, and our employers (or customers) severely limit the amount of time we can devote to the IETF. But that's a subject for another thread.

Time to get back to that bug now...

Yoav

On Nov 2, 2010, at 5:09 PM, Martin Rex wrote:

> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf
> 
> Scanned by Check Point Total Security Gateway.

---

From: Martin Rex <[email-masked]>
Date: Tue, 2 Nov 2010 19:16:29 +0100 (MET)

Yoav Nir wrote:
> 
> My conclusion is that we can't just ignore industry and keep polishing
> away, but that we have to do things in a timely manner.  One thing
> we've learned from the TLS renegotiation thing was that it is possible
> to get a document from concept to RFC in 3 months.  Yes, you need
> commitment from ADs and IETFers in general (IIRC you and I were among
> those pushing to delay a little), but it can be done.

Funny that you mention TLS renego.  This document was actually a
severe abuse of most IETF process rules.  And a significant part
of the document contents was written and published against
WG consensus and without IETF consensus.

We could have easily finished the work in two month with a simpler
and more robust solution and higher quality spec if there had not
been so much resistence from a group of conspiring vendors plus AD,
WG co-chair and document editor.

I realized it is *MUCH* easier to write a new I-D from scratch than getting
the problems of the original proposal fixed by discussion in the WG because
of the existing bias -- which is why I think that both, the document
editing process and the "WG item, documented progress hobbled by
consensus process" is the most significant roadblock in IETF document
progression if there is strong bias in the system.

If there is a need to ship draft versions of a spec, there should
be a field in the protocol identifying which version of a draft a
particular implementation is based on -- or the "early adopters"
should voluntarily bear the burden to either limit control or
field update distributed implementations of early drafts.

Another worrysome event was with the TLS channel bindings document,
when the definition of "TLS-unique" channel bindings had to be changed
underneath the RFC Editors feet to match the installed base of some
vendor (who failed to point out this issue earlier, although
the original definition had been like that for a few years).

Or the gssapi zero length message protection, which had been explicitly
described for GSS-APIv2 (rfc-2743,Jan-2000) because it is a non-expendable
protocol element required by the FTP security extensions (rfc2228,oct-1997)
specified in rfc-2743, section 2.3 last paragraph of page 62:

https://tools.ietf.org/html/rfc2743#page-62

but foobar'ed in Microsoft Windows Vista & Windows7
(fails decryption of gss_wrap(conf=TRUE) tokens with SEC_E_ALTERED_MESSAGE).

This features is tested and the error is reported by the OpenSource
gssapi test tool that I've been providing since 2000.

But as I've previously commented:
    http://www.ietf.org/mail-archive/web/kitten/current/msg00640.html

testing seems not very high up on some vendors priority lists.
It is a real pity that either Microsoft didn't bring any
Vista betas to this event:
    https://lists.anl.gov/pipermail/ietf-krb-wg/2006-March/005885.html

or there was no interest in performing rfc-4121 interop tests with a
GSS-API interop test tool that works quite well for interop testing
an MIT Kerberos for windows with Microsoft Win2K&Win2K3 Kerberos SSP.

-Martin

---

From: Martin Rex <[email-masked]>
Date: Tue, 2 Nov 2010 20:39:26 +0100 (MET)

Andrew Sullivan wrote:
> are later rendered broken by updates to the specification.

No, I'm more worried about ending up with a set of half a dozen
specs for any _new_ technology that are considered a valid "standard"
for _ALL_ vendors to implement and ship rather than one single document.

If there is only one "proposed standard", then it is much easier
for me as an implemetor, especially if I am _not_ an early adopter
to _not_ implement certain stuff shipped by some early adopters,
and it is up to the early adopters to fix broken implementations
that they've aggressively shipped without sufficiently planning ahead.
Standardization only makes sense if there is convergence on ONE
standard, rather than a half a dozen variants of every spec.

>
> Note that this method is actually the one we claim potentially to have;
> the two-maturity-levels draft does not change that.  The idea is that you

There are early adopters of new technogies.
There are early adopter which ship defective implementations of early specs.
There are early adopters which recklessly ship implementations of early
specs with no sensible planning ahead for potential changes in the
final spec and how to deal with it _gracefully_ (which in some cases
means distributing updates to the entire installed base).

Personally, I do not have a problem with early adopters in general,
but only with those that do not plan ahead -- i.e. build an implementation
that is (a) highly likely to be fully and flawlessly interoperable with the
final spec and (b) plan for updating *ALL* shipped products when the spec
is finalized (which is more about the date of IESG approval than the
date when the RFC is issued).

> problem you're talking about.

The IETF can not stop early adopters from doing stupid things,
but the IETF can refuse to create a mess instead of a single
proposed standard for later implementors of the protocol.
Documenting characteristics of early-adopter implementations
in an appendix (as well as non-negligable breakage) would
also be helpful to later implementors.

For every aggressive early adopter implementor within the IETF
there are 10 more thoughtful and conservative implementors outside
of the IETF, and it is really them for which the IETF should be
producing the standards (as well as for those who want to create
a successor protocol version and a transition/migration plan
to the new version.

>
> That attempt, which is not documented anywhere, amounts to a high
> bar of difficulty in getting an RFC published at all.

Getting a good and consistent document published is likely easy.
Obtaining a good and consistent document can be quite difficult
when you start with a very thin outline and make this thin outline
a highlander proposal by making it a WG document early in its pity shape
and subject every change to "WG consensus" change control.
If there are not enough engineers and not enough implementation
experience involved in the WG discussion, document progression
may become quite slow.

An approach that might work much better would be that a vendor
that is very interested in a particular technology starts documenting
his prototyping work early on as an I-D and announces that I-D in
the relevant working group -- and continues improving the I-D
and the prototype while discussing ideas.

I really think the bigggest problem in today's IETF is that
vendors are walling off their engineering from the IETF
WG discussion too much, and that is the root cause of the problem.

An IETF working group is not a crowd of highly skilled engineers
that is just sitting there twiddling their thumbs and waiting for
opportunities to start implementing new proposals and have fruitful
discussions on running code.  If vendors do not involve their
engineers in the IETF standardization work, then WG discussions
have a much higher likelyhood to become heated, contentious and
unproductive at a purely theoretical level.

>
> I'm not actually sure I have the empirical data to support a claim
> that it really is harder to get that initial-version RFC published;
> but people seem to think that it is, anyway.

Converging on the initial proposal for a new technology is harder
than revisioning an existing technology.  And there is experience
with the installed base, even for those that are otherwise
participating discussion free of implementation experience.

> that iterations will, or ought to, improve things.

Only if the early implementations are completely ditched as often as
the document is revisioned.

Look at the Linux Kernel versions of the "Enterprise" Linux Distros.
If your target market is business customers and the useful lifetime
of your software is 10+ years, maturity is much more important
than bleeding edge bells and whistles.

> Imagine: in some
> clean interoperation layer with the "legacy" IPv4 networks.

My impression is that we are around IPv10 -- not with the official protocol
version numbering, but with the concept revisions how to make it work
and usable for the public internet.

-Martin

---

From: SM <[email-masked]>
Date: Wed, 03 Nov 2010 04:37:35 -0700

>     interoperable without first proving that it will never need to

Some people view that level as an immature-ish level; some people may 
view it as a mature as they have overcome the barriers to publication.

>     4.  Most of the world thinks "RFC" == "Internet Standard".

Yes.

>as Proposed Standard once it gets published") by simply eliminating

That's somewhat similar to the Working Group Snapshot proposal.  Such 
proposals are mainly about addressing point 4.

I don't think that this community would support having "Proposed 
Standard" renamed to "Alleged Standard" or that such a change is in 
accordance with the IETF's sense of humor.

>least for a while), because nobody thinks that TRFCs are standards.

The RFC brand worked too well.  The people who have to decide on the 
issue are the same people who have authored documents which are 
currently at Proposed Standard level.  At a different level, this is 
like asking the IETF to give up the RFC brand.

>Note that this is not to denigrate SM's suggestion, which also doesn't

I view your comments as constructive criticism.

>not actually the product of it), the blockage might be removed.

People commented that there were not one issue but multiple 
issues.  RFCs published in the IETF Stream differ from RFCs from 
other streams in one aspect; they go through the IETF Standards 
Process.  I am over-simplying here.  The label is also about archival 
of the consensus decision [1].

At 14:03 29-10-10, Martin Rex wrote:
>Essentially, you seem to be asserting that IETF community feedback
>should be considered harmful and delayed to much later in the process
>where it can have even less impact.

I did not describe community feedback as "harmful".  I prefer not to 
provide an example here to avoid conflating this with matters that 
are subject to appeal.

>To me, that sound a little like giving up.

Maybe.  I doubt that the ideal solution would gain 
consensus.  Anyway, what's ideal is subjective.

>Changing solutions, fixing protocols and fixing documents is exhausting
>and painful, so let's just skip all of that.  Let vendors and implementors

I agree that it can be exhausting and painful.  But then there is a 
price to pay for getting free reviews.  Authors who would like to 
have their protocols and documents fixed should realize that it 
cannot happen without effort on both sides.

>wiggle out how to create interoperable products from shoddy specs
>all by themselves -- which is what some of them have been doing for
>some time -- implementing defective specs and shipping interoperability-
>impaired products long before the standardization work has converged
>on a moderately stable Proposed Standard.

We might call it shoddy specs; other view it as a matter of doing 
business.  "It works for me" is a bad idea if we are concerned about 
interoperability and clarity.

At 01:39 30-10-10, John C Klensin wrote:
>If that is true --and it may be-- it would indicate that not
>even we can keep track of the difference between "RFC" and
>"Standard".  If that were to be the case, discussion of maturity
>levels is basically a waste of time.

Yes, and the end result could be giving up.

>I think there are other issues with your outline, but the key
>one is that it would really, really, depend on "do or die"
>working.  If it didn't, the IETF would rapidly acquire a
>reputation for producing garbage as Standards, and that would
>be, IMO, really bad.

Yes.

>But the odds are against "do or die".  We've had that provision
>(automatic moves to Historic for Proposed (or Draft) Standards
>that are not advanced within a set period) for a very long time.

Agreed.

>I also suggest that the odds are against us, if only because the
>IESG will always have higher priorities than reviewing the
>status of documents no one cares about any more (either because
>they didn't get traction or because they got so much traction
>that they represent established practice that no one has
>motivation to update them).  In addition, I'm cynical enough to
>believe that IESG members would hesitate to kill off documents
>that have a few supporters who might put a lot of effort into
>complaining to the Nomcom... at least without strong documented
>evidence of community support.

Yes.

>Note that we have also made killing things hard: the idea that
>it takes putting together and publishing an RFC with details,
>justification, and all of the usual sections and boilerplate to
>move another RFC to Historic is a post-2026 innovation.  I think

It is called bureaucracy.

>languish.  IMO, if we wanted "do or die", we'd have to change
>that culture too.

Yes.

Picking up a comment from a different thread:

At 01:17 30-10-10, John C Klensin wrote:
>There is lots of time to change the written procedures if such
>an effort works, even experimentally.  After all, we've been out
>of synch with 2026 for 14 years now and it hasn't shut us down.

Please feel free to correct me; the Internet Standards Process has 
been out of sync for 18 years.  The RFC publication rate jumped from 
100 to approximately 300 a year.  Most documents are under 75 
pages.  The process requires more a reading of the documents, e.g. 
tracking discussions, cat herding, etc.  At another level, there is 
the question of what guidance can be given to people that are new to 
the IETF.  Is this what 2026 is about, i.e. what the process attempts 
to achieve?

I am not going to comment on [2]:

   "Proposed Standard (PS). The first official stage, but many standards
    never progress beyond this level (probably because IETFers don't like
    bureaucracy)."

Anyone asserting that IETFers do not understand the process faces the 
risk of being flamed by other participants.  There is a divergence of 
opinions about how the process works at times [3].

There isn't any pressing need for a change to 2026.  There is an 
opportunity to make a change.  There is also the option of doing 
nothing and put this to the list of periodic discussions that come up 
every now and then.

Regards,
-sm

1. See RFC 5741.  Michael StJohns posted some questions ( 
http://www.ietf.org/mail-archive/web/ietf/current/msg64191.html ) 
about a document that was approved for publication.
2. http://www.ietf.org/newcomers.html
3. http://www.ietf.org/mail-archive/web/ietf/current/msg64445.html

---

From: "t.petch" <[email-masked]>
Date: Wed, 3 Nov 2010 12:42:00 +0100

----- Original Message -----
From: "Yoav Nir" <[email-masked]>
To: <[email-masked]>
Cc: "t.petch" <[email-masked]>; <[email-masked]>
Sent: Tuesday, November 02, 2010 5:08 PM

The fact that there were many implementations based on drafts of standards shows
that industry (not just us, but others as well) does not wait for SDOs to be
"quite done".  They are going to implement something even we label them
"danger - still a draft pretty please don't implement"

Everybody in our industry has heard of Internet Drafts. They know that these are
the things that end up being RFCs, which are, as others have said, synonymous
with standards. If we don't get the drafts reviewed well enough to be considered
"good enough to implement" fast enough, industry is just going to ignore us and
implement the draft.

My conclusion is that we can't just ignore industry and keep polishing away, but
that we have to do things in a timely manner.  One thing we've learned from the
TLS renegotiation thing was that it is possible to get a document from concept
to RFC in 3 months. Yes, you need commitment from ADs and IETFers in general
(IIRC you and I were among those pushing to delay a little), but it can be done.

It's a shame that we can't summon that energy to regular documents, and that's
how we get the SCEP draft which has been "in process" for nearly 11 years, and
it's still changing. But that is partially because we (IETFers) all have day
jobs, and our employers (or customers) severely limit the amount of time we can
devote to the IETF. But that's a subject for another thread.

<tp>
Perhaps we should step back a little further, and refuse to charter work that
will become an RFC unless there are two or more independent organisations that
commit to producing code.  There is nothing like interoperability for
demonstrating the viability (or not) of a specification, and likewise, two
independent organisations are likely to bring two rival views of what should and
should not be specified.  Those not implementing can watch the two slugging it
out, and provide a balanced judgement when something needs consensus.

And two organisations with an interest might want to see a ROI sooner rather
than later.

Tom Petch
</tp>

Yoav

---

From: Yoav Nir <[email-masked]>
Date: Wed, 3 Nov 2010 15:12:36 +0200

> And two organisations with an interest might want to see a ROI sooner rather

That's being a killjoy. Organizations never commit to producing code. Besides, sometimes people get ideas and would like to get them published even before they have convinced their management that implementing is a good idea.

Now if someone produces an RFC just to convince management that it's a good idea to implement (because there's an RFC) that's a different thing.

Anyway, I have followed three cases where there were competing proposals for doing the same thing, one in NEA, the other two in IPSECME. As it turned out, those not implementing watched the two slugging it out, and provided no judgement whatsoever. In all three cases. I get a feeling that working groups are much better at polishing a single proposal than they are at choosing between competing proposals. I think Martin pointed out a similar thing yesterday. The RI proposal was not the only one on the table, but it was essential to have just one proposal to get the process running.

RFCs have one big advantage over all kinds of "blessed" internet drafts. The process of publishing an RFC gets the IANA allocations. Every implementation you make based on a draft will ultimately not work with the finished RFC because you have to use some "private" ranges of numbers. I have a feeling (not backed by any evidence) that part of the reason people rush to publish documents is the need to get IANA assignments for their implementations.

If we could have some kind of "viable", "promising" or "1st review" status for an Internet Draft, where the IANA assignments can be done on a temporary basis, I think this could allow for better review later on. I have no idea, though, how to get rid of the "need to support legacy implementations" argument that will arise later on, if changes to the protocol are proposed as part of the review.

Yoav

---

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 3 Nov 2010 09:48:46 -0500

Hi, Yoav,

Recognizing that we all work in different parts of the IETF, so our 
experiences reflect that ...

> RFCs have one big advantage over all kinds of "blessed" internet drafts. 
> The process of publishing an RFC gets the IANA allocations. Every 
> implementation you make based on a draft will ultimately not work with the 
> finished RFC because you have to use some "private" ranges of numbers. I 
> have a feeling (not backed by any evidence) that part of the reason people 
> rush to publish documents is the need to get IANA assignments for their 
> implementations.

I know that getting IANA allocations is a major consideration for one of the 
SDOs I'm liaison shepherd for, so my experience matches this (of course, 
there are various IANA policies - if a registry is first-come first-served, 
this isn't a consideration).

> If we could have some kind of "viable", "promising" or "1st review" status 
> for an Internet Draft, where the IANA assignments can be done on a 
> temporary basis, I think this could allow for better review later on. I 
> have no idea, though, how to get rid of the "need to support legacy 
> implementations" argument that will arise later on, if changes to the 
> protocol are proposed as part of the review.

Again, this depends on the instructions to IANA - some policies are easier 
to accommodate than others.

Thanks,

Spencer

---

From: Gonzalo Camarillo <[email-masked]>
Date: Sat, 06 Nov 2010 15:59:17 +0800

Hi,

I remember talking with Henning about the review process in the IETF
some time ago. I have pasted below the relevant section of his draft:

http://tools.ietf.org/html/draft-tschofenig-rai-reducing-delays-01#section-2

Review marathon:

      Getting a document through the IETF process does not happen
      without reviews.  There are reviews before the document becomes a
      WG item, reviews while the document is within the group, some
      chairs demand reviews before they issue a WGLC to probe
      'readiness', someone in between reviews by special expert groups
      are done (also by other SDOs and external groups, if necessary),
      then comes the WGLC, review by the responsible AD, review by the
      IESG, review by directorates (and there are many of those) and
      (depending on the type of document and history) an IETF Last Call.
      Everybody has an opinion, often not necessarily a technical
      opinion, on how documents should be written and why other solution
      approaches have not been explored.  Reviewers need time and then
      the review comments often cannot be ignored but need to be
      discussed and resolved.  When reviews happen later in the process
      then text changes are often expected to keep the reviewer happy.
      IESG members frequently put DISCUSSes on reviews and this
      increases their priority allowing a single person to, for example,
      delay the publication of a document for an extended period of
      time.  From a psychological point of view reviewers are in the
      unfortunate position that they have the feeling that something
      must be improved as an outcome of the review activity.  As soon as
      documents leave the working group the transparency is largely
      lost, despite IESG comments being sent to the authors, WG chairs
      and responsible ADs and despite information being available in the
      I-D tracker.  Mentally, many working group members consider
      documents to be 'done' when they leave the working group.

      The authors are not arguing that reviews are unnecessary but there
      has to be balance with respect to the goal that is about to be
      accomplished.

Note that this draft was written in the context of the RAI area but
seems to be equally applicable to other areas. Interestingly, the draft
also discussed the "Degeneration of the 3-level Standards Track process".

Cheers,

Gonzalo

On 31/10/2010 11:48 PM, Dave CROCKER wrote:
> 
> 
> On 10/30/2010 11:31 AM, Joel M. Halpern wrote:
>>      One of the positive effects
>> of our current system is taht because WG knows tha tthye have to clear all the
>> ADs, not just their own, they actually think about all these issues. And usually
>> manage to cope with them
> 
> A working group is diligent or it isn't.  It gets a range of feedback and 
> responds constructively to it... or it doesn't.
> 
> The working group behaviors that I have seen that pay explicit attention to the 
> specific question of satisfying AD reviews have nothing to do with quality of 
> the work and more to do with guessing what will personally bother an AD.  In 
> other words, it's about dealing with AD idiosyncrasy rather than with quality.
> 
> It is now common to get cross-area reviews and my own observation is that these 
> are a) typically quite reasonable and diligent, and b) dealt with constructive 
> by the working group.
> 
> ADs do sometimes come up with interesting and even important points, but AD 
> review is an extremely expensive and often frustrating mechanism that we already 
> have a vastly superior replacement for.  Its timing is better and it distributes 
> the work far better.
> 
> The fact that an AD sometimes catches some important problem is typically taken 
> as proof that the AD review and Discuss mechanism is essential.  This is highly 
> flawed logic, on two counts.
> 
> One is that it does not represent meaningful cost/benefit evaluation.  The cost 
> is actually quite high in energy, delay and frustration, and the significant 
> benefit overall is quite low (if the wg has been diligent and has gotten 
> cross-area reviews.)
> 
> The other is that protocol specs have a statistical likelihood of bugs, even 
> with the AD review.  We talk about AD review almost as if it ensures perfection, 
> but of course we know it does not.
> 
> Ultimately, we have to trust the real world to evaluate the safety and efficacy 
> of a protocol.  That fact ought to give us permission to balance the cost and 
> benefit of the quality assurance efforts we require during specification 
> development and approval.
> 
> 
>> Yes, it would be very good to spot all of these things sooner. I have not yet
>> seen a proposal that actually works for doing so. But letting WGs or WGs + ADs
>> approve documents for general advancement is a step likely to lead to problems.
>> If all our WGs handed their ADs high quality documents that they had checked for
>> all these issues, then maybe we could look at this differently.
> 
> We do need quality assurance efforts.  The basic idea that working group efforts 
> are subject to outside review prior to approval is a significant value-add by 
> the IETF, IMO.  The question is how to provide sufficient review in a reasonable 
> way.
> 
> I believe that cross-area reviews largely satisfy that requirement.  If 
> within-area reviews are also needed, the AD should commission them, not do them 
> directly.
> 
> d/