Subject: call for ideas: tail-heavy IETF process

From: Jari Arkko <[email-masked]>
Date: Wed, 1 May 2013 08:33:47 -0700

I wrote a blog article about how we do a fairly significant amount of reviews and changes in the late stages of the IETF process. Next week the IESG will be having a retreat in Dublin, Ireland. As we brought this topic to our agenda, Pete and I wanted to raise the issue here and call for feedback & ideas for improving the situation with all of you.

http://www.ietf.org/blog/2013/05/balancing-the-process/

Jari

---

From: Michael Richardson <[email-masked]>
Date: Wed, 01 May 2013 12:33:47 -0400

<#part sign=pgpmime>

>>>>> "Jari" == Jari Arkko <[email-masked]> writes:
    Jari> I wrote a blog article about how we do a fairly significant
    Jari> amount of reviews and changes in the late stages of the IETF
    Jari> process. Next week the IESG will be having a retreat in
    Jari> Dublin, Ireland. As we brought this topic to our agenda, Pete
    Jari> and I wanted to raise the issue here and call for feedback &
    Jari> ideas for improving the situation with all of you. 

    Jari> http://www.ietf.org/blog/2013/05/balancing-the-process/

I'll repeat what has been said repeatedly in the newtrk and related
discussions.  The step from ID to "RFC" is too large because we are
essentially always aiming for "STD" rather than "PS".

If we are unwilling to bring "RFC" back to a place were it does not
equal STD, then we need to create a new category of document which
amounts to "fully baked ID".  Things like IANA allocations could occur
at that point.

In the days of dot-boom it was not unusual to see widespread
implementation very early in the ID process and even interop and
experimental deployment.   While this still occurs for quite a number of
things (and sometimes it's a problem that the ID can not be changed as a
result), there is an equal amount of "wait for the RFC to come out".

I believe that we probably need to simply do less.  
Or perhaps we've reached the n^2 overhead problem, and since resources
are less(%), if we can't increase resources allocated to overhead, then
it's time to reduce n:  the IETF should fork and/or shard somehow.

(%)-it's not just about $$ invested, it's also, I think, that after
    many years of caffeine and sugar, many of us are simply 
    immune to their effects, and/or have given them up.

(2)-by adding an intermediate step in the "ID" process, I haven't
    removed the "heavy" part of the process, I've just redefined the
    process so that it's no longer at the "tail" of the process.
    This is, I admit, akin to adjusting the definition of unemployment.

    But, we can all agree when an ID is baked enough for the WG to
    consider it deployable, then we will actually get to the "running
    code" part sooner, which frankly is the only real way to get
    real experience.

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 1 May 2013 18:39:09 +0200

On May 1, 2013, at 18:33, Michael Richardson <[email-masked]> wrote:

> we need to create a new category of document which
> amounts to "fully baked ID"

Yes.  (I'm not sure it's a category, it might just be a stage.)

There is a stage in the development of a protocol where I no longer want to hear "it's just a draft, so we can change anything at will".

This would also be a good time to bring in the cross-area review.

Grüße, Carsten

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 01 May 2013 11:11:11 -0600

On 5/1/13 10:39 AM, Carsten Bormann wrote:
>> we need to create a new category of document which amounts to
>> "fully baked ID"
> 

Isn't that Proposed Standard?

Peter

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 1 May 2013 10:27:03 -0700

> Isn't that Proposed Standard?

No, it very clearly isn't. To get from "the last version of this draft" to "Proposed Standard" takes many months, significant involvement of an AD, and some involvement of all ADs. All of that has costs.

--Paul Hoffman

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 01 May 2013 11:35:28 -0600

On 5/1/13 11:27 AM, Paul Hoffman wrote:
> No, it very clearly isn't. To get from "the last version of this
> draft" to "Proposed Standard" takes many months, significant
> involvement of an AD, and some involvement of all ADs. All of that
> has costs.

Sorry: oughtn't that be Proposed Standard?

Peter

---

From: Sam Hartman <[email-masked]>
Date: Wed, 01 May 2013 13:44:02 -0400

>>>>> "Michael" == Michael Richardson <[email-masked]> writes:

    Michael> If we are unwilling to bring "RFC" back to a place were it
    Michael> does not equal STD, then we need to create a new category
    Michael> of document which amounts to "fully baked ID".  Things like
    Michael> IANA allocations could occur at that point.

Hi.  Could you clearly articulate why you want this category and what
you hope it will do and not do?  I tried to respond with my thoughts
about this but realized that I don't understand your goals well enough
to provide more than a poorly considered reaction.

---

From: Dave Crocker <[email-masked]>
Date: Wed, 01 May 2013 10:59:58 -0700

The blog nicely classes the problem as being too heavy-weight during 
final stages.  The quick discussion thread seems focused on adding a 
moment at which the draft specification is considered 'baked'.

I think that's still too late.

Certainly it could be useful, but it's still very late in the process.

Specification development usually does -- and certainly ought to -- 
formulate basic design ideas or approaches that motivate the details in 
a specification.  A good specification will include discussion of this, 
and I think the typical and most significant hashing and re-hashing that 
happens in later stages is about the design issues.

While basic design revisions can happen at any time in a process, I'll 
suggest that a specification effort should be asked to document its 
interesting design choices early, and circulate /these/ for external 
review, iterating later if basic choices are changed.

I suspect that an earlier exercise at summarizing functional goals and 
design approaches and issues will have a number of benefits, beyond 
enabling earlier external review.

d/
-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Michael Richardson <[email-masked]>
Date: Wed, 01 May 2013 14:11:30 -0400

>>>>> "Sam" == Sam Hartman <[email-masked]> writes:

    Sam> Hi.  Could you clearly articulate why you want this category and what
    Sam> you hope it will do and not do?  I tried to respond with my thoughts
    Sam> about this but realized that I don't understand your goals well enough
    Sam> to provide more than a poorly considered reaction.

It's what Carsten said.

1) this idea is baked enough for cross-area review to make sense.
2) the protocol is not going to change significantly, one could
   implement.
3) any future changes need thus to take into account impact on
   existing implementations... BUT that doesn't mean that we can't
   change things.

It's what PS *ought* to have been, and what "RFC"s were prior to
1990 or so.

---

From: Michael Richardson <[email-masked]>
Date: Wed, 01 May 2013 14:16:25 -0400

>>>>> "Dave" == Dave Crocker <[email-masked]> writes:
    Dave> I think that's still too late.

    Dave> Certainly it could be useful, but it's still very late in the
    Dave> process. 

It's too late for Internet Changing Things (BGP4, TCP, IPv6 Header Options)..

It's still too early for things we know little about
(SPF, IPsecME AD VPN, all the LLN stuff).. 

just to name a few, there are lots of examples I know.

The problem is partly that we apply the same process regardless of
impact... The review is therefore too late for Internet Changing Things,
and way to heavy for new things that are more self-contained.

---

From: Sam Hartman <[email-masked]>
Date: Wed, 01 May 2013 14:19:38 -0400

OK.

>   existing implementations... BUT that doesn't mean that we can't

I'm very nervous about changes to the process of this form.

I'd support doing this sort of categorization as an advisory label.
For example if people wanted to get together and somehow track what
documents were fully baked as a service to the community and see how
useful it was, I'd support that.

I am strongly against making something like this a formal part of the
process without something informal like that first and without
significant experience with the informal something.

---

From: SM <[email-masked]>
Date: Wed, 01 May 2013 12:19:15 -0700

At 08:33 01-05-2013, Jari Arkko wrote:
>I wrote a blog article about how we do a fairly significant amount 
>of reviews and changes in the late stages of the IETF process. Next 
>week the IESG will be having a retreat in Dublin, Ireland. As we 
>brought this topic to our agenda, Pete and I wanted to raise the 
>issue here and call for feedback & ideas for improving the situation 
>with all of you.
>

Here's my write-up:

  (a) Why should the document be a Proposed Standard?

      Because the intended status says so.

  (b) Describe the depth and breath of review.

      Yes.

  (c) Does the document require review from other areas?

      No.

  (d) Describe the consensus.

      Yes.  I ignored the stupid comments.

There IESG is faced with the inevitable choice of either turning a 
blind eye or discussing about the document.  The later raises late 
surprises.  The simple fix is for the IESG not to discuss about the 
document. :-)  The not-so-simple fix is for the IESG to find out 
where the WG failure occurred and take remedial action.  It's 
politically incorrect to fire WG Chairs.  That constrains the 
remedial action to a slap on the wrist [1].

Who is going to do all these early cross-area reviewing?  Is all that 
work for free?

The executive summary is that the working group is not working.  The 
Last Call does not work.  The directorate/team reviews is a 
work-around [2] to fix that.

Regards,
-sm

1. If someone gives you a slap on the wrist, it is a mild punishment.
2. a plan or method to circumvent a problem without eliminating it.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 1 May 2013 21:32:45 +0200

> It's what PS *ought* to have been, and what "RFC"s were prior to

One problem is certainly the cognitive barrier imposed by the RFC process.
-- RFCs never change, so you want to get them right;
-- there is a two-month editorial process in front of the publication;
etc. etc.

So I don't think changing the process leading up to the RFCs is really going to change that much.

Having a label for a "baked" I-D, maybe some publicly visible directory for them, but retaining the I-D's fast change capability for editorial changes (and fixes that turn out to be necessary), would work better.

I also like what Sam said: Try this out first as an informal addendum to what we have and what works.

Grüße, Carsten

---

From: "Moriarty, Kathleen" <[email-masked]>
Date: Wed, 1 May 2013 15:41:25 -0400

How about having a running list (or registry) of IETF RFCs that have become the "de-facto" standards?  

Best regards,
Kathleen

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Carsten Bormann
Sent: Wednesday, May 01, 2013 3:33 PM
To: Michael Richardson
Cc: Sam Hartman; IETF list
Subject: Re: call for ideas: tail-heavy IETF process

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Wed, 01 May 2013 15:58:08 -0400

>>>>> "Moriarty," == Moriarty, Kathleen <[email-masked]> writes:
    Moriarty> How about having a running list (or registry) of IETF RFCs
    Moriarty> that have become the "de-facto" standards?    

So, the STD series is pretty exactly this.

Newtrk also proposed more than one level of this.

-- 
Michael Richardson <[email-masked]>, Sandelman Software Works

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 02 May 2013 08:05:34 +1200

> I think that's still too late.

What, you agree with your younger self?

http://tools.ietf.org/html/draft-carpenter-icar-sirs-01

Apart from the non-diverse acronym, I still think that proposal
was a good one.

    Brian

---

From: Dave Crocker <[email-masked]>
Date: Wed, 01 May 2013 13:17:07 -0700

> http://tools.ietf.org/html/draft-carpenter-icar-sirs-01

I really dislike being consistent, and always have.

> Apart from the non-diverse acronym, I still think that proposal

Me too.  Could be interesting to, ummmm, review it for updating...

d/

---

From: Scott Brim <[email-masked]>
Date: Wed, 1 May 2013 17:00:08 -0400

A draft does get cross-area review, at least once, often more than once.
 Some drafts in some WGs get it earlier than others, by explicit
invitation.  Others don't get it until the latest they can, when they go
to last call ... but a process point for cross-area review during WG
handling, already exists.  It's just optional.  I did like SIRS, and the
equivalent of SIRS exists -- in the informal relationships that are all
through the IETF.

You can make the option more explicit, you could even make it mandatory,
but doing so won't speed up draft processing at all and will probably
make it slower.  One of the IETF's problems is not the problem under
discussion but unconscious assumptions about the standards process.
People think of last call as near the end of the process.  It's time to
abandon that outdated view and recognize that so-called "last" call is
not as close to the end as it used to be, that it's a bit closer to the
middle ... and that that's really okay.  Really.  Some other SDOs have
their process structured that way.  Let's rename "last call" to
something like "IETF review" and stop giving people the wrong
expectations.  Review outside the WG is vital, can be done repeatedly,
and must be done by the whole IETF at least once.

Scott

---

From: Ted Lemon <[email-masked]>
Date: Wed, 1 May 2013 21:10:45 +0000

On May 1, 2013, at 5:00 PM, Scott Brim <[email-masked]> wrote:
> Let's rename "last call" to

Yup.   The term "last call" is traditional, and I feel a bit of attachment to it, but I think your observation here is exactly correct: we ought not to think of the IETF last call as the end of the process.

However, that is a bit of a problem, because I think it's fairly rare for documents to get additional "review" at last call time.   Changing the name probably won't fix that.

---

From: Ralph Droms <[email-masked]>
Date: Wed, 1 May 2013 17:43:19 -0400

On May 1, 2013, at 1:59 PM 5/1/13, Dave Crocker <[email-masked]> wrote:

> 
> The blog nicely classes the problem as being too heavy-weight during final stages.  The quick discussion thread seems focused on adding a moment at which the draft specification is considered 'baked'.
> 

...and not useful unless the diverse review and changes to the document take place much earlier in the process to make sure the document is "baked."  Declaring done doesn't make it so.

> Certainly it could be useful, but it's still very late in the process.

As Jari wrote, we often bury the heavy tail of the process in a limited discussion among IESG and the document authors.

The tail is heavy in two different ways:

* significant review and modification takes place in IESG review, after the WG and the IETF have declared the document done
* the burden of the review, managing the discussion, making sure any changes fix the problem and don't break anything else often falls on the IESG and even a single AD

> 
> Specification development usually does -- and certainly ought to -- formulate basic design ideas or approaches that motivate the details in a specification.  A good specification will include discussion of this, and I think the typical and most significant hashing and re-hashing that happens in later stages is about the design issues.
> 
> While basic design revisions can happen at any time in a process, I'll suggest that a specification effort should be asked to document its interesting design choices early, and circulate /these/ for external review, iterating later if basic choices are changed.
> 
> I suspect that an earlier exercise at summarizing functional goals and design approaches and issues will have a number of benefits, beyond enabling earlier external review.

Dave - excellent idea.  A summary that explains the "why" first, that promotes early cross-area review or otherwise expands the set of people that look at and understand the nature of the problem and the structure of the specification will allow for earlier review and lighter-weight correction of problems.

> d/

- Ralph

---

From: Ralph Droms <[email-masked]>
Date: Wed, 1 May 2013 17:47:17 -0400

> and must be done by the whole IETF at least once.

At the same time, we have to emphasize that "IETF review" means "a significant group of people have to look at this document really, really carefully to make sure it's right and report the results of their review".  It doesn't mean "well, it looks OK and the IESG is going to give a really, really careful review anyway so I don't need to worry about it".

Perhaps some of the directorate reviews ought to happen much earlier in the process?

- Ralph

---

From: Elwyn Davies <[email-masked]>
Date: Wed, 01 May 2013 23:13:32 +0100

> Apart from the non-diverse acronym, I still think that proposal
Rereading the background document that lead to the SIRS proposal is also 
worthwhile:
http://tools.ietf.org/html/draft-ietf-problem-issue-statement-00

I think the views in sections A.2.6. to A.2.9 are still pretty much 
applicable; judging from recent comments some people still think the 
situation with the IESG is a 'battle of wills'.

I guess one way of summarising this whole problem might be:
We set on a group of people (a WG) who by definition are supposed to 
work in 'common mode' on a particular problem in what is mostly a closed 
bubble. It's perhaps not entirely surprising that what comes out at IETF 
Last Call time has more than its fair share of 'common mode failures', 
symptoms of groupthink and blinkered views of the overall requirements 
for effective operation in the wider Internet.

The comments in the 'problem' draft and the SIRS proposal reflect an 
attempt to suggest how the closed bubble can be kept in contact with the 
wider world.  However, getting the relationship between outside 
reviewers and the WG teams right would be non-trivial - not to mention a 
significant organisational effort and a load on the reviewers.  To be 
honest, AFAIK we have never attempted to quantify how much extra effort 
(if any) a SIRS-like exercise would require as compared with the 
existing directorate reviews.

Personally, I don't believe that we should attempt to categorise some 
I-Ds as more significant than others.  This requires 20-20 hindsight 
with which we are certainly not equipped.

/Elwyn
BTW
Re the diversity discussion: take a look at the editorial panel on the 
problem draft - ok, its only along one of the dimensions.

Jari might also remember a contribution he made to 'problem':
>     o  WG process is too slow, because of feeping creaturism, deadlocked
>        conflicts or lack of worker bandwidth (Jari Arkko)
I can certainly agree on the feeping creaturism!

/E

---

From: Elwyn Davies <[email-masked]>
Date: Wed, 01 May 2013 23:11:33 +0100

/Elwyn
BTW

/E

---

From: Scott Brim <[email-masked]>
Date: Wed, 1 May 2013 18:46:33 -0400

> Dave - excellent idea.  A summary that explains the "why" first, that promotes early cross-area review or otherwise expands the set of people that look at and understand the nature of the problem a…

I forgot to say I like that a lot too.  The proponents should be doing
that by some means anyway within the WG.

---

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 2 May 2013 05:21:54 +0000

> Sorry: oughtn't that be Proposed Standard?

Yep, it ought to.

Crazy idea: Call a draft PS when it completes WG last-call and give it an RFC number, call it something else when it passes IESG review (draft standard? :-) and republish the RFC. When there is interop experience, call it full standard and republish as needed.

Yeah, all kinds of issues, but if we created a new thing here in between WGLC and PS, the broader industry would never understand.

Lars

---

From: t.p. <[email-masked]>
Date: Thu, 2 May 2013 11:46:34 +0100

----- Original Message -----
From: "Jari Arkko" <[email-masked]>
To: "IETF list" <[email-masked]>
Cc: "Pete Resnick" <[email-masked]>
Sent: Wednesday, May 01, 2013 4:33 PM
Subject: call for ideas: tail-heavy IETF process

I wrote a blog article about how we do a fairly significant amount of
reviews and changes in the late stages of the IETF process. Next week
the IESG will be having a retreat in Dublin, Ireland. As we brought this
topic to our agenda, Pete and I wanted to raise the issue here and call
for feedback & ideas for improving the situation with all of you.

<tp>

Well, yes, any engineer knows that the later in the process changes are
made, the more costly they are, so the key is getting the early stages
right.  But what are we doing wrong?  We need feedback about what the
DISCUSSes are, what might we have done at eg WG Adoption of an I-D to
forestall the later DISCUSS.

Are some Areas or WG better than others?  A statistical analysis would
help here, who to learn from, who not to; the graph suggests to me that
there is a steadily rising trend of DISCUSSes per document except when
there is a recession, when there is a step fall in them, but I do not
see any action I can take to make a difference from this observation.

Tom Petch

Jari

---

From: Francis Dupont <[email-masked]>
Date: Thu, 02 May 2013 15:47:39 +0200

>  amounts to "fully baked ID".  Things like IANA allocations could occur

=> IMHO the last point (IANA allocations) is a critical detail.

>  result), there is an equal amount of "wait for the RFC to come out".

=> as an implementor of early specs I very often get a problem with
the to-be-done IANA allocations: if it is not a concern for a lab
prototype, I enforce the strict policy to *never* make an official
distrib with support, etc, using not officially allocated by IANA
code points. I believe I have not to explain the reason of this policy.
Note this concern does not go very well with the "running code" principle,
this is why I'd like to see Jarri's issue solved, even currently I
don't agree (or disagree) with Michael's proposal.

Thanks

[email-masked]

PS: I like the SIRS idea even the gen-art review team (which can be seen
as a lightweight prototype of it) can lead me to believe it won't happen
in the real world (i.e., I'd like to be wrong :-), mainly because it
supposes an amount of skilled manpower we don't have.

---

From: Andy Bierman <[email-masked]>
Date: Wed, 1 May 2013 10:19:09 -0700

> essentially always aiming for "STD" rather than "PS".

I don't think this helps.  We want the highest quality documents
possible for developers to translate into implementations.

IMO, the answer is to identify cross-area issues early,
and more importantly, get early cross-area reviewers to help
avoid bad decisions that won't make it through IESG review.
Maybe a cross-area expert needs to join a design team
for awhile to make sure good decisions are made.

The ADs need to make sure these early reviews get
staffed and completed.  Maybe additional people
(directorate?) helps manage this process.

Andy

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 2 May 2013 18:17:13 +0200

> Yeah, all kinds of issues, but if we created a new thing here in between WGLC and PS, the broader industry would never understand.

That is a matter of naming and marketing ("candidate RFC"?).

The "this is baked, go and implement it" signal to the outside is only one aspect, though.

The other two things that were mentioned:
-- get the cross-area review going;
-- get the numbers (IANA).

As we are seeing e.g., with MPL (ROLL WG) right now, the latter can go wrong (aspects of the protocol still change, the allocation has to change with it).
But that isn't too much of a cost in most cases.

Re the cross-area review:  Doing it earlier doesn't mean doing more.  In theory.
In practice, not-really-baked stuff comes up and needs to be reviewed n times.
(As an appsdir reviewer, I have seen that happen.)

I think the WGLC is the wrong process step to define something as baked.
-- too late, because there may still be some (identifiable) random corner that the WG wants to fix;
-- too early, because the WG may be deluded or mismanaged.
Much better would be a simple collusion between WG chairs and sponsoring AD.
(In any sensible WG, you would still involve the WG, of course.)

Grüße, Carsten

---

From: Hannes Tschofenig <[email-masked]>
Date: Thu, 02 May 2013 21:14:39 +0300

Hi Jari, Hi Pete,

three issues come to my mind:

a) From several past discussions it is clear that we are not really able 
(maybe not willing) to change our processes. Even the smallest change 
faces a lot of resistance. Due to the resistance our response is to 
back-off and we solve a different problem instead.

Recent example: Look at the discussion from late last year that was 
triggered by the interactions regarding the WhatWG and Ian Hickson. This 
was a discussion about our long document production process. Some time 
later the discussions shifted to the discussion of an experiment to 
shorten the last call when running code is available, which then did not 
go anywhere and was only recently resurrected.

b) There is no interest to research where delay really happen. Your 
statistics just tell that there is delay but not why (of course). From 
my own experience I noticed that there are many reasons for delay and I 
am not sure I can blame it to the IESG reviews for most of the delay. It 
is only during the IESG review phase when the problems surface that 
could have been tackled much earlier.

Without looking at a number of cases we might focus on the wrong issues. 
Maybe, but I am not sure, there is something that can be generalized 
from some individual cases.

I am happy to work with someone else to go through a couple of different 
IETF activities that did not really go as planned to "reconstruct" what 
went wrong. I suggested this in the past but it is of course not 
fashionable and exciting.

c) For a number of IETF activities it does not matter whether they take 
a long time because the deployment phase that happens afterwards will 
take 5 times longer (in the positive case). The timing matters and even 
if something takes much longer than expected it might still be too early 
or does not meet the expectations of the community. I would rather 
prefer to spent more time on a topic and do it properly (and by properly 
I do not only focus on the standardization work but also on the 
outreach) than to publish documents quickly to have good statistics. 
Finding a common understanding and reaching conclusions with many 
different parties just takes a long time. Finding an agreement among 
your close friends is easy. In essence, we do not only want "good" 
standards but we also want those that make the Internet work better.

Ciao
Hannes

PS: I just want to add that I still dislike DISCUSSes that make me 
re-write documents just because an English native speaker believes 
(probably rightfully so) that he knows how to write the text better (or 
at least in a different way). (hint, hint)

On 05/01/2013 06:33 PM, Jari Arkko wrote:
> the IESG will be having a retreat in Dublin, Ireland. As we brought
> this topic to our agenda, Pete and I wanted to raise the issue here
> and call for feedback & ideas for improving the situation with all of
> you.
>

---

From: Pete Resnick <[email-masked]>
Date: Thu, 2 May 2013 14:18:10 -0500

One quick thing:

On 5/2/13 1:14 PM, Hannes Tschofenig wrote:
> From several past discussions it is clear that we are not really able 
> (maybe not willing) to change our processes.

Note that although we did ask the bigger question, the more central 
question relates to what we on the IESG can do all by ourselves (without 
making changes to the formal processes) that we can discuss during our 
IESG meeting next week. So don't limit your thinking to things that 
require process changes; suggested behavior changes on the part of ADs 
are also useful.

> (probably rightfully so) that he knows how to write the text better 
> (or at least in a different way). (hint, hint)

We were supposed to have gotten that hint before. Sometimes we are slow. 
Reminding us is a good thing. :-)

Thanks for your other comments.

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478

---

From: Martin Thomson <[email-masked]>
Date: Thu, 2 May 2013 13:25:08 -0700

Hannes Tschofenig <[email-masked]> wrote:
> There is no interest to research where delay really happen.

This is an important point.  We might want to measure before we cut.

> [...] From my
> own experience I noticed that there are many reasons for delay and I am not
> sure I can blame it to the IESG reviews for most of the delay.

My experience is similar, though not uniformly.  For the most part,
time spent in IESG processing has been short relative to the value
provided, in my opinion.

---

From: John Leslie <[email-masked]>
Date: Thu, 2 May 2013 16:29:36 -0400

> require process changes; suggested behavior changes on the part of ADs 

   The ADs could do worse than starting from the SIRS draft:

tools.ietf.org/html/draft-carpenter-icar-sirs-01

but as I read through it, I find a number of issues which I'd advise
_not_ trying to resolve. For example:

- All reviews public: not always a good idea;

- Normally posted to WG list: often a bad idea -- to easy for them to
  start a flame-war;

- Standardized summary message: these may be a very poor choice for
  early reviews;

- Selection process: it's far more important that the IESG be comfortable
  with each member.

   OTOH it contains items which SHOULD be taken seriously:

- No change to formal process;

- Formal process to add SIRS members;

- Inactive members fall off the SIRS list;

- WG may request a particular member;

- WG should request more than one reviewer;

- Earlier reviews should consider architectural questions;

- Three review cycle per document should be the minimum.

====

   I also note the overlapping discussion of DNS type SPF...
From: Doug Barton <[email-masked]>
> 
> Given that you can be 100% confident that the issue will be raised
> during IETF LC, wouldn't it be better to hash it out in the WG (as we
> have attempted to do)? Or is the WG's position, "we have no intention of
> dealing with this unless we're forced to?"

   In this case, hashing it out on the WG list at this stage seems a
_bad_ idea. If we had gotten a SIRS review right at the beginning, it
could perhaps be discussed on the WG list; but by now it's doubtful
any discussion would change anyone's mind. (And, of course, if the WG
had chosen only SIRS reviewers that agreed with their preferred way
of resolving the issue, the issue _wouldn't_ have been resolved early
in the process...)

   This is on it's way to being a poster-child "late surprise". :^(

> I'm fully sympathetic with your collective desire to move the process
> forward, and finish your document. The problem is that as it stands the
> document contains a course of action that is not only bad on its face,
> but contrary to a basic architectural principle adopted 4 years ago in
> 5507.

  BTW, I agree with Doug Barton that deprecating type SPF has some serious
negatives. The IESG will have to balance WG rough-consensus against
architectural principles; and I see no resolution that won't invite
appeals. :^(

   In a properly designed early-review situation, the issue would have
surfaced early; and it's possible it could have been resolved before
too many folks' positions had hardened...

--
John Leslie <[email-masked]>

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 03 May 2013 08:59:36 +1200

Would people like to see a new version of the SIRS draft?

In addition to the questions John raised below, Francis
and others mentioned: lack of reviewers. Also there is the
question of overlap with Area review teams such as secdir,
and there is accumulated experience from Gen-ART (RFC 6385).

Regards
   Brian

---

From: Francis Dupont <[email-masked]>
Date: Thu, 02 May 2013 23:38:10 +0200

In your previous mail you wrote:

>  We want the highest quality documents possible for developers to
>  translate into implementations.

=> I am afraid this statement is not fully consistent.

Regards

PS: note it is not an attack against you: in fact you gave a good summary
of the deeper issue.

---

From: joel jaeggli <[email-masked]>
Date: Thu, 02 May 2013 14:49:17 -0700

On 5/2/13 11:14 AM, Hannes Tschofenig wrote:
> my own experience I noticed that there are many reasons for delay and 
> I am not sure I can blame it to the IESG reviews for most of the 
> delay. It is only during the IESG review phase when the problems 
> surface that could have been tackled much earlier.
>
> Without looking at a number of cases we might focus on the wrong 
> issues. Maybe, but I am not sure, there is something that can be 
> generalized from some individual cases.
>
> I am happy to work with someone else to go through a couple of 
> different IETF activities that did not really go as planned to 
> "reconstruct" what went wrong. I suggested this in the past but it is 
> of course not fashionable and exciting.
>
The big delays on documents in my queue (e.g. one's that I inherited) 
are almost exclusively of author or wg holds the token again or blocked 
on some other document variety.  the tail is pretty long, whether it's 
heavy or not for a visibile minority of drafts.

---

From: Jari Arkko <[email-masked]>
Date: Thu, 2 May 2013 18:25:59 -0400

Hannes,

Regarding your point about process changes I agree that we've struggled there. But for some reason I'm quite optimistic that we can do the right changes. Regarding your point on deployment time being even longer, my observation has been that most changes have the "right" time, and that doing things too far before or too far after makes a big impact on the deployment time. In other words, a long IETF development time can and has increased deployment time, or even caused a new function to not be deployed. But I agree that mere time at the IETF is not what we want to look at.

Anyway, I wanted to respond more in depth on the your below point:

> b) There is no interest to research where delay really happen. Your statistics just tell that there is delay but not why (of course). From my own experience I noticed that there are many reasons for delay and I am not sure I can blame it to the IESG reviews for most of the delay. It is only during the IESG review phase when the problems surface that could have been tackled much earlier.

I agree of course that looking at where the issues are is very important, and would support work on that.

However, I did want to point out that when I said "tail-heavy", I did *not* necessarily mean delay. I meant that a lot of activity is going on, many document changes, and much review is going on. Obviously in some cases this translates to delay as well.

But the delay was really not my main concern. Primarily because I think other issues such as transparency to the working group or late surprises are more fundamental issues than mere timing. But also because I actually *do* have some statistics that seem to indicate that, overall, the last phase still goes through pretty quickly. Look at http://www.arkko.com/tools/lifecycle/wgdocs.html and compare the WG, IESG, and RFC editor times in the first graph. The WG time dominates. (I said "seem to indicate" because the results are pretty dated and I'm not really sure how valid they are, but they match at least my intuitive experience.) Not saying delay reduction wouldn't be useful, the overall times are still very long, and the IETF last call - IESG time is still a significant component. Just that delay would not be my primary concern.

Joel wrote:

> The big delays on documents in my queue (e.g. one's that I inherited) are almost exclusively of author or wg holds the token again or blocked on some other document variety.  the tail is pretty long, whether it's heavy or not for a visibile minority of drafts.

That matches also my intuition, as well as some other statistics that I've collected. In other words, most of the time we are waiting for a document change as the tracker state is revised ID needed. But of course, I don't know why we really are waiting for a document change, it could of course be that there are cases where the request for a change was unnecessary.

Jari

---

From: Dave Crocker <[email-masked]>
Date: Thu, 02 May 2013 15:58:43 -0700

On 5/2/2013 3:25 PM, Jari Arkko wrote:
> But the delay was really not my main concern. Primarily because I think other issues such as transparency to the working group or late surprises are more fundamental issues than mere timing. But also because I actually*do*  have some statistics that seem to indicate that, overall, the last phase still goes through pretty quickly. Look athttp://www.arkko.com/tools/lifecycle/wgdocs.html  and compare the WG, IESG, and RFC editor times in the first graph. The WG time dominates. (I said "seem to indicate" because the results are pretty dated and I'm not really sure how valid they are, but they match at least my intuitive experience.) Not saying delay reduction wouldn't be useful, the overall times are still very long, and the IETF last call - IESG time is still a significant component. Just that delay would not be my primary

Jari,

Very interesting set of graphs.  Thanks!

Doing very rough eyeballing of the "left side" averages against the 
"right side" averages -- that is, considering how things have changed 
over the last 10 years -- it looks like:

      Working groups were taking around 500 days and now take around 600.

      The IESG was taking around 200 days and now takes around 110.

      The RFC then and now takes around 100 days (with lots of variation 
between the then and the now, of course.)

      Considering the 'now' set of relationships among the phases, The 
IESG is adding about 20% on top of the working group, and the RFC Editor 
is adding another 20% on top of the working group.  In other words, once 
a working group considers itself done, they are probably only around 70% 
done...

      The total, today appears to be around 3 years to get a 
specification developed and published.  That's for one document, not the 
set of them that might be needed to produce a useful service...

A basic question, then, is whether we think these absolute numbers and 
these proportions of time are reasonable and appropriate for the IETF to 
be/remain effective?

If we don't, what should reasonable numbers look like?

d/

---

From: Randy Bush <[email-masked]>
Date: Thu, 02 May 2013 16:25:50 -0700

>  Considering the 'now' set of relationships among the phases, The IESG
> is adding about 20% on top of the working group, and the RFC Editor is
> adding another 20% on top of the working group.  In other words, once
> a working group considers itself done, they are probably only around
> 70% done...
> 
> The total, today appears to be around 3 years to get a specification
> developed and published.  That's for one document, not the set of them
> that might be needed to produce a useful service...
> 
> these proportions of time are reasonable and appropriate for the IETF
> to be/remain effective?

seems pretty reasonable to me.  from personal experience, the iesg and
rfced add useful value.  

being in a many-year process of getting a technology through the
saussage machine, it's the wg that feels to me to be the most
inefficient part of the process.  i attribute this to email not being
the best medium, and meetings being too short and too far between.  but
that is purely subjective.

randy

---

From: Ted Lemon <[email-masked]>
Date: Fri, 3 May 2013 01:47:17 +0000

On May 2, 2013, at 6:58 PM, Dave Crocker <[email-masked]> wrote:
>     The RFC then and now takes around 100 days (with lots of variation between the then and the now, of course.)

Bear in mind that one of the delays that can occur and is credited to the RFC editor is author delays in AUTH48; I think another is document dependencies: a document that has passed IESG review may wait indefinitely in the RFC editor queue until the documents it depends on are published.

Jari, does your graph include that time?

---

From: Jari Arkko <[email-masked]>
Date: Thu, 2 May 2013 22:14:09 -0400

Ted,

> Jari, does your graph include that time?

Yes - RFC editor time includes AUTH48 time, just as the IESG time includes waiting for revised ID from the authors.

In any case, please do not look at the numbers in too much detail. They haven't really been validated too much, and again, are from an old run. Working to get some better numbers when I have spare time…

Jari

---

From: Jouni Korhonen <[email-masked]>
Date: Fri, 3 May 2013 14:11:20 +0300

Jari,

This was an interesting (and a needed) writeup. I also want to share
my view as an IETF newbie who has had a chance to experience IETF
document process a few times. Sorry for chiming in late..

For the most part I got the feeling that we have the right tools and
a working process already in place. It is mostly about how we treat
the process and adjust our habits to it.

The thing I dislike is when you ship a camel into IESG and a horse
comes out.. and everybody is like "what happened?" ;-) My approach
would be simple. If a document gets x DISCUSSes out of y or even a
single DISCUSS would substantially change the technical solution of
the document, ship it back to the WG - always, no questions asked. The
document is not ready and we are wasting IESG's time. The technical
work belongs to the WG. Obviously this begs for a much higher
threshold for an AD to give a document a DISCUSS instead of a COMMENT.

Then about the habits. From my limited experience, folks are used to
the think "the document will anyway be reviewed, reworked and can be
finished in the IESG" and that occasionally shows. I want to believe
this habit changes if the document must be ready for real before
leaving the WG.

The cross area directorate reviews are great and should have even
more weight than they have today. And here is what I would like to
see to change slightly. The directorate & expert reviews should all
be done before sending the document out of the WG into the IESG.
Probably the IETF LC should also happen before sending the document
out of the WG.

My Z$0.02,
	Jouni

---

From: Thomas Narten <[email-masked]>
Date: Fri, 03 May 2013 08:59:04 -0400

Just a few points...

> result), there is an equal amount of "wait for the RFC to come out".

I suspect there is a bit of rose colored reminiscing of history.

The world has changed, significantly.

For example, there has been massive consolidation in industry. There
simply are fewer entities doing implementations today. 15 years ago,
it was common to see a half dozen or more implementations from
different (often smaller) entities, even before a document got out of
a WG. Nowadays, the cost of doing an implementation (for a company) is
in some sense much higher, and business realities make companies
*very* selective in what they implement and put into a product.

I suspect that the notion that if the IETF published documents more
quickly industry would implement them more quickly/frequently is
wishful thinking.

>    existing implementations... BUT that doesn't mean that we can't

Like it or not, 2 and 3 are contradictory. From an interoperability
perspective, the difference between "not change significantly" and
"change a lot" is irrelevant once you have deployments. Change (in the
behavior or wire format) is change and breaks interoperability, no
matter how big or small.

Hannes Tschofenig <[email-masked]> writes:

> b) There is no interest to research where delay really happen.

I don't think that is true. Jari has pointed to his work. I think
there is actually quite a lot of understanding of where the delays
are. But fixing them is really really hard. Blaming the "tail end" or
"the IESG" for the problems has always been the easy target. The
reality (IMO) is that the place where improvements are needed is
within the WG and on having authors be more responsive. But history
suggests there are no easy fixes here.

> seems pretty reasonable to me.  from personal experience, the iesg and

+1

Like everyone, I wish things moved more quickly, but every attempt
I've ever seen that tries to speed things up ends up reducing the
quality or having some other undesirable side effect.

The bottom line is that getting a good spec requires iteration, and
reviews from a broad set of parties. That is best done
*sequentially*. And given the limited cycles the (volunteer) community
as a whole has, you can't easily change these dynamics. We've seen
many attempts to reduce the overall time by trying to overlap reviews
and steps, but that has the side effect of losing *sequential*
*iteration* (where each new review reviews the previous set of
additions/changes). IMO, overlapping steps is dangerous and leads to
poor quality.

> that is purely subjective.

If you want to speed up the process, focus on how to *increase* the
amount of iteration you get (i.e., revised drafts) while at the same
time *reducing* the time between revised drafts.

If you look at the delays documents encounter (both in WG and in IESG
review), the killer is long times between document revisions. Focus on
understanding the *why* behind that and what steps could be taken to
make improvements.

And finally, a big reason the IESG review is where things happen is
that it *really* is the last time one has to verify that a document is
ready to go. With the limited cycles we all have, there will always be
a tendancy to not deal with a document until the point of "this is
your last chance". Nothing like a *real* deadline to motivate people
to finally deal with a document, before it's too late.

Thomas

---

From: Stephen Farrell <[email-masked]>
Date: Fri, 03 May 2013 14:27:53 +0100

> understanding the *why* behind that and what steps could be taken to

Good point. I guess the obvious answers are "not enough
cycles" and, for newer authors, uncertainty about how to
get stuff done, but are there other less obvious answers?
(Input here might really help the IESG discussion btw since
in the nature of things, we're less likely to realise what
newer or less frequent participants find problematic.)

S.

---

From: Thomas Narten <[email-masked]>
Date: Fri, 03 May 2013 10:13:58 -0400

> newer or less frequent participants find problematic.)

One possible step is to have WG Chairs be *managers*, like they are
supposed to be. That means the equivalent of having project plans. For
a given document, come up with a (reasonable, pragmatic, workable)
schedule for getting reviews, discussion, revision, repeat. And then
get committments from parties (authors, reviewers, etc.) to
deliver. And followup if they don't...

The current cycle too often seems to be more like "new version
posted". Wait if anyone reviews. Some reviews eventually, maybe. Oh,
IETF meeting coming, time for a revision. But with meeting
approaching, there are a zillion docs and cycles are limited.  Rather
haphazard, with too many documents effectively only being revised once
per meeting cycle.

WGs make the most progress when authors respond quickly (within days)
to reviews, and do so on the list discussing possible text
revisions. You get much quicker and substantive discussion that way
and it becomes clear whether folk are converging.

Contrast that with "ask for reviews". Wait a month or two. New
document appears that says "this reflects comments we got, go
read". Delays between review/response tend to feed upon
themselves. Folk forget context, etc., making it more of an effort to
go back and remember their reviews, etc.

Too much of WG activites are "best effort" where volunteers (who are
busy and may not actually have the experience to do things optimally)
are expected magically to just know how things are supposed to work
make progress. Reality is a bit different.

Two concrete suggestions:

1) have WGs do the managing role more proactively
2) mentor authors and others a bit more to encourage them how best to
operate

Thomas

---

From: "Adrian Farrel" <[email-masked]>
Date: Fri, 3 May 2013 15:20:16 +0100

Well said, Thomas. 

> 2) mentor authors and others a bit more to encourage them how best to

Which I suspect means...

0) have ADs manage/mentor the WG chairs more proactively.

Almost certainly a case of "if I had less work to do I would have more time to
do the things that would reduce my workload."  :-)

WG chairs might like to comment, but I suspect that one lunchtime training
session every four months does not constitute proactive management.

Adrian

---

From: Ray Pelletier <[email-masked]>
Date: Fri, 3 May 2013 10:29:22 -0400

>> 1) have WGs do the managing role more proactively

Provide WG Chairs the monitoring tools they need to be proactive - Action Tracker, what do I need to do today data tracker  views.  Same for AD. 

Same for authors and their mentors, if any. 

Ray

---

From: Thomas Narten <[email-masked]>
Date: Fri, 03 May 2013 10:32:50 -0400

> 0) have ADs manage/mentor the WG chairs more proactively.

Absolutely. But that is why the nomcom is supposed to select ADs that
have *management* capabilities as well as technical skills.

> Almost certainly a case of "if I had less work to do I would have
> more time to do the things that would reduce my workload."  :-)

Well, this is why you get paid the big bucks.

Not to put too fine a point on in, but ADs need to manage their time,
and they need to balance between the immediate and the long term. They
have to do *both*, and especially NOT neglect longer-term stuff that
will have real and signficant, but not immediate benefits.

> session every four months does not constitute proactive management.

This would help and could be part of a comprehensive approach. But ADs
also need to lay down expectations on WG chairs, and *grade* and
*evaluate* WG Chair performance. WG chairs also don't have
cycles... And ADs seem reluctant to step in and make management
changes that really often are needed to change the dynamices within
WGs that are not delivering...

Thomas

---

From: Dave Crocker <[email-masked]>
Date: Fri, 03 May 2013 07:37:04 -0700

> Same for authors and their mentors, if any.

Probably none of the above.

The IETF already provides pretty good tools.  They could always be 
better, of course, but they almost certainly aren't essential.

The issue with providing management assistance is to focus on managing 
the work.  That's an organizational orientation, not just a tracking 
thing.  It's about getting clarity of the work to be done and of getting 
folks to do the work of contributing, writing, reviewing, and debating 
in a timely manner, and achieving forward progress in a timely manner.

Few working groups have enough detail to juggle to make this something 
that hinges on the tools.

d/

---

From: Elwyn Davies <[email-masked]>
Date: Fri, 03 May 2013 15:51:13 +0100

On Fri, 2013-05-03 at 14:27 +0100, Stephen Farrell wrote:
> cycles" 
and not enough "concentrated cycles" - designing and documenting a
protocol in my experience requires a longish period of undisturbed
concentration by the person with the editing crayon, and responsive,
deep and immediate review by co-authors to generate fast turn round
reviews of versions.
Its hard enough keeping the in-brain implementation of the protocol
running accurately day-by-day.  If there are gaps of weeks rebooting the
implementation wastes precious concentrated time.  Probably putting a
few people with no other commitments in a isolated space generates
fastest results. Maybe we could get ISOC to provide a retreat space (or
one per continent) for protocol designers?
> and, for newer authors, uncertainty about how to
One thing that might help:
We have directorates and review panels.  Can we distil any/more of their
wisdom into checklists/guidance for protocol writers?
Some is already there (not a complete list):
BCP 72 for security considerations 
BCP 41 on congestion control
BCP 61 on Strong Security Requirements
Some stuff in RFC 6385 for gen-art

[Some of this is getting a bit long in the tooth.]

So what do you ...
- think about when designing the security aspects/transport congestion
aspects/... of a new protocol?
- what triggers alarm bells/gold stars when you are reviewing the
general principles/security aspects/congestion/ABNF/state
machines/mib/xml/extensibility/.... of a draft?

Regards,
Elwyn

---

From: Pete Resnick <[email-masked]>
Date: Fri, 3 May 2013 10:12:11 -0500

> will have real and signficant, but not immediate benefits.

Interesting. For the most part, I agree with the above. And I think we 
have been neglecting some of the longer-term stuff. But earlier:

> quality or having some other undesirable side effect.

Can you talk a bit more about this, including some examples?

I'm trying to reconcile the concern that we "end up reducing the 
quality" and  that we "NOT neglect longer-term stuff that will have real 
and significant, but not immediate benefits." ADs have finite resources 
and can't accomplish both immediate and longer-term stuff maximally all 
of the time. My inclination would be to say, "If they are in conflict, 
weight the longer-term over the immediate, even if that might result in 
some reducing of quality on any single document." Am I over-reading your 
earlier message to say that you think the weighting should go toward the 
immediate? I think we agree that doing the longer-term stuff will end up 
with increased quality over the long term, but doing more of the 
longer-term stuff is sure to reduce the amount of time spent on 
immediate, which means document reviews and therefore some short term 
lowering of quality. Is that OK, or is that an example of the kind of 
thing we've attempted that ends up with bad results?

pr

---

From: "Adrian Farrel" <[email-masked]>
Date: Fri, 3 May 2013 16:59:15 +0100

> Provide WG Chairs the monitoring tools they need to be proactive - Action
> Tracker, what do I need to do today data tracker  views.  Same for AD.
> 

Wouldn't work for me. YMMV.

Adrian

---

From: John Leslie <[email-masked]>
Date: Fri, 3 May 2013 12:52:26 -0400

> better, of course, but they almost certainly aren't essential.

   That much is probably true.

> in a timely manner, and achieving forward progress in a timely manner.

   That much is true.

> that hinges on the tools.

   That, alas, is not true.

   I subscribe to a number of WG lists where they find a issue tracker
_quite_ essential to their work. And to most folks, the issue tracker
output is more confusing than helpful: you're never quite sure which
issue number something belongs under.

   The lists I subscribe to have as work items drafts where nothing
happens until IETF-week deadlines (and sometimes not even then!).

   It seems _very_ likely that some automated tools to point out the
inactivity would help...

---

From: Randy Bush <[email-masked]>
Date: Fri, 03 May 2013 09:08:30 -0700

> supposed to be.
> ...
> IETF meeting coming, time for a revision.

with wg chairs taking multiple months to do the simplest of button
pushes, authors slow down so that they get the most out of that oh so
rare button push.

randy

---

From: Thomas Narten <[email-masked]>
Date: Fri, 03 May 2013 13:14:36 -0400

Ray Pelletier <[email-masked]> writes:

> Provide WG Chairs the monitoring tools they need to be proactive -
>  Action Tracker, what do I need to do today data tracker views.
>  Same for AD.

> Same for authors and their mentors, if any.

Tools may help in some cases (IESG, chairs), but the fundamental
problem is not tools. It's about work culture, setting proper
expectations (and communicating them to everyone), and having
accountability and followup to fix things when stuff isn't happening.

Thomas

---

From: t.p. <[email-masked]>
Date: Fri, 3 May 2013 18:14:59 +0100

----- Original Message -----
From: "Ray Pelletier" <[email-masked]>
To: <[email-masked]>
Cc: "Thomas Narten" <[email-masked]>; "IETF list" <[email-masked]>
Sent: Friday, May 03, 2013 3:29 PM

Provide WG Chairs the monitoring tools they need to be proactive -
Action Tracker, what do I need to do today data tracker  views.  Same
for AD.

<tp>

Such as a list once a month, sent to the list of each active WG, of the
state change, if any, in the I-Ds which have been accepted by the WG as
work items, giving the date at which the last state change occurred.
State is not just the IESG type tracker state, but also a new revision
of the I-D.

I know how useful I find this from the WG Chairs who produce it already.

Tom Petch

Ray

---

From: Thomas Narten <[email-masked]>
Date: Fri, 03 May 2013 13:29:22 -0400

Dave Crocker <[email-masked]> writes:

> The issue with providing management assistance is to focus on
> managing the work.  That's an organizational orientation, not just a
> tracking thing.  It's about getting clarity of the work to be done
> and of getting folks to do the work of contributing, writing,
> reviewing, and debating in a timely manner, and achieving forward
> progress in a timely manner.

> Few working groups have enough detail to juggle to make this
> something that hinges on the tools.

Strongly agree here.

Let me expand further on "work plan" and  "project management".

It needs to be done on a per-document basis. I.e., Wouldn't it be nice
if every WG document was revised 3 times between meetings? I bet that
alone would change dynamics (assuming there were substantive
changes)...

But it extends to the WG as a whole.  WGs have a finite number of
cycles. There are always a small number of core players (whether
editor, reviewer, etc.), and if you spread their resources too thinly,
a WG starts having problems.

Few WGs acknowledge that and try to manage it. Managing it means:

1) identifying core documents that are priority, and minimizing
distractions that will take cycles away from the priority
deliverables.

2) It may mean making concious decisions not to work on some documente
*yet* in order to concentrate resources where it matters

3) it means tying the work plan directly back to the charter. If the
two aren't in sync, one or both need changing.

4) It means recognizing that there are real limits to how many
documents a WG can (credibly) work on at any one time. (Too many WGs
have lots of marginal documents that suck cycles away from the
critical ones.) How many WGs say "no" to marginal documents?

Thomas

---

From: Michael Richardson <[email-masked]>
Date: Fri, 03 May 2013 13:31:20 -0400

>>>>> "Thomas" == Thomas Narten <[email-masked]> writes:
    Thomas> The current cycle too often seems to be more like "new
    Thomas> version posted". Wait if anyone reviews. Some reviews
    Thomas> eventually, maybe. Oh, IETF meeting coming, time for a
    Thomas> revision. But with meeting approaching, there are a zillion
    Thomas> docs and cycles are limited.  Rather haphazard, with too
    Thomas> many documents effectively only being revised once per
    Thomas> meeting cycle.

I would like to suggest that the IESG Review be done in public on the WG
mailing list.    I've been a WG co-chair for just over a year now, and
I'm truly astounded by what happens behind the scenes.  

It's not the substance, it's the quantity, and the lack of WG view of
it.   I think that this substantially and quite negatively contributes
to the "fix it during IESG review", and therefore to the IESG workload.

---

From: "Tony Hain" <[email-masked]>
Date: Fri, 3 May 2013 10:41:21 -0700

Pete Resnick wrote:
...
> I'm trying to reconcile the concern that we "end up reducing the quality"
and
> that we "NOT neglect longer-term stuff that will have real and
significant, but
> not immediate benefits." ADs have finite resources and can't accomplish
> both immediate and longer-term stuff maximally all of the time. My
> inclination would be to say, "If they are in conflict, weight the
longer-term
> over the immediate, even if that might result in some reducing of quality
on
> any single document." Am I over-reading your earlier message to say that
> you think the weighting should go toward the immediate? I think we agree
> that doing the longer-term stuff will end up with increased quality over
the
> long term, but doing more of the longer-term stuff is sure to reduce the
> amount of time spent on immediate, which means document reviews and
> therefore some short term lowering of quality. Is that OK, or is that an
> example of the kind of thing we've attempted that ends up with bad
results?

Pete,

I would agree with you that weighting longer term is the 'right thing', but
given that people wait for an RFC number to implement, and then take the
position that RFC (PS) == STD, you never get to the longer term because the
bar is set so high on the first hurdle that it becomes impossible for people
to justify the effort to go for the next one.

When the I-D series was introduced specifically to deal with the issue of
RFC == STD, we didn't get it right. In hind sight we should have done
something more like the IEEE does with 802.11, and explicitly put a big
DRAFT: in the title. Industry doesn't seem to get confused about DRAFT: N or
DRAFT: AC, and still deploys, then moves on with full standards. We could
still do that, and likely get motivation to move a DRAFT: to PS (remove the
DRAFT: from the title) by wider review and deployment. I realize this is
really doing nothing more than introducing an extra step in the process, but
in some ways it is more of a formalization of the expected behavior at the
first RFC step. One could go so far as to not remove DRAFT: until the spec
met DS, and move the default  from PS to DS at the same time as removing the
extra step from the process. One could really get crazy and just remove DS
from the list, because nobody understands what that one is for anyway, and
then when the DRAFT: gets removed from the title it is really the STD people
are expecting ... ;)

In any case I don't see 60/40 as a tail-heavy process. If the WG time were
cut in half and the IESG/RFC-editor time stayed the same, maybe it would be
tail-heavy, but realistically more 'balanced'.  At the end of the day, the
only way to reduce the time at IESG review is to have a document with clear
objectives up front that the document can be evaluated against, and for the
WG to put a quality document in their hands. Too often the WG takes so long
that the objectives drift, if they were even clearly stated up front, and
comments like "you have to have been involved in the mail thread" show up
during review. If something is not clear in the document, and it was cleared
up in a meeting or on the mail list, that clarification needs to get moved
into the document or subsequent reviewers / implementers will never get it,
and the result is delay and/or poor quality documents. 

Tony

---

From: Ralph Droms <[email-masked]>
Date: Fri, 3 May 2013 13:43:36 -0400

> *very* selective in what they implement and put into a product.

Adding a little detail - those implementations have to be prototypes, with the implementing entities willing to make changes in the code prior to shipping product. Once the code has been shipped to customers, there is a lot of resistance to making changes to the specification.  

> behavior or wire format) is change and breaks interoperability, no

Right, so declaring a document a "fully baked ID" might have the perverse effect of freezing specifications at a point where there are still problems.

The key issues are the investment in an implementation, the cost of updating an implementation and, perhaps most importantly, the cost of making changes to code that has been shipped in products.

>> seems pretty reasonable to me.  from personal experience, the iesg and

It would be interesting to understand the cost of that useful value, and whether the same useful value could be provided elsewhere at reduced cost.

"Cost" in this case can be tricky - the useful value in IESG review comes at the cost of asking the members of the IESG to invest 10s of hours per week in careful review of every document.  That time requirement contributes to the notion that being an AD is a full time job, which in return reduces the diversity of the pool of candidates willing to volunteer for an AD position.

So, there is useful value, but can we get that same value by distributing the load out to the larger community?

> understanding the *why* behind that and what steps could be taken to

This claim sounds right to me.  Can we document it?

The long delays, I'll bet, can be a result of a feedback loop - a long delay before a new revision is might tend to increase the number of changes in the new rev and the details of the draft have dropped out of short-term memory for those who are following the draft.  So, reviewing the new rev becomes lower priority and there's a longer delay before comments are published.  In general, the draft seems less important and gets a lower priority...
> to finally deal with a document, before it's too late.

And, I'll suggest there is a lack of accountability toward everyone who was supposed to have contributed to a document that is ready to publish when it gets to the IESG .

Suppose, for a moment, the IESG approved a document for publication with a serious technical flaw?  Who would be accountable?

A large percentage of documents reviewed by the IESG receive at least one DISCUSS position.  What are the consequences to the document authors, the WG and the IETF?

- Ralph

---

From: Pete Resnick <[email-masked]>
Date: Fri, 3 May 2013 13:04:25 -0500

Not disagreeing with your message, but a couple of clarifying points:

> to justify the effort to go for the next one.

When Thomas said "longer-term", I took him to mean *all* of the things 
that are longer time horizon, like management/mentoring of WGs and 
chairs, doing more early cross-area participation (including by ADs) in 
the design of and work on protocols (not just review), etc. In 
particular, I didn't understand him to mean longer-term as having 
anything to do with advancement on the standards track. I think the 
above longer-term goals are important. I think movement along the 
standards track will be the outcome if we get to working well, but not a 
goal in-and-of itself.

> tail-heavy, but realistically more 'balanced'.
>    

I'll point you to Jari's excellent reminder that "time" is probably the 
least of the heaviness on the tail end. What is tail heavy is the work 
investment: We do WG last call reviews, IETF-wide Last Call reviews, 
directorate reviews, IESG reviews, all at the end of the process. We 
spend scant little energy (of the wider IETF community and leadership) 
on early design reviews and protocol development early and in the middle 
of the document process. As Jari said, that model ends us up with late 
surprises, lack of transparency, etc., and probably uses much more 
energy overall than if we invested it earlier in the process. Sure, this 
can translate to delays, but that's only one (maybe minor) effect of 
being tail-heavy.

pr

---

From: "Tony Hain" <[email-masked]>
Date: Fri, 3 May 2013 12:03:22 -0700

Pete Resnick wrote:
> > I would agree with you that weighting longer term is the 'right
> > thing', but given that people wait for an RFC number to implement, and
> > then take the position that RFC (PS) == STD, you never get to the
> > longer term because the bar is set so high on the first hurdle that it
> > becomes impossible for people to justify the effort to go for the next
one.
> When Thomas said "longer-term", I took him to mean *all* of the things
that
> are longer time horizon, like management/mentoring of WGs and chairs,
> doing more early cross-area participation (including by ADs) in the design
of
> and work on protocols (not just review), etc. In particular, I didn't
understand
> him to mean longer-term as having anything to do with advancement on the
> standards track. I think the above longer-term goals are important. I
think
> movement along the standards track will be the outcome if we get to
> working well, but not a goal in-and-of itself.

I actually view all of the housekeeping activities as short term. Thomas
will have to state what he meant.

> 
> > In any case I don't see 60/40 as a tail-heavy process. If the WG time
> > were cut in half and the IESG/RFC-editor time stayed the same, maybe
> > it would be tail-heavy, but realistically more 'balanced'.
> >
> 
least of
> the heaviness on the tail end. What is tail heavy is the work
> directorate reviews, IESG reviews, all at the end of the process. We spend
> scant little energy (of the wider IETF community and leadership) on early
> design reviews and protocol development early and in the middle of the
> document process. As Jari said, that model ends us up with late surprises,
> lack of transparency, etc., and probably uses much more energy overall
than
> if we invested it earlier in the process. Sure, this can translate to
delays, but
> that's only one (maybe minor) effect of being tail-heavy.

While I agree that it would appear to be a heavy workload, that is just
because it is concentrated enough to see. If you actually did all the work
across all the participants to review every document at every iteration,
nothing would get done. Even when the IETF was ~60 people creating a few
documents a year, reviews happened late, and changes were made. Yes earlier
comments from outside the WG will reduce overall surprise and workload, but
finding the right reviewers is tricky.

One could require that every BOF be presented to every Open Area meeting,
then required public comment from every other AD before being allowed to
form a WG. This would front load the review and raise general awareness
about what is being discussed. Unfortunately since that event and resulting
documents are half a decade apart, potential reviewers have long since
forgotten any concerns they might have had, and need to start fresh anyway.
For long standing WGs one could consider doing the same at every charter
change, but again, document movement within the WG needs to be faster for
any of that to matter. 

The only way I can see to reduce the time from BOF to RFC is to reduce the
public expectation on the quality of that initial document which will get
wide review and initial implementation.  We should be talking about ~ 3
years to get to STD, not the initial draft that gets review beyond the WG.
In practice what we have today is functionally STD at initial publication,
so there is a very concentrated effort at one point in time. If the process
to get from WG I-D to initial RFC was basically a sanity-check, the detailed
IESG and cross-area review could be spread out during implementation prior
to the update that removed the DRAFT: status. Realistically though, human
nature would take over and nothing would happen until the WG tried to take
that step and we would be having the same discussion about tail work load. 

Bottom line, we need a signal to other areas that a WG document is ready for
serious in depth review, and a document title change seems to be what people
are looking for. At the same time WGs need to do a better job of spreading
awareness earlier in the process to get comments from those that have
different perspectives, then actually listen to them during document
development. Surprise comes from being isolated, and/or shouting down voices
that later come back during review to highlight flaws. If the process to get
to DRAFT: status was less burdensome, documents would move more quickly and
less effort would have gone into the flawed document before it got seriously
reviewed. 

Tony

---

From: Thomas Narten <[email-masked]>
Date: Fri, 03 May 2013 15:40:20 -0400

> Can you talk a bit more about this, including some examples?

Sorry, what I had in mind here was what I'll call process
tweaks. Attempts at changing the process to speed reviews, dispense
with reviews, overlap reviews, etc.

No, our processes aren't perfect. But we have bigger problems that are
not due to process that we'd do better to focus on rather than trying
to tweak the process in (IMO) minor ways. We should focussing more on
*root* *causes* and what concrete, pragmatic steps can be taken to
mitigate them.

> quality"

Again, I was thinking of proposals for process (or operational)
changes that attempt to shorten things by overlapping or removing
reviews or steps, trying to get the IESG to do less review (e.g., "ADs
should only review looking at topics that are defined by their area")
and otherwise focus on the *process* as being the problem.

> and  that we "NOT neglect longer-term stuff that will have real 
> immediate?

You have it right. And I meant "longer-term stuff" in a general
sense. There are many topics that are not immediate, but could be
important long term.

Spend enough time on long term so that it is not neglected. You have
to do both. But you must also MUST do long term stuff.

> I think we agree that doing the longer-term stuff will end up 

It's easy to fall back to the comfort zone, which is doing the
bi-weekly document reviews and such. I.e., the day-to-day operational
stuff.

What's more important to the IETF? Spending time on one questionable
(but quite possibly irrelevant) document to make it better before it gets
published? Or getting the IETF as a whole to work better and more
efficiently and produce better product?

Thomas

---

From: Barry Leiba <[email-masked]>
Date: Fri, 3 May 2013 17:39:37 -0400

> to the "fix it during IESG review", and therefore to the IESG workload.

Pete and I suggested this to the App chairs in Orlando (see here:
http://www.ietf.org/proceedings/86/slides/slides-86-appsawg-4.pdf ),
and we are currently trying it out with draft-ietf-core-coap, which is
scheduled for the 16 May IESG telechat (see
http://datatracker.ietf.org/doc/draft-ietf-core-coap/ and note that
I've set the "Send notices to" field to include the working group
mailing list).

It appears to be working very well, and I'd like to try it on more
documents in more working groups.  Perhaps some day it will be the
normal way to operate; we'll have to see.

Barry, Applications AD

---

From: Hector Santos <[email-masked]>
Date: Fri, 03 May 2013 18:10:29 -0400

May I suggest that the IETF produce a web site for gathering IETF 
participants profiles and even resumes.  It can have a questionnaire to 
extract the valuable information that will help maximize the IETF/IESG 
efforts.  The "Business Information" (BI) can then be leveraged by ADs, 
WGs (and other IETF/IESG leaders) to recruits/invite participants for 
various projects, reviews, etc.  It will have many win-wins for all.  We 
should consider there are as much, and perhaps even more IETF man-years 
in people that never attended meetings but were active for many years, 
nonetheless, in their various IETF protocol interest areas.

The structure of this questionnaire will be important to be successful 
and beneficial.

Sincerely,

Hector Santos

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 04 May 2013 00:49:23 +0200

+1
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Hannes Tschofenig <[email-masked]>
Date: Sun, 05 May 2013 12:12:08 +0300

> suggests there are no easy fixes here.

The type of data I am looking for is different to what Jari provides.
I want to know what the real reasons are, which requires you to look at 
the individual documents and their history. Ultimately there are people 
working on these documents and each individual topic has it's own 
challenges contributing to delay, quality, etc.

Regarding your statement about the lower number of implementations it 
really depends what technology you look at. There are differences in 
terms of layers (with lower layer protocols you are less likely to see 
lots of implementations than with high layer protocols), and the 
dependencies a specific technology has (a feature that depends on 10 
other things to be implemented and deployed will see fewer 
implementations).

An example: in JOSE and in OAuth you see a larger number of 
implementations already at an early stage of the document work. How many 
of those will then ultimately get updated to the published RFCs remains 
to be seen.

---

From: Hannes Tschofenig <[email-masked]>
Date: Sun, 05 May 2013 12:23:27 +0300

Thomas, this all sounds very easy and reasonable. Unfortunately, it does 
not work that way (as you might recall from your own WG chair experience).

Both chairs and WG participants are very busy and so they do not 
follow-up on the tasks they had agreed on earlier.

Since everything is done on a volunteer basis you cannot force anyone to 
do something. Additionally, there are lots of unexpected developments 
happening during the WG time.

Just look at the efforts on getting slides for WG sessions ahead of time 
and meeting minutes soon after the meeting.

Ciao
Hannes

PS: I personally found regular conference calls quite helpful to talk to 
folks and to check the current status.

---

From: Hannes Tschofenig <[email-masked]>
Date: Sun, 05 May 2013 12:23:46 +0300

On 05/03/2013 01:25 AM, Jari Arkko wrote:
> However, I did want to point out that when I said "tail-heavy", I
> did*not*  necessarily mean delay. I meant that a lot of activity is
> going on, many document changes, and much review is going on.
> Obviously in some cases this translates to delay as well.

In my own document writing experience I agree that there have been a lot 
of document changes because you have to address the review comments. The 
question is (and I wouldn't want to answer that without surveying 
various documents) what difference these changes make in terms of 
interoperability and easier deployment.

> But the delay was really not my main concern. Primarily because I
> think other issues such as transparency to the working group or late
> surprises are more fundamental issues than mere timing.

While the DISCUSSes are available in the tracker most working group 
participants do not pay attention to the tracker itself but they rather 
follow the mailing list. Unless someone (e.g., draft author or WG 
chairs) forward the comments to the list they do not get a lot of 
attention.

For example, I am active in the GEOPRIV working group and I did not 
realize that the technical solution for conveying location in DHCP was 
re-written during the IESG review phase. Maybe I missed the notes to the 
mailing list from the draft authors but I just didn't realize it. It was 
a surprise to me.

I personally would prefer to have all the discusses also sent to the 
working group automatically.

---

From: Hannes Tschofenig <[email-masked]>
Date: Sun, 05 May 2013 12:27:16 +0300

>     That much is probably true.

I would have always hoped to have a tools where I can register the 
persons who said that they do a review and then get them reminded 
automatically.

> issue number something belongs under.

I got the same impression. It may work fine for software projects...
> happens until IETF-week deadlines (and sometimes not even then!).

Yup.

---

From: joel jaeggli <[email-masked]>
Date: Sat, 04 May 2013 14:26:41 -0700

> However, that is a bit of a problem, because I think it's fairly rare for documents to get additional "review" at last call time.   Changing the name probably won't fix that.
It feels like unless something is particularly controversial that the 
likelihood of it getting useful review other than solicited ones during 
last call has actually gone down. If it's fully cooked, and sometime it 
seems that they aren't, that lack of commentary does not indicate 
consensus  or readiness in particular.

---

From: Benoit Claise <[email-masked]>
Date: Sun, 05 May 2013 12:37:43 +0200

> That is a matter of naming and marketing ("candidate RFC"?).
There is already some misconception between I-RFC and Standards Tracks RFC.
I don't believe that adding a new name/category would help: instead it 
would add to the confusion.

Regards, Benoit

---

From: Ted Lemon <[email-masked]>
Date: Sun, 5 May 2013 10:44:41 +0000

On May 4, 2013, at 10:26 PM, joel jaeggli <[email-masked]> wrote:
> It feels like unless something is particularly controversial that the likelihood of it getting useful review other than solicited ones during last call has actually gone down. If it's fully cooked, and sometime it seems that they aren't, that lack of commentary does not indicate consensus  or readiness in particular.

By "last call time" I was referring to IETF last call, not WGLC.   I am painfully familiar with "we'll do a WGLC to see if we can get some review."

---

From: Benoit Claise <[email-masked]>
Date: Sun, 05 May 2013 12:50:17 +0200

Thomas,
> deliver. And followup if they don't...
Replace WG chairs by document shepherd in your email, and I would say "+ 
1" to your entire email.

Regards, Benoit

---

From: Benoit Claise <[email-masked]>
Date: Sun, 05 May 2013 13:47:22 +0200

> * the burden of the review, managing the discussion, making sure any changes fix the problem and don't break anything else often falls on the IESG and even a single AD
When the write-up is done, the WG perception is that their work is done.
Some reviews that come after that, and specifically the IESG ones, are 
considered as coming from trouble-makers.
We should change that perception.
I'm all in favor to keep the WG involved for any reviews: directorate, 
IETF LC, IESG. Currently, we lack consistency wrt where the feedback is 
sent (we're working on that). Now, I feel it's going to imply more 
follow-up work for the ADs as the discussions might take longer. This is 
where the document shepherd could help.

Regards, Benoit

---

From: Hannes Tschofenig <[email-masked]>
Date: Sun, 05 May 2013 14:54:48 +0300

On 05/05/2013 01:37 PM, Benoit Claise wrote:
>>> Yeah, all kinds of issues, but if we created a new thing here in
>>> between WGLC and PS, the broader industry would never understand.

Interesting that you mention this.

A note from a recent experience: Together with Olaf we are participating 
in a European Multistakeholder Platform for ICT standardization, see 
http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:C:2011:349:0004:0006:EN:PDF

The aim of this group is to find out how to reference IETF RFC (and 
standards from other organizations, like the W3C) since the European 
Commission seems to be unable to just reference standards beyond a small 
set of organizations (such as ETSI).

As you can imagine, the different types of RFCs are not that easy to 
understand for those who do not participate actively in the IETF. 
Getting others to understand the different streams, the various document 
types and different standards is already difficult and maybe there is 
room for simplification here.

---

From: Hannes Tschofenig <[email-masked]>
Date: Sun, 05 May 2013 15:00:01 +0300

On 05/05/2013 02:47 PM, Benoit Claise wrote:
>> * significant review and modification takes place in IESG review,
>> after the WG and the IETF have declared the document done
>> * the burden of the review, managing the discussion, making sure any
>> changes fix the problem and don't break anything else often falls on
>> the IESG and even a single AD

Benoit,

while it is desirable to get wider reviews happen earlier in the process 
there is obviously a challenge: You don't want to ask for reviews before 
the document is stable and you cannot ask many times since good reviews 
are "expensive".

As you indicate there is the perception that the document work is 
finished when the document gets sent to the IESG (maybe because the 
discussions that happen afterwards are less visible).

One approach is to increase visibility of the IESG discussions (and the 
feedback from various review teams) to the working group and the other 
approach is to get AD feedback earlier to the working group.

Ciao
Hannes

---

From: Benoit Claise <[email-masked]>
Date: Sun, 05 May 2013 14:17:46 +0200

Hi Jouni,
> document is not ready and we are wasting IESG's time.
Agreed. This is something that the IESG should be more often.
> The technical
Agreed.
This is a practical solution for the previous email I sent:

    When the write-up is done, the WG perception is that their work is
    done.
    Some reviews that come after that, and specifically the IESG ones,
    are considered as coming from trouble-makers.
    We should change that perception.
    I'm all in favor to keep the WG involved for any reviews:
    directorate, IETF LC, IESG. Currently, we lack consistency wrt where
    the feedback is sent (we're working on that). Now, I feel it's going
    to imply more follow-up work for the ADs as the discussions might
    take longer. This is where the document shepherd could help. 

Regards, Benoit

---

From: Scott Brim <[email-masked]>
Date: Sun, 5 May 2013 08:38:56 -0400

> the document is stable and you cannot ask many times since good reviews

There have been several attempts at defining "snapshot" review points,
but afaict working groups rarely feel like a draft is ready for review
until they think it's finished.

---

From: John C Klensin <[email-masked]>
Date: Sun, 05 May 2013 10:34:11 -0400

--On Friday, May 03, 2013 18:10 -0400 Hector Santos
<[email-masked]> wrote:

> May I suggest that the IETF produce a web site for gathering
> IETF participants profiles and even resumes.  It can have a
> questionnaire to extract the valuable information that will
> help maximize the IETF/IESG efforts.  The "Business
> Information" (BI) can then be leveraged by ADs, WGs (and other
> IETF/IESG leaders) to recruits/invite participants for various
> projects, reviews, etc.  It will have many win-wins for all.
> We should consider there are as much, and perhaps even more
> IETF man-years in people that never attended meetings but were
> active for many years, nonetheless, in their various IETF
> protocol interest areas.
> 
> The structure of this questionnaire will be important to be
> successful and beneficial.

Hector,

I can see the advantages of statements of interests and maybe
qualifications.   Moving beyond that to profiles and especially
resumes would give me fairly serious concerns about privacy for
those who supply the information and about inappropriate
pressures (even if unintentional) on those who do not.   There
might even be some analogies to the reasons we keep information
flow to the Nomcom confidential.  There would be clear
advantages to not doing so but, so far, the community has
concluded that, on balance, confidentiality is the wiser course.

best,
   john

---

From: John C Klensin <[email-masked]>
Date: Sun, 05 May 2013 17:23:24 -0400

--On Friday, May 03, 2013 13:29 -0400 Thomas Narten
<[email-masked]> wrote:

> But it extends to the WG as a whole.  WGs have a finite number
> of cycles. [...] if you spread their
> resources too thinly, a WG starts having problems.
> 
> 2) It may mean making concious decisions not to work on some
> documente *yet* in order to concentrate resources where it
> matters
> 
> 3) it means tying the work plan directly back to the charter.
> If the two aren't in sync, one or both need changing.
> 
> documents a WG can (credibly) work on at any one time. (Too
> many WGs have lots of marginal documents that suck cycles away
> from the critical ones.) How many WGs say "no" to marginal
> documents?

If one replaces "WG with AD" and then "document" with "WG", one
sees another part of the problem.  It seems to me that we need
to either:

	* reduce expectations of what a given AD is supposed to
	do with WGs and documents  or
	
	* reduce the number of workload-weighted WGs that an AD
	is expected to "manage".

If we don't do one or the other, this discussion's increased
expectations for mentoring, management, and keeping track of the
big picture and long-term issues are unrealistic: neither
ongoing overload nor lurching from one near-crisis to the next
are a good basis for getting either good management or strategic
thinking done.

SIRS has been mentioned as an old idea that we haven't tried.
But, when one starts looking at AD load, there are other
proposals around too: ones to put limits on the number of WGs
and others to change the review process so that we don't expect
the same people to manage WGs, advocate for them, take the lead
in advocating their output, and then do the evaluations.  There
are reasons to believe that combination would not be ideal under
the best of circumstances but, if we are going to expect more
attention to WG oversight and mentoring, it gets worse.

Along the same lines, it seems to me that we need to be able to
trust WG Chairs to correctly understand and monitor both WG
consensus and agreements reached after Last Call.  If that isn't
realistic, we probably need better-trained WG Chairs or
replacement WG Chairs, not more work for the ADs.  In
particular, as another piece of the end game, I think AUTH48
ought to be able to be managed by WG Chairs and authors, with a
general understanding that substantive issues have to go back to
the WG and no AD involvement except in very exceptional cases.

Finally, there are a few things that we used to do, that were
helpful, and that were abandoned due to industry evolution and
changes in priorities.  The original idea of a Proposed Standard
as a fairly rough specification that would be used for study and
evaluation on the basis of implementation experience, not a spec
from which products were built, is one that has been mentioned
(although not quite in that way).   The other one that occurs to
me --done rarely but to good effect-- was to identify WGs who
are doing design work with broad implications and give them
plenary time to explain their goals, work plans, and general
functional ideas.  Done well and timed correctly, that allows
very early and very broad review by the community independent of
what is committed to writing in particular documents and alerts
to issues that had best be considered and resolved earlier
rather than later.  Perhaps we should review our priorities for
plenary time and try that again.

best,
   john

---

From: Jari Arkko <[email-masked]>
Date: Mon, 6 May 2013 01:50:30 +0100

Hannes,

> The aim of this group is to find out how to reference IETF RFC (and standards from other organizations, like the W3C) since the European Commission seems to be unable to just reference standards beyond a small set of organizations (such as ETSI).
> 
> As you can imagine, the different types of RFCs are not that easy to understand for those who do not participate actively in the IETF. Getting others to understand the different streams, the various document types and different standards is already difficult and maybe there is room for simplification here.
> 

This may be going to a side-track from the purposes of the original discussion, but thank you Hannes and Olaf working the EC to better accept our standards. It is very important work. And I understand it has not been easy at all. If I have understood correctly, there is a process to accept standards (or groups of standards) one at a time, and that we've wanted to push IPv6 as one such specification but despite two years and your heroic efforts, we still do not have acceptance.

#rant on

But I'd separate two different reasons for difficulties of this nature. First, there is some actual confusion. For instance the distinction between different types of RFCs is usually lost on most people outside the IETF. But I'd say it is often mixed (and might be so in this case) with other issues, such as desire to push other types of standards. I really don't know how out of touch you'd have to be to not understand that there are some agreed standards in the Internet. Things that are glaringly obvious to any user who stares at a screen. Like "http". Or IP, if you took even one course in the university about networking technology. Actively ignoring reality?

#rant off

Jari

---

From: Bill McQuillan <[email-masked]>
Date: Mon, 6 May 2013 00:26:17 -0700

On Sun, 2013-05-05, John C Klensin wrote:

> (although not quite in that way).   

FWIW, this leads me to the thought that the IETF may have a 
terminology problem. The word "standard" is used too soon in the 
maturity levels of RFCs.

I think that our skills are primarily in producing "protocols"
rather than "standards". Perhaps it would have been better if the
names of the maturity levels were something like:

  Proposed Protocol
  Test Version 1 Protocol
  Test Version 2 protocol
   .
   .
   .
  Standard Protocol

Only using the word "standard" when it was determined to be 
stable and recommended for wide usage.

Anyway, my 2 cents.

-- 
Bill McQuillan <[email-masked]>

---

From: Olaf Kolkman <[email-masked]>
Date: Mon, 6 May 2013 04:35:04 -0400

On May 5, 2013, at 7:54 AM, Hannes Tschofenig <[email-masked]> wrote:

> A note from a recent experience: Together with Olaf we are participating in a European Multistakeholder Platform for ICT standardization, see http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:C:2011:349:0004:0006:EN:PDF
> 

FWIW.

I think the difference between Informational and Standards track is fairly easily explained (in that context), having all the information in the headers and boilerplates helps. Where things become difficult is at the point where the maintenance of our standards need to be explained and questions about progression on the standards ladder get asked. 

Personally I hope that RFC 6410 has the effect that we, as a community, get serious about promoting our proposed standards, or obsolete them. 

I wonder how many standards got promoted after 6410 was published.

--Olaf

---

From: Andy Bierman <[email-masked]>
Date: Fri, 3 May 2013 11:34:06 -0700

..

> session every four months does not constitute proactive management.
+1 !!!

It works on down the line too.
WG Chairs meeting with I-D editors once every 4 months isn't so great
either.

If the total time has gone up 100 days, and the IESG time has gone down
100 days, then clearly the WG process is the main problem.

I'd like to see some stats on average delta between revisions of an I-D,
with some metric for the number of changes per revision.  I suspect
it would show way too many drafts that have a few edits once every
4 months. (One of mine right now even ;-)

The WG Chairs need more training on how to avoid the "slow tweak mode"
problem. The ADs need to manage their WG chairs so this doesn't happen.

Adrian
Andy

---

From: SM <[email-masked]>
Date: Mon, 06 May 2013 13:28:50 -0700

>session every four months does not constitute proactive management.

One lunch every four months does not look like proactive management. :-)

At 11:34 03-05-2013, Andy Bierman wrote:
>WG Chairs meeting with I-D editors once every 4 months isn't so great either.

Yes.

Yes.

I posted a message over a year ago ( 
http://www.ietf.org/mail-archive/web/urn/current/msg01653.html ).  I 
looked at the WG status page a few minutes ago.  The WG did not 
publish any document.  I looked at the mailing list archive.  There's 
hasn't been a lot of activity.

>problem. The ADs need to manage their WG chairs so this doesn't happen.

The choice is whether to shut down the working group or not.  It was 
suggested (other thread) that WG Chairs have to find at least X 
reviewers [1].  I doubt that anyone is going to pay these reviewers 
[2].  This raises the question of whether the people who showed 
interest in starting the work were a bunch of cheerleaders. :-)

The current mode of operation is:

   (i)   The IESG will fix that

   (ii)  The RFC Editor will fix that

   (iii) More tools will fix that

There isn't any hurdle to try reviews by Senior IETF Reviewers 
(SIRS).  As an experiment the SIRS could commit to post public 
reviews of every draft going through a Working Group Last Call (WGLC) 
until the end of the year.  I assume that a report of the experiment 
will be provided after that so that the IETF community can assess for 
itself whether the experiment was a success.

Regards,
-sm

1. http://www.asterix.com/galerie/smailix/images/i11c.gif
2. http://www.asterix.com/galerie/smailix/images/i8c.gif

---

From: John C Klensin <[email-masked]>
Date: Mon, 06 May 2013 18:55:40 -0400

--On Monday, May 06, 2013 04:35 -0400 Olaf Kolkman
<[email-masked]> wrote:

> Personally I hope that RFC 6410 has the effect that we, as a
> community, get serious about promoting our proposed standards,
> or obsolete them. 
> 

According to my rough count (well, not that rough), the number
of RFCs numbered above 6410 that represent full Standards is
zero.  I note that DKIM is now in Last Call for promotion
without revision; if it succeeds, it would be the first.  

RFC 6410 was published in October 2011.  Between, e.g., October
2009 and May 2011, seven documents were published as Internet
Standards (RFCs 5652, 5730 - 5734, and 6152).  Because of the
five documents in the EPP series, that is not quite as large a
count as one would expect but even if one counts that as only
three, three is larger than zero.    

That is definitely not a comparison on which I'd want to make
any statistical assertions, but it suggests that 6410 has not
been an overwhelming success is promoting documents.

    john

---

From: John C Klensin <[email-masked]>
Date: Mon, 06 May 2013 18:57:22 -0400

--On Monday, May 06, 2013 00:26 -0700 Bill McQuillan
<[email-masked]> wrote:

>> changes in priorities.  The original idea of a Proposed
>> Standard as a fairly rough specification that would be used
>> for study and evaluation on the basis of implementation
>> experience, not a spec from which products were built, is one
>> that has been mentioned (although not quite in that way).   
> 
> terminology problem. The word "standard" is used too soon in
> the  maturity levels of RFCs.
> 
> rather than "standards". Perhaps it would have been better if
> the names of the maturity levels were something like:
> 
>   Proposed Protocol
>   Test Version 1 Protocol
>   Test Version 2 protocol
>    .
>    .
>    .
>   Standard Protocol
> 

Not the first time this has been proposed.  Or even only the
fourth or fifth.  That doesn't necessarily make it a bad idea,
just one the community has not been willing to adopt.

   john

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 07 May 2013 14:48:22 -0600

On 5/2/13 4:58 PM, Dave Crocker wrote:
>> surprises are more fundamental issues than mere timing. But also
>> because I actually*do*  have some statistics that seem to indicate
>> that, overall, the last phase still goes through pretty quickly. Look
>> athttp://www.arkko.com/tools/lifecycle/wgdocs.html  and compare the
>> WG, IESG, and RFC editor times in the first graph. The WG time
>> dominates. (I said "seem to indicate" because the results are pretty
>> dated and I'm not really sure how valid they are, but they match at
>> least my intuitive experience.) Not saying delay reduction wouldn't be
>> useful, the overall times are still very long, and the IETF last call
>> - IESG time is still a significant component. Just that delay would
>> not be my primary
> 
> 
> Jari,
> 

I'm curious what exactly falls under the RFC Editor phase. My impression
from recent plenaries is that the purely RFC Editor responsibilities
(not including states like MISSREF and AUTH48) has been running around
6-7 weeks. That's a far cry from 100 days.

Peter

---

From: "Heather Flanagan (RFC Series Editor)" <[email-masked]>
Date: Wed, 08 May 2013 10:07:47 +0100

> 6-7 weeks. That's a far cry from 100 days.

You are correct, Peter.  MISSREF and AUTH48 are not part of the RFC
Editor timed states, and the RFC Editor timed states have been largely
under 7 weeks for the last year.

If anyone is curious about current stats, which includes average times
in AUTH48 and MISSREF, see http://www.rfc-editor.org/CurrQstats.txt

Heather Flanagan, RSE

---

From: Jari Arkko <[email-masked]>
Date: Wed, 8 May 2013 16:50:29 +0100

Heather, all,

> under 7 weeks for the last year.

Indeed. The actual time for what RFC Editor does for documents is quite short (and thank you and others at the RFC Editor side for that!).

My stats counted both MISSREF and AUTH48, I think.

Jari

---

From: [email-masked] (Martin Rex)
Date: Wed, 8 May 2013 18:54:40 +0200 (CEST)

Olaf Kolkman wrote:
> 
> Where things become difficult is at the point where the maintenance
> of our standards need to be explained and questions about progression
> on the standards ladder get asked. 
> 
> Personally I hope that RFC 6410 has the effect that we, as a community,
> get serious about promoting our proposed standards, or obsolete them. 
> 

One significant problem is that in the few working groups (in the security
area) that I'm tracking, there appears to be a constituency behind every
(formally provable) breakage and every non-trivial omission that makes
it impossible to ever progress the documents beyong proposed.

And the IESG seems OK with waiving the breakage when documents are
recycled on proposed, rather than requiring WGs to fix it when
problems are identified during LC of a bis document.

-Martin

---

From: Spencer Dawkins <[email-masked]>
Date: Thu, 09 May 2013 03:32:13 -0500

> My stats counted both MISSREF and AUTH48, I think.

I remember that when this was previously discussed (mid-2000s), we noted 
that the states didn't map 1:1 with who had to actually do something. 
Sounds like this is still true.

So in this case, we're looking at "RFC Editor state" = "Heather, please 
do something" + "some working group, please do something" + "author(s), 
please do something", and we can't tell how much time to attribute to 
each of these ...

Spencer

---

From: John C Klensin <[email-masked]>
Date: Thu, 09 May 2013 09:13:06 -0400

--On Thursday, May 09, 2013 03:32 -0500 Spencer Dawkins
<[email-masked]> wrote:

> So in this case, we're looking at "RFC Editor state" =
> "Heather, please do something" + "some working group, please
> do something" + "author(s), please do something", and we can't
> tell how much time to attribute to each of these ...

You could further add to that list "RFC Production Center,
please do something" (different from an RSE wait, which is, or
at least ought to be, more significant) and "IESG or appropriate
AD, please do something", which does happen.

But the RFC Editor's numbers try (almost always successfully) to
separate the two "waiting on the RFC Editor Function to do
something" (Heather plus Production Center plus, in principle,
Publisher) from the other states.  Those other states could,
from their point of view, be aggregated into "stuck, someone
else's problem".   If we are looking for issues with IETF
end-game process, we need to parse those, but that is a
different sort of question in terms of data-gathering and
reporting.

best,
   john

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 10 May 2013 08:09:38 +1200

> different sort of question in terms of data-gathering and

All of which suggests that, ideally, the tracker would include
a variable "onTheHook" for each draft, containing at all times
the person or role responsible for the next step. That isn't
necessarily implied by the state machine. For example,
AUTH48 doesn't always imply that the author is on the hook -
it may be that the author has asked the WG Chair to ask the
WG for a quick review of a proposed last-minute change. At
that point, the WG as a whole is on the hook.

    Brian

---

From: Elwyn Davies <[email-masked]>
Date: Fri, 10 May 2013 10:28:38 +0100

Similarly, AFAICS the 'IESG time' includes IETF last call and the
inevitable delay caused by the quantized nature of IESG teleconferenes.

On the average, this will be somewhere around 28-30 days (2 or 4 weeks
in Last call according to document type plus an average of 1 week until
the earliest possible teleconference and a fudge factor for weekends)
which is an irreducible minimum imposed by our processes.   There is not
much the IESG can do to help on that.

/Elwyn

---

From: Jari Arkko <[email-masked]>
Date: Fri, 10 May 2013 13:07:39 +0100

I wanted to send an update, after having discussed this topic in the IESG retreat that we just had here in Dublin. The overall plan is to start with three specific changes listed below. Note that these are approaches that we have discussed, and more detailed plans will be developed in the coming weeks.

1. Send more documents back to the WG to complete the necessary changes. While the responsibility for changes has always been at the WG, the documents were still in the IESG process. The IESG felt that particularly in situations where changes are large or multiple, it would be useful to make the situation about WG ownership even clearer. We will do this by handing the document back to the WG and getting the IESG out of the loop, with the instruction that the chairs should manage the process and address the issues raised in the IETF last call and by the ADs.

2. Moving some directorate reviews earlier, to WG last call time as opposed to IETF last call or IESG review. With the intent of not increasing the number of reviews.

3. Start inviting the document shepherds (i..e., usually WG chairs) to some of the telechats where the IESG discusses their documents.

While these changes do not by themselves decrease the amount of work done in the final stages, we believe they make the role of the working group more central, and as such, contribute to improving the tail-heavy situation in the long run. We will work further in the IESG and affected parties on the details. One obvious question is how various directorates feel about the changes, and how we avoid increasing workload for them.

Jari

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 14 May 2013 13:58:39 +0000

Few thoughts. 

1) don't get wrapped around the axel of STD, PS, Foo bar label, it has nothing to do with the problem that that IESG believes many drafts need changes to fix significant problems. Lots of people imply that the IESG is setting the bar too high but when you look at the changes that are made to drafts from point they go in, to point they come out of IESG it seems to be a rare example where people don't agree that major changes were an improvement and needed.  

2) On the point of what the IESG should be doing, I would like to see the whole IESG say they agree with the Discuss Criteria document and will stay within that (or change it if they disagree). The cross area review teams might want to also provide comments within this context. 

3) Require each pub request  from a WG to come with names of 3 nom com eligible people that are willing to say "I have carefully reviewed this draft and I believe it is ready to be published as an RFC". If a WG can't find 3 people to do this, it should probably be closed. You might consider something similar for AD sponsored drafts but I am more interested in the WG ones. 

4) Get the ADs involved earlier. My experience is that many ADs won't say much because they are worried that they would be over influencing the WG. ADs got the job because we think they are the right people to provide influence and obviously it would be better to get that early rather than late. 

5) Given how long a WG takes, I have no problem with time IESG / RFCEd takes. I would suggest mitigating this concern by clearly telling the community that if there were a spec that truly needed to be fast tracked for some real reason, you would make it priority. And on the rare occasion that is needed, do it and track the stats for fast track separately. I would guess that less than 1 in 100 needs this and you could do theses with delays in the small numbers of weeks instead of months. How fast you can move when really needed is more important to me than the average and I know that the IESG/IANA/RFCEd team works well in parallel and can move fast when needed. 

6) Over the long term, set up the tools to separately track IESG / RFCEd time where the ball is in the authors court. 

7) Remember that in the end, the goal is not a standard. The goal is an feature rich interopeable internet that is a fertile ground for inovation. High quality and relevant standards that think about the future are the way to get there. The goal is not to publish lots of specification that aren't used and don't work. 

Cullen

---

From: Ted Lemon <[email-masked]>
Date: Tue, 14 May 2013 16:34:12 +0000

> 2) On the point of what the IESG should be doing, I would like to see the whole IESG say they agree with the Discuss Criteria document and will stay within that (or change it if they disagree). The…

I am not entirely convinced that the DISCUSS criteria are complete.   There are some rules in the criteria that are intended to curb abuse, and that I think do have that effect, but that also would make some very appropriate DISCUSSes fail to meet the criteria.   I don't really know how to address that problem.   E.g. the rule about not coming up with new DISCUSSes: if a DISCUSS winds opening up a can of worms, it ought to be possible to enlarge the DISCUSS, but I think that the rule about not adding new DISCUSSes after the first DISCUSS tends to forbid that, and the reason given is a good one.   Having said that, I don't think the right answer is to ignore that requirement.  I don't actually have a good answer.

It's worth noting that ADs are not omniscient, and hence the DISCUSS criteria apply to what the AD entering the DISCUSS _knows_, not to the full state of all knowledge in the world.   If someone other than the AD has knowledge that they think means that the DISCUSS doesn't meet the criteria, that doesn't mean the DISCUSS doesn't mean the criteria.   In this case, the critic needs to communicate it to the AD, who may or may not agree with the critic's point of view.   This is not to say that it's never correct to say "this doesn't meet the DISCUSS criteria."   But the reason given should be that it would be obvious to anyone reading the DISCUSS that it didn't meet the criteria.   The critic's expert knowledge can't be given as a reason.

I have also noticed that some authors have the impression that a DISCUSS means the AD doesn't like the document, or doesn't want the document to advance, or is a non-negotiable pronouncement from on high that the authors should not question.   This is certainly not my motivation when I enter a DISCUSS.   I'm just some guy who got nominated by the nomcom.   Hopefully I'm qualified, but I don't claim to be right.   I have seen DISCUSSes I've raised improve documents, and I've seen DISCUSSes I've raised turn out not to require any change, but just some discussion to clear up a misunderstanding on my part.  I very much hope that in the latter case, the author will argue back, and not just make a change to shut me up!

The reason I raise a DISCUSS rather than a comment is that at the time I'm writing the DISCUSS, it appears _to me_ to be the case that there is a problem that ought to be addressed before the document is published.   I may think it's generally a great document, but I want the concern I've raised to be addressed before it moves forward.   That is _all_ a DISCUSS from me means.   If I think the document is an irretrievably bad idea, I will abstain, and say why I think so.

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Tue, 14 May 2013 13:45:21 -0300

Just to echo in some form what others have said, I believe that an
intermediate stage between I-D and RFC is needed.

I don't have a name for it, but conceptually would be something like
'feature freeze', e.g. no more tweakings to the protocol, or base spec
are to be introduced (unless a major showstopper is discovered), and
just nits and maybe IANA considerations are left to add / discuss.

This would help by sending a clear signal to developers when it's
reasonable to start implementing a given I-D.

A couple of days ago I was part of an email exchange in WEIRDS where
something along this lines happened, when someone proposed an addition
to a base spec that, at least myself, considered already 'frozen'.

The confusion is understandable. Someone entering a WG at a later stage
during the development of an I-D has no way of knowing if the base spec
/ protocol has been agreed and considered stable by the rest of the
participants.

Yes, sure, he/she could comb the ML archives, but I don't think that's
either productive nor even viable in every case, since this 'freezing'
is (in my experience) more like a 'shared feeling' in the ML and not
(usually) explicitly signaled.

IMO, this could also help reducing the length of the overall process and
I would think such a signal would lead to more 'running code' earlier in
the process.

cheers!

~Carlos

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 14 May 2013 16:55:06 +0000

inline

On May 14, 2013, at 10:34 AM, Ted Lemon <[email-masked]>
 wrote:

> I am not entirely convinced that the DISCUSS criteria are complete.  

agree but I like and think they help even thought they are not complete 

>  There are some rules in the criteria that are intended to curb abuse, and that I think do have that effect, but that also would make some very appropriate DISCUSSes fail to meet the criteria.  

If you could give some examples of DISCUSSes that you think are reasonable but fail to meet the criteria, it would be really great if you could provide some examples of that as I think it would help refine DISCUSS criteria. I think everyone agrees there could be things in that category but - that was part of the reason DISCUSS criteria did not end up as a BCP. 

>  I don't really know how to address that problem.   

The IESG needs to keep using common sense and keep doing the right thing. I'm happy that the IESG deal with the unexpected.

> E.g. the rule about not coming up with new DISCUSSes: if a DISCUSS winds opening up a can of worms, it ought to be possible to enlarge the DISCUSS, but I think that the rule about not adding new DISCUSSes after the first DISCUSS tends to forbid that, and the reason given is a good one.   Having said that, I don't think the right answer is to ignore that requirement.  I don't actually have a good answer.

I think the IESG has generally followed a good path of not adding new things, and at the same time if some huge problem was found later, still dealing with it. A long time ago it was hard to know when the IESG might actually finish review, that is not longer a problem but I don't want to go back to it being a problem. 

> 
> It's worth noting that ADs are not omniscient, and hence the DISCUSS criteria apply to what the AD entering the DISCUSS _knows_, not to the full state of all knowledge in the world.  

Of course and that is part of why the name DISCUSS. The AD wants to DISCUSS it with people and improve their knowledge of state of work and what happened in WG and why it is the way it is and resolve it. I'm perfectly happy with DISCUSSes being resolved with "once it got explained to me, I see it is not a problem and cleared". 

>  If someone other than the AD has knowledge that they think means that the DISCUSS doesn't meet the criteria, that doesn't mean the DISCUSS doesn't mean the criteria.   In this case, the critic needs to communicate it to the AD, who may or may not agree with the critic's point of view.   This is not to say that it's never correct to say "this doesn't meet the DISCUSS criteria."   But the reason given should be that it would be obvious to anyone reading the DISCUSS that it didn't meet the criteria.   

Sure but as everyone knowledge increases, I'd expect that AD would update the DISCUSS or remove it if it becomes clear it is not longer relevant. My experience has been IESG does do this.  

> The critic's expert knowledge can't be given as a reason.
> 

+1 to that. 

> The reason I raise a DISCUSS rather than a comment is that at the time I'm writing the DISCUSS, it appears _to me_ to be the case that there is a problem that ought to be addressed before the docum…

Again, +1 to that and my experience is most the IESG feels the same way. 

Thanks for sending your email. I think it help people understand how ADs think about DISCUSSes.

---

From: Dave Crocker <[email-masked]>
Date: Tue, 14 May 2013 10:18:45 -0700

On 5/14/2013 6:58 AM, Cullen Jennings (fluffy) wrote:
>    when you look at the
> changes that are made to drafts from point they go in, to point they
> come out of IESG it seems to be a rare example where people don't
> agree that major changes were an improvement and needed.

This is an important assertion; probably fundamental.  Hence it needs 
some sort of objective substantiation.

> 2) On the point of what the IESG should be doing, I would like to see
> the whole IESG say they agree with the Discuss Criteria document and
> will stay within that (or change it if they disagree).

+1

And a Discuss should be required to assert which criteria apply and how.

FWIW, I assume the formal list will need to change; this means that 
there will be times when it is incomplete.  Hence any specific AD 
concern might not fit into the existing list. So I think it's important 
for that to be ok, as long as the AD justifies the exception carefully 
and thoroughly.

>  The cross area
> review teams might want to also provide comments within this
> context.

Interesting idea.  Reviews don't carry the force of law, as a Discuss 
does, but the discipline of distinguishing specific discuss criteria 
from other comments in a review could be helpful.

d/

---

From: Stephen Farrell <[email-masked]>
Date: Tue, 14 May 2013 18:41:28 +0100

Hi Cullen,

On 05/14/2013 02:58 PM, Cullen Jennings (fluffy) wrote:
> I would like to see the whole IESG say they agree with the Discuss Criteria document and will stay within that (or change it if they disagree). 

That I'm pretty sure is the case. When I started as a new AD
one of the first things I did was put on stupid discuss that
didn't meet the discuss criteria, at which I was very quickly,
but nicely, pointed, (thanks Robert:-), so I learned that
lesson early.

I've seen it happen a fair few times since, with new and not
so new ADs - the IESG as a whole do beat one another up with
the discuss criteria, and not that infrequently. (There was
mail making exactly this point today for example.)

If it helps, I personally think the discuss-criteria are very
good, and work well and I do my best to follow 'em. (Though
since most of my discusses are security related that's maybe
easier for me than for ADs in other areas.)

> The cross area review teams might want to also provide comments within this context.

No harm. But all cross-area reviews are explicitly adopted by
an AD before becoming discuss points. And the IESG also beat
one another up if an AD posts a discuss that only says "see review
foo" without saying more, so all discusses are the AD's and need
to be argued by that AD. Again, it doesn't happen every week
that we beat up an AD for that, but maybe a few times a year.

On 05/14/2013 05:34 PM, Ted Lemon wrote:
> the rule about not coming up with new DISCUSSes:

I've not found that a real problem. When its happened that we
did turn up something bigger than we thought after the telechat
(and updating your discuss points before or during the telechat
is considered fair game) then I think the authors/chairs have
always also agreed that stuff needed to be done.

Cheers,
S.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 15 May 2013 08:10:09 +1200

I think this exchange between Cullen and Ted says it all, except
for one tweak: the IESG is allowed, even encouraged, to apply common
sense when considering the DISCUSS criteria. They are guidance,
not rules.

Also, everybody needs to take the word "discuss" literally. An
entirely possible outcome is that after discussion, the AD says
"Oh. You're correct. Pretend I never spoke!". Or the author says
"Oh. You're correct. We'll change it." Either outcome is good.

Regards
   Brian

---

From: Joe Touch <[email-masked]>
Date: Tue, 14 May 2013 13:45:54 -0700

Brian, et al.,

> "Oh. You're correct. We'll change it." Either outcome is good.

The trouble with this assumption is the IESG review process.

COMMENTS are appropriate for feedback, but the IESG review process is 
too often considered an *opportunity* for the IESG to "make the document 
better", or (in some case) have an opportunity for their input.

As important as the DISCUSS criteria are, there are NON-DISCUSS criteria 
that ought to be more carefully followed - including the point that 
disagreements with the WG or clarifications are not justification for 
DISCUSS.

Ted pointed out that DISCUSS doesn't mean the IESG doesn't like a 
document - agreed, but it *does* hold up a document until the IESG 
member clears it.

That can - has - and is - being used as leverage to force changes to 
documents that are sometimes clearly out of scope (e.g., listed as 
NON-DISCUSS).

DISCUSS is a heavyweight mechanism that holds up document resolution; it 
should be used only where absolutely appropriate. If the IESG wants to 
have a "discussion" with the authors, they are welcome to participate in 
the WGs or IETF LC, or to contact them out of band.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 14 May 2013 13:46:21 -0700

+1

Joe

---

From: Stephen Farrell <[email-masked]>
Date: Tue, 14 May 2013 21:59:34 +0100

Joe,

> disagreements with the WG or clarifications are not justification for

I had assumed that the term discuss-criteria meant [1] which includes
both. Not sure if that's also what you meant but worth adding the URL
here just in case some folks aren't familiar with it.

   [1] https://www.ietf.org/iesg/statement/discuss-criteria.html

> DISCUSS is a heavyweight mechanism that holds up document resolution; 

Agreed. But its a keystone in the current process. So getting
rid of it would be fairly revolutionary. (Not that I'm against
a bit of revolving now and then:-)

> it
> should be used only where absolutely appropriate. 

s/absolutely appropriate/appropriate/ would be better.

> If the IESG wants to

With our current tail-heavy process, and ~100 WGs that's impossible
in almost all cases.

On 05/14/2013 09:46 PM, Joe Touch wrote:>
>> And a Discuss should be required to assert which criteria apply and how.

That'd be -1 from me fwiw. There aren't enough disputes about that
to make it worthwhile and as I said the IESG is, believe it or not,
pretty good at policing itself in this respect.

S.

---

From: Joe Touch <[email-masked]>
Date: Tue, 14 May 2013 14:12:59 -0700

> a bit of revolving now and then:-)

I am *not* suggesting getting rid of it.

I *am* suggesting that it needs to be used only where necessary, and 
that 'necessary' ought to be clearly proved by:

	- citing the specific DISCUSS criteria involved

	- providing explicit information on what would
	be required to clear the DISCUSS

> With our current tail-heavy process, and ~100 WGs that's impossible

The NON-DISCUSS criteria imply that DISCUSS is not an opportunity for 
late input. I appreciate that early input isn't practical, but that does 
NOT provide a rationale for violating the NON-DISCUSS criteria 
(technical disagreement with the WG, etc. - see the list).

The IESG can offer its advice in COMMENTS in other positions. It can 
make suggestions, and let the authors and ADs decide what to do.

But it should NEVER hold a document hostage with a DISCUSS without 
specific merit.

Since some of you asked, here's a *current* example:

------

Ted Lemon issued a DISCUSS on a doc I have pending - here's the link:

http://datatracker.ietf.org/doc/draft-ietf-tcpm-experimental-options/ballot/

Note that his DISCUSS raises the point that 32-bit ExIDs "don't add 
value" and are "pointless" (despite having several paragraphs of 
justification in the doc). That's second-guessing the WG, which is 
listed among the explicit NON-DISCUSS criteria.

There's no clear no collateral damage to the Internet, just damage to 
those who might use a longer TCP option, and there are implementations 
that already use this (as well as other TCP options that are much less 
frugal in their use of option space). It affects only shared use of 
experimental option codepoints, a situation that's intended to be 
transient if an option becomes widely used. The choice of 32-bit ExIDs 
has pros and cons listed in the doc, and both 16-bit and 32-bit ExIDs 
are allowed.

Right now, he's waiting for the doc to be changed to recommend 16-bit 
ExIDs. I don't disagree with that suggestion, but is it really 
appropriate to hold a document hostage using a DISCUSS this way?

I don't think so.

I questioned this process, and he suggested that debating the validity 
of a DISCUSS was out of scope during IESG review. That's not acceptable 
to me, and I hope to others.

Joe

---

From: Ted Lemon <[email-masked]>
Date: Tue, 14 May 2013 21:39:26 +0000

> always also agreed that stuff needed to be done.

Yup, I'm not convinced that it's a problem in practice.   I just happened to notice that it could have become an issue the other day when discussing a DISCUSS.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 14 May 2013 18:00:53 -0400

It seems to me that if it is really a discussion, then there may be many 
possible things which could resolve it, and the AD raising the question 
may not know exactly what is feasible to clear it.  Otherwise it is a 
demand, not a discussions.  And in my experience while ADs can be pushy 
(like the rest of us), they are generally prepared to have discussion.

Thus, I find your second item below to be inappropriate.

At the same time, discussions do have to be resolvable.  If there is no 
way to address it, then it is not a discuss.  But "required to clar" is 
the wrong picture as far as I can tell.

Yours,
Joel

---

From: Joe Touch <[email-masked]>
Date: Tue, 14 May 2013 15:04:05 -0700

> the wrong picture as far as I can tell.

Point taken - at least some indication on what is expected to be changed 
toward a path of resolution.

As you note, otherwise it's not a DISCUSS.

Joe

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 14 May 2013 18:11:13 -0400

Below:

> As you note, otherwise it's not a DISCUSS.
Thanks Joe.  I am glad we are on the same apage about the kind of 
clarity of description needed in discusses.  We may have different 
experiences, but the vast majority of the discusses I have dealt with 
have been sufficiently clear on the question of how to approach 
resolving them.  Sometimes an actual conversation with the AD has been 
necessary to understand the issue and reach a practical point for 
resolving it, but that is a communication style issue, not a discuss 
substance issue.

Yours,
Joel

---

From: Ted Lemon <[email-masked]>
Date: Tue, 14 May 2013 22:12:37 +0000

On May 14, 2013, at 6:00 PM, "Joel M. Halpern" <[email-masked]> wrote:
> At the same time, discussions do have to be resolvable.  If there is no way to address it, then it is not a discuss.  But "required to clar" is the wrong picture as far as I can tell.

Exactly right.   It would actually be pretty presumptuous for an AD to say what is required to clear the DISCUSS.   That would tend to imply that the DISCUSS is a directive, not an invitation to, well, discuss.   Of course we have to _try_ to say what we think would clear the discuss, but I don't think we can go beyond that; it's virtually impossible for us to have complete information.

---

From: Dave Crocker <[email-masked]>
Date: Tue, 14 May 2013 15:30:52 -0700

On 5/14/2013 3:12 PM, Ted Lemon wrote:
>> At the same time, discussions do have to be resolvable.  If there
>> is no way to address it, then it is not a discuss.  But "required
>> to clar" is the wrong picture as far as I can tell.
>
> Exactly right.   It would actually be pretty presumptuous for an AD
> to say what is required to clear the DISCUSS.   That would tend to
> imply that the DISCUSS is a directive, not an invitation to, well,
> discuss.

It isn't an 'invitation'.  It's an exercise in political authority by
blocking progress of the document.

We came up with the term "Discuss" when I was an AD because, at the
time, the IESG had little authority and wanted to encourage a
constructive tone; we didn't want to sound like we were saying 'no'.

And of course, that's still everyone's preference.  But the reality is
that the imposition of the Discuss is an assertion that changes are
being required.

For reference, that milder uses of Discuss, which is something akin to
"I'd like something clarified" does not require a Discuss.  It requires
a query to the working group and some dialogue.

   Of course we have to _try_ to say what we think would
> clear the discuss, but I don't think we can go beyond that; it's
> virtually impossible for us to have complete information.

That makes no sense.  The AD is the one choosing to block progress.  It
will be the AD who decides to clear the discuss.

How can it be reasonable for the AD to provide no basis for knowing what
it will take to get the AD to do this?

I suspect you are confusing 'what' with 'how'.  The issue is not one of
imposing the details of the engineering or documentation changes that
are being demanded by the AD, but of the criteria that will be applied
in evaluated the adequacy of the changes.  Othrwise, the authors have to
play a guessing game, trying to figure out what will please the AD well
enough to clear the Discuss.

d/

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 14 May 2013 18:46:05 -0400

> a query to the working group and some dialogue.

To be fair, for what it's worth as a WG chair I've had the latter
experience at least as often as the former in the use of DISCUSS, and
I've observed some DISCUSSes cleared without any change at all to the
document in question.

A

-- 
Andrew Sullivan
[email-masked]

---

From: Ted Lemon <[email-masked]>
Date: Tue, 14 May 2013 23:03:56 +0000

> that the imposition of the Discuss is an assertion that changes are

No, it absolutely is not.   That may have been the theory when you were AD, but I can tell you from personal experience dealing with DISCUSSes on drafts for which I am the responsible AD that that is not the theory now.

> a query to the working group and some dialogue.

One of the DISCUSS criteria requires that you not keep heaping on DISCUSSes, so it seems to me that escalating a No Objection to a DISCUSS would be worse, but perhaps you disagree?

> will be the AD who decides to clear the discuss.

I don't agree with your characterization here, but certainly it's true that the AD has to clear the DISCUSS.

> it will take to get the AD to do this?

It's not.   But that's not what Joe said.   What Joe said was that the AD should say _precisely_ what will clear the DISCUSS, and that is a very difficult thing to do with incomplete information.

> enough to clear the Discuss.

If the authors think that the goal is to "please the AD," something's wrong.  This would suggest that they will just do what the AD says without debate, which is exactly the wrong thing.  The whole point of a DISCUSS is to have a discussion.   Frankly, it's pretty disrespectful both to the AD and to the working group if the authors make changes to "please the AD."

---

From: Dave Crocker <[email-masked]>
Date: Tue, 14 May 2013 16:15:27 -0700

> I've observed some DISCUSSes cleared without any change at all to the

We suffer a continuing logic error in the IETF.  We use "sometimes it
happens the other way" as if that negates the existence and problem
cause by what is being criticized.

So, yeah, of course a Discuss /sometimes/ causes a small delay with no
changes.  /Sometimes/ ADs use the sledgehammer of the Discuss to ask for
a bit of conversation.  That's all irrelevant.

What's relevant is the nature of the mechanisms, its capability, and the
cost it can and does impose on authors and the working group.

When a serious defect is identified, it's entirely worth the cost.

When it isn't, it isn't.

In all cases, the person imposing the cost has an obligation to
facilitate closing it, including making clear the criteria for closing
it.  It is unreasonable to have those who must do the work to clear it
play a guessing game.

> On 5/14/2013 4:03 PM, Ted Lemon wrote:> On May 14, 2013, at 6:30 PM,
>>> And of course, that's still everyone's preference.  But the
>>> reality is that the imposition of the Discuss is an assertion
>>> that changes are being required.
>>
>> No, it absolutely is not.   That may have been the theory when you
>> were AD, but I can tell you from personal experience dealing with
>> DISCUSSes on drafts for which I am the responsible AD that that is
>> not the theory now.

It's the practice now.  It's also often/typically the intent now.

      "'DISCUSS' is a blocking position; the document cannot proceed
until any issues are resolved to the satisfaction of the Area Director
who issued the DISCUSS."

This, of course, sounds very conversational (except for the blocking
part.)  And sometimes that's how it is used.  Sledgehammer to demand
polite conversation.  Talk with me, or else.

But look over the Discuss Criteria (Section 3.1) and at least half of
the bullets pertain to core technical or procedural failings.  These are
not matters of misunderstanding, resolved with some polite
conversations.  They require change.

d/

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 14 May 2013 19:57:48 -0400

And your bottom line is exactly what te rules say, what I said, what Ted 
said, and what Joe agreed is reasonable.  It also matchesthe practice I 
have seen.  Even the discuss that I had a lot of arguments with did 
include proposals for paths forward.  Sometimes they were ard to 
understand.  That's probably inevitable with all these opinionated 
humans doing this.

Yours,
Joel

---

From: Joe Touch <[email-masked]>
Date: Tue, 14 May 2013 17:27:48 -0700

On 5/14/2013 4:03 PM, Ted Lemon wrote:
> If the authors think that the goal is to "please the AD," something's
> wrong.  This would suggest that they will just do what the AD says
> without debate, which is exactly the wrong thing.  The whole point of
> a DISCUSS is to have a discussion.   Frankly, it's pretty
> disrespectful both to the AD and to the working group if the authors
> make changes to "please the AD."

That is what happens exactly because the DISCUSS holds up the document, 
and most ADs don't want to burn time stalling their documents if there's 
a way around that delay.

If the suggestion is a suggestion, then change your DISCUSS to some 
other position, and issue the suggestion as a COMMENT. Yes, that's both 
general advice and specific to the case we have pending, Ted.

Joe

---

From: Ted Lemon <[email-masked]>
Date: Wed, 15 May 2013 00:53:16 +0000

On May 14, 2013, at 8:27 PM, Joe Touch <[email-masked]> wrote:
> That is what happens exactly because the DISCUSS holds up the document, and most ADs don't want to burn time stalling their documents if there's a way around that delay.

It can only happen if an author values getting their document through the process more than getting it right, in which case one has to wonder why they tried to publish the document in the first place.   (I assume you meant "authors," not "ADs" above).   The IESG processes documents quite quickly; I don't think it's valid to say that there is some terrifying stall in the document process as a result of the IESG, such that an author needs to chew off their leg to finally get the thing through.

---

From: Sam Hartman <[email-masked]>
Date: Tue, 14 May 2013 22:42:04 -0400

I'll say that about a year and a half ago I found myself pushing back on discusses
that in my opinion clearly were not within the discuss criteria
significantly more than I ever had to do as an AD. My role was as WG
chair/editor.

Interestingly it's been less of an issue in my experience lately.

Definitely not statistically significant data set for any of this.

---

From: Keith Moore <[email-masked]>
Date: Wed, 15 May 2013 00:54:30 -0400

> blocking progress of the document.

It's a mistake to inherently define "progress" in terms of advancing the 
document.   Many documents should not be published in the form in which 
they first reach IESG; a few documents that reach IESG should probably 
never be published.

> that the imposition of the Discuss is an assertion that changes are

That's neither what Discuss means, or what it should mean.   Though I've 
seen many cases where WGs or authors demanded that ADs tell them what to 
fix.   It's understandable that they should want clarity from the AD, 
and yet fixing the document is not the AD's job.
> a query to the working group and some dialogue.
Disagree.   I've seen a number of cases where informal conversations 
caused more problems and delay than voting DISCUSS.   By the time a 
document has reached final IESG review, it's generally much better to 
have the level of formality and transparency that comes with voting 
DISCUSS.

> will be the AD who decides to clear the discuss.

I disagree in the strongest possible terms.

If an AD believes that there is something wrong with a document, even 
something that needs clarification, the proper thing for the AD is to 
vote DISCUSS.   This is NOT "choosing to block progress". Publishing 
broken or unclear documents is not progress.

Keith

---

From: Keith Moore <[email-masked]>
Date: Wed, 15 May 2013 01:05:58 -0400

On 05/14/2013 04:45 PM, Joe Touch wrote:
> too often considered an *opportunity* for the IESG to "make the 
> document better", or (in some case) have an opportunity for their input.

For that matter, working groups are too often echo chambers where a set 
of people manage to isolate themselves from input from those whose work 
they might otherwise effect.    Some people seem to think that working 
group output should be sacrosanct.   There's absolutely no reason to 
believe that.  WGs often make technical mistakes that are uncovered by 
external review.   Even when no such mistakes are encountered, WG output 
rarely represents rough consensus of all interested parties, and WGs 
often fail to do due diligence in considering the interests of the broad 
spectrum of those potentially affected by their work.

Of course IESG isn't infallible either, and shouldn't behave as if it 
is.  But review by experts from outside of the WG generally seems to 
improve the IETF's output.

> As important as the DISCUSS criteria are, there are NON-DISCUSS 
> criteria that ought to be more carefully followed - including the 
> point that disagreements with the WG or clarifications are not 
> justification for DISCUSS.

Strongly disagree.  First, DISCUSS only means that there's something the 
AD wants to discuss.    In particular, disagreements with the WG about 
technical quality are always appropriate for IESG to raise. The same is 
true of requests for clarification.

> document - agreed, but it *does* hold up a document until the IESG 

But there are also procedures within IESG to ignore a single DISCUSS 
vote.   So ultimately it takes multiple DISCUSS votes to hold up a 
document indefinitely.

> it should be used only where absolutely appropriate. If the IESG wants 
> to have a "discussion" with the authors, they are welcome to 
> participate in the WGs or IETF LC, or to contact them out of band.

DISCUSS is not supposed to be a heavyweight mechanism, and actually it's 
hard to imagine a lighter weight mechanism that gives IESG review any 
weight.   Informal communication doesn't generally work well in practice 
because it lacks transparency, and it can cause additional delay without 
resolving the problem.   Voting DISCUSS puts pressure on BOTH the AD and 
the WG to resolve the issue.

Keith

---

From: Jari Arkko <[email-masked]>
Date: Wed, 15 May 2013 13:52:18 +0300

I feel that the discussion is stuck on the different perceptions on whether an AD's actions are either blocking reasonable progress, or an essential correction to a mistake that went undetected.

I'd like to make a couple of observations. First of all, we at the IESG process 10-25 documents every second week, and several points (comment or discuss) are raised for each one. Given that both working groups and the IESG consist of humans, I'm guessing that none of us are making perfect decisions all the time. While the Discuss Criteria document is very useful, I'd warn against trying to codify the proper behaviour too much. It would be very difficult, and ultimately it comes down to making a call on whether an issue is really crucial for the Internet or not.

I feel that the discussion and pressure from other ADs and authors and the rest of the working group is a more useful avenue to ensure that we really are fixing the necessary bugs and only those. And I know you are doing that - lets make sure we keep doing it. "If you see something, say something". It is everyone's responsibility to try to do the right thing, and if you feel it is not happening, say so. I think the current IESG has a very good mood for receiving feedback and acting on it. (Speaking as the longest serving member in the current IESG, who frequently gets shown to be wrong by other ADs or WG folk.) 

In addition, as discussed separately, moving more of the issue resolution to the open and to the WG is a good thing.

Jari

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 May 2013 07:25:21 -0700

On 5/14/2013 5:53 PM, Ted Lemon wrote:
>> That is what happens exactly because the DISCUSS holds up the
>> document, and most ADs don't want to burn time stalling their documents
>> if there's a way around that delay.
>
> It can only happen if an author values getting their document
> through  the process more than getting it right, in which case one has to wonder
> why they tried to publish the document in the first place. (I assume you
> meant "authors," not "ADs" above).

No; there are times when the document authors are pressured by ADs to do 
anything to resolve pending DISCUSSes, rather than stand up to the fact 
that the issue is either incorrect or the DISCUSS is invalid.

> The IESG processes documents quite
> quickly; I don't think it's valid to say that there is some terrifying
> stall in the document process as a result of the IESG, such that an
> author needs to chew off their leg to finally get the thing through.

Well, there are IESG members who stand their ground even when it's 
incorrect, such as:

	- claiming that determining WG consensus is up to the AD,
	then repeatedly demanding evidence of that consensus

	- failing to drop a DISCUSS even when it meets their own
	criteria

Those hold up a document too, as you should know (these are examples 
from your review of my doc). As does demanding a document revision while 
there remain open ballot positions, as was done today - on this 
document, to address your pending DISCUSS.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 May 2013 07:30:02 -0700

On 5/14/2013 4:57 PM, Joel M. Halpern wrote:
> said, and what Joe agreed is reasonable.

Unfortunately, it's not what happens/is happening right now.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 May 2013 07:35:05 -0700

On 5/14/2013 4:03 PM, Ted Lemon wrote:
> The whole point of a DISCUSS is to have a discussion.

The *whole* point of a DISCUSS is to hold a document in IETF review 
until a discussion is *resolved*.

There are thus three parts:
	- having a discussion
	- pausing the document
	- waiting for resolution

Nothing limits the IESG to having a "discussion" by issuing a DISCUSS; 
they can issue COMMENTs in any position and simply ask for a dialogue.

Because DISCUSS includes these other properties, it influences authors 
to make changes simply to make a DISCUSS go away, due to pressure by the 
IESG or their own ADs.

That's why they need to be used only where that pressure is appropriate 
and not inappropriate; that's the reason there are both DISCUSS and 
NON-DISCUSS criteria, and why it's very important for those in IESG 
review to hold the IESG to that distinction.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 May 2013 07:39:37 -0700

On 5/14/2013 9:54 PM, Keith Moore wrote:
> Publishing broken or unclear documents is not progress.
>
> Keith

Broken, agreed.

Unclear, nope - please review the NON-DISCUSS criteria, notably:

The motivation for a particular feature of a protocol is not clear 
enough. At the IESG review stage, protocols should not be blocked 
because they provide capabilities beyond what seems necessary to acquit 
their responsibilities.

The DISCUSS isn't there to make documents "better" - that's for 
COMMENTs. A DISCUSS there to catch a set of problems and to *block* the 
document's progress until that problem is resolved.

Joe

---

From: Keith Moore <[email-masked]>
Date: Wed, 15 May 2013 10:41:36 -0400

On 05/15/2013 10:39 AM, Joe Touch wrote:
> because they provide capabilities beyond what seems necessary to 
> acquit their responsibilities.
>
> COMMENTs. A DISCUSS there to catch a set of problems and to *block* 
> the document's progress until that problem is resolved.

I strongly disagree with what the NON-DISCUSS criteria say. DISCUSS 
isn't just for blocking documents.   And document quality is as 
important (in the sense that poor document quality can lead to as many 
interoperability or other problems) as technical correctness.

Why are people trying to sabotage IESG?

Keith

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 May 2013 07:47:59 -0700

On 5/14/2013 10:05 PM, Keith Moore wrote:
...
> improve the IETF's output.

Sure, but note that there is a specific NON-DISCUSS criteria on this point:

Disagreement with informed WG decisions that do not exhibit problems 
outlined in Section 3.1 (DISCUSS Criteria). In other words, disagreement 
in preferences among technically sound approaches.

Finding technical mistakes is good, but imposing the IESG's preferred 
technical solution over the WG's preference is inappropriate, but happens.

> AD wants to discuss.

As noted in another post, it means "hold this doc until the DISCUSS is 
resolved". Discussions can happen as a result of COMMENTs in any IESG 
position.

>  In particular, disagreements with the WG about
> technical quality are always appropriate for IESG to raise.

Yes.

> same is

Sometimes - there's a specific NON-CRITERIA about clarification, however:

> vote.   So ultimately it takes multiple DISCUSS votes to hold up a

Those procedures take time, so even a single DISCUSS holds things up.

> weight.

Oh, I'm not suggesting removing the DISCUSS mechanism.

First, it ought to have a new name - one that makes clear that this is 
heavyweight. Perhaps "HOLD FOR REVISION", which is the net effect it 
already has.

The key bug is that the IESG can't easily issue a question without 
casting a ballot position, but that may also be a feature. It might be 
useful to be able to issue a QUESTION withouth changing a ballot 
position, though.

 > Informal communication doesn't generally work well in practice

Which is appropriate only when the IESG should have the right to force 
that resolution, and when the path to that resolution is sufficiently 
clear. If there is no path, then vote NO. If forcing resolution and 
putting pressure isn't warranted, issue a COMMENT on any other ballot 
position.

Joe

---

From: Ralph Droms <[email-masked]>
Date: Wed, 15 May 2013 10:49:44 -0400

On May 15, 2013, at 10:39 AM 5/15/13, Joe Touch <[email-masked]> wrote:

> The motivation for a particular feature of a protocol is not clear enough. At the IESG review stage, protocols should not be blocked because they provide capabilities beyond what seems necessary to acquit their responsibilities.
> 
> The DISCUSS isn't there to make documents "better" - that's for COMMENTs. A DISCUSS there to catch a set of problems and to *block* the document's progress until that problem is resolved.

I'll agree with you *if* you consider an unclear description of a feature of a protocol, severe enough that reader of the specification are not able to build interoperable implementations, as a problem for which a DISCUSS can be posted.

In my opinion, there are many ways in which document can be unclear beyond the "motivation for a particular feature of a protocol is not clear enough."  

- Ralph

---

From: Ted Lemon <[email-masked]>
Date: Wed, 15 May 2013 14:55:43 +0000

On May 15, 2013, at 10:41 AM, Keith Moore <[email-masked]> wrote:
> I strongly disagree with what the NON-DISCUSS criteria say. DISCUSS isn't just for blocking documents.   And document quality is as important (in the sense that poor document quality can lead to as many interoperability or other problems) as technical correctness.

The interpretation of this particular NON-DISCUSS criterion that Joe has given is simply wrong.   The key word to pay attention to to see the error is "motivation."   The point of this criterion is to eliminate a very specific sort of stall that has been known to happen in the past: the stall where the AD doesn't understand why the document is being put forward at all, and therefore blocks the document until the authors explain the motivation behind the document to the satisfaction of the AD who is holding the DISCUSS.

This is a real issue that has created real problems in the past, and that is why it is in the NON-DISCUSS criteria.   But this criterion _does not_ mean that a criticism that the document itself is unclear is not a valid reason to hold a DISCUSS on it.   In fact, it's an excellent reason to hold a DISCUSS on it.   A lack of clarity in a document can result in it being implemented incorrectly, or in the case of a BCP, interpreted incorrectly.   Or in extreme cases, not read at all.   This is a bad outcome, worth spending time on, even if the authors would rather be quit of it.

---

From: Keith Moore <[email-masked]>
Date: Wed, 15 May 2013 11:06:22 -0400

IMO, IESG should have grounds to reject any document that isn't specifically authorized in a WG's charter.

- Keith

---

From: Jari Arkko <[email-masked]>
Date: Wed, 15 May 2013 18:15:29 +0300

Joe,

> Broken, agreed.

Yep.

> The DISCUSS isn't there to make documents "better" - that's for COMMENTs. A DISCUSS there to catch a set of problems and to *block* the document's progress until that problem is resolved.

Yes, but note that there are multiple aspects of "unclear". You cite above the motivation aspect. There's also a DISCUSS criteria for other forms of unclear, e.g., if I can't figure out what I should do in the implementation, it would be an issue. The criteria document confirms:

"The specification is impossible to implement due to technical or clarity issues."

> Disagreement with informed WG decisions that do not exhibit problems outlined in Section 3.1 (DISCUSS Criteria). In other words, disagreement in preferences among technically sound approaches.
> 
> Finding technical mistakes is good, but imposing the IESG's preferred technical solution over the WG's preference is inappropriate, but happens.

If you are hit with a Discusss that is about preferring another technical solution, you should push back.

Jari

---

From: Andy Bierman <[email-masked]>
Date: Tue, 14 May 2013 17:56:10 -0700

Hi,

The evidence seems to show that the IESG is getting faster
at their job and WGs are getting slower at theirs.  I don't
see any need for "DISCUSS Rules".  All the AD reviews I've
experienced have improved the document, sometimes a lot.
All DISCUSS issues got cleared with reasonable (even good)
solutions.

IMO, there is no question that applying the right expertise at the
appropriate time in the WG draft process would improve the
entire system and avoid surprises during I* Last Call.
The question is the best way to do that.

Andy

On Tue, May 14, 2013 at 4:57 PM, Joel M. Halpern <[email-masked]>wrote:

> have seen.  Even the discuss that I had a lot of arguments with did include
> proposals for paths forward.  Sometimes they were ard to understand.
>  That's probably inevitable with all these opinionated humans doing this.
>
> Yours,
> Joel
>

---

From: Yoav Nir <[email-masked]>
Date: Wed, 15 May 2013 15:33:28 +0000

> IMO, IESG should have grounds to reject any document that isn't specifically authorized in a WG's charter.

Why? There's definitely a process failure there, and it should be blamed on the WG chairs and/or the AD, who should have either moved the work out of the working group or worked on updating the charter.

But the IESG handles documents from both working groups and from individuals, and individuals don't have charters.  Why should an otherwise good document be rejected after it has gone through IETF last call and has been brought to the IESG by one of their own?

Yoav

---

From: Thomas Narten <[email-masked]>
Date: Wed, 15 May 2013 17:55:27 +0200

+1 to what Jari says below.

>From my perspective, the important things to keep in mind:

1) Discuss criteria should be principles, not rigid rules. The details
of the issue at hand always matter, and it will sometimes come down to
judgement calls where reasonable individuals just might disagree. And
if we try to replace judgement with rigid, overly specified rules, we
surely won't like the results. i.e, the process requires we do X, when
X is really a bad outcome overall.

2) There will always be judgement calls, where an AD and an author (or
WG) don't necessarily agree. Fine tuning the rules won't fix that. By
default, when there is disagreement, the presumption needs to be the
AD could be right and needs to be convinced. That is the essence of
what a DISCUSS is intended to be. If the dialog between author/WG/AD
is not progressing, bring in someone else. I.e., the way to overrule a
"stubborn" AD is to convince other ADs (and/or WG members) to weigh in
based on the merits. If an author/WG can't do that, that says
something...

Thomas

Jari Arkko <[email-masked]> writes:

> I feel that the discussion is stuck on the different perceptions on
> whether an AD's actions are either blocking reasonable progress, or
> an essential correction to a mistake that went undetected.

> I'd like to make a couple of observations. First of all, we at the
> IESG process 10-25 documents every second week, and several points
> (comment or discuss) are raised for each one. Given that both
> working groups and the IESG consist of humans, I'm guessing that
> none of us are making perfect decisions all the time. While the
> Discuss Criteria document is very useful, I'd warn against trying to
> codify the proper behaviour too much. It would be very difficult,
> and ultimately it comes down to making a call on whether an issue is
> really crucial for the Internet or not.

> I feel that the discussion and pressure from other ADs and authors
> and the rest of the working group is a more useful avenue to ensure
> that we really are fixing the necessary bugs and only those. And I
> know you are doing that - lets make sure we keep doing it. "If you
> see something, say something". It is everyone's responsibility to
> try to do the right thing, and if you feel it is not happening, say
> so. I think the current IESG has a very good mood for receiving
> feedback and acting on it. (Speaking as the longest serving member
> in the current IESG, who frequently gets shown to be wrong by other
> ADs or WG folk.)

> In addition, as discussed separately, moving more of the issue
> resolution to the open and to the WG is a good thing.

---

From: Thomas Narten <[email-masked]>
Date: Wed, 15 May 2013 18:25:43 +0200

I don't think the IETF needs to be in the profile/resume
business. There are plenty of other places that do a fine job already.

What I do think the IETF should do is *require* that participants
identify themselves. That means knowing who they are (a name and email
contact) and an affiliation. For 80% of the participants, this info is
not very hard to figure out (see below). But we also have participants
that use obscure email handles that don't correlate to anything
obvious, whether a real person or to a name in the list of registered
attendees, etc. I suspect these folk are *not* intenending to be
anonymous participants, but in practice they are.

And yes, knowing who someone is, their background and who they work
for is important to me in figuring out how to guage their input. E.g.,
I would likely pay more attention to an operator's comments on a
proposed use case than from someone else.

How to figure out who someone is:

1) look at the list of registered attendees. (but that doesn't include
email addresses, so no clean way to map attendee name into email
addresses being used).

Also, for some reason, some people who register don't bother giving an
affiliation. In some cases this is intentional, but there are others
where it doesn't make sense (e.g., someone who has worked for the same
employer for 10+ years and is still working for that employer).

E.g., if you look at the registration list for Orland0, fully 180 names
don't list affiliations -- and there are a number pretty obvious
surprises in that list...

2) look at email addresses. But nowadays they are often generic (e.g.,
gmail) and don't correlate back to an obvious sponsor.

3) Google names, look at authorship info in RFCs, linked in,
etc. Works in a lot of cases, but is sometimes more work than seems
appropriate. And for those with less history in the IETF, knowing
where to look for this stuff is trickier.

But even doing the above, there are people participating (e.g.,
posting on the IETF list) who I don't know who they are, even after
spending some time trying to figure who who they are and what their
background is. For an open standards organization, that somehow
doesn't seem quite right.

Thomas

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 May 2013 09:36:54 -0700

On 5/15/2013 7:55 AM, Ted Lemon wrote:
>>> The motivation for a particular feature of a protocol is not
>>> clearenough. At the IESG review stage, protocols should not be blocked
>> I strongly disagree with what the NON-DISCUSS criteria say.
>> DISCUSS isn't just for blocking documents. And document quality is
>> as important (in the sense that poor document quality can lead to
>> as many interoperability or other problems) as technical
>> correctness.
>
> The interpretation of this particular NON-DISCUSS criterion that Joe
> has given is simply wrong. The key word to pay attention to to see the
> error is "motivation." The point of this criterion is to eliminate a
> very specific sort of stall that has been known to happen in the past:
> the stall where the AD doesn't understand why the document is being put
> forward at all, and therefore blocks the document until the authors
> explain the motivation behind the document to the satisfaction of the AD
> who is holding the DISCUSS.

I'm impressed that you have such a specific interpretation that this
criteria refers to the entire document, even when it talks about the
"feature of a protocol".

So now we're supposed to accept your interpretation of this rather than 
the explicit text?

> This is a real issue that has created real problems in the past, and
> that is why it is in the NON-DISCUSS criteria.   But this criterion
> _does not_ mean that a criticism that the document itself is unclear
> is not a valid reason to hold a DISCUSS on it.

Agreed - this does not refer to the document. It refers to a specific 
feature of a protocol.

> In fact, it's an
> excellent reason to hold a DISCUSS on it.   A lack of clarity in a
> document can result in it being implemented incorrectly, or in the
> case of a BCP, interpreted incorrectly.

I agree that THESE are good reasons for a DISCUSS, but simply not being 
clear is NOT.

> Or in extreme cases, not
> read at all.

There's nothing you can do about that; RFCs are not read all the time.

Joe

---

From: Ted Lemon <[email-masked]>
Date: Wed, 15 May 2013 17:15:24 +0000

> criteria refers to the entire document, even when it talks about the

"The motivation for a feature of a protocol is not clear enough."   What's ambiguous or subject to interpretation about that?   The commentary exactly echoes what I said.   This does not mean that all lacks of clarity are not DISCUSS criteria: only that a lack of clarity with respect to motivation is not.

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 May 2013 10:23:45 -0700

On 5/15/2013 10:15 AM, Ted Lemon wrote:
> "The motivation for a feature of a protocol is not clear enough."
> What's ambiguous or subject to interpretation about that? The commentary
> exactly echoes what I said. This does not mean that all lacks of clarity
> are not DISCUSS criteria: only that a lack of clarity with respect to
> motivation is not.

And yet that is the precise issue of your pending DISCUSS on my current 
document.

You don't agree that the motivation for the difference between using 
16-bit vs. 32-bit ExIDs is sufficient, even though that is already 
discussed in the document.

Joe

---

From: SM <[email-masked]>
Date: Wed, 15 May 2013 10:32:12 -0700

At 08:06 15-05-2013, Keith Moore wrote:
>IMO, IESG should have grounds to reject any document that isn't 
>specifically authorized in a WG's charter.

I read a few charters:

core:
   Dec 2099 - HOLD (date TBD) Constrained security bootstrapping specification
               submitted to IESG
ancp:
   Sep 2010 - Access Node Control Protocol (ANCP) Last Call
   Dec 2010 - ANCP MIB Last Call
   Dec 2010 - ANCP Multicast Extensions last call
   Jan 2011 - ANCP applicability to PON last call
   Mar 2011 - Re-charter or conclude Working Group

6man:
  Jan 2008 - Submit PPP Compression Negotiation specification to IESG as a
             Proposed Standard
  Mar 2008 - Determine way forward for ULA-C specification

l2tpext:
   Mar 2008 - Submit Internet-Draft of PPP over L2TPv3 to IESG
   Done     - WG Last Call on TDM over L2TPv3
   Jun 2008 - WG Last Call on IP over L2TPv3

drinks:
   Apr 2012 - Request publication of SPPP over SOAP Document
   Apr 2012 - Request publication of Framework Document

straw:
   Nov 2011 - Specification for a SIP overload control mechanism based on
              implicit/explicit feedback to IESG for publication as 
Proposed Standard
   Feb 2012 - Specification for a SIP load filtering mechaism to IESG 
for publication
              as Proposed Standard
idr:
   Mar 2010 - Solicit work items for scalability improvements
   Aug 2010 - Submit AS-wide Unique BGP Identifier for BGP-4 to IESG 
as a Proposed Standard
   Aug 2010 - Submit Dynamic Capability for BGP-4 to IESG as a 
Proposed Standard
   Aug 2010 - Submit ASpath ORF draft to IESG as a Proposed Standard
   Aug 2010 - Submit MIB v2 for BGP-4 to IESG as a Proposed Standard
   Nov 2010 - Submit BGP Support for Four-octet AS Number Space 
(revised version) to
              IESG as a Proposed Standard
   Nov 2010 - Submit Revisions to the BGP 'Minimum Route 
Advertisement Interval' to
              IESG as a Proposed Standard
   Nov 2010 - Submit Advertisement of Multiple Paths in BGP to IESG 
as a Proposed Standard
   Nov 2010 - Submit BGP Link Bandwidth Extended Community to IESG as 
a Proposed Standard
   Nov 2010 - Submit Advertisement of the best external route in BGP to IESG as
               a Proposed Standard
   Dec 2010 - Submit Multisession BGP to IESG as a Proposed Standard
   Dec 2010 - Submit Error Handling for Optional Transitive BGP Attributes to
              IESG as a Proposed Standard
   Dec 2010 - Submit ASpath ORF to IESG as a Proposed Standard
   Dec 2010 - Revise WG charter

pim:
   Aug 2011 - First WG version of udated RFC 4601
   Aug 2011 - Submit a more reliable PIM solution (refresh reduction) 
to the IESG
   Nov 2011 - Submit a population count extension to PIM to the IESG
   Dec 2011 - Submit update of RFC 4601 to IESG for advancement to 
Draft Standard

pkix:
   Sep 2007 - Progression of CRMF, CMP, and CMP Transport to DRAFT Standard
   Sep 2007 - Progression of Qualified Certificates Profile RFC to 
DRAFT Standard
   Sep 2007 - Progression of Certificate & CRL Profile RFC to DRAFT Standard
   Sep 2007 - Progression of Time Stamp Protocols RFC to DRAFT Standard
   Sep 2007 - Progression of Logotype RFC to DRAFT Standard
   Nov 2007 - Progression of Proxy Certificate RFC to DRAFT Standard
   Nov 2007 - Progression of Attribute Certificate Profile RFC to 
DRAFT standard
   Feb 2008 - Update to CMC approved as PROPOSED Standard
   Mar 2008 - ECC Algorithms approved as PROPOSED Standard RFC
   Mar 2008 - Progression of CMC RFCs to DRAFT Standard
   Mar 2008 - SCVP approved as PROPOSED Standard RFC

ippm:
   Nov 2010 - Final version of process draft
   Nov 2010 - Implementation report based on process draft
   Mar 2011 - Revise charter

ppsp:
   Dec 2010 - Submit problem statement to IESG as Informational
   Apr 2011 - Submit architectural survey to IESG as Informational
   Apr 2011 - Submit requirements document to IESG as Informational
   Aug 2011 - Submit PPSP peer protocol to IESG as Proposed Standard
   Aug 2011 - Submit PPSP tracker protocol to IESG as Proposed Standard
   Dec 2011 - Submit usage guide to IESG to IESG as Informational

I did not verify whether the drafts mentioned about left the working 
group or not.  The IESG would be rejecting a lot of documents if it 
looked into what was authorized by the charter.

At 08:33 15-05-2013, Yoav Nir wrote:
>Why? There's definitely a process failure there, and it should be 
>blamed on the WG chairs and/or the AD, who should have either moved 
>the work out of the working group or worked on updating the charter.

There would be a lot of WG chairs and/or ADs to blame as there are 
dates up to five years in the past in the charter extracts mentioned above.

>         then repeatedly demanding evidence of that consensus

If there was WG consensus it shouldn't be much of a problem to 
provide evidence.  I read a write-up recently where "strong 
consensus" is mentioned.  I took a quick look at the mailing list 
archives and I saw the WGLC message; the next message about the draft 
was the AD review.

Regards,
-sm

---

From: Ted Lemon <[email-masked]>
Date: Wed, 15 May 2013 18:08:32 +0000

On May 15, 2013, at 1:23 PM, Joe Touch <[email-masked]> wrote:
> You don't agree that the motivation for the difference between using 16-bit vs. 32-bit ExIDs is sufficient, even though that is already discussed in the document.

I don't think this is a topic that the IETF as a whole is likely to find very interesting.   However, if anyone is curious, they are welcome to read the DISCUSS here and see if they agree with your characterization of my question: http://datatracker.ietf.org/doc/draft-ietf-tcpm-experimental-options/ballot/

For those who may be interested, the last sentence of the first paragraph is the motivation for this being a DISCUSS position (as opposed to a comment).

---

From: John C Klensin <[email-masked]>
Date: Wed, 15 May 2013 14:38:13 -0400

--On Wednesday, May 15, 2013 18:25 +0200 Thomas Narten
<[email-masked]> wrote:

> What I do think the IETF should do is *require* that
> participants identify themselves. That means knowing who they
> are (a name and email contact) and an affiliation. For 80% of
> the participants, this info is not very hard to figure out
> (see below). But we also have participants that use obscure
> email handles that don't correlate to anything obvious,
> whether a real person or to a name in the list of registered
> attendees, etc. I suspect these folk are *not* intenending to
> be anonymous participants, but in practice they are.

Thomas, 

I completely agree, but..

> Also, for some reason, some people who register don't bother
> giving an affiliation. In some cases this is intentional, but
> there are others where it doesn't make sense (e.g., someone
> who has worked for the same employer for 10+ years and is
> still working for that employer).
> 
> E.g., if you look at the registration list for Orland0, fully
> 180 names don't list affiliations -- and there are a number
> pretty obvious surprises in that list...

Ok.  I'm probably one of the 180, although I would be surprised
if I were one of the "obvious surprises".  I had a corporate
affiliation until somewhat over a decade ago and showed that
affiliation when asked for it on registration, IAB and IESG
mini-biographies, etc.  Since then, I have been an independent
consultant.  I don't consider the way my business is organized,
whether it has other employees or not, etc., to be any of the
IETF's business.  In order to respond to an implied part of your
question, in that last decade or so, I have never been paid by a
client to attend IETF or to represent that client's interests.
That has been largely by choice and driven by a desire to stay
absolutely independent.   I have accepted travel money and
waivers of registration fees on a few (very few) occasions, but
never, in my post-corporate life, compensation for my time spent
of IETF.

So, what would you have me (and others like me) put on
registration forms so that I'm not part of that undifferentiated
"180 names"?  I objected several years ago to the Secretariat
listing me as "Independent" because I know of multiple
organizations/ enterprises with "Independent" in their names
(including, apparently, six separate ones with second-level
domain names in six popular gTLDs I checked).  You tell me how
to fill in that form a way that doesn't misrepresent my
situation or disclose information that is irrelevant to the IETF
and I will happily do it.  Up through the last IETF meeting, I
could most closely approximate that condition by leaving the
information box blank.

To pursue this a tad further, for the purposes for which you
want affiliations, someone who is legally a consultant or
independent contractor rather than an employee but who is being
paid by a firm to attend IETF may have no less obligation to
consider the interests of that firm in their IETF actions than
you have to IBM.  They might even have more of an obligation.
If you want affiliation information in the interest of openness
--and, again, I think that would be a really good idea-- than
such a person is not "independent" or representing only
themselves.   If we are trying to determinate affiliation to
evaluate positions, he or she is definitely affiliated with that
firm and should be listing it (ideally as "consultant to..." or
"contractor to...").

On of the other side of that coin, in prior professional lives,
I've been in situations in which an employer or research sponsor
has essentially said "you are welcome to attend IETF as long as
you can show (in a way that will satisfy auditors) that you did
so entirely on your own time, with no portion of your salary
during that time or your expenses attributable to the company.
In most cases, that person is as independent of company control
over positions taken as I am -- far more so than a consultant
who is being paid to represent a company at IETF or to come to
IETF in order to advise the company on IETF strategy.  She may
even be constrained to not mention the company's name in
conjunction with IETF efforts for fear that mention will be
taken as endorsement.   What would you like to see on the
registration form?

Finally, do any of the answers change if the consultant or
contractor either is employed directly by a consulting firm that
pays a salary or is an individual but more organized as a
business than I have chosen to be? Is the affiliation you want
the name of the nominal employer of the organizational identity
of whomever is really paying the bills and/or calling the tune?

Relative to overall IETF participation, all of the above may be
edge cases.  For reasons similar to those of the "diversity"
debates, I have no way to guess how many there actually are in
the registrant or participant communities.  But, if one were to
look only at present and past WG Chairs, document authors, and
IAB and IESG members, I wouldn't be surprised to find that
people in those ambiguous situations were somewhat
overrepresented.

I think it is all very well to ask for affiliations in principle
and, also in principle, I agree that it is a good idea.  But, in
practice, I think there are a lot of clarifications and other
changes that would be required and that might or might not be
practical.  

 best,
   john

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 May 2013 11:48:21 -0700

On 5/15/2013 11:08 AM, Ted Lemon wrote:
> I don't think this is a topic that the IETF as a whole is likely to
> find very interesting. However, if anyone is curious, they are welcome
> to read the DISCUSS here and see if they agree with your
> characterization of my question:
> For those who may be interested, the last sentence of the first
> paragraph is the motivation for this being a DISCUSS position (as
> opposed to a comment).

Which is "I think that using a 4-byte ExID runs a real risk of
overflowing the available space in the TCP header in real-world 
circumstances."

Except that the document already describes the ExID as either 16-bit or 
32-bit:

    >> All ExIDs MUST be either 16-bits or 32-bits long.

Motivation for the additional two bytes is already explained in the 
document in several places, notably:

    The second two bytes serve as a "magic number".
...
    Using the additional magic number bytes helps the option contents
    have the same byte alignment in the TCP header as they would have if
    (or when) a conventional (non-experiment) TCP option codepoint is
    assigned. Use of the same alignment reduces the potential for
    implementation errors, especially in using the same word-alignment
    padding, if the same software is later modified to use a
    conventional codepoint. Use of the longer, 32-bit ExID further
    decreases the probability of such a false positive compared to those
    using shorter, 16-bit ExIDs.
...
    Use of the longer, 32-bit ExID consumes more
    space, but provides more protection against false positives.

Which is why I feel this motivation isn't sufficient for a DISCUSS.

I'd be glad to hear others' view of this as well.

Joe

---

From: Keith Moore <[email-masked]>
Date: Wed, 15 May 2013 15:35:52 -0400

On 05/15/2013 02:48 PM, Joe Touch wrote:
> Except that the document already describes the ExID as either 16-bit 
> or 32-bit:
>

It certainly seems like a valid topic for an AD to want to discuss with 
a WG.    And that's all that DISCUSS inherently means.

Keith

---

From: Keith Moore <[email-masked]>
Date: Wed, 15 May 2013 16:24:18 -0400

> Why? There's definitely a process failure there, and it should be blamed on the WG chairs and/or the AD, who should have either moved the work out of the working group or worked on updating the cha…

ADs shouldn't have to micro-manage the WGs and keep track of whether 
every single document that a WG is working on is authorized by its 
charter.  Fundamentally, it's the WG chair's job to stay within the charter.

What I was addressing with my above statement is that there seems to be 
a presumption on the part of some people that a WG can produce anything 
it wants, and that the IESG is under an obligation to approve such work 
unless it can object on some very specific grounds.    I can understand 
something resembling such a presumption for work that the WG is 
specifically chartered to do.   We don't want WGs investing their 
members' time and energy to produce something that will never see the 
light of day, and WG members need some assurance that their efforts are 
likely to bear fruit at least as far as publication is concerned.   But 
I see no reason that a WG should be able to presume that the IESG should 
ultimately accept something that they weren't specifically chartered to 
do in the first place.

Though probably a better remedy than to reject the document outright, 
would be for IESG to treat such documents as individual submissions.

Keith

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 May 2013 13:49:42 -0700

On 5/15/2013 7:49 AM, Ralph Droms wrote:
>> >The DISCUSS isn't there to make documents "better" - that's for COMMENTs. A DISCUSS there to catch a set of problems and to*block*  the document's progress until that problem is resolved.

> I'll agree with you *if*  you consider an unclear description of a
> feature of a protocol, severe enough that reader of the specification
> are not able to build interoperable implementations, as a problem for
> which a DISCUSS can be posted.
>
> In my opinion, there are many ways in which document can be unclear
> beyond the "motivation for a particular feature of a protocol is not
> clear enough."
>
> - Ralph

Absolutely. Note that issues interfering with interoperability or 
correctness are among the explicit DISCUSS criteria.

Joe

---

From: Thomas Narten <[email-masked]>
Date: Thu, 16 May 2013 00:39:50 +0200

Hi John.

I agree there are a number of specific cases where there is no
simple/obvious way to handle. I'd be OK with something fairly simple
as "unaffiliated" or "consultant" or something more nuanced. But I'd
like to think those are edge cases (but could of course be wrong in
how common they are). I was specifically thinking of much more obvious
cases where I'm pretty sure that their main employers is sending them.

One thing I've done at times is look at the attendance figures at
meetings to try to identify some of the entities that send the most
attendees to meetings (e.g.,, to track trends). As a result of that,
I've noticed that the number of "no affiliation given" seems high to
me, and I've seen obvious cases where I know the person and where they
were working in the past, and then when I check later, it turns out
they are still working at the same place (e.g. a Cisco, Google or
Huawei). Maybe its just the case that the registration form makes it
to easy to leave it blank.

Just loooking at the list
https://www.ietf.org/registration/ietf86/attendance.py?sortkey=3&login=
I easily counted 20 names of folk I know that don't list an
affiliation, where if any of them have changed employers, that number
would likely be very small.

Thomas

---

From: Stephen Farrell <[email-masked]>
Date: Thu, 16 May 2013 00:33:23 +0100

On 05/15/2013 07:38 PM, John C Klensin wrote:
> "180 names"? 

How about 7 densely worded paragraphs?

Sorry, couldn't resist:-)

S.

---

From: "Edwin A. Opare" <[email-masked]>
Date: Thu, 16 May 2013 00:14:09 +0000

In all earnestness I don't think a resume will be necessary at all :).

--
Regards,

Edwin A. Opare

On Wed, May 15, 2013 at 11:33 PM, Stephen Farrell <[email-masked]

---

From: Pete Resnick <[email-masked]>
Date: Wed, 15 May 2013 18:07:52 -0700

I initially replied to address Keith's comment. But a few things on Joe's:

On 5/15/13 7:41 AM, Keith Moore wrote:
>> The DISCUSS isn't there to make documents "better" - that's for COMMENTs.

Exactly right. Sometimes we forget; it's a good thing to remind us.

>> A DISCUSS there to catch a set of problems and to *block* the 

Mostly correct. However:

- If there is only one AD who wishes to DISCUSS and no other AD agrees 
with the DISCUSS holder, at the next telechat the document is unblocked. 
(See <http://www.ietf.org/iesg/voting-procedures.html>.)
- Even if others agree with the DISCUSS, the chair can be asked to use 
the alternate procedure, which requires 2/3 of non-recused ADs to agree 
to publication.

Which is to say, if there is only a single AD "blocking" a document, 
that block is essentially a 2 week affair if you are willing to push the 
point. No need for negotiating; if the WG decides that the AD is totally 
off base, tell your sponsoring AD that you're waiting the two weeks. 
People (unfortunately IMO) don't push the point nearly enough.

(For the record, the IESG has *never* used the latter of the two 
procedures.)

That said, I am also of the view that there is a third way, but I have 
never seen a WG attempt it:

DISCUSS should in fact require discussing. Assuming there was some good 
faith effort on the part of the WG to figure out what the AD was on 
about and they really assess that the AD has it wrong, a WG could say, 
"Sorry, we got this right. You are confused (or wrong). We are not 
changing the document. We are done discussing." At that point, I am of 
the opinion that the AD cannot hold the DISCUSS any longer. The AD must 
move to ABSTAIN. The discussion is, for all intents and purposes, over 
and to continue to DISCUSS is (IMO) an appealable offense. Then it is a 
matter of the IESG deciding whether there are enough ADs supporting the 
document (YES or NO OBJECTION) to count as consensus of the IESG. We 
ostensibly use 2/3 of non-recused ADs to mean "consensus", since (I 
think the theory goes) if you can't get 2/3 of ADs to agree that it's OK 
to publish a document, that is a sign of a lack of rough consensus in 
the IETF (and probably a serious problem in WG operation). Indeed, if 
the ADs are so off of their rockers that more than 1/3 of them are 
against a perfectly reasonable document, it's time for the appeals and 
recall procedures to be used.

(This is, BTW, where I disagree with Dave Crocker: Yes, a DISCUSS can be 
used as an exercise in authority, but only if it is allowed to be an 
"assertion that changes are being required". If the rest of the IETF 
started saying, "Sorry, no change is going to be made" instead of making 
changes simply to satisfy the AD, we'd actually be better off and the 
DISCUSS would stop being seen -- and used -- as authoritative.)

Finally, do keep in mind that, although there have been times where the 
numbers have been much different, currently there are 16 documents with 
outstanding DISCUSSes (and I think the total number has been pretty 
stable for a while), and 7 of those are on tomorrow's telechat and 
therefore should hopefully be cleared within a few days of when the 
document could most quickly have been approved anyway. So I'm not sure 
(and we should look at the statistics) how much of a problem this is for 
most documents we are producing.

> Why are people trying to sabotage IESG?

I'm sorry Keith, but your last line is rubbish. To claim that what 
people in this thread are talking about amounts to an attempt to 
sabotage the IESG is the height of hubris. In my opinion, the IESG has 2 
jobs as far as document review goes: (1) Check for serious technical 
problems that the WG might not have considered and make sure they get 
considered; and (2) Make sure that our processes have been followed to 
assure that the WG products are truly the product of (IETF-wide) 
consensus and fit within the policies of the IETF. I am *positive* that 
I have failed to limit my DISCUSSes on documents to one of those two 
reasons; I expect I will err and do so in the future. I should be 
corrected. We reject kings, and should.

As for the rest: The IESG should be looking for technical correctness 
problems and identify those that do not appear to have been properly 
considered by the WG, but trying to police document quality beyond that, 
including all of those listed in the NON-DISCUSS criteria, really are 
out of bounds for the IESG to be concerning themselves with. Sure, there 
are issues of clarity that can result in interoperability problems, but 
every one of those I have seen is because the words written can be read 
in a technically incorrect way. My take is you are solidly in the rough 
on the question of the DISCUSS/NON-DISCUSS criteria.

  Our documents are never going to be perfect and attempts to make them 
so actually lessen the quality of our output over the long term.

pr

---

From: Keith Moore <[email-masked]>
Date: Thu, 16 May 2013 00:55:41 -0400

On 05/15/2013 09:07 PM, Pete Resnick wrote:
>>> COMMENTs.
> with the DISCUSS holder, at the next telechat the document is 
> unblocked. (See <http://www.ietf.org/iesg/voting-procedures.html>.)
> the alternate procedure, which requires 2/3 of non-recused ADs to 
> agree to publication.
>
> that block is essentially a 2 week affair if you are willing to push 
> the point. No need for negotiating; if the WG decides that the AD is 
> totally off base, tell your sponsoring AD that you're waiting the two 
> weeks. People (unfortunately IMO) don't push the point nearly enough.

I think it's very unfortunate that IESG has adopted rules that work this 
way.   Part of IESG's job is to provide independent review of WG 
output.   It that review can be circumvented merely by waiting two 
weeks, that's a bug in the process.   And if an AD raises a DISCUSS 
about a matter of technical or document quality (or for that matter, 
about a process violation), and the WG isn't even willing to discuss the 
point, but instead relies on the two week timeout, I think that's 
grounds for appeal to the IAB.

> DISCUSS should in fact require discussing. 

We agree on that much.
> Assuming there was some good faith effort on the part of the WG to 
> figure out what the AD was on about and they really assess that the AD 
> has it wrong, a WG could say, "Sorry, we got this right. You are 
> confused (or wrong). We are not changing the document. We are done 
> discussing." At that point, I am of the opinion that the AD cannot 
> hold the DISCUSS any longer. The AD must move to ABSTAIN. 

I disagree with this in the strongest possible terms.  I believe that 
for IESG to have rules that insist on or encourage voting ABSTAIN when 
the AD really means "this document is not acceptable", is both 
irresponsible and dishonest on the IESG's part.  I also believe that it 
tends to mean that ADs who didn't actually review the document get more 
say than ADs who did.   The proper thing to do in that case is for 
either side to appeal to the IAB.

> The discussion is, for all intents and purposes, over and to continue 
> to DISCUSS is (IMO) an appealable offense. Then it is a matter of the 
> IESG deciding whether there are enough ADs supporting the document 
> (YES or NO OBJECTION) to count as consensus of the IESG. We ostensibly 
> use 2/3 of non-recused ADs to mean "consensus", since (I think the 
> theory goes) if you can't get 2/3 of ADs to agree that it's OK to 
> publish a document, that is a sign of a lack of rough consensus in the 
> IETF (and probably a serious problem in WG operation). Indeed, if the 
> ADs are so off of their rockers that more than 1/3 of them are against 
> a perfectly reasonable document, it's time for the appeals and recall 
> procedures to be used.

I don't think IESG voting should be thought of as a consensus process, 
except perhaps in the deadlock breaking procedure.   The typical number 
of reviewers on the IESG is too small for a consensus process to be 
meaningful.   With so few thorough reviewers outside of the WG, you need 
a procedure that at least initially assumes that all review comments 
have to be taken seriously.

>> important (in the sense that poor document quality can lead to as 
>> many interoperability or other problems) as technical correctness.
>>
> sabotage the IESG is the height of hubris. 

"sabotage" is probably not the best word I could have chosen.   But I do 
have the strong impression that people are clamoring for IESG reviews to 
be toothless, and that frustrates me.   Every time I attend an IETF 
meeting and look at what a broad spectrum of WGs are doing, I find that 
most of the ones I see are producing output of very dubious quality.   
While trying to fix this after WGLC almost never works well, the threat 
of IESG rejection of WG documents is one of the few incentives that WGs 
have to actually consider interests outside of their narrow areas.   
External review after a WG has completed its work is absolutely 
essential.   And in our current process, the only even marginally 
effective mechanism that we have for that is IESG review.

I understand that ADs can sometimes be capricious and sometimes simply 
be wrong, and I'm supportive of the idea that there should be some way 
to keep a single AD from holding things up.   But expecting ADs to 
change their reviews just because others don't agree with them is 
completely inappropriate.   So is letting people who didn't thoroughly 
reviewed a document have more sway in the review process than people who 
have reviewed it.

> In my opinion, the IESG has 2 jobs as far as document review goes: (1) 
> Check for serious technical problems that the WG might not have 
> considered and make sure they get considered; and (2) Make sure that 
> our processes have been followed to assure that the WG products are 
> truly the product of (IETF-wide) consensus and fit within the policies 
> of the IETF. I am *positive* that I have failed to limit my DISCUSSes 
> on documents to one of those two reasons; I expect I will err and do 
> so in the future. I should be corrected. We reject kings, and should.

I agree that we should reject kings.   But we should also reject 
stubborn working groups and stubborn authors.   We shouldn't presume 
that a WG knows better than the IESG.  The two have different 
points-of-view, but neither is inherently superior to the other.

> considered by the WG, but trying to police document quality beyond 
> that, including all of those listed in the NON-DISCUSS criteria, 
> really are out of bounds for the IESG to be concerning themselves 
> with. Sure, there are issues of clarity that can result in 
> interoperability problems, but every one of those I have seen is 
> because the words written can be read in a technically incorrect way. 

I disagree with this also.   Ambiguities in technical specifications 
that could affect interoperability need to be corrected no matter who 
finds them.   But I've rarely seen significant delays result from fixing 
minor ambiguities here and there.   The bigger delays come from the 
occasional documents that are so poorly written that small changes can't 
clarify them.    Those documents, frankly, need to be rewritten.   
(Perhaps the best thing that IESG could do in such cases is approve such 
documents as Experimental and commission a rewrite with a competent 
technical writer as co-author.)

> so actually lessen the quality of our output over the long term.

I think I understand what you mean, and probably agree.   But the 
problem with stating things in that way is that it can be taken to mean 
that we should just write things as quickly and sloppily as possible.

Keith

---

From: John C Klensin <[email-masked]>
Date: Thu, 16 May 2013 01:44:37 -0400

--On Thursday, May 16, 2013 00:55 -0400 Keith Moore
<[email-masked]> wrote:

>> Which is to say, if there is only a single AD "blocking" a
>> document,  that block is essentially a 2 week affair if you
>> are willing to push  the point. No need for negotiating; if
>> the WG decides that the AD is  totally off base, tell your
>> sponsoring AD that you're waiting the two  weeks. People
>> (unfortunately IMO) don't push the point nearly enough.
> 
> I think it's very unfortunate that IESG has adopted rules that
> work this way.   Part of IESG's job is to provide independent
> review of WG output.   It that review can be circumvented
> merely by waiting two weeks, that's a bug in the process.
> And if an AD raises a DISCUSS about a matter of technical or
> document quality (or for that matter, about a process
> violation), and the WG isn't even willing to discuss the
> point, but instead relies on the two week timeout, I think
> that's grounds for appeal to the IAB.

Keith,

I generally agree with Pete although I share your low opinion of
a lot of current IETF work, but your comment above seems to call
for comment from someone, like myself, who has often been
critical of the IESG.  I don't think the current rules are
ideal, but the effect of the ones that you and Pete cite isn't
really that a WG can circumvent a review by waiting two weeks.
First, the dissenting AD has those same two weeks to convince
others on the IESG that his or her position is reasonable or at
least needs more extensive consideration.  If that is possible,
then there is no longer a single AD objecting and the two week
rule does not apply.  If it is not possible, then there is
either something wrong with the objection or the AD making it
and probably it is reasonable for the process to move forward.

Conversely, a WG that decides to avoid actually engaging on an
issue in the hope of letting the two-week clock run out is
putting itself at considerable risk.  The IESG still have the
ability to fire WG leadership and even to close WGs.  Under any
reasonable circumstances, I assume that the IESG would respond
very strongly if an AD pointed out the a WG had ignored an
effort to discuss a substantive issue even if the rest of the
IESG disagreed with that particular AD about that issue.  I
can't remember any instances of a WG Chair being fired, or a WG
being shut down entirely, while documents were post IETF Last
Call and under IESG discussion, but I presume that is because
the IESG has never been provoked quite enough rather than
because there is any rule that would prevent it.  Firing a Chair
or closing a WG on the grounds that they refused to engage on a
substantive issue would, I assume, be sustained on any appeal.

Similarly...
 
> I agree that we should reject kings.   But we should also
> reject stubborn working groups and stubborn authors.   We
> shouldn't presume that a WG knows better than the IESG.  The
> two have different points-of-view, but neither is inherently
> superior to the other.

I don't think we should presume that a WG knows better than the
IESG either.  I do think it is reasonable to assume that a WG
knows better than a single AD who cannot convince others on the
IESG of the merit of his or her position or even that the issue
is important enough that they should read the document carefully
and form their own informed opinions.

    best,
      john

---

From: Keith Moore <[email-masked]>
Date: Thu, 16 May 2013 02:05:12 -0400

> and probably it is reasonable for the process to move forward.

I don't think I agree.   I do agree that a single AD shouldn't be able 
to indefinitely delay a WG's output, and that you want IESG to be able 
resolve such issues internally in the vast majority of cases (because 
appeals to IAB are very time- and resource-consuming). But I don't think 
two weeks is long enough in general to get other ADs to thoroughly 
review a document.  Two months would be much better.

Now it might be the case that IESG members will help each other out, and 
collaborate to log multiple DISCUSS votes to give everybody more time to 
review a document.   People of good will acting in good faith can 
usually find ways to work around buggy processes to make the right thing 
happen, at least in the most important cases.   But it's not ideal, and 
it relies on ADs "getting along" with each other - which creates 
incentives for ADs to not do an adequate job of reviewing some 
documents, so that they'll be able to call in favors from other ADs when 
they need them.   So overall I think the balloting process that IESG 
currently follows is seriously flawed.

> ability to fire WG leadership and even to close WGs.

In theory, yes.   But that hardly seems like a good tool for resolving 
issues in a single document, especially when the WG has other ongoing 
work that might turn out to be useful.    Also, at least when I was on 
IESG all of us seemed to be aware that though we did have the authority 
and responsibility to push back on poor work, we also had to be mindful 
of the potential for the community to mutiny.

> Under any
> IESG disagreed with that particular AD about that issue.

I'd hope so.  But I also think that the process should work even for an 
objection raised by an AD who was unpopular with the rest of IESG.  Of 
course I hope that ADs get along well with one another and work together 
to make a better Internet.  But I also know from experience that some 
ADs tend to have more clout than others, and that the ones with the most 
clout aren't always the ones with the best technical judgment.

Keith

---

From: Keith Moore <[email-masked]>
Date: Thu, 16 May 2013 02:19:58 -0400

On 05/15/2013 12:25 PM, Thomas Narten wrote:
> contact) and an affiliation.

I have mixed feelings about this.   On one hand I would like to know who 
(if anyone) is paying someone to do IETF work, because there's always 
some potential for inappropriate bias even among people with a high 
degree of integrity - and of course not everyone has such integrity.

On the other hand I don't think that a contributor's affiliation should 
mean anything at all when evaluating that contributor's input to IETF.   
If people treat contributors from major companies as having more weight 
than other contributors, it makes a joke out of the whole notion of 
consensus-based decision making.   (And we all know that people do this 
sometimes.)

I also don't think that anyone should automatically presume that a 
contributor's input is representing his employer's interests.

On balance, I do sometimes find it helpful when IETF contributors 
disclose their employers.   But I don't think it should be required.   
And if everybody stopped disclosing their employers in the context of 
IETF conversations, it might be a good thing.

Keith

---

From: Ted Lemon <[email-masked]>
Date: Thu, 16 May 2013 12:22:55 +0000

I must say that I have enjoyed reading the discussion between the three of you, and think it is immensely valuable in explaining what the IESG ought to be doing.   You three should write it up.

---

From: [email-masked] (Dale R. Worley)
Date: Thu, 16 May 2013 11:47:00 -0400 (EDT)

> From: Thomas Narten <[email-masked]>
> 

I believe that this is a less good idea than first appears.

One objection is that the concept of "affiliation" is less simple than
it appears.  People normally expect it to mean "Who is your employer?"

As John mentions, it can mean "What is the organization whose
interests you are promoting?"  But in regard to your observation about
use cases, it can also mean "What area of technology do you have
experience in?"

In many cases, "affiliation" is related to "Who is paying for your
attendance?"  But there are situations where your employer may be
willing to pay for your attendance but doesn't want you to be seen as
officially representing them.

And (at least in common-law countries) one can simply invent an
(unincorporated!) company name and claim it as your affiliation.

In practice, the way we deal with these problems is that we consider
everyone to be individuals, rather than as "the representative of
company XYZ", and it takes time to build a reputation.  One is thus
unwilling to sacrifice that expensive reputation to advocate a poor
solution because one's employer favors it.

    3) Google names, look at authorship info in RFCs, linked in,
    etc. Works in a lot of cases, but is sometimes more work than
    seems appropriate.

If you can't find any information about them, they have no reputation.

Dale