Subject: Is this an elephant? [Was: call for ideas: tail-heavy IETF process]

From: "Adrian Farrel" <[email-masked]>
Date: Wed, 15 May 2013 21:30:26 +0100

The claim (or one of the claims) is that some ADs may place Discusses that are
intended to raise a discussion with the authors/WG that could equally have been
raised with a Comment or through direct email. This, it is claimed, may
unnecessarily delay the document from completing the publication process.

Now the dangerous bit,

Suppose the AD raised her concern by writing a Comment or sending an email and
balloting "No Objection." That would mean that the I-D would be approved for
publication.

At this point either:
- the discussion goes on, but the document becomes an RFC anyway
or
- the responsible AD holds the document pending satisfactory completion of the
discussion.

I suggest that the former is a bad result. Not that the authors/WG will ignore
the discussion, but if they disagree on something the AD considers very
important, the authors/WG have no incentive to participate in the discussion. Of
course, all participants in this thread so far would never behave like that, but
there is a possibility that this will happen for some authors.

I also suggest that the latter introduces exactly the same amount of delay as
the Discuss.

Personally (but this may reflect my Discusses) I find that an active engagement
by the authors and the Discussing AD on the issue and the potential resolution,
always leads to speedy progression of the document either with the AD feeling
stupid, or the document improved. Both are adequate results.

Adrian

---

From: Ted Lemon <[email-masked]>
Date: Wed, 15 May 2013 20:39:05 +0000

> there is a possibility that this will happen for some authors.

This can also happen purely by accident—it need not involve malice on anyone's part.

---

From: Dave Crocker <[email-masked]>
Date: Thu, 16 May 2013 09:23:09 -0700

On 5/15/2013 1:30 PM, Adrian Farrel wrote:
> I suggest that the former is a bad result.

> stupid, or the document improved. Both are adequate results.

Adrian,

I suggest we all take a look at the original text of the Subject field.

The problem here is that basic reviewing is being done by the ADs too 
late in the process.  We are making the mistake of having ADs be exempt 
from IETF Last Call, and allowing them to be unprepared for the IESG 
vote.  So we are combining "education" with "voting".  That's a paradigm 
error.

By the time the IESG schedules the vote, ADs need to already have 
educated themselves about the document.

Of course, the IESG discussion during the voting process well might 
uncover an actual, serious issue with the document; this should be 
exceptional and it should be an issue that the /IESG/ agrees needs to be 
resolved and it means that voting should not take place until it is. 
But that is quite different from the usual "let's talk about it" kind of 
Discuss imposed by individual ADs.

So here's a simple proposal that pays attention to AD workload and 
includes a simple efficiency hack:

      When the IETF Last Call is issued, wait a few days, to see whether 
any serious issues are raised by the community.  The really serious ones 
usually are raised quickly.  If there are none, it's pretty certain the 
document will advance to an IESG vote.  That leaves 7-10 days of IETF 
Last Call for ADs to get educated and ask questions, just like everyone 
else.

Jari has expressed the goal of having AD concerns be raised more 
publicly.  Moving AD review and comment to the IETF Last Call venue 
nicely accomplishes this, too.

On 5/15/2013 8:55 AM, Thomas Narten wrote:
 > 1) Discuss criteria should be principles, not rigid rules. The details
 > of the issue at hand always matter, and it will sometimes come down to
 > judgement calls where reasonable individuals just might disagree.

We've been doing protocol specification for 40 years.  If we can't 
codify the major concerns that warrant blocking advancement of a 
protocol, we're just being lazy.  Really.  That a given situation might 
cause the IESG to decide to enhance the list does not mean that the list 
shouldn't seek to be complete and precise and that ADs be held to it.

At every turn, the approach we've taken to the IESG review and approval 
process is to limit actual accountability to the community.  Everyone is 
well-intentioned, but really this makes the process a matter of 
personalities and not the rigor of serious professionalism.

The IESG's process is far, far better than it was 10-15 years ago, but 
it still lacks meaningful predictability and serious accountability; it 
is not reasonable for the process to require that authors and chairs 
take the initiative of complaining when an AD is being problematic.

In terms of quality assurance, the idea that we have a process that 
relies on the sudden insight of a single AD, at the end of a many-month 
process, is broken.  It's fine if that person sees something that 
everyone else has missed until then, but that is quite different from 
designing a process that is claimed to rely on it.

And of course, the reality is that we allow bad specs out the door all 
the time; we just allow fewer of them than many/most other standards 
bodies...

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: "Adrian Farrel" <[email-masked]>
Date: Thu, 16 May 2013 17:40:15 +0100

That's a good question Dave.

The community might like to comment. 

On the whole, I am told that if an AD weighs in with her comments during working
group last call, her fearsome personality may overwhelm some of the WG
participants and she may dominate the WG consensus.

Is it possible that the same would happen in IETF last call? I know, of course,
that a number of you have a robust ability to defend your opinions, but how
would you (the community, not Dave) feel if ADs (speaking as ADs, not speaking
as individual contributors) made their review comments during IETF last call?

I agree that this would bring the tail forward a little (not a lot).
I don't believe it would reduce AD work-load (which is another issue entirely).
I have mixed feelings about whether it would change the processing time for a
draft. That might depend on how the review is handled in IETF last call.

Lastly, I think I disagree with you about "really serious" IETF last call
comments coming in the first few days. It seems to me that we also get an number
of late comments of substance resulting from people who are not familiar with
the document reviewing it (such as directorate reviews and people who didn't
notice the I-D sneaking through).

Still thinking,
Adrian

> -----Original Message-----
> From: Dave Crocker [mailto:[email-masked]]
> Sent: 16 May 2013 17:23
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: Is this an elephant? [Was: call for ideas: tail-heavy IETF
process]

---

From: Dave Crocker <[email-masked]>
Date: Thu, 16 May 2013 10:01:26 -0700

> participants and she may dominate the WG consensus.

Only women ADs? [1]

But seriously...

> Is it possible that the same would happen in IETF last call?

I think that the public sway of ADs on the IETF list is less of a risk 
than on wg mailing lists.  And note that my suggestion has ADs waiting 
to weigh in until community-generated active disagreement with the spec, 
or its passive agreement, has been established.

In any event, the current reality of having an AD weigh in with a a 
process-blocking Discuss is not exactly /under-/whelming...

Simply put:  We are starting with the reality that an AD is going to be 
expressing their opinions.  The question is when and how.  It's not 
going to be /less/ intimidating to have it done as a Discuss.

Contrary to some of the mythology that has been expressed about this, 
the frequent reality is that the typical wg goal is to clear the 
Discuss, not really to engage in debate with an AD who is blocking 
document progress.  (I've watched this reality many times over the 
years.)  That is far less healthy than having the AD's concern become 
part of the /public/ review process.

> I don't believe it would reduce AD work-load (which is another issue entirely).

Yes, the timing change is small.  But the context change is enormous.

> comments coming in the first few days. It seems to me that we also get an number

Some, sure.  But I said it was an efficiency hack.  The goal isn't for 
it to be perfect, just helpful.

d/

[1] The question of proper referential language is a continuing surprise 
to me, given that the currently-popular conventions create more problems 
than they solve -- and it's even a question on a PSAT preparatory 
example that I saw yesterday.  There's a pretty straightforward 
alternative that works nearly all the time:

    http://dcrocker.net/#gender

---

From: Ted Lemon <[email-masked]>
Date: Thu, 16 May 2013 17:14:44 +0000

>   http://dcrocker.net/#gender

That's what I do.  It gets a bit awkward with verb agreement and constructs like "themself," which elicits the dreaded red snake underline of doom.   But I find it more comfortable than just subverting the sexist paradigm with a differently sexist paradigm.

---

From: [email-masked] (Martin Rex)
Date: Thu, 16 May 2013 20:06:15 +0200 (CEST)

> the time; we just allow fewer of them than many/most other standards 

But different to (at least some) other standards bodies, we lack an
official means to publish defect reports (aka errata) to document defects
in a _timely_(!!) fashion.  (Timely = can be found where the RFC says
that it can be found, and within at most a few weeks after the
defect/omission has been found by an implementor).

In theory, we have the errata process, and recent RFC even include
a direct URL pointer to the RFC-Editors errata page on the title page:

   Information about the current status of this document, any errata,
   and how to provide feedback on it may be obtained at
   http://www.rfc-editor.org/info/rfcXXXX.

("XXXX" containing the real number of the RFC on which this appears).

However, the Errata process is currently working poorly, primarily
because a number of folks (including some IETF leadership) currently
thinks that posting something a trivial as a missing vital one-line
clarification to a published RFC as a "substantial change" that can
only be performed by publishing a whole new RFC.

-Martin

---

From: Scott Brim <[email-masked]>
Date: Thu, 16 May 2013 14:08:35 -0400

On Thursday, May 16, 2013, Dave Crocker wrote:

> By the time the IESG schedules the vote, ADs need to already have educated
> themselves about the document.
>

Oh, so you're suggesting adding another phase to the process: IESG
education.  OK.

> document will advance to an IESG vote.  That leaves 7-10 days of IETF Last
> Call for ADs to get educated and ask questions, just like everyone else.
>

Placing a time limit on some phase of the process doesn't produce quality.
 The process should take as long as it must in order to produce a good
outcome.  The question shouldn't be how long the process takes (that's just
a cheap shortcut), it should be how to make it more efficient in doing
well.  I have an idea: let's place a time limit on working groups: they
need to finish drafts in six weeks or there will be penalties.  Why not?

Scott

---

From: joel jaeggli <[email-masked]>
Date: Thu, 16 May 2013 13:49:26 -0700

>> participants and she may dominate the WG consensus.
If Tiresias prophet of Thebes is in your wg, I'd listen.
> to weigh in until community-generated active disagreement with the 
> spec, or its passive agreement, has been established.
>
Early participation AD's weighing in on draft's generally takes the form 
of just another participant unless you're asking for an opinion on a 
process point. wearing different hats on the course of the lifecycle is 
normal imho. opinion at the mic or on the list is not iesg review. The 
sponsoring AD (who is presumably the most involved) is not likely the 
one with the discuss later in the process since they had to do the 
initial review, put it through the ietf last call and then ballot it, so 
fundamentally they should have a document they can live with when it 
gets to that stage.
> Simply put:  We are starting with the reality that an AD is going to 
> be expressing their opinions.  The question is when and how. It's not 
> [1] The question of proper referential language is a continuing 
> surprise to me, given that the currently-popular conventions create 
> more problems than they solve -- and it's even a question on a PSAT 
> preparatory example that I saw yesterday.  There's a pretty 
> straightforward alternative that works nearly all the time:
>

---

From: Yoav Nir <[email-masked]>
Date: Thu, 16 May 2013 20:46:12 +0000

On May 16, 2013, at 9:08 PM, Scott Brim <[email-masked]<mailto:[email-masked]>> wrote:

On Thursday, May 16, 2013, Dave Crocker wrote:
By the time the IESG schedules the vote, ADs need to already have educated themselves about the document.

Oh, so you're suggesting adding another phase to the process: IESG education.  OK.

I don't speak for Dave, but it makes sense that when a document reaches the IESG telechat, the questions to be asked are (1) Did this get sufficient review by sufficiently capable people, and (2) Has the process been followed, including have all issues been addressed.

The time for asking whether the group has considered making this field fixed length instead of variable, or whether RFC 2119 language is used in an appropriate way, or whether the protocol is extensible enough is at IETF last call. This is true of any participants, and ADs can do this too, regardless of whether their presence might intimidate the crowd. It could even be that if ADs voice their concerns at that time, others might join in, making the LC time more useful and less about just waiting two or four weeks.

There is a problem, though, that this will increase the load on ADs. Other concerns raised during IETF LC may lead to revised I-Ds, which the ADs will need to re-read (or at least look at the diff). I don't know how significant this extra work would be, but it would come at a time that we're thinking of ways to reduce AD workload. It might also require prolonging the LC time, because there would be actual discussion in it.

Yoav

---

From: Stephen Farrell <[email-masked]>
Date: Thu, 16 May 2013 21:55:23 +0100

I think Dave's idea is worth looking at, but have one comment:

On 05/16/2013 09:46 PM, Yoav Nir wrote:
> There is a problem, though, that this will increase the load on ADs.

There is that. But don't forget that ADs mostly read everything
in IESG review and often comment. Even leaving aside DISCUSSes,
if every IETF LC draft got a bunch of AD review comments, then
the IETF LC would be much busier than now. And its in the nature
of IETFers to chime in. So this would I suspect increase everyone's
load, if it works, and not just ADs.

Still worth a look though.

S.

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Thu, 16 May 2013 21:00:19 +0000

> participants and she may dominate the WG consensus.

There may be places where that happens, but I would be surprised if it happened in my working group. I think it is fair to say that the AD (or an IAB member, or someone who has recognized expertise on the topic) is likely to be listened to more carefully than some others might. Heck, I'm careful when I make a technical comment on a document in my working group, flagging it with "</chair>" to ensure that it is seen as intended - a comment by a competent practitioner of the art, not a process remark or an attempt to trump some other view. Speaking personally, I would prefer to see those comments in the WGLC, not IETF Last Call, if we can make that happen. For example, I'd like to get directorate reviews done (gen-art, security directorate, etc) in the timeframe of WGLC.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 17 May 2013 09:04:38 +1200

Dave,

On 17/05/2013 04:23, Dave Crocker wrote:
...
> late in the process.  

You are making a lot of assumptions in that sentence. At least these:

1. "Basic" reviewing means ....

2. At some stage before approval, ADs should perform basic reviewing.

3. Doing basic reviewing after the document has completed IETF LC
   is too late.

Now, if "basic" reviewing means (A) "looking for fundamental flaws"
that is one thing. If it means (B) "getting a general understanding
of the topic" it's another. I'm really not sure which you mean.

> We are making the mistake of having ADs be exempt

Ah, maybe you mean basic (B). Well, I think this is a totally
unrealistic expectation. There are too many drafts on too diverse
a range of subjects for ADs to educate themselves at the time
of IETF LC, which may be weeks or months before a draft hits
the agenda. (I'm not talking about the sponsoring AD or her
co-AD, of course: they are presumed to be aware of what's going
on in their area.) Busy people are deadline driven, and the
IESG agenda is an imminent deadline for ADs in a way that an
IETF LC in another Area isn't.

> uncover an actual, serious issue with the document; 

And that's basic (B). I think we all agree that it's unfortunate
if a basic flaw shows up during the IESG ballot phase, but
please don't blame the IESG for that. Blame the WG and the
IETF as a whole for failing to pick it up during WG LC and
IETF LC. Thank the IESG for finding it.

> this should be
> resolved and it means that voting should not take place until it is. But
> that is quite different from the usual "let's talk about it" kind of

I'm not quite sure what you're saying here, but I think you're
saying that the majority of clarity and cross-area issues should
be picked up earlier. I couldn't agree more, but don't expect the
over-worked IESG to fix that. The rest of us need to fix that.

> Last Call for ADs to get educated and ask questions, just like everyone

Which, as I said above, is a totally unrealistic expectation.

> nicely accomplishes this, too.

Thanks but no thanks. My mail is full enough with discussion of drafts
I will never read as it is. AD reviews should certainly go to the
WG list unless they are only nits, but isn't that what everybody does
anyway?

> shouldn't seek to be complete and precise and that ADs be held to it.

I agree with Thomas. We need to be able to apply common sense. Rules
and targets are the enemy of common sense.

> it still lacks meaningful predictability and serious accountability; it

I find that the tracker in its present state has substantially improved
both visibility and predictability.

> take the initiative of complaining when an AD is being problematic.

Huh? Who else will complain?

> designing a process that is claimed to rely on it.

It's supposed to be a backstop, and I think we concur that too many
issues are being left for the backstop.

It's not the IESG that is falling down on the job. It's the rest
of us, who keep sending defective documents to the IESG.

> the time; we just allow fewer of them than many/most other standards

Hubris alert!

   Brian

---

From: Ralph Droms <[email-masked]>
Date: Thu, 16 May 2013 17:56:40 -0400

> There may be places where that happens, but I would be surprised if it happened in my working group. I think it is fair to say that the AD (or an IAB member, or someone who has recognized expertise…

I think Fred is returning to an earlier theme here, when he asks for earlier review.

Perhaps, as has been already suggested in this thread, we should think about SIRSbis? 

First, from draft-carpenter-icar-sirs-01:

     The procedure described in this document is intended to
     solve, or palliate, a number of related problems that
     have been observed in the IETF process [PROBLEM]:

          *    submission of documents to the IESG that
               still have significant problems (leading
               to delay)

          *    failure to detect fundamental problems
               and Internet- wide issues at an early
               stage

     Particularly because of the second point, it is
     impossible to resolve these problems simply by
     giving additional responsibility to working groups
     themselves. An additional procedure is needed.

In my opinion, it's important to assign responsibility (and accountability) to all WGs for producing publication-ready documents.  I agree that some additional work is needed before WGs send documents to the IESG.  Perhaps we can accomplish these goals through reorganizing the work we are 

I suggest we might want to combine the need for more responsibility with the discussion of a new "really close to being ready" document state.  However, rather than a new document state, suppose we codify the expectation that a document that has passed WG last call is essentially ready-to-publish?  Correspondingly, any significant problems found in a document after WG last call would be considered a serious defect.

   Discussion:  I realize that, elsewhere in this thread, it has been
   asserted (or at least implied), that WGs already have this responsibility
   and DISCUSSes on document are usually unnecessary.  In practice, while
   there may still be unnecessary DISCUSSes, my experience as AD was that
   most DISCUSSes were appropriate and each one referred to a problem that
   the WG had missed.

Let's get all the expert review possible - directorate, AD, cross-area - in the WG last call review.  What pops out *should* be ready for publication.  Any issues raised by these reviews in WG last call will be exposed to and can be discussed by the WG at large, rather than being buried in the noise of IETF last call discussions or being fixed in more focused discussions among the IESG and the document authors.  This procedure diverges some from draft-carpenter-icar-sirs-01, in that it doesn't add a new form of review process.  Instead, it reschedules reviews that were going to take place anyway earlier in the process, so there is little or no new work added to the document publication process.

Perhaps the WG chairs would want to assign document shepherds earlier in the process, as well, investing the document shepherds with the responsibility of getting the right reviews and advising the WG chairs as to the readiness of the document for advancement.

Any WGs willing to volunteer as experimental subjects?  There is really no new process to invent ... it's mostly a matter of realigning expectations and responsibilities out to 

- Ralph

---

From: Keith Moore <[email-masked]>
Date: Thu, 16 May 2013 17:58:29 -0400

On 05/16/2013 04:46 PM, Yoav Nir wrote:
> The time for asking whether the group has considered making this field 
> fixed length instead of variable, or whether RFC 2119 language is used 
> in an appropriate way, or whether the protocol is extensible enough is 
> at IETF last call. 

Actually the time for asking these questions is long before IETF-wide 
Last Call.  We need widespread review of proposals for standards-track 
documents long before a WG thinks it's finished with those documents.   
It's a gaping hole in our process.

Fix that problem, and most of the conflicts between IESG and WGs that 
surround DISCUSS votes will go away.

Keith

---

From: Ralph Droms <[email-masked]>
Date: Thu, 16 May 2013 18:06:33 -0400

Dave - I hope you'll indulge my selective quoting as I have a couple of specific points to address.  My apologies if I end up quoting you out of context...

On May 16, 2013, at 12:23 PM 5/16/13, Dave Crocker <[email-masked]> wrote:

> [...]
> 
> So here's a simple proposal that pays attention to AD workload and includes a simple efficiency hack:
> 
>     When the IETF Last Call is issued, wait a few days, to see whether any serious issues are raised by the community.  The really serious ones usually are raised quickly.  If there are none, it's pretty certain the document will advance to an IESG vote.  That leaves 7-10 days of IETF Last Call for ADs to get educated and ask questions, just like everyone else.
> 
> Jari has expressed the goal of having AD concerns be raised more publicly.  Moving AD review and comment to the IETF Last Call venue nicely accomplishes this, too.

I just posted elsewhere a suggestion to move this review even earlier, to WG last call.  Accomplishes most of the same ends, while putting the discussion in front of the IETF participants who are, presumably, most invested in the resulting document.

> 
> 
> [...]
> In terms of quality assurance, the idea that we have a process that relies on the sudden insight of a single AD, at the end of a many-month process, is broken.  It's fine if that person sees something that everyone else has missed until then, but that is quite different from designing a process that is claimed to rely on it.

As you and I have discussed in person, I am 100% in agreement with this comment.  As much as I liked to think of myself, when I was an AD, as a rock-god Network Expert with complete and in-depth insight into every document I reviewed, I know the reality was that any problems I might have found were related to the old observation that "even a blind squirrel finds an acorn occasionally".
> 
> And of course, the reality is that we allow bad specs out the door all the time; we just allow fewer of them than many/most other standards bodies...

You're such an optimist.

- Ralph

---

From: Ralph Droms <[email-masked]>
Date: Thu, 16 May 2013 18:07:50 -0400

On May 16, 2013, at 5:58 PM 5/16/13, Keith Moore <[email-masked]> wrote:

>> The time for asking whether the group has considered making this field fixed length instead of variable, or whether RFC 2119 language is used in an appropriate way, or whether the protocol is extensible enough is at IETF last call. 
> 
> Actually the time for asking these questions is long before IETF-wide Last Call.  We need widespread review of proposals for standards-track documents long before a WG thinks it's finished with those documents.   It's a gaping hole in our process.

Hear, hear.

> 
> Fix that problem, and most of the conflicts between IESG and WGs that surround DISCUSS votes will go away.

Well, you might be a little optimistic here, but I agree in theory.

- Ralph

---

From: joel jaeggli <[email-masked]>
Date: Thu, 16 May 2013 15:09:32 -0700

On 5/16/13 2:58 PM, Keith Moore wrote:
>> The time for asking whether the group has considered making this 
>> field fixed length instead of variable, or whether RFC 2119 language 
>> is used in an appropriate way, or whether the protocol is extensible 
>> enough is at IETF last call. 
>
> documents long before a WG thinks it's finished with those 
> documents.   It's a gaping hole in our process.
>
As a Chair and as an AD I have asked for external and cross area reviews 
of some documents before they were considered for WG acceptance. this 
doesn't apply to all work we processed but it does apply to those where 
we were clear that such input was going to be useful.  One case you can 
see for that today is with capwap extensions that are potentially in 
opsawg.
> surround DISCUSS votes will go away.
Maybe but I wouldn't take that as an article of faith. You're going to 
get pressure for more changes when fresh eyes review something.

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 16 May 2013 16:12:28 -0600

>> Actually the time for asking these questions is long before IETF-wide Last Call.  We need widespread review of proposals for standards-track documents long before a WG thinks it's finished with th…

One suggestion I've heard several times is a kind of cross-area buddy
system (e.g., ask or assign someone with clue in Area X to help WG Y in
Area Z early and often with regard to specific issues that are often
addressed in Area X). Unfortunately, this suffers from the same problem
that too many WGs suffer from on their own: participant burnout. But I
still think it's a good idea...

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Thu, 16 May 2013 22:38:36 +0000

> There is a problem, though, that this will increase the load on ADs. Other concerns raised during IETF LC may lead to revised I-Ds, which the ADs will need to re-read (or at least look at the diff)…

If they raise the issue later in a "discuss", will they not have to do this anyway? How does this relate to the timing of the comment or the vehicle by which it is conveyed?

----------------------------------------------------
The ignorance of how to use new knowledge stockpiles exponentially. 
   - Marshall McLuhan

---

From: Keith Moore <[email-masked]>
Date: Fri, 17 May 2013 00:11:41 -0400

> get pressure for more changes when fresh eyes review something.

Yeah, every new set of eyes that looks at a document is going to have 
some new ideas for what the document should be.   The trick is to get 
those eyes to look at the document earlier in the process, when it's 
easier to fix problems.     Maybe if we did the early review well 
enough, the scope of Last Call comments could be limited in some way.

Keith

---

From: Yoav Nir <[email-masked]>
Date: Fri, 17 May 2013 09:18:53 +0000

> load, if it works, and not just ADs.

More community participation at IETF last call is a good thing. 

The problem with AD's time is that they will need to review at IETF last call, and then may need to review again after some changes have been made (including changes not related to their own issue). That's where the increased AD load comes from.

Yoav

---

From: Yoav Nir <[email-masked]>
Date: Fri, 17 May 2013 09:31:08 +0000

> Actually the time for asking these questions is long before IETF-wide Last Call.  We need widespread review of proposals for standards-track documents long before a WG thinks it's finished with tho…

Sure. But we have opinionated ADs who read every draft that comes to the IESG. There is no way they have time to participate in all of the working groups. I, as a participant, can read drafts as they are discussed in working groups, because I'm free to ignore all the drafts that are not interesting to me. ADs don't have that luxury.

> Fix that problem, and most of the conflicts between IESG and WGs that surround DISCUSS votes will go away.

Good reviewers are a scarce resource, and there are 500(*) working group drafts competing for their attention. That's a hard problem to fix.

Yoav

(*) Went to datatracker, and searched for active drafts that start with  "draft-ietf-". I probably hit some things that have already progressed, but OTOH the 500 number is too round, and may be a limitation of datatracker.

---

From: Yoav Nir <[email-masked]>
Date: Fri, 17 May 2013 09:32:10 +0000

> If they raise the issue later in a "discuss", will they not have to do this anyway? How does this relate to the timing of the comment or the vehicle by which it is conveyed?

If you review early, you later might feel like you need to review again, because the document has changed some. Hence - more work.

Yoav

---

From: Stephen Farrell <[email-masked]>
Date: Fri, 17 May 2013 10:32:28 +0100

> The problem with AD's time is that they will need to review at IETF last call, and then may need to review again after some changes have been made (including changes not related to their own issue)…

I'm not so sure that's a problem. The additional time I think
(for me) would be in reading the 100's of ietf-discuss mails
that'd get triggered by my or some other AD's comments. Not
all comments would trigger that of course, but if you look at
the minutes for any IESG telechat [1] and imagine that all
those comments had been sent to ietf-discuss during IETF LC
then you might worry about what'd happen on this list.

I'm not against considering this idea at all, but we should
bear in mind that any random thread on this list has a non-zero
probability of turning into major amounts of traffic. Most of
that traffic would doubtless be excellent IETF LC stuff, but
we maybe need to be careful in what we wish for too.

Cheers,
S.

[1] https://www.ietf.org/iesg/minutes/2013/index.html

---

From: Jari Arkko <[email-masked]>
Date: Fri, 17 May 2013 16:34:18 +0300

Dave, Ralph,

> I just posted elsewhere a suggestion to move this review even earlier, to WG last call.  Accomplishes most of the same ends, while putting the discussion in front of the IETF participants who are,…

We've also said that we'd like to move the directorate reviews earlier, to WGLC time. (Subject to us finding a way to do that without increasing directorate workload too much.) See the thread "ways forward with the tail-heavy aspects of the IETF process".

But overall, we have to be careful that we don't move the same processes to a process step that has a different name but in reality will not impact how things are done. Part of the idea for moving directorate reviews earlier is that more of the responsibility for dealing with these could reside with the WGs rather than ADs. And that is a change that affects substance and may improve scalability. Similar reasons are needed for other changes.

I'm not opposed to moving AD reviews to IETF LC time or even earlier, but we'd have to find a way to accommodate both technical comments as well as those relating to taking LC feedback into account. ("draft-foo LC feedback from Joe was was not appropriately addressed" and similar end-of-process checks.)

Jari

---

From: Keith Moore <[email-masked]>
Date: Fri, 17 May 2013 10:01:56 -0400

> Sure. But we have opinionated ADs who read every draft that comes to the IESG. There is no way they have time to participate in all of the working groups. I, as a participant, can read drafts as th…

Unless things have changed a great deal since I was on IESG, ADs do have 
the luxury of not reading drafts.   When I was an AD I tried to read 
every draft that was in my area (Applications), and every draft that 
seemed to have the ability to affect applications developers.    The 
lower in the protocol stack, the less likely that I'd feel like I'd have 
anything useful to say about a draft.   Even when I "read" a draft 
outside of my area, in many cases it was just skimming the draft looking 
for red flags.   I developed a pretty good sense of whether a group had 
done due diligence or whether there were serious technical omissions 
that they were trying to ignore.   Only in the latter (rare) cases did I 
feel like such drafts needed more of my attention.

I certainly agree that ADs don't have time to participate in all working 
groups, or even probably 10% of our working groups.   But WGs should be 
able to periodically summarize what they're doing - what problem they're 
trying to solve, what approach they're taking, what technologies they're 
using, what major decisions they've made, what the current sticking 
points seem to be, what problems are as yet unresolved, what potential 
for cross-group and cross-area effects have been identified, and what 
efforts have been made to get the affected parties in the loop.   For 
most groups that summary should be maybe 2-3 pages.   The ADs should be 
able to verify that those summaries are accurate and reasonably 
complete, or appoint a trusted WG observer other than the chair to 
review each summary. ADs and other members of the community should be 
able to view those summaries and comment on their accuracy.   And I 
think it would be reasonable for everyone on IESG to read through those 
summaries once in awhile - at least for groups that seemed likely to 
affect their areas of concern.   I think that such summaries could 
actually lessen IESG's workload.
> Good reviewers are a scarce resource, and there are 500(*) working group drafts competing for their attention. That's a hard problem to fix.

That's a related but IMO somewhat different problem.   Working groups 
are producing far too many documents.   That's one reason (far from the 
only one) why WGs shouldn't undertake documents that aren't specifically 
authorized in their charters.

Keith

---

From: Keith Moore <[email-masked]>
Date: Fri, 17 May 2013 10:22:14 -0400

> If you review early, you later might feel like you need to review again, because the document has changed some. Hence - more work.
Yes, but you only need to review what has changed.     It _can_ be more 
work, particularly for documents that have changed a lot, or if it's 
been too long since the last review.    But I really wouldn't suggest 
that ADs should do several reviews of each document.    I suspect that 
the trick is to review a document at the time the WG thinks that it's 
maybe 70-80% done (which is to say the WG is probably closer to 40-50% 
done), but when some issues still seem unresolved.   That's when the 
ADs' input, and external input in general, can be the most valuable.    
That way, ADs should be able to say "yes, but you're totally ignoring 
this very important issue here, and you really have to deal with it" at 
a time when the WG isn't yet so exhausted or off in the weeds that it 
can't focus on it.   Then the ADs just need to track the changes from 
that point forward, to make sure that the issues identified were dealt 
with satisfactorily.

Of course new issues can still be identified, and WGs can address 
previously identified issues in unsatisfactory ways.   But at some point 
(well prior to WGLC) it might be appropriate to raise the bar for new 
issues.   At some point it should take a serious defect to significantly 
delay publication of a document, and at that point it might make more 
sense to consider other remedies for identified less-serious issues, 
especially if the existing protocol isn't seen to be actually harmful 
and it appears that the protocol can be extended to address the issues 
in a manner that is compatible with existing implementations.   In those 
cases, it might make sense to go ahead and publish, but charter the WG 
to extend the protocol to address those issues.

Keith

---

From: Dave Crocker <[email-masked]>
Date: Fri, 17 May 2013 08:37:34 -0700

On 5/17/2013 7:01 AM, Keith Moore wrote:
> But WGs should be able to periodically summarize what they're doing -
> what problem they're trying to solve, what approach they're taking, what
> technologies they're using, what major decisions they've made, what the
> current sticking points seem to be, what problems are as yet unresolved,
> what potential for cross-group and cross-area effects have been
> identified, and what efforts have been made to get the affected parties
> in the loop.   For most groups that summary should be maybe 2-3 pages.
> The ADs should be able to verify that those summaries are accurate and
> reasonably complete, or appoint a trusted WG observer other than the
> chair to review each summary. ADs and other members of the community
> should be able to view those summaries and comment on their accuracy.

The idea that working groups should be required to issue periodic 
project progress reports seems strikingly reasonable and useful.

This makes the folks who are the most knowledgeable responsible for 
assessing their work, and should facilitate public review. Recording the 
sequence of reports into the wg datatracker could nicely allow 
evaluating progress over time.

It also, of course, nicely distributes the work.

  d/

---

From: Brian Haberman <[email-masked]>
Date: Fri, 17 May 2013 11:44:29 -0400

Dave,

> It also, of course, nicely distributes the work.

We (the INT ADs) have been trying to do that for the past year.  See 
http://trac.tools.ietf.org/area/int/trac/wiki/IETF86 as an example.  The 
goal is to provide a place for INT area WGs to summarize what they have 
been doing so that others within INT can get an idea of what is going 
on.  It is not to the level of a "progress report", but it does provide 
a useful summation (at least based on the feedback that I have received 
to date).

Regards,
Brian

---

From: t.p. <[email-masked]>
Date: Fri, 17 May 2013 18:28:16 +0100

----- Original Message -----
From: "Dave Crocker" <[email-masked]>
To: <[email-masked]>
Sent: Friday, May 17, 2013 4:37 PM
> project progress reports seems strikingly reasonable and useful.

Some WG Chairs already do this and I find it most helpful.  Even at the
most basic level, of what documents have changed status in the past
month and what have not, this provides a valuable measure of progress,
or lack thereof.

And what is management but finding out what is going on and intervening
when intervention is needed?  And how can you know if you do not get
progress reports?  And, who knows, at a basic level, this could be
automated from the I-D Tracker, no extra work for AD or WG Chair (except
for intervening when intervention is needed).

Tom Petch

---

From: Dave Crocker <[email-masked]>
Date: Fri, 17 May 2013 11:12:04 -0700

> month and what have not, this provides a valuable measure of progress,

That's good, but it highlights a point I also noticed in the examples we 
were given for INT:  There's a basic difference between activity and 
progress.

Except for in-depth, detailed tracking situations, I think that 
reporting 'activity' is actually noise.

Progress is about decisions and output, not about movement and input.

Since the goal of the kind of reporting being discussed here is to 
inform the 'community' it needs to be very concise and semantic:  It 
needs to talk about the accomplishments that have /completed/ things.  A 
decision finishes a question.  Agreement on the text of a section 
finishes that part of a document.

Iterating on a document and holding meetings are activity that might or 
might not contain the kernel of progress.

For the current discussion, we need to hear about that kernel, IMO.  Not 
the rest.

d/

---

From: Yoav Nir <[email-masked]>
Date: Fri, 17 May 2013 20:36:01 +0000

On May 17, 2013, at 6:37 PM, Dave Crocker <[email-masked]> wrote:

> The idea that working groups should be required to issue periodic project progress reports seems strikingly reasonable and useful.
> 
> This makes the folks who are the most knowledgeable responsible for assessing their work, and should facilitate public review. Recording the sequence of reports into the wg datatracker could nicely allow evaluating progress over time.
> 

"
From: WG Chair
To: [email-masked]
Sunbject: Progress Report - Foo WG

There has been zero activity on the FOO list in the last three months (except for that "Fake Conference" message everybody got last month). I've tried emailing the WG document authors twice, but they're not answering my emails.

So, the WG has 2 documents: draft-ietf-foo-use-cases-03, and draft-ietf-foo-proto-01.

The use case document is just about done, but we haven't really started discussing the proto document. We haven't met in Orlando, and are unlikely to meet in Berlin

That's it for this report.

Cheers

WGC

"

---

From: Keith Moore <[email-masked]>
Date: Fri, 17 May 2013 21:29:59 -0400

> That's it for this report.

Instead of a WG progress report, what I had in mind was a separate 
report for each work item.   The report should briefly describe

1. The purpose of the work being undertaken
2. A description of the work being undertaken (including mention of 
major technologies on which the work is based)
3. All known parties and interests likely to be affected by the work
4. The current state of the work (what's been done, what hasn't been done)
5. Any major issues that have been identified but not resolved
6. Pointers to the WG's charter, the datatracker pages for each of the 
current draft document(s) associated with that work item, and any other 
useful material (e.g. open issues list, summaries of design decisions 
already taken and the rationale for each)

A person who is knowledgeable about current Internet protocols should be 
able to read that single document and understand what the WG is doing 
with this particular work item, what state it's in, whether or not it 
will affect that person's are of interest, and where to look for 
detailed information.

Keith

---

From: Keith Moore <[email-masked]>
Date: Fri, 17 May 2013 22:29:58 -0400

On 05/17/2013 10:21 PM, Andy Bierman wrote:
>
> I notice that nowhere on this list is any mention of the charter 
> milestones
> or dates.  Is the Foo Proto draft due in 14 months or is it 14 months 
> behind
> schedule?  If the latter, why isn't the Foo WG meeting at the IETF?
>

I don't think milestones will be useful unless and until:

(a) they're defined in terms of not only concrete but also meaningful 
goals (e.g. "complete problem definition", "identify affected parties 
and groups representing their interests", "complete outline of initial 
design", but NOT "revise document X");
(b) we start automatically suspending the activities of groups that fail 
to meet them (no meetings, no new I-Ds accepted, mailing list traffic 
blocked), until such groups are formally rechartered; and
(c) IESG is reluctant to recharter groups that have repeatedly failed to 
meet milestones, especially if those groups haven't produced evidence of 
significant progress.

Keith

---

From: Keith Moore <[email-masked]>
Date: Fri, 17 May 2013 22:51:54 -0400

On 05/17/2013 10:37 PM, Andy Bierman wrote:
>
> On Fri, May 17, 2013 at 7:29 PM, Keith Moore 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>
>     (a) they're defined in terms of not only concrete but also
>     meaningful goals (e.g. "complete problem definition", "identify
>     affected parties and groups representing their interests",
>     "complete outline of initial design", but NOT "revise document X");
>     (b) we start automatically suspending the activities of groups
>     that fail to meet them (no meetings, no new I-Ds accepted, mailing
>     list traffic blocked), until such groups are formally rechartered; and
>     (c) IESG is reluctant to recharter groups that have repeatedly
>     failed to meet milestones, especially if those groups haven't
>     produced evidence of significant progress.
>
>
> I think we can find some middle ground between "ignore charter 
> milestones completely"
> and "autobot to terminate WGs behind schedule". :-)
>
Actually I think it might require an autobot.   Because someone 
(probably the responsible AD) has to evaluate a WG's progress, and ADs 
don't want to take the heat for shutting WGs down.   Better to put the 
responsibility on the chairs for completing the milestones and reporting 
to the AD before the shutdown deadline.

(of course, there could be a generous grace period between the milestone 
deadline and the actual shutdown, with warning messages sent to the WG 
chairs and ADs, etc.)

Keith

---

From: Andy Bierman <[email-masked]>
Date: Fri, 17 May 2013 19:21:04 -0700

On Fri, May 17, 2013 at 6:29 PM, Keith Moore <[email-masked]>wrote:

>>> sequence of reports into the wg datatracker could nicely allow evaluating
>>> progress over time.
>>>
>> There has been zero activity on the FOO list in the last three months
>> (except for that "Fake Conference" message everybody got last month). I've
>> tried emailing the WG document authors twice, but they're not answering my
>> emails.
>>
>> The use case document is just about done, but we haven't really started
>> discussing the proto document. We haven't met in Orlando, and are unlikely
>> to meet in Berlin
>>
> Instead of a WG progress report, what I had in mind was a separate report
> for each work item.   The report should briefly describe
>
> 2. A description of the work being undertaken (including mention of major
> technologies on which the work is based)
> able to read that single document and understand what the WG is doing with
> this particular work item, what state it's in, whether or not it will
> affect that person's are of interest, and where to look for detailed
> information.
>
>
This seems like a good idea, although maybe a bit formal.
IMO, big surprises at the tail end that cause lots of work to
be thrown out are evidence of a management failure.
The IESG and WG chairs should be more proactive wrt/ avoiding
late surprises from ever happening.

I notice that nowhere on this list is any mention of the charter milestones
or dates.  Is the Foo Proto draft due in 14 months or is it 14 months behind
schedule?  If the latter, why isn't the Foo WG meeting at the IETF?

Keith
Andy

---

From: Andy Bierman <[email-masked]>
Date: Fri, 17 May 2013 19:37:23 -0700

On Fri, May 17, 2013 at 7:29 PM, Keith Moore <[email-masked]>wrote:

> goals (e.g. "complete problem definition", "identify affected parties and
> groups representing their interests", "complete outline of initial design",
> but NOT "revise document X");
I think we can find some middle ground between "ignore charter milestones
completely"
and "autobot to terminate WGs behind schedule". :-)

Keith
Andy