Subject: IONs & discuss criteria

From: Ted Hardie <[email-masked]>
Date: Thu, 6 Mar 2008 12:01:25 -0800

The call for comments on IONs seems to have ended without
clarifying the effect of the end of the experiment on the standing
of current IONs.  For most of them, I honestly don't think the
standing is much of an issue.  But for the "discuss criteria" ION,
I believe it is a serious issue.  At this point, it is difficult to know
whether the discuss criteria document is in force or not, and the
extent to which the issuing body is bound by it.

I think this is a very bad thing.

I call on Russ to restore this document to its original status as
an Internet Draft and to process it as a BCP.  IESG DISCUSSes are
a very serious part of our process at this point.  Having a community
agreed standard to which IESG members  could be held was always a better
path than than a document approved only by the IESG.  Now that
the ION experiment is over and the status of its document is in
limbo, things are even worse.

The current document is here:

http://www.ietf.org/IESG/content/ions/ion-discuss-criteria.html

for those readers playing the home game.  

				Ted  Hardie
_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 07 Mar 2008 09:42:12 +1300

Ted,

Firstly, it's not for me to prejudge the IESG's conclusions
about IONs, but I would suggest that any ION issued by the
IESG implicitly carries the same status as any other IESG
statement, unless rescinded, so I don't quite share your concern.

However, the deeper question is whether the "discuss criteria"
*should* be promoted to BCP (which effectively binds future
IESGs as well as the current IESG). That is worth some
discussion. My experience on both sides of the fence is that
having the criteria spelled out has been extremely valuable
to authors, WGs, reviewers and the IESG itself. I'm a bit less
certain whether they should be made binding. Flexibility
leaves space for applying common sense.

    Brian

---

From: Ted Hardie <[email-masked]>
Date: Thu, 6 Mar 2008 13:01:25 -0800

>statement, unless rescinded, so I don't quite share your concern.

I went and looked at RFC 4693 before I posted my first note.
It says:

   If the IESG decides that the feedback warrants terminating the
   series, the repository will be closed for new documents, and the
   existing ION documents will be returned to having the same status as
   any other Web page or file on the IETF servers -- this situation will
   closely resemble the situation before the experiment started.

The status this document had prior to being approved
as an ION was "Internet draft", which means it had
no formal status at all and was followed by the IESG
as a matter of lore.

>leaves space for applying common sense.

There is no reason for a community-agreed document not to
have flexibility.  There are strong reasons to make this a community
agreed document.   Making it something that the community
can hold the IESG to, rather than something the IESG can modify
by issuing an updated ION, is a critical part of this.

			Ted Hardie

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 6 Mar 2008 13:04:12 -0800

Ted,

Speaking for myself here but I suspect that other ADs are in the same  
boat ... I'm keen to make sure my Discusses are within the parameters  
of the discuss criteria ION regardless of the official status of this  
document. Agree we need to sort out what we the end result is of  
several experiments. I believe Russ is working to get that some IESG  
agenda time.

Cullen

---

From: Russ Housley <[email-masked]>
Date: Thu, 06 Mar 2008 15:52:49 -0500

Ted:

The call for comments has resulted in some input, and the IESG plans 
to discuss that input at our meeting on Sunday.  In fact there is 
also an experiment on mail list suspension that we will be discussing 
as well.  The two experiments are listed on the web page:

http://www.ietf.org/IESG/content/experiments.html

Once this discussion is over, the future of IONs should be clear, and 
I will share with the whole IETF community the outcome of the experiment.

If IONs are not part of the future, then we need to figure out the 
best home for each of the things that has been posted as part of the 
experiment:

http://www.ietf.org/IESG/content/ions/

Russ

---

From: Sam Hartman <[email-masked]>
Date: Thu, 06 Mar 2008 16:14:27 -0500

>>>>> "Cullen" == Cullen Jennings <[email-masked]> writes:

    Cullen> Ted,

    Cullen> Speaking for myself here but I suspect that other ADs are in the same
    Cullen> boat ... I'm keen to make sure my Discusses are within the parameters
    Cullen> of the discuss criteria ION regardless of the official status of this
    Cullen> document. Agree we need to sort out what we the end result is of
    Cullen> several experiments. I believe Russ is working to get that some IESG
    Cullen> agenda time.

Ted, as one of the ADs you have most recently interacted with on the
discuss criteria document, I believe that I'm bound by that document
and assume it has the same force as an IESG statement.

---

From: Russ Housley <[email-masked]>
Date: Thu, 06 Mar 2008 16:17:35 -0500

Ted:

Not oall of the IONs were "approved" for posting by the IESG.  There 
is one from the IAOC, for example.  That was the point of "figure out 
what to do."

Russ

---

From: John C Klensin <[email-masked]>
Date: Thu, 06 Mar 2008 16:24:50 -0500

--On Thursday, 06 March, 2008 12:01 -0800 Ted Hardie
<[email-masked]> wrote:

> I call on Russ to restore this document to its original status
> as an Internet Draft and to process it as a BCP.  IESG
> DISCUSSes are a very serious part of our process at this
> point.  Having a community agreed standard to which IESG
> members  could be held was always a better path than than a
> document approved only by the IESG.  Now that the ION
> experiment is over and the status of its document is in limbo,
> things are even worse.
> 

Hmm.   If people believe that this document should be processed
as a BCP, thereby presumably constraining long-term IESG
behavior and adding to our procedural core, should it be added
to the PUFI agenda for preliminary discussion?

      john

---

From: Ted Hardie <[email-masked]>
Date: Thu, 6 Mar 2008 13:34:18 -0800

>I will share with the whole IETF community the outcome of the experiment.

Russ,
	Whatever the fate of IONs in general, it is clear to me that
this document does not belong in the series.  It needs community
consensus that binds the IESG and the rest of the community; as
a statement of the IESG (and which is subject to update by the IESG)
it does not have the force it needs.  At the moment it is completely
lacking in such force.
	I look forward to the IESG statements on the series
as a whole.  But please understand that if the IESG indicates an
intent to retain this document at the end of that in this series I would
appeal.
	I respect your work, but I believe the IESG has recently
relaxed the vigilance it once held toward adherence to these criteria.
I have seen at least two recent discusses that amounted to
"go satisfy that guy" and several cut and paste external reviews where
it was blindingly obvious the AD had not even looked at the
most recent version of the text.  I have also seen quite a few
that amount to "Disagreement with informed working group
decisions" where the AD is putting their preferences over
any real acknowledgement that a working group has considered
the issues.
	The only way I know of to make sure the IESG restores
the focus on this issue (which took a lot of our energy several years
ago) is to make it binding on the IESG.   I hope that you, personally,
agree that it should be community-based and binding on the IESG
and that we are simply discussing the mechanism by which that
occurs.  If you do not agree that it should be binding on the IESG
and a consensus statement of the community, I am interested to know
why.
			Ted Hardie
_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: Lakshminath Dondeti <[email-masked]>
Date: Thu, 06 Mar 2008 13:35:04 -0800

Cullen,

Thank you for your statement that you are keen to make sure your 
DISCUSSes are within the parameters of the discuss criteria ION.  I 
appreciate it.  Perhaps I am naive or my understanding of the English 
language is poor (they are both probably true), but could you explain 
how one of your most recent DISCUSSes:

"Cullen Jennings:

Discuss [2008-03-05]:
There has been a lot of discussion about keying modes for
SRTP, so I'm glad to see a document that covers this topic
for MIKEY. For that reason, I think it's really important
to get this right. It looks to me like some of the issues
EKR raises need to be fixed in order to achieve that."

does not fit into the DISCUSS non-criteria?

"Unfiltered external party reviews. While an AD is welcome to consult 
with external parties, the AD is expected to evaluate, to understand and 
to concur with issues raised by external parties. Blindly 
cut-and-pasting an external party review into a DISCUSS is inappropriate 
if the AD is unable to defend or substantiate the issues raised in the 
review."

You chose to not even cut-and-paste the comments.

I also wonder which of the DISCUSS criteria fit to advance that specific 
document to an informational RFC.   Are we to guess which of the "the 
issues EKR raises" the authors need to fix?

Needless to say, we don't need to debate the specifics of that document 
here, but whereas your intent is honorable, the externally observable 
behavior is unfortunately different.  Sorry for picking on you; I can 
probably find other similar examples on other ADs, but I was thinking 
that the ION is not a BCP and so I have no basis to raise the issue.

Perhaps I am misunderstanding the ION or perhaps as Brian says that 
there should be some flexibility in the application of the rules.

regards,
Lakshminath

---

From: Sam Hartman <[email-masked]>
Date: Thu, 06 Mar 2008 16:43:25 -0500

>>>>> "Ted" == Ted Hardie <[email-masked]> writes:

    Ted> I respect your work, but I believe the IESG has recently
    Ted> relaxed the vigilance it once held toward adherence to these criteria.
    Ted> I have seen at least two recent discusses that amounted to
    Ted> "go satisfy that guy" and several cut and paste external reviews where
    Ted> it was blindingly obvious the AD had not even looked at the
    Ted> most recent version of the text.  I have also seen quite a few
    Ted> that amount to "Disagreement with informed working group
    Ted> decisions" where the AD is putting their preferences over
    Ted> any real acknowledgement that a working group has considered
    Ted> the issues.
    Ted> The only way I know of to make sure the IESG restores
    Ted> the focus on this issue (which took a lot of our energy several years
    Ted> ago) is to make it binding on the IESG.   I hope that you, personally,
    Ted> agree that it should be community-based and binding on the IESG
    Ted> and that we are simply discussing the mechanism by which that
    Ted> occurs.  If you do not agree that it should be binding on the IESG
    Ted> and a consensus statement of the community, I am interested to know
    Ted> why.
    Ted> Ted Hardie

If someone believes that a discuss is inappropriate, I recommend that
they start both by contacting the discussing AD *and* the shepherding
AD.

I know that I would treat a request to rethink whether a discuss I
held was consistent with the discuss criteria document from another
IESG member very seriously.  I would treat such a request from an
author seriously, although not as seriously as from another IESG
member.  There are a number of reasons for this.  The authors are
often much more emotionally involved in a document than ADs.  The ADs
are likely more familiar with the discuss criteria than the author and
are definitely more familiar with the current interpretation of the
discuss criteria.  No matter how frustrating it is, it's simply true
that procedures like the discuss criteria are subject to
interpretation, and the IESG is going to have an evolving
interpretation of any such procedural document.

I think it is reasonable for an author to expect to get a response
back from an shepherding AD that either they think the discuss is
reasonable, or they think it is unreasonable.

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 06 Mar 2008 13:55:41 -0800

At Thu, 06 Mar 2008 13:35:04 -0800,
> You chose to not even cut-and-paste the comments.

Doesn't this fall into the category of "evaluate and concur"?

> issues EKR raises" the authors need to fix?

If only some other area director had excerpted my review
and identified the sections that he felt most clearly needed
correction.

Oh, wait, Sam did.

-Ekr

---

From: Sam Hartman <[email-masked]>
Date: Thu, 06 Mar 2008 16:51:11 -0500

>>>>> "Lakshminath" == Lakshminath Dondeti <[email-masked]> writes:

    Lakshminath> Cullen,
    Lakshminath> Thank you for your statement that you are keen to make sure your
    Lakshminath> DISCUSSes are within the parameters of the discuss criteria ION.  I
    Lakshminath> appreciate it.  Perhaps I am naive or my understanding of the English
    Lakshminath> language is poor (they are both probably true), but could you explain
    Lakshminath> how one of your most recent DISCUSSes:

    Lakshminath> "Cullen Jennings:

    Lakshminath> Discuss [2008-03-05]:
    Lakshminath> There has been a lot of discussion about keying modes for
    Lakshminath> SRTP, so I'm glad to see a document that covers this topic
    Lakshminath> for MIKEY. For that reason, I think it's really important
    Lakshminath> to get this right. It looks to me like some of the issues
    Lakshminath> EKR raises need to be fixed in order to achieve that."

Presumably Cullen is agreeing with the discuss position that I'm
holding and that Russ is holding.  If Cullen plans to hold his discuss
position past the resolution of Russ's discuss (Russ has agreed to
take on mine), then I agree his discuss is inappropriate.  I'm not sure that Cullen made the best use of the tool, but I'm not sure he did anything wrong either.  I believe
that my discuss is consistent with the discuss criteria because while
it is based on an external review, I've explained what parts of the
review I consider blocking.  I haven't read Russ's discuss.  I believe
that if he selected what parts of the review he considers are a valid
discuss,or if he simply asked you to respond to Eric's comments
(saying that he believes last call discussion is still ongoing), then
it is a valid discuss.  The second discuss (please respond to Eric and
conclude the last call discussion) is a process discuss not a content
discuss; he would be asking you to actually engage in a discussion.
If he later believed that Tim had incorrectly evaluated the consensus
of that discussion, he might change his position to another process
discuss about a consensus problem.

---

From: Lakshminath Dondeti <[email-masked]>
Date: Thu, 06 Mar 2008 14:21:46 -0800

Sam,

I fail to understand why this has to be a guessing game.  I also don't 
understand the argument about resolving DISCUSSes sequentially (in 
reference to your point about Cullen holding his DISCUSS beyond 
resolution of Russ's).

I have seen better examples where for instance your DISCUSS quotes what 
you think needs to be resolved from the external review.  Likewise, Russ 
states what needs to be resolved from the external review.  Recently Tim 
put a DISCUSS on another document based on my comments stating what his 
specific concerns are.  In all those cases, it is at least somewhat 
clear as to what the AD might want.

In closing, perhaps some of us would like to be behind a DISCUSS that 
states "resolve some of the issues of a 3rd party."  I for one find it 
very hard to work in that kind of an environment.

 From my view point, here is how the process looks: First we have to beg 
to know what the issues are,  then propose text, only to have it 
dismissed after some time, propose text again, repeat that for a 
non-deterministic number of times and eventually hope that the AD is 
satisfied; repeat that for the next AD, and so on.

That is seriously broken!!  All I am asking for though is to just remove 
the first step for starters.  We shouldn't have to ask to know what the 
DISCUSS is about.

best regards,
Lakshminath

On 3/6/2008 1:51 PM, Sam Hartman wrote:
> Lakshminath> Cullen, Lakshminath> Thank you for your statement that
> you are keen to make sure your Lakshminath> DISCUSSes are within the
> parameters of the discuss criteria ION.  I Lakshminath> appreciate
> it.  Perhaps I am naive or my understanding of the English 
> Lakshminath> language is poor (they are both probably true), but
> could you explain Lakshminath> how one of your most recent DISCUSSes:
> 
> 
> Lakshminath> Discuss [2008-03-05]: Lakshminath> There has been a lot
> of discussion about keying modes for Lakshminath> SRTP, so I'm glad
> to see a document that covers this topic Lakshminath> for MIKEY. For
> that reason, I think it's really important Lakshminath> to get this
> right. It looks to me like some of the issues Lakshminath> EKR raises
> need to be fixed in order to achieve that."
> 
> holding and that Russ is holding.  If Cullen plans to hold his
> discuss position past the resolution of Russ's discuss (Russ has
> agreed to take on mine), then I agree his discuss is inappropriate.
> I'm not sure that Cullen made the best use of the tool, but I'm not
> sure he did anything wrong either.  I believe that my discuss is
> consistent with the discuss criteria because while it is based on an
> external review, I've explained what parts of the review I consider
> blocking.  I haven't read Russ's discuss.  I believe that if he
> selected what parts of the review he considers are a valid discuss,or
> if he simply asked you to respond to Eric's comments (saying that he
> believes last call discussion is still ongoing), then it is a valid
> discuss.  The second discuss (please respond to Eric and conclude the
> last call discussion) is a process discuss not a content discuss; he
> would be asking you to actually engage in a discussion. If he later
> believed that Tim had incorrectly evaluated the consensus of that
> discussion, he might change his position to another process discuss
> about a consensus problem.
> 
>

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 6 Mar 2008 14:23:07 -0800

Part of the reason I replied so quickly on this thread  is that I  
think I currently have two discuss that do not meet the discuss  
criteria (this being one of  them the other being on Lost). Totally  
fair to pick on me here. Both were entered as, excuse the pun, fairly  
fluffy comments because I believe fully stating each point by point  
things  would have actually make it significantly harder for the  
editor of the documents to find a good solution. In both cases I  
believe the editor pretty much understands the issue and if I get  
requests to please rewrite to be a reasonable discuss, I'm glad to do  
that after the meeting.

On Mar 6, 2008, at 1:35 PM, Lakshminath Dondeti wrote:

> appreciate it.  Perhaps I am naive or my understanding of the  
> English language is poor (they are both probably true), but could  
> you explain how one of your most recent DISCUSSes:
>
> "Cullen Jennings:
>
> Discuss [2008-03-05]:
> "Unfiltered external party reviews. While an AD is welcome to  
> consult with external parties, the AD is expected to evaluate, to  
> understand and to concur with issues raised by external parties.  
> Blindly cut-and-pasting an external party review into a DISCUSS is  
> inappropriate if the AD is unable to defend or substantiate the  
> issues raised in the review."
>
> I also wonder which of the DISCUSS criteria fit to advance that  
> specific document to an informational RFC.   Are we to guess which  
> of the "the issues EKR raises" the authors need to fix?
>
> Needless to say, we don't need to debate the specifics of that  
> document here, but whereas your intent is honorable, the externally  
> observable behavior is unfortunately different.  Sorry for picking  
> on you; I can probably find other similar examples on other ADs, but  
> I was thinking that the ION is not a BCP and so I have no basis to  
> raise the issue.
>
>> Speaking for myself here but I suspect that other ADs are in the  
>> same  boat ... I'm keen to make sure my Discusses are within the  
>> parameters  of the discuss criteria ION regardless of the official  
>> status of this  document. Agree we need to sort out what we the end  
>> result is of  several experiments. I believe Russ is working to get  
>> that some IESG  agenda time.
>> Cullen

---

From: Lakshminath Dondeti <[email-masked]>
Date: Thu, 06 Mar 2008 14:29:47 -0800

Thanks for the clarification Cullen.  I appreciate it.  Speaking from 
the view point of someone on the other side, more often than not, a 
detailed DISCUSS is much more helpful.

Thank you again.

best wishes,
Lakshminath

On 3/6/2008 2:23 PM, Cullen Jennings wrote:
> 
> Part of the reason I replied so quickly on this thread  is that I think 
> I currently have two discuss that do not meet the discuss criteria (this 
> being one of  them the other being on Lost). Totally fair to pick on me 
> here. Both were entered as, excuse the pun, fairly fluffy comments 
> because I believe fully stating each point by point things  would have 
> actually make it significantly harder for the editor of the documents to 
> find a good solution. In both cases I believe the editor pretty much 
> understands the issue and if I get requests to please rewrite to be a 
> reasonable discuss, I'm glad to do that after the meeting.
> 
>> with external parties, the AD is expected to evaluate, to understand 
>> and to concur with issues raised by external parties. Blindly 
>> cut-and-pasting an external party review into a DISCUSS is 
>> inappropriate if the AD is unable to defend or substantiate the issues 
>> raised in the review."
>>
>> specific document to an informational RFC.   Are we to guess which of 
>> the "the issues EKR raises" the authors need to fix?
>>
>> observable behavior is unfortunately different.  Sorry for picking on 
>> you; I can probably find other similar examples on other ADs, but I 
>> was thinking that the ION is not a BCP and so I have no basis to raise 
>> the issue.
>>

---

From: Sam Hartman <[email-masked]>
Date: Thu, 06 Mar 2008 17:32:39 -0500

>>>>> "Lakshminath" == Lakshminath Dondeti <[email-masked]> writes:

    Lakshminath> Sam,
    Lakshminath> I fail to understand why this has to be a guessing game.  I also don't
    Lakshminath> understand the argument about resolving DISCUSSes sequentially (in
    Lakshminath> reference to your point about Cullen holding his DISCUSS beyond
    Lakshminath> resolution of Russ's).

I guess I was unclear.  I think it's reasonable for Cullen to say "I
agree with that other discuss," and that's how I interpret his current
position.  I think it's kind of odd for him to stick that in the
discuss box rather than the comment box, but I don't think it is
particularly harmful provided that his discuss never blocks the
document.  I.E. he needs to make sure his discuss is removed before
Russ clears.

Put another way, it's fine for Cullen to tell other IESG members that
he agrees with a discuss.  It's fine for him to agree so strongly that
he'd like to be given an opportunity to take on the discuss if for
example the person holding the discuss gives up and wants to drop the
issue.  It's not fine for him to expect you to do anything based on a
discuss that vague.  It's not fine for his inaction to cause your
document to get stuck based on a discuss that vague.

---

From: Pete Resnick <[email-masked]>
Date: Thu, 6 Mar 2008 16:48:08 -0600

On 3/6/08 at 4:24 PM -0500, John C Klensin wrote:

>Hmm.   If people believe that this document should be processed as a 
>BCP, thereby presumably constraining long-term IESG behavior and 
>adding to our procedural core, should it be added to the PUFI agenda 
>for preliminary discussion?

The PUFI BOF chair, who has not completed his list of currently 
desired items on the grand list of things to cover in this BOF, 
hereby groans at the thought of adding another.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102
_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: Ted Hardie <[email-masked]>
Date: Thu, 6 Mar 2008 14:50:49 -0800

>requests to please rewrite to be a reasonable discuss, I'm glad to do

Speaking as one of the editors of LoST, and not as a general statement
on this problem, a clear discuss from the beginning would have been
helpful.  I have been willing to have phone calls, conduct a series of email
exchanges, and chat via IM on this; I want this document to be right.
But the amount of effort you are requiring from the authors to
work towards a statement of the problem, much less a solution,
has been pretty significant:  10 emails from me alone on your issues alone, as
we worked toward an understanding of the issue you have.
That boiled down to a DISCUSS that should be as simple as
"In non-emergency use cases, I am concerned that the provisions
of 12.2 allow servers to use algorithms that will needlessly return
null results.   Please add guidance for non-emergency use cases."

Speaking again as someone who thinks this is general problem, the
issue I am raising is not that there are bad discusses.  The issue I
am raising is that the document which describes what discusses
are or should be has no force at the moment at all, and should be
a community document rather than a statement of the body
which may hold discusses.  Only the latter allows the community
to hold the IESG accountable adequately.

				regards,
					Ted Hardie

---

From: Lakshminath Dondeti <[email-masked]>
Date: Thu, 06 Mar 2008 14:57:13 -0800

Sam,

There is no need to prolong this particular side of the discussion now 
that Cullen clarified his position.  But, I have to say that this thread 
is but one example that we often don't clearly understand each other's 
positions.

You interpret Cullen's DISCUSS as :  "I think it's reasonable for Cullen 
to say "I
agree with that other discuss," and that's how I interpret his current 
position. "

Cullen clarifies it as: "I believe the editor pretty much understands 
the issue and if I get requests to please rewrite to be a reasonable 
discuss, I'm glad to do that after the meeting. "

Most of the IESG members' names have four letters or less :).  It is not 
very hard to type "Agree with XXXX" even if someone is in a hurry.

Next, I can't read Steffen and Dragan's minds, and so I don't know what 
their understanding of the issue is and whether they understand it as 
Cullen agreeing with the other discuss or something else.

At this point, we have that additional step of saying "please rewrite 
your discuss to be a reasonable discuss."  It looks like my 
interpretation was right that I have to beg for clarification to go 
forward here.

best,
Lakshminath

---

From: Sam Hartman <[email-masked]>
Date: Thu, 06 Mar 2008 17:59:19 -0500

>>>>> "Ted" == Ted Hardie <[email-masked]> writes:

    Ted> Speaking again as someone who thinks this is general problem, the
    Ted> issue I am raising is not that there are bad discusses.  The issue I
    Ted> am raising is that the document which describes what discusses
    Ted> are or should be has no force at the moment at all, 

Ted, I'd like to disagree with this point.  I believe that you could
appeal a discuss because it does not meet the discuss criteria.  I
believe you could ask the iesg as a body to evaluate whether a discuss fit the criteria.

If you did appeal, I believe you could carry the appeal to the IAB; I
believe that they would conclude that the IESG has chosen to bind
itself to the discuss criteria document.

There is some complexity.  The IESG could in theory use some other
mechanism rather than its balloting procedure to reach consensus on
what to do with a document.  Especially if that consensus were strong,
I think it would be reasonable for the IESG to do that.  In my time on
the IESG that's never happened; I don't see it starting soon.

---

From: Ted Hardie <[email-masked]>
Date: Thu, 6 Mar 2008 15:00:38 -0800

>interpretation of any such procedural document.

I can think of no stronger statement of why the discuss criteria
document should be a community statement that represents the
agreement of the IETF as a whole than that above.  If the IESG
can have an "evolving interpretation" of the criteria, then the
community needs to know it and agree to the interpretation.
That interpretation being evolved without community input
in the context of an inward-looking IESG is not appropriate.
And an inward-looking IESG is exactly what is described,
with the trust given to IESG members being greater simply
because they share that common context.

As a side note, this view seems to relegate the document shepherd's
role to "invisible friend", something I would press further on if
Sam were not so immanently leaving the IESG.

				Ted Hardie

---

From: Dave Crocker <[email-masked]>
Date: Thu, 06 Mar 2008 16:04:52 -0700

> to the PUFI agenda for preliminary discussion?

Yes.

A series of postings by sitting area directors about their commitment to 
following a document says nothing about the commitment of any future area director.

If the document is merely a reference to be used internally by the IESG, then it 
needs no formal standing.

If the document is meant as formal criteria to ensure transparency and 
accountability of the IESG, then it needs formal standing, which means formal 
adoption by the IETF community.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net
_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: Sam Hartman <[email-masked]>
Date: Thu, 06 Mar 2008 18:06:55 -0500

>> interpretation of any such procedural document.

    Ted> I can think of no stronger statement of why the discuss criteria
    Ted> document should be a community statement that represents the
    Ted> agreement of the IETF as a whole than that above.  If the IESG
    Ted> can have an "evolving interpretation" of the criteria, then the
    Ted> community needs to know it and agree to the interpretation.

Ted, we have an evolving interpretation of every document we've ever
written--BCPs, standards, IONs, webpages, email messages,
presentations.  I don't mean that the iesg should be able to interpret
the discuss criteria document in a manner inconsistent with the text.
however there are a lot of ways to read the text.

Making it a BCP will make the interpretation problem worse not better.
_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: Dave Crocker <[email-masked]>
Date: Thu, 06 Mar 2008 16:34:02 -0700

> Making it a BCP will make the interpretation problem worse not better.

How?

d/
-- 

   Dave Crocker

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 07 Mar 2008 13:10:35 +1300

Dave,

>> Making it a BCP will make the interpretation problem worse not better.

To some extent that depends on how carefully the putative BCP
is crafted, with "should" and when to disregard "should" being
very precise. What I think we've seen, with 2026 over the years,
and apparently this year with 3777, is that it's virtually
impossible to write precise procedural text that deals with
completely unexpected circumstances. Yet if the text has the
force of a BCP, it becomes very hard to interpret it flexibly
when flexibility is clearly needed.  I don't know if that
is Sam's point, of course.

    Brian

---

From: Russ Housley <[email-masked]>
Date: Thu, 06 Mar 2008 19:09:26 -0500

Ted, Lakshminath, and the Rest of the IETF Community:

>I fail to understand why this has to be a guessing game.

The handling of reviews by non-IESG members seems to be an important 
part of this discussion.  So, I'll tell everyone how I deal with 
Gen-ART Reviews.  Other General ADs may have done things slightly different.

When I use a Gen-ART Review as the basis of a DISCUSS, I put it in 
one of two categories.

(1)  The Gen-ART Review was ignored.  Like any other Last Call 
comment, it deserves an answer.  So, this is a procedural 
objection.  In this situation, I've been careful to say that the 
authors do not need to accept all of the comments, but then need to 
answer them.

(2)  I agree with one or more concerns raised in the Gen-ART Review 
that has not been resolved.  I often break the unresolved review 
comments into DISCUSS and COMMENT.  AD judgement is needed, and I 
consider the DISCUSS Criteria in making that judgement.

Russ

---

From: Lakshminath Dondeti <[email-masked]>
Date: Thu, 06 Mar 2008 16:59:37 -0800

Hi Russ,

Thanks for your response.  Some notes inline:

On 3/6/2008 4:09 PM, Russ Housley wrote:
> part of this discussion.  

I agree and have contributed to that part of the solution, only now I am 
realizing that it may be becoming part of the problem, so to speak.  The 
concern is that over time it seems to be degenerating into, I will use 
Ted's phrase here because that is what it feels like, "go satisfy that 
guy."  Consider how it sounds when trying to explain to an outsider: the 
document is held up in IESG processing because X, who is not an IESG or 
IAB member, does not like it.

I think your own word is "answer" (I have heard "respond") in lieu of 
"satisfy."  Some notes on that below:

> So, I'll tell everyone how I deal with Gen-ART 
> Reviews.  Other General ADs may have done things slightly different.
> 
> When I use a Gen-ART Review as the basis of a DISCUSS, I put it in one 
> of two categories.
> 
> (1)  The Gen-ART Review was ignored.  Like any other Last Call comment, 
> it deserves an answer.  So, this is a procedural objection.  In this 
> situation, I've been careful to say that the authors do not need to 
> accept all of the comments, but then need to answer them.

I have reviewed documents as a Gen-ART reviewer (during Brian's tenure I 
think), sec-dir reviewer and also provided IETF LC comments on some 
documents.  As a reviewer, I am not sure whether I was expecting answers 
all those times.  I am pretty sure I have not always stated whether or 
not the answers are satisfactory.

Next, I can imagine an author not wanting to respond to something I may 
have said because it was totally bogus or inappropriate and does not 
deserve a response.  That might very well happen when I review documents 
on a topic that I am not familiar with and haven't had the time to read 
related references (that varies depending on the time available, etc.). 
  Perhaps that is not such a bad thing; being blissfully ignorant on 
some topics keeps me, well, blissful.  I use somewhat of a hyperbole for 
obvious reasons.  I am sure many other situations are much more nuanced. 
  I hope ADs don't continue to hold a DISCUSS in those situations 
waiting for a dialog to take place or waiting for a consensus to emerge. 
  I sometimes hint in my reviews that the topic may be at the border of 
my knowledge and if I have a bias.  Perhaps that is helpful.

regards,
Lakshminath

> 
> (2)  I agree with one or more concerns raised in the Gen-ART Review that 
> has not been resolved.  I often break the unresolved review comments 
> into DISCUSS and COMMENT.  AD judgement is needed, and I consider the 
> DISCUSS Criteria in making that judgement.
> 
> Russ
> 
>

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 6 Mar 2008 17:05:41 -0800

I believe Sam's discuss cover the issues I was concerned about and I  
have removed my discuss.

On Mar 6, 2008, at 2:57 PM, Lakshminath Dondeti wrote:

> Sam,
>
> There is no need to prolong this particular side of the discussion  
> now that Cullen clarified his position.  But, I have to say that  
> this thread is but one example that we often don't clearly  
> understand each other's positions.
>
> You interpret Cullen's DISCUSS as :  "I think it's reasonable for  
> Cullen to say "I
> agree with that other discuss," and that's how I interpret his  
> current position. "
>
> Cullen clarifies it as: "I believe the editor pretty much  
> understands the issue and if I get requests to please rewrite to be  
> a reasonable discuss, I'm glad to do that after the meeting. "
>
> Most of the IESG members' names have four letters or less :).  It is  
> not very hard to type "Agree with XXXX" even if someone is in a hurry.
>
> Next, I can't read Steffen and Dragan's minds, and so I don't know  
> what their understanding of the issue is and whether they understand  
> it as Cullen agreeing with the other discuss or something else.
>
> At this point, we have that additional step of saying "please  
> rewrite your discuss to be a reasonable discuss."  It looks like my

---

From: Lakshminath Dondeti <[email-masked]>
Date: Thu, 06 Mar 2008 17:06:54 -0800

Brian,

A small clarification below on the reference to the interpretation 
problems related to 3777:

> and apparently this year with 3777, is that it's virtually

I am not sure whether you have made it to the appendix in my report, but 
the disagreements in interpretation of 3777 have a history (see Page 
37).  The only thing special about the current nomcom is that we chose 
to bring it to the community's attention.  In Ralph's case, he brought 
it to the IESG and IAB's attention in March 2006.

thanks,
Lakshminath
Nomcom 2007-8 Chair

---

From: Lakshminath Dondeti <[email-masked]>
Date: Thu, 06 Mar 2008 17:07:30 -0800

Thanks Cullen.

regards,
Lakshminath

On 3/6/2008 5:05 PM, Cullen Jennings wrote:
>> that Cullen clarified his position.  But, I have to say that this 
>> thread is but one example that we often don't clearly understand each 
>> other's positions.
>>

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 6 Mar 2008 17:48:22 -0800

The part of the discuss on lost that I have problems with as a discuss  
was text that said:

"Ted and I have discussed this and he is going to propose some  
clarifying text before I try to evaluate this."

I put that in before the IESG call where this document was on the  
Agenda - This was put in as the document editor, Ted in this case, had  
asked me not to put in a discuss until we tried to figure out a way to  
resolve this that did it without opening too many old wounds (this WG  
has plenty of wounds). Ted sent me the text before the call (he send  
it Tuesday) - I should have updated the discuss before the call this  
morning however for some odd reason there have been some other things  
using up my time this week. I have now updated the discuss and removed  
this part as I should have done this morning.

Cullen

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 07 Mar 2008 14:55:10 +1300

> it to the IESG and IAB's attention in March 2006.

That's true, from my personal knowledge since I was in the IESG
at that time. However, that supports my point ;-) .

(Not to be defensive, but the only changes in RFC 3777 that Ralph
specifically recommended were the ones covered in RFC 5078).

    Brian

---

From: Ralph Droms <[email-masked]>
Date: Thu, 6 Mar 2008 21:15:40 -0500

> specifically recommended were the ones covered in RFC 5078).

Brian - you might be right, but only on a technicality.  I noted that  
a clarification in RFC 3777 in the definition of the term of a mid- 
term appointment was needed, but didn't give a specific  
recommendation.  More to the point of Lakshminath's observation, I  
explicitly pointed out the conflict between RFC 3777 and the IAB  
requirements statement to the IAB and the IESG; I didn't recommend a  
change to RFC 3777 mostly because I thought it was the IAB  
requirements that needed to change.

- Ralph

---

From: John C Klensin <[email-masked]>
Date: Thu, 06 Mar 2008 21:16:24 -0500

--On Thursday, 06 March, 2008 16:48 -0600 Pete Resnick
<[email-masked]> wrote:

>> Hmm.   If people believe that this document should be
>> processed as a  BCP, thereby presumably constraining
>> long-term IESG behavior and  adding to our procedural core,
>> should it be added to the PUFI agenda  for preliminary
>> discussion?
> 
> The PUFI BOF chair, who has not completed his list of
> currently desired items on the grand list of things to cover
> in this BOF, hereby groans at the thought of adding another.

The PUFI BOF chair has the sympathy of the former POISSON chair.

However, if the community is ready to take on a significant
package of changes to our basic procedures, then, IMO, it ought
to have an opportunity to prioritize which ones it wants to
address and to discuss how it wants to discuss those and what,
if anything, it wants to do about the others.   To the extent to
which the General Area AD has decided that PUFI is the best way
to explore those issues, I think it is incumbent on PUFI that it
have the whole list on its agenda, not just one document that,
because of either timing or authorship, was given Last Call
treatment not afforded to any of the others.

   john

---

From: Ted Hardie <[email-masked]>
Date: Thu, 6 Mar 2008 18:43:39 -0800

>this part as I should have done this morning.

For the record, I suggested we talk after Cullen DEFERed on the previous
telechat to see whether it was a misunderstanding on his part,
rather than a serious issue.  It was later that I suggested someone else
hold the discuss, because I thought Cullen would want to recuse,
since he is a patent author on a patent his company has filed related to
this document.

Given that I have now seen the discuss text, I can see that
he has managed to open the old ones, bring in ones from a related
working group that aren't really salient, and open significant new ones,
all without having paid much attention to all of the text and effort that
flowed in the attempt to get early discussion of this.

To quote ekr, "Outstanding!"

				Ted Hardie

---

From: Andrew Newton <[email-masked]>
Date: Thu, 6 Mar 2008 22:10:18 -0500

On Mar 6, 2008, at 7:10 PM, Brian E Carpenter wrote:

> very precise.

What does this mean?  Is it an argument that as a BCP the "shoulds"  
carry weight whereas now they can be obeyed more conveniently?  Or is  
it a general comment regarding the futility of formalizing procedures?

-andy

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 07 Mar 2008 16:32:44 +1300

On 2008-03-07 16:10, Andrew Newton wrote:
> carry weight whereas now they can be obeyed more conveniently?  Or is it
> a general comment regarding the futility of formalizing procedures?

I think it's both. It's harder to disregard a "should" in a BCP;
it's easier to update an IESG-issued document than a BCP,
and it's very hard to get either of them 100% right.

We also have to remember that a "DISCUSS" position is not a formal
part of the IETF process. It's simply the current method used by the
IESG for logging lack of consensus. There's a lot of work in
turning it into formal process language, and I wonder who has
the appetite for that work?

    Brian

---

From: Dave Crocker <[email-masked]>
Date: Fri, 07 Mar 2008 00:43:05 -0700

Brian E Carpenter wrote:
> impossible to write precise procedural text ...

I see your point.

What I don't see is why the IETF's efforts at writing and enforcing procedural 
rules would be so much less successful than the efforts to create laws, 
contracts, and the formal rules that govern so many other organizations.

The real irony, of course, is that we are community whose main work is to write 
formal procedures and make then into formal standards.  So we probably ought to 
view it as a tad embarrassing that we can't do that adequately for our governing 
body.

d/

d/

d/

-- 

   Dave Crocker

---

From: "Spencer Dawkins" <[email-masked]>
Date: Fri, 7 Mar 2008 07:56:31 -0600

Wow. This was an interesting thread (that developed quickly). Why aren't you
guys all on airplanes yet?

So, to summarize a couple of points that other people made, but I didn't
want to lose in the forest...

... DISCUSS has no BCP process standing today. I'm not sure that giving it
process standing is the best use of any time we're willing to spend on
process discussions.

... if people would feel better if the IESG reissued the discuss criteria
document as an IESG statement, go for it.

... I understand what Ted is saying about "this needs to be a BCP". If we
were, in any way, capable of making changes to BCP text during the same
decade that we discover a problem, I might even agree. We're not. Pete's
heavy sigh about the amount of stuff that's stacking up for PUFI is real,
but is also the result of the community's decision to ignore process topics
after Montreal. That existing stack does not make me want to put anything in
BCP text that's not already in BCP text.

Look. We can't even come up with new names for three levels of standards
track that aren't stupid, and that really doesn't matter. What makes us
think we can come up with a set of "non-criteria" that will be solid enough
to put in a BCP that might last for more than a decade (please check the
dates on most text in 2026, 2418, etc), especially if the downside is that
an AD can then say "but that's not on the list of non-criteria"?

If the concern really is a rogue AD, I'm pretty sure our ability to predict
and preempt the path of rogue-ivity isn't up to this challenge (it would be
the IETF BCP equivalent of proposing a "Law Prohibiting Cooking Intelligence
to Justify Invading the Country Next to the Country that Hosted the Group
that Attacked Us" for the USA in 2002).

... I'm thinking that unreasonable DISCUSSes will be unreasonable whether
they specifically map onto a "non-criteria" in the current ION, or in some
future BCP, or not. Unreasonable DISCUSSes should always be challenged.

... there is a pre-existing escalation process for dealing with unreasonable 
DISCUSSes, but it doesn't sound like we're particularly good at using it.

... there is a pre-existing process for dealing with an IESG that goes
against the current in the community, whether they are actually violating
BCP text or not.

Various folks (who I consider smart friends) have pointed out that we've
never "used" the recall process (also in RFC 3777). That's true, but in 
private
discussions, I've heard about two recall petitions that were filled out and
then shown to the subjects of the petitions, and the recall petition
initiators were satisfied with the results, without "going public". So I'm
not as concerned about a rogue IESG that says "yeah, that's what the ION
says, but we don't care" (or even "yeah, that's what the IESG Statement
says, but we don't care") as other people seem to be. If you're rogue 
enough, you can blow off BCP text, too.

On the specific topic of "unfiltered" reviews, (as one of the original
Gen-ART reviewers) I do see authors talking about "satisfying" reviewers.
Please don't. "Satisfy" the COMMENTing/DISCUSSing AD, and "satisfy" the
community.

Always remember that http://www.alvestrand.no/ietf/gen/art/gen-art-FAQ.html
explicitly addresses the question, "What if the Gen-ART reviewer is just
flat wrong?". That is not an accident...

I think it would be nice for the IESG to collect, and maybe even put in one
place, "the way reviews work".

It's worth noting that even for Gen-ART, the process has changed
significantly since we started writing reviews for Harald. Those reviews
were done ONLY for documents on a telechat agenda, and they went ONLY to
Harald, who either laughed, COMMENTed, or DISCUSSed. The reviewers weren't
involved in further resolution, and I'm not even sure authors knew that
Harald's comments came from someone else's review. Eventually, we started
reviewing at IETF Last Call time, and copying shepherding ADs, and then
authors, and then WG chairs, but that's not where we started out.

So, for example, it probably IS worth finding out if the rest of the ADs who 
sponsor reviewing bodies also use Russ's division into

- "did you consider the reviewer's comments?", especially when the review 
was issued as
part of IETF Last Call, and

- "how did you address this specific comment, which I agreed with?", whether 
the AD entered the comment as a non-blocking COMMENT or as a DISCUSS.

Thanks,

Spencer

---

From: Andrew Newton <[email-masked]>
Date: Fri, 7 Mar 2008 11:37:42 -0500

On Mar 6, 2008, at 9:43 PM, Ted Hardie wrote:

> It was later that I suggested someone else

This is a reasonable action given the conflict of interest here.  If  
another IESG member could take up the discuss, the author team and  
document shepherd would work to resolve it.  But Cullen should recuse.

-andy

---

From: Russ Housley <[email-masked]>
Date: Fri, 07 Mar 2008 13:56:43 -0500

Lakshminath:

>>So, I'll tell everyone how I deal with Gen-ART Reviews.  Other 
>>General ADs may have done things slightly different.
>I have reviewed documents as a Gen-ART reviewer (during Brian's 
>tenure I think), sec-dir reviewer and also provided IETF LC comments 
>on some documents.  As a reviewer, I am not sure whether I was 
>expecting answers all those times.  I am pretty sure I have not 
>always stated whether or not the answers are satisfactory.
>
>Next, I can imagine an author not wanting to respond to something I 
>may have said because it was totally bogus or inappropriate and does 
>not deserve a response.  That might very well happen when I review 
>documents on a topic that I am not familiar with and haven't had the 
>time to read related references (that varies depending on the time 
>available, etc.).  Perhaps that is not such a bad thing; being 
>blissfully ignorant on some topics keeps me, well, blissful.  I use 
>somewhat of a hyperbole for obvious reasons.  I am sure many other 
>situations are much more nuanced.  I hope ADs don't continue to hold 
>a DISCUSS in those situations waiting for a dialog to take place or 
>waiting for a consensus to emerge.  I sometimes hint in my reviews 
>that the topic may be at the border of my knowledge and if I have a 
>bias.  Perhaps that is helpful.

Even if the response does not go to the person making the comments, 
ADs need to see a response.  Silence does not help us understand if 
consensus has been achieved.  Last Call is the only point in the 
development and review of many documents where review from other IETF 
Areas takes place.  It is very important that this cross-Area review 
take place.

Russ

---

From: Thomas Narten <[email-masked]>
Date: Fri, 07 Mar 2008 14:18:20 -0500

> not the answers are satisfactory.

On this one point.

IMO, one of the biggest causes of problems (and most under-appreciated
process weakness) in the IETF (and any consensus based organization
for that matter) is poor handling of review comments.

The ideal way to deal with them is to always respond, and to get an "I
am satisfied with your response" to close  the thread. Anything else
runs the risk of:

 - author says "I dealt with those comments in the revised ID", with
   the reviewer saying "Nope, I was ignored".

 - author says "I dealt with those comments in the revised ID", and
   this actually was the case, except that they accidentally missed
   one or two important issues. Reviewer is left wondering "was I
   blown off, or was this just an oversight, or..."

 - author thinking they are "done", because they "responded on the
   list". But, no changes in the document and/or reviewer is still not
   happy.

 - reviewer having invested a not-insignificant amount of time doing a
   quality review feeling "what's the point", which doesn't help to
   motivate a volunteer organization. This is  especially problematic
   from the cross-area review perspective.

Repeat above several times and intersperse with long periods of time
where nothing happens on a document. You now have an idea of why it
seems to take a long time to get documents through the system.

One of the reasons I'm such a fan of issue trackers is that it tends
to remove a lot of the above stuff by simply not allowing stuff to
fall through the cracks. Sure, trackers have overhead and are overkill
in some cases. But if one could somehow analyze the number of
documents that have been delayed for some time due to poor handling of
review comments...

Thomas

---

From: "Eric Gray" <[email-masked]>
Date: Fri, 7 Mar 2008 13:45:23 -0600

Thomas, et al,

	Please see one (set of) comment(s) below...

--
Eric Gray
Principal Engineer
Ericsson  

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Thomas Narten
> Sent: Friday, March 07, 2008 2:18 PM
> To: Lakshminath Dondeti
> Cc: [email-masked]; [email-masked]
> Subject: Re: IONs & discuss criteria
> 

Actually both of these last points are especially problematic from a
generalist review perspective.  Those of us doing Gen-ART reviews can
not possibly subscribe to every list and it is sometimes (often?) the
case that some of the review comments are discussed at great length
on the mailing list - completely (though unintentionally) leaving the 
commenter out of it.

In addition to raising the "what's the point" attitude flag, this can
even lead to the feeling that the comments actually went black-hole
surfing, possibly even leading to those harrassing trouble tickets for
lost E-Mail our IT people love to see so much.  This is very disturbing
given the non-insignificant amount of effort occasionally involved in
just trying to understand a draft enough to make (presumed) intelligent
comments about it and not even getting a sense of whether or not you
actually succeeded.

Minimally, as one of the people to whom that has happened, it would be
nice if at least an initial ("thanks for the review and comments") mail
included the commenter, in every case.  Even a "I wish you would stop 
bothering us with all of these silly comments" would be a response.  
Of course, I presonally would prefer that that sort of response was not 
addressed to the list, with or without me on it.  :-)

By the way, I agree with the comments below on issue tracker usage, and
I feel there must have been some significant, noticeable, improvements
in draft processing since they started being used.  Perhaps with more
(and wider) education on what is available, the gains may be even more
appreciable.

---

From: Andrew Newton <[email-masked]>
Date: Fri, 7 Mar 2008 15:21:25 -0500

To Eric, Spencer, and all the other Gen-ART reviewers:  Thank you.

My experience with Gen-ART reviews has been very positive, and I  
appreciate your work and effort.  I realize you weren't seeking public  
praise, but your volunteer contribution to the good of the IETF should  
be recognized.

---

From: Tim Polk <[email-masked]>
Date: Fri, 7 Mar 2008 16:00:57 -0500

Spencer,

On Mar 7, 2008, at 8:56 AM, Spencer Dawkins wrote:

> (stuff deleted)
>
> So, for example, it probably IS worth finding out if the rest of  
> the ADs who sponsor reviewing bodies

As an AD who sponsors a reviewing body (the Security Directorate), I  
guess it is my
turn to step into the fray.  Yikes!

Before I get started, I would like to note that I would not have  
survived my first year as an AD without
the support of the Security Directorate.  I continue to be impressed  
by the quality of the reviews that
are performed.  It is not an easy task when you are assigned a  
document from an unfamiliar area.   I
am sure that ADs sponsoring other reviewing bodies feel the same way.

> also use Russ's division into
>
> - "did you consider the reviewer's comments?", especially when the  
> review was issued as
> - "how did you address this specific comment, which I agreed  
> with?", whether the AD entered the comment as a non-blocking  
> COMMENT or as a DISCUSS.
>

I considerate it my responsibility to ensure that reviews from the  
security directorate
are considered thoughtfully.  I requesting that members of the  
community devote their
precious time to these reviews, and I don't want to see them  
ignored.  So yes, I have
filed process discusses of the form "The authors have not responded  
to John Doe's
secdir review."  If the review had unsuccessfully attempted  to  
initiate a conversation
with the authors and they were unresponsive, that may even be the  
only appropriate
path to take.  If the review was in the form of concrete suggestions  
("the security
considerations section needs to address man-in-the-middle attacks") I  
may choose to
issue a DISCUSS using their text instead of involving the reviewer.

If the review was acknowledged, then I review the email thread.  I  
attempt to verify that
any agreed upon actions are included in the current draft or  
implemented by a note to
the RFC Editor.  Modifications promised but not executed merit a  
discuss just to ensure
they aren't forgotten.  (I believe that this is consistent with the  
DICUSS criteria ION, under
the IETF process for document advancement.  Regardless, it doesn't  
present a burden or
unduly complicate the process.  At least, no one has complained about  
this type of discuss.)
If issues were raised in the review but agreement was not reached, I  
try to decide whether
I agree with the reviewer's comment *and* its relative importance.  
So, these "residual"
issues get addressed using the second method.

The more difficult problems come when a review is submitted just days  
before a telechat.
IETF Last Call has typically closed, and the authors may not have  
even seen the review
yet.  This is dangerous territory, since the temptation is to cut and  
paste the entire review
into my discuss to ensure that it isn't overlooked.   Of course, the  
more appropriate and
more helpful course of action is to determine which comments I  
support and separate
them into non-blocking comments and discuss worthy buckets.

[Confession time:  The temptation of cut-and-paste is sometimes too  
strong for a mere mortal,
though. I just revised my discuss on a document from yesterday's  
telechat, where I had cut
and pasted a secdir review, to separate the issues into the discuss  
and comment buckets.
Knowing better isn't the same as doing better.  My thanks to the  
sponsoring AD, who kept
me honest and asked me to review and revise!]

I am not particularly methodical by nature, so I can't claim I  
perform this exact process in
every case.  However, that is a good overview of the process I try to  
follow.

Tim Polk

---

From: Lakshminath Dondeti <[email-masked]>
Date: Sat, 08 Mar 2008 00:12:16 -0800

On 3/7/2008 11:18 AM, Thomas Narten wrote:
> am satisfied with your response" to close  the thread. 

"Ideal" being the keyword though.  Not everyone, for any number of 
reasons, including cultural reasons, will come out and state "all 
clear."  It is also asking too much to ask the reviewer to get into a 
debate with the authors.  It also fosters an environment where the 
reviewer starts becoming an authority.

>Anything else

I am not sure this is the "right" kind of motivation here.  As one such 
reviewer, all I look for is thanks from the AD whose directorate or team 
I am serving on.  And they have always been thankful.  No, I do not 
constitute "representative population."  I am just offering one data point.

> seems to take a long time to get documents through the system.

Indeed.  What started out as a great idea -- I volunteered to be a 
GenART reviewer 3-4 years ago now -- is beginning to become yet another 
burden in the process.

> documents that have been delayed for some time due to poor handling of

Yeah, it would be interesting.  Although, I wonder what we will do with 
that information.  Reviewers are not accountable for delays.  There is 
no expectation of time commitment from them, for instance.

---

From: Lakshminath Dondeti <[email-masked]>
Date: Fri, 07 Mar 2008 23:52:22 -0800

On 3/7/2008 10:56 AM, Russ Housley wrote:
>>> comment, it deserves an answer.  So, this is a procedural objection.  
>>> In this situation, I've been careful to say that the authors do not 
>>> need to accept all of the comments, but then need to answer them.
>>
>> I have reviewed documents as a Gen-ART reviewer (during Brian's tenure 
>> I think), sec-dir reviewer and also provided IETF LC comments on some 
>> documents.  As a reviewer, I am not sure whether I was expecting 
>> answers all those times.  I am pretty sure I have not always stated 
>> whether or not the answers are satisfactory.
>>
>> situations are much more nuanced.  I hope ADs don't continue to hold a 
>> DISCUSS in those situations waiting for a dialog to take place or 
> Even if the response does not go to the person making the comments, ADs 
> need to see a response.  Silence does not help us understand if 
Russ,

Thank you for your note.

It's a fair thing to say that the ADs need to see a response.  I also 
agree that cross-area review is important and at times unearths issues 
that may not have been raised in WG-level reviews.  Personally, I prefer 
cross-area reviews to take place prior to the LC process and hope that 
the the LC process is for those issues that may have been really 
overlooked despite the best efforts of the WG chairs and ADs.

I do not however quite understand the idea that we have to get consensus 
in the context of each GenART/Sec-dir/<fill-in-the-blank> review.  It is 
of course quite plausible that one or two of those reviewers will never 
be satisfied with any level of revision of a given specification.  In 
other cases, it may be that the reviewer has his or her personal 
preference on how to write documents and will never come out and say 
that the document they reviewed is "ready."

I have winced when some authors wrote to me after a review saying 
something along the lines of "does the following revised text sound 
better."  I would have been merely pointing out something I thought was 
not clear or might cause interoperability issues that they may have 
overlooked or missed.  How they might fix it is entirely up to them (or 
the ADs involved).  If their take was that they considered the comment 
and thought what they wrote is much more meaningful in the context of 
their work or the interoperability issue I raised does not apply within 
the scope of their specification, well so be it.  Next, I trust them to 
do that.  I don't need to see a dialog.  I am willing to clarify my 
comments, and if I cannot articulate the issues I am raising clearly, 
well then the document needs to move on in the process.

Either the shepherding AD or the AD who solicited the review needs to 
determine what if anything specific needs to be done in all these cases. 
  They are the judges of consensus.  No matter who the reviewer might be 
they were not selected by the community to do the AD's job.

Creating an environment where all these additional reviews and reviewers 
essentially block document progress is not the right direction.

best regards,
Lakshminath

---

From: Michael Thomas <[email-masked]>
Date: Sat, 08 Mar 2008 07:03:48 -0800

> praise, but your volunteer contribution to the good of the IETF should  

I have to say that I really like these and the cross area reviews a lot. As
an author/editor having to digest zillions of posts on the lists mixed with
time and entropy, it gets really hard to look at the draft with the 
critical eye
of somebody who might have to actually try to make sense of it. Not to
mention trying to determine what the draft says versus the lore that's
built up around it.

If I could make a suggestion, what might really be useful as well is 
collecting
implementation notes, and especially what throws people off when coding
up the draft, or implements things in new and unintended ways. Unintended
can sometimes be very cool, but often it's that the draft isn't 
sufficiently
precise.

          Mike, who tries to do this but given a cookie cutter form 
might do better

---

From: Lakshminath Dondeti <[email-masked]>
Date: Sat, 08 Mar 2008 08:53:17 -0800

And my thanks, specifically to Pasi, Miguel(twice), Vijay and Joel, who 
as GenART reviewers, provided thorough reviews of documents I recently 
shepherded or co-authored, and followed up diligently after revised 
documents have been published and sent a "ready for publication" or 
"addresses all comments" emails.  Thanks guys.

(I picked the recent instances).

best,
Lakshminath

---

From: John C Klensin <[email-masked]>
Date: Sat, 08 Mar 2008 11:56:07 -0500

--On Saturday, 08 March, 2008 00:12 -0800 Lakshminath Dondeti
<[email-masked]> wrote:

>> The ideal way to deal with them is to always respond, and to
>> get an "I am satisfied with your response" to close  the
>> thread. 
> 
> "Ideal" being the keyword though.  Not everyone, for any
> number of  reasons, including cultural reasons, will come out
> and state "all  clear."  It is also asking too much to ask the
> reviewer to get into a  debate with the authors.  It also
> fosters an environment where the  reviewer starts becoming an
> authority.
>...
 
>> Repeat above several times and intersperse with long periods
>> of time where nothing happens on a document. You now have an
>> idea of why it seems to take a long time to get documents
>> through the system.
> 
> Indeed.  What started out as a great idea -- I volunteered to
> be a  GenART reviewer 3-4 years ago now -- is beginning to
> become yet another  burden in the process.

I have to agree with this.  One of the biggest risks we have to
quality in standards is the dark side of the review process, a
situation in which the effort to get a document as nearly
correct as is reasonable given its maturity level and possible
turns into "deal with that objection".   A half-dozen years ago
we had extensive discussions of a concern in which ADs would
object to particular text and authors, exhausted from the
process of getting documents produced, would agree to any
suggested changes --as long as they were not really offensive--
in order to make the objection go away.   Put differently, there
is a tendency for "satisfy him (or her) and make the DISCUSS go
away" to become a more important objecting in practice than "get
things right".

In at least some ways, the DISCUSS criteria were an attempt to
constrain that problem, at as as far as the ADs were concerned.

If we replace the "opportunity" to have to individually satisfy
a dozen or so ADs with the opportunity to have to individually
satisfy them plus a dozen more area-related reviewers, we are in
big trouble.

I am greatly in favor of these invited reviews if they ensure
that every document is carefully reviewed by someone who was not
part of its development process.  I think that is where we
started out.

But the IESG has been selected by the community to take
responsibility for these evaluations.   If the IESG isn't going
to do it, or can't, we need to be looking at our basic processes
and structure, not at who is reviewing what for whom.  If a
review is done for a particular area, I expect that review to be
treated primarily as advice to the relevant ADs.  I expect those
ADs to evaluate that advice carefully, not just to critically
accept it.

I definitely do not want to see a discussion between authors and
reviewers --especially Area-selected reviewers-- during Last
Call.  It too easily deteriorates into a "satisfy him"
situation, and those reviewers are not anything special (or,
unlike the ADs themselves, selected by some community mechanism).

I think this whole process needs a little refocusing.
Especially for WG documents, no matter how many reviewers,
shepherds, and lions, tigers, and bears we introduce into the
system, the IESG should have one primary focus, which is making
a "go / no-go" decision on whether a document is ready for
approval and publication given maturity levels and any other
relevant issues.  If the answer is "no-go" for anything but an
obvious matter about which there is no dissent or ambiguity, the
document ought to go back to the WG for resolution of issues
(which should clearly be identified as clearly as possible), not
turned into a negotiation process between whomever happened to
generate a comment and the authors about whether that person's
view of the comment can be satisfied.   There is just too much
risk in the "satisfy comments" model of getting something
important wrong or of responding to the comment but missing the
main point of which the comment is a symptom.

I certainly don't object to Thomas's idea of using issue
trackers more and I think that making reviews public is an
important safeguard.  But those issue trackers should be used to 

	* inform the IESG's decision about whether the document
	is ready and to
	
	* help inform the WG, presumably along with a careful
	IESG summary, about the issues (not the specific
	comments) it needs to address if the document is bounced
	back to it.

Except perhaps for editorial matters and for clarification, a
dialogue between an author and someone who makes a comment
should have no place in the consensus process.

    john

---

From: "Frank Ellermann" <[email-masked]>
Date: Sun, 9 Mar 2008 05:35:05 +0100

> To Eric, Spencer, and all the other Gen-ART reviewers:  Thank you.

+1  

And thanks for offering a public list allowing to see what is going
on, for a boilerplate stating "treat it just like any other review",
and for picking the nice name "review team" instead of "directorate" <shudder />

 Frank

---

From: Sam Hartman <[email-masked]>
Date: Sun, 09 Mar 2008 09:38:45 -0400

>>>>> "Dave" == Dave Crocker <[email-masked]> writes:

    >> Making it a BCP will make the interpretation problem worse not better.

    Dave> How?

You can update an IESG statement mor easily than a BCP.  As you find
areas where the text is unclear and you have to interpret you can
actually go back and update the text.

---

From: Russ Housley <[email-masked]>
Date: Sat, 08 Mar 2008 16:42:44 -0500

Lakshminath:

>It's a fair thing to say that the ADs need to see a response.  I 
>also agree that cross-area review is important and at times unearths 
>issues that may not have been raised in WG-level 
>reviews.  Personally, I prefer cross-area reviews to take place 
>prior to the LC process and hope that the the LC process is for 
>those issues that may have been really overlooked despite the best 
>efforts of the WG chairs and ADs.
>
>I do not however quite understand the idea that we have to get 
>consensus in the context of each GenART/Sec-dir/<fill-in-the-blank> 
>review.  It is of course quite plausible that one or two of those 
>reviewers will never be satisfied with any level of revision of a 
>given specification.  In other cases, it may be that the reviewer 
>has his or her personal preference on how to write documents and 
>will never come out and say that the document they reviewed is "ready."

I think you completely misunderstand my point.  A reviewer can make a 
comment, and the authors or WG can say that they disagree.  This is 
important for an AD to see.  The AD now needs to figure out whether 
the reviewer is in the rough part of the rough consensus or whether 
the reviewer is representative of a larger part of the community.  If 
the review is ignored, then the AD get no indication that further 
investigation is needed.

>better."  I would have been merely pointing out something I thought 
>was not clear or might cause interoperability issues that they may 
>have overlooked or missed.  How they might fix it is entirely up to 
>them (or the ADs involved).  If their take was that they considered 
>the comment and thought what they wrote is much more meaningful in 
>the context of their work or the interoperability issue I raised 
>does not apply within the scope of their specification, well so be 
>it.  Next, I trust them to do that.  I don't need to see a 
>dialog.  I am willing to clarify my comments, and if I cannot 
>articulate the issues I am raising clearly, well then the document 
>needs to move on in the process.

In the example you give, I think the response is good as it ensures 
that the comment was understood.

>Either the shepherding AD or the AD who solicited the review needs 
>to determine what if anything specific needs to be done in all these 
>cases.  They are the judges of consensus.  No matter who the 
>reviewer might be they were not selected by the community to do the AD's job.

True.  As you have heard from Tim and me, these reviews are very 
helpful to the ADs, and I am saying that the discussion that follows 
such a review is helpful in judging consensus.

>Creating an environment where all these additional reviews and 
>reviewers essentially block document progress is not the right direction.

I agree, and requiring a response to the review is substantially 
different than blocking a document.  The alternative is silence, and 
silence cannot aid in judging consensus.

Russ

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 10 Mar 2008 09:30:52 +1300

Lakshimnath,

On 2008-03-08 21:12, Lakshminath Dondeti wrote:
...
> Reviewers are not accountable for delays.

Well, at least for Gen-ART there is a deadline:
the end of Last Call for LC reviews, and a day or
so before the telechat for pre-IESG reviews.
Obviously, reviewers are human and sometimes miss
those deadlines, but they exist.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 10 Mar 2008 09:46:29 +1300

John,

> unlike the ADs themselves, selected by some community mechanism).

Well, speaking as a reviewer who has often exchanged email
with authors during LC, I find that hard to accept. Very often
the dialogue is about clarity issues (either in the the draft,
or in the review) and ends with either the reviewer or the
author saying "Oh, I see what you mean." But there's a reason
why the Gen-ART review boilerplate says "Please resolve these
comments along with any other Last Call comments you may receive."
You're correct that there is no special status, and I'd be
rather disturbed if my review was the only community review.

     Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 10 Mar 2008 09:54:45 +1300

On 2008-03-09 17:35, Frank Ellermann wrote:
> on, 

Just to be clear - the list has a public *archive*
http://www.ietf.org/mail-archive/web/gen-art/current/maillist.html

    Brian

for a boilerplate stating "treat it just like any other review",

---

From: Ted Hardie <[email-masked]>
Date: Sun, 9 Mar 2008 22:45:33 -0400

>actually go back and update the text.

No, Dave cannot update an IESG statement or an ION issued by the IESG.
He can, however, put forward a proposed replacement document for
a BCP as an Internet draft.  So could the IESG.

See the difference?

This particular ION had a life as an internet-draft with an intent to publish
it as an RFC before the ION series existed.  It was draft-iesg at one time,
and no one came up with  a draft-ietfer- counter proposal.  But our
mechanisms for allowing that kind of publication have years of
experience behind them.  Not so much for community commentary
on  IONs, IESG statements, or the like, which have tended to be perceived
as changeable only by replacing the sitting IESG.

					Ted

				Ted

---

From: Ted Hardie <[email-masked]>
Date: Sun, 9 Mar 2008 22:56:06 -0400

>the review is ignored, then the AD get no indication that further

<snip>
>silence cannot aid in judging consensus.

I think you and Tim (and potentially other ADs in areas that have review
teams) are missing an opportunity here.  Over time, these  review teams
have been grown to the point where they do their reviews at Last Call
or before.  That's a very good thing. One of the reasons it *could* be a good thing
is to foster a culture of general cross area review.  If the Last Call reviews by
SAAG, Transport, Applications, and so on were seen as positive activities of the
areas, they could help encourage even earlier cross area review, either by
those teams or the areas as a whole.  Since that is one of the main selling
points of the IETF, that would be, let us say, nice.

To make that happen, though, you'd have to see them as your areas feeding
Last Call comments into the general Last Call commentary stream.  Those
are resolved by the shepherds and  the area advisor, not by the area directors
for the areas.  The way you're doing it now treats these reviews differently,
as advice to the area director of a relevant area, to be resolved differently.
In other words, it continues to make the individual IESG folks the focus of the
activity.  That limits the benefits this review can provide, pretty much, to
the benefit the IESG can absorb.  If the IESG isn't doing the early review,
the review teams don't either.

To put this another way, having vibrant, active review teams for an area
could be an area of leadership. Right now, it looks like they are being used
soley as time-management aids for the ADs instead.  That's a real opportunity
missed.
					Ted

---

From: Lakshminath Dondeti <[email-masked]>
Date: Sun, 09 Mar 2008 20:42:06 -0700

Brian,

Thanks for the clarification.  I know of those deadlines. Sam W also 
sets deadlines on sec-dir deadlines.

My reference to the delays was in the context of overall delays that 
Thomas was also referring to, that is if and when a dialog ensues 
between one or more reviewers and authors and the goal becomes 
'achieving some kind of a consensus with the reviewers'  and there is 
really no accountability for delays and such.

regards,
Lakshminath

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 10 Mar 2008 09:49:07 -0400

--On Sunday, March 09, 2008 22:45:33 -0400 Ted Hardie <[email-masked]> 
wrote:

>
> This particular ION had a life as an internet-draft with an intent to
> publish it as an RFC before the ION series existed.  It was draft-iesg at
> one time, and no one came up with  a draft-ietfer- counter proposal.  But
> our mechanisms for allowing that kind of publication have years of

my memory says that "discuss criteria" started out as an I-D, but with no 
consensus (or even a clear idea) on how to get a referenceable publication 
of it. It was one of the exampels that made me suggest IONs, exactly for 
that reason - RFC just didn't seem to fit.

                       Harald

---

From: Ted Hardie <[email-masked]>
Date: Mon, 10 Mar 2008 10:13:53 -0400

At 9:18 PM -0700 3/9/08, Russ Housley wrote:
>
>I really disagree.  Gen-ART Reviews begin this way:
>
>    I have been selected as the General Area Review Team (Gen-ART)
>    reviewer for this draft (for background on Gen-ART, please see
>    _http://www.alvestrand.no/ietf/gen/art/gen-art-FAQ.html_).
>
>    Please resolve these comments along with any other Last Call comments
>    you may receive.
>
>This tells the recipients that the review fits exactly the role you
>describe above.

But your behavior does not tell the recipient that.  If they were
being treated as general Last Call comments, it would be up to
the shepherd and sponsoring AD to resolve them, not up to
the General Area AD.  That leaves one set of people on the hook
for making sure they are done and deciding when they are,
and it is the same set no matter how the Last Call comment
is generated.  Your mechanism privileges one set
over others (in that they are more likely to be held as blocking
until resolved), is likely to be slower (since yet another busy person
must be informed that something is resolved, and may miss it
when it was),  and does not encourage things to push earlier than
Last Call (which is the opportunity I think you're missing).

				Ted Hardie

---

From: Russ Housley <[email-masked]>
Date: Mon, 10 Mar 2008 00:18:24 -0400

Russ

---

From: Russ Housley <[email-masked]>
Date: Mon, 10 Mar 2008 10:42:51 -0400

>Last Call (which is the opportunity I think you're missing).

I disagree with this characterization.

IETF Last Call (hopefully) generates comments.  These are usually 
resolved before IESG Evaluation, which is what you advocate in your 
note.  This is the normal case in my experience.  The issue seems to 
come up when they are not resolved.  As I said in my previous note, 
there are two cases.

(1)  The Gen-ART Review or other Last Call comments were ignored.  If 
someone takes the time to review the document at Last Call, they 
deserve the respect of a response.  Failure to respond is a 
procedural objection.  This is usually handled by the PROTO Shepherd, 
WG Chair, or document author.  If by the time the document reaches 
IESG Evaluation, I have put a DISCUSS on documents to ensure that a 
response does happen. (I did not say that the comments are accepted; 
I said that a response is provided.)  I have entered DISCUSS 
positions like this for Gen-ART Reviews, SecDir Reviews, and reviews 
from individual IETF participants.  I've been careful to say that the 
authors do not need to accept all of the comments, but then need to 
answer them.

(2)  I agree with one or more concerns raised in the Last Call 
comments that was not resolved.  Thus, a very  often a very small 
portion of last Call comments become blocking comments.  I tend to 
break the unresolved review comments into DISCUSS and COMMENT, giving 
credit to the source of the review.  (I'm not trying to take credit 
for someone else's work.)  AD judgement is needed here, and I 
consider the DISCUSS Criteria in making that judgement.

Russ

---

From: "Spencer Dawkins" <[email-masked]>
Date: Mon, 10 Mar 2008 13:50:40 -0400

Speaking only as a Gen-ART reviewer, what Russ said is how I think it works, 
and Ted's concern that I might be privileged as a Gen-ART reviewer at last 
call time is the reason we're having that conversation.

Gen-ART reviewers have had that concern since we were writing reviews for 
Harald. We don't WANT to be privileged, and we've worked consistently to 
head that off.

I provided this text that's in the Gen-ART FAQ: 'And always remember that 
the IESG ballot position is called "DISCUSS", not "IMPERIAL EDICT" or 
"BLACKMAIL"'.

This should be doubly so, when a review team reviewer raised a concern.

Thanks,

Spencer

From: "Russ Housley" <[email-masked]>

---

From: Jari Arkko <[email-masked]>
Date: Mon, 10 Mar 2008 14:31:48 -0400

It is my experience as well that Gen-ART or other organized reviews are
not given any more weight than other Last Call comments. However, I at
least weight different comments in different ways, based on whether I
agree with the issue, whether I believe the issue is a major problem or
a minor nit, whether the RFC Editor would take care of the edit anyway,
whether the fix can be just done or if it requires careful guidance and
discussion with someone, what the implications of the problem are if
left unsolved, etc.

At the end of the day, for some comments I would ask the documents to go
back to the WG, some require further discussion with the reviewer and
draft revision, some require an ack from the authors, some can be left
for the authors to decide what they do with it, and some can be ignored
(with a response, of course).

Jari

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 11 Mar 2008 09:38:06 +1300

On 2008-03-11 03:42, Russ Housley wrote:
> note.  This is the normal case in my experience.  

FWIW, that's my experience as General AD too - I left the handling
of LC reviews to the responsible people, but I certainly looked back
to check that they had been responded to, when the draft reached the
IESG.

I believe that one reason for this apparent disagreement is a missing
tool: an IETF-wide system for logging Last Call comments (including
reviews) on each draft that's in Last Call. The last time we discussed
this, we ended up slightly tuning the text of the Last Call message,
to say

" Please send substantive comments to the
  ietf at ietf.org mailing lists by <date>. Exceptionally,
  comments may be sent to iesg at ietf.org instead. In either case, please
  retain the beginning of the Subject line to allow automated sorting. "

But that just dumps comments into a messy email archive, at best.

I'd really like to see a more organised tool for this, and perhaps
some more precision about when a comment to the IESG alone is
appropriate.

    Brian

---

From: Tim Polk <[email-masked]>
Date: Mon, 10 Mar 2008 19:29:02 -0400

Ted,

There is no intention to treat Last Call comments from individuals  
differently than
those that come from a review team.   ADs *do* submit the same type  
of procedural
discuss to ensure a response to Last Call comments that *weren't*  
generated by
a review team.  I'll agree that such discusses are more commonly  
associated with
Last Call comments from review teams.  That shouldn't be surprising.   
For many
documents, the only cross area reviews come from the review teams.

However, I also expect that Last Call comments that *weren't*  
generated by a review
team are more likely to fall through the cracks.   I know that I am  
more sensitive to the
security directorate reviews than Gen-ART reviews or other Last Call  
comments.   In
addition to being assigned more or less on my behalf, they are  
focused on issues near
and dear to my heart.  I try to read all the Last Call comments, but  
when reviews focus
on issues I don't understand or don't find compelling, I move on.  In  
that case, I will not
notice that a response did not occur.  I suspect that other ADs  
suffer from similar human
foibles.

Thanks,

Tim Polk

---

From: Scott Brim <[email-masked]>
Date: Mon, 10 Mar 2008 09:59:26 -0400

All of this depends on the quality of the review and how it's followed 
up on.  Having to push back on insistent nonsense is a problem.  A good 
review that engenders a lot of discussion on substantial issues is very 
worthwhile.  We should foster those -- they are important.  This is no 
different than what happens within a WG.
_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: Dave Crocker <[email-masked]>
Date: Tue, 11 Mar 2008 13:53:06 -0400

> for that matter) is poor handling of review comments.

Whereas all of my own experiences with groups having problematic handling of 
reviews (except for a recent one where I happened to be the reviewer), is with 
the reviewer.

In all of the groups I've been involved with, reviewers were taken and pursued 
seriously by the group.

Problems occurred when the reviewer was vague, misguided and/or intractable.

Diligent reviews are hugely helpful, especially so the earlier they occur.

But not all reviews (including AD Discuss vetoes, which frequently are part of a 
form of review) are offered so helpfully.

These repeated discussions about reviews are forceful in demanding 
acknowledgment of the former, helpful type, while vigorously denying the 
damaging reality of the latter and the need to deal with the pattern of 
strategic problems they cause.

> documents that have been delayed for some time due to poor handling of

Mostly, we agree on these points.  Handled properly, placing review items in an 
issues list can be helpful to all parties, as long as each issue is clearly 
stated and possible resolutions or constructive guidance are included.

One caveat:  Sometimes it is the aggregate review that is most significant and 
breaking it into constituent 'issues' loses the broader concerns.

d/
-- 

   Dave Crocker

---

From: Michael Thomas <[email-masked]>
Date: Tue, 11 Mar 2008 11:16:39 -0700

> breaking it into constituent 'issues' loses the broader concerns.

I can't imagine dealing with something even vaguely contentious
without the issue tracker.  However as I've seen it used, there's a fair
amount of power in being able to declare on a wg list that something
is an ISSUE that needs to be tracked. In particular, I've seen a lot of
cycles expended on 'issues' that clearly have little or no following. It
can be pretty time consuming and disruptive.

Maybe we need some better gating functions to the issue tracker, but
it would have to be careful to not squash comments/issues from left field
(ie, from cross-area, disinterested, etc) with some crushing new process.

       Mike