Subject: Re: text suggested by ADs

From: Ralph Droms <[email-masked]>
Date: Tue, 03 May 2005 20:32:24 -0400

On Sat, 2005-04-30 at 11:12 -0700, Dave Crocker wrote:
> > > >  1. A Discuss may be asserted only when it pertains to a normative
> > > >  concern that involves the viability of the specification.
> >
> >  As a practical matter, the line between normative and informative is
> >  likely grey enough to make this suggestion unworkable...
> 
> interesting point.  first question, then, is why has the ietf been finding it 
> important to make the distinction between the two?

I can't speak for the IETF as a whole; for the most part, classification
as normative/informative is pretty obvious.  But, if a reviewer feels
strongly that something in a spec needs to be fixed, seems likely that
the reviewer will classify the issue as "normative".

> second question is how do we distinguish between Discuss items that really do 
> pertain to "it won't work" and "it's unacceptably deficient" concerns, versus 
> an AD's personal preferences and whims?

I suggest we depend on the IETF as a whole ... by publishing and
discussing the Discuss comments on a widely read mailing list.

> > > >  2. The AD raising the Discuss must post the details of their concern
> > > >  to the mailing list targeted to that specification and must provide
> > > >  clear direction as to how to cure the problem.  Failing the ability to
> > > >  provide the detail about how to From [email-masked] Wed May 04 00:29:29 2005
From: Ralph Droms <[email-masked]>
To: Dave Crocker <[email-masked]>
Date: Tue, 03 May 2005 20:32:24 -0400
Cc: [email-masked]
Subject: Re: text suggested by ADs
Sender: [email-masked]

On Sat, 2005-04-30 at 11:12 -0700, Dave Crocker wrote:
> > > >  1. A Discuss may be asserted only when it pertains to a normative
> > > >  concern that involves the viability of the specification.
> >
> >  As a practical matter, the line between normative and informative is
> >  likely grey enough to make this suggestion unworkable...
> 
> interesting point.  first question, then, is why has the ietf been finding it 
> important to make the distinction between the two?

I can't speak for the IETF as a whole; for the most part, classification
as normative/informative is pretty obvious.  But, if a reviewer feels
strongly that something in a spec needs to be fixed, seems likely that
the reviewer will classify the issue as "normative".

> second question is how do we distinguish between Discuss items that really do 
> pertain to "it won't work" and "it's unacceptably deficient" concerns, versus 
> an AD's personal preferences and whims?

I suggest we depend on the IETF as a whole ... by publishing and
discussing the Discuss comments on a widely read mailing list.

> > > >  2. The AD raising the Discuss must post the details of their concern
> > > >  to the mailing list targeted to that specification and must provide
> > > >  clear direction as to how to cure the problem.  Failing the ability to
> > > >  provide the detail about how to fix the specification, the AD must
> > > >  engage in a dialogue that has the goal of specifying that detail.
> >
> >  I agree with the first clause; the concern must be explained and motivated
> >  in detail.  The WG - not the AD or the doc authors in isolation - should
> >  develop the solution.
> 
> This raises two issues.  One is that the focus of the suggestion is making 
> sure that an AD who asserts a late-stage veto is meaningfully obligated to 
> work constructively to remove it.

Agreed and such obligation (which is the usual case now) would be a good
thing.

>   The other is that working groups rarely develop solutions.  Participants or 
> small sub-groups develop solutions; working groups review and approve.

Good point.

> When a random participant raises a concern during specification development, 
> the working group can readily acknowledge the issue and add it to the 
> workload, or it can fail to gain traction.  In the former case, the working 
> group takes responsibility for finding the solution.  In the latter, the issue 
> is, effectively, turned back to the person with the concern.  It is up to them 
> to find some way to get the working group to embrace the concern; the usual 
> way to do this is to propose a solution, so that the working group has a more 
> solid sense of the topic.
> 
> Now we move to a late-stage AD veto.  The working group has put years of 
> effort in and lots of review.  Here comes an AD -- typically one who has not 
> been involved until this point -- blocking progress by stating some concern.
> 
> If the concern is obviously valid to everyone, then there is no issue.  
> Everyone goes wow, we sure are glad you caught that, and goes off to fix it.
> 
> The problem is when the AD's concern is not obviously valid, or at least not 
> obviously valid as a valid reason for blocking progress.
> 
> Today, there is almost no cost to the AD in these situations and, therefore, 
> no pressure on them to be reasonable and constructive to resolve it.

So, without meaning any offense to the ADs, I suggest we lump random
participants, WG members, doc editors and ADs together when the spec is
reviewed - and ensure that all comments are published in the same forum
and given appropriate weight based on technical merit, as supported by
explanatory text when the comments are published.

- Ralph

> 
> We need to change limits and incentives, to fix this.
> 
>  
> > > >  In order to deal with the issue of a pocket veto, whereby the AD is
> > > >  intractable but maintains the veto, there needs to be a mechanism to
> > > >  force review of the Discuss, either to assert that, indeed, it
> > > >  involves a valid showstopper (failure) of the specification or that it
> > > >  can be ignored.
> > > >
> > >  such a mechanism already exists.
> 
> If you are referring to a classic Appeal, then that is too heavyweight and 
> onerous.  The cost to the participant, of making an appeal, is significant.
> 
> If you are referring to something else, what is is and where is it documented?
> 
> 
> 
>  d/
>  ---
>  Dave Crocker
>  Brandenburg InternetWorking
>  +1.408.246.8253
>  dcrocker  a t ...
>  WE'VE MOVED to:  www.bbiw.net

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Dave Crocker <[email-masked]>
Date: Wed, 4 May 2005 12:31:22 -0700

>  explanatory text when the comments are published.

Ralph,

  This isn't likely to succeed.  

  It is too straightforward, reasonable and fair.

and with my sarcasm mode turned off:

  This sounds like an excellent suggestion.

  It is straightforward, reasonable and fair.

  d/
  ---
  Dave Crocker

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Wed, 04 May 2005 16:28:20 -0400

In message <[email-masked]>, Ralph Droms writes

>explanatory text when the comments are published.

Then what?  How are the comments resolved?  Who makes the final 
decision about whether or not a document has met certain standards?

		--Prof. Steven M. Bellovin, http://www.cs.columbia.edu/~smb

---

From: Keith Moore <[email-masked]>
Date: Wed, 4 May 2005 18:24:56 -0400

>   It is straightforward, reasonable and fair.

why am I reminded of "Harrison Bergeron" ?  (if you haven't read it,  
google for it)

or for that matter _Atlas Shrugged_?  (not that I agree with Rand on  
everything, but she had this one pegged)

---

From: Ralph Droms <[email-masked]>
Date: Thu, 05 May 2005 08:30:21 -0400

Keith - thanks for the pointer to "Harrison Bergeron".  Coincidentally,
I was trying to recall this story in a conversation recently and had
forgotten the details and the author...

But, I don't see how it applies here.  I'm not claiming "Nobody was
smarter than anybody else."  Yakov explained it better than I have: "for
each AD there is more than one person in the IETF who is more
technically astute than that AD. So, why should the IETF decision
process favor opinion of such AD more than the opinion of these other
individual who are more astute that the AD ?"

- Ralph

---

From: Ralph Droms <[email-masked]>
Date: Thu, 05 May 2005 08:32:37 -0400

Steve - Final decision is made as it is today; proposed change is timing
and context for review...

- Ralph

---

From: Keith Moore <[email-masked]>
Date: Thu, 5 May 2005 11:41:07 -0400

> Keith - thanks for the pointer to "Harrison Bergeron".   
> Coincidentally, I was trying to recall this story in a conversation  
> recently and had
> smarter than anybody else."  Yakov explained it better than I have:  
> "for each AD there is more than one person in the IETF who is more  
> process favor opinion of such AD more than the opinion of these  
> other individual who are more astute that the AD ?"

Ralph,

While for each AD there may be more than one person in the IETF who  
is technically astute than that AD, it is difficult to reliably  
identify those people, and those people will typically _not_ be the  
other individuals who are reviewing a document in Last Call.   
Technically astute people who are able to review a significant number  
of documents probably end up on IESG sooner or later anyway -  
presuming that they have the other skills and attributes that make  
for good ADs.  The other thing is that given a set of people who are  
technically astute in a general sense, the ones who specialize in a  
particular activity probably end up being better at that activity  
than their peers.  So for instance the ones who specialize in  
reviewing large numbers of documents and trying to ensure some  
consistency between those documents are likely to be better at that  
skill than the other IETF participants.  It is not expected that ADs  
know more about each WG's topic than the WG.  But WGs sometimes have  
trouble seeing the forest for the trees.  It's part of the AD's job  
to examine things from a broader perspective.

There is no way to have meaningful external review of WG output  
without somehow "favoring" the opinion of those doing the review.   
And in order for such review to maintain some consistency between  
documents, there need to be a relatively small number of selected  
reviewers.  Peer review cannot work as well, both because the average  
clue level of a larger number of reviewers will inherently be lower,  
and because peers will not be as adept at looking at the document  
from a big picture perspective.

One inherent consequence of this for IETF is that it is possible for  
a single AD to have a large say in whether a document is found  
acceptable.  This is both good and bad.  Often a single AD will spot  
a significant problem that nobody else has seen.   Sometimes that  
problem is a subtle problem that others in IESG and/or the WG don't  
immediately accept or understand.  The other side of this coin is  
that when a single AD gives a document a good review, other ADs are  
less likely to state an objection to that document.   Giving a single  
AD a large say often has the effect of speeding a document's approval.

In other cases an AD will act capriciously or out of unreasonable  
prejudice, and there are sometimes accusations that an AD acted out  
of malice.  IETF and IESG processes try to provide multiple remedies  
to those situations.   It may be that the processes need to be  
tweaked or that other remedies are needed, and we should constantly  
be looking for ways to make our processes better.  But I don't see  
how to dispense with either external review or giving a single  
reviewer's opinion a lot of weight, at least on an initial review,  
without drastically reducing the quality and/or volume of our output.

Thinking about the disagreements between ADs and WGs that I've seen,  
they generally fall into two (overlapping) categories:

1. There is a legitimate difference of opinion about how to reconcile  
the conflict between the interests of the community of users that the  
WG hopes to serve, and some other community of users.  In this case I  
think "favoring" IESG is the right thing to do, because WGs  
frequently produce results that will cause problems for other  
communities - though naturally both IESG and the WG should attempt to  
work out a compromise that serves both sets of interests while  
minimizing the conflict between them.

2. The AD has a legitimate concern, but the change suggested by the  
AD is either (a) lame or (b) worse than the problem the AD wishes to  
remedy.  Part of this problem is that in practice the AD has  
relatively little ability to fix a document.  No matter how bad the  
document is, the most an AD can generally manage to do is to insist  
on small changes to the text.  Yes, in theory the AD could insist  
that the document undergo significant revision, but the pressure from  
both inside and outside the IESG to move even bad documents along is  
considerable.  [*].  Sometimes there's no way to fix a document with  
small textual changes.  But because of the pressure to either approve  
a document or to ask for fairly small changes to the document text,  
ADs sometimes suggest small changes to the text that end up being  
poor fixes.

The best cure for both of these problems, I suspect, is not to give  
ADs less "favor" in reviewing documents, but to identify such  
problems earlier, at a time when there is less investment in the WG's  
output and less sense of urgency to get the document out the door.

The connection with the Vonnegut story is this: an insistence on  
making things "fair" has the effect of discouraging excellence.

Keith

[*] Bad documents take more time to review than good documents --  
both because they tend to be poorly written and because the AD is  
struggling to think of a simple fix while reading the document -- and  
nobody on IESG wants to review a bad document over and over.

---

From: Pekka Savola <[email-masked]>
Date: Thu, 5 May 2005 18:48:32 +0300 (EEST)

> individual who are more astute that the AD ?"

What is the context of technical astuteness?  How do you compare 
people with different technical focuses?  You can't.

If an AD raises an issue about a document (from area X) that it 
conflicts or causes serious problems (from the perspective of area Y), 
how do you ensure that the more technically astute people 
(particularly on Y but also a bit on X) participate in the discussion?

The key point here is that the AD from area Y might be much more 
technically astute about that area than anyone in the WG producing a 
document.

Are you proposing that the IETF list is turned into generic discussion 
board for all the documents, generating 20-50+ messages for every 
IETF-LC'd document (every AD's "IESG review" plus clarifications, 
follow-ups, outsiders opinions, etc.)?

What about Informational/Experiemental WG submissions which so far 
have not been IETF LC'd?

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Joe Touch <[email-masked]>
Date: Thu, 05 May 2005 09:11:57 -0700

> decision about whether or not a document has met certain standards?

"Rough consensus and running code"

Even at this level.

Joe

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Fri, 06 May 2005 01:26:22 +0200

>"Rough consensus and running code"

Unfortunately this is not always true. When a "best common practice" is 
imagined or a IANA registry created. They can have impact beyond repairs. 
This is why I underline how Ted Hardie's question and the answers to it are 
important.

Everyone can forgive and forget an RFC. No one can do that with a IANA 
registry or with the momentum induced in telling people the world uses to 
do something it does not. Once a registry has been created or a non 
existing practice endorsed on matter "A", on wrong premises or not, if the 
matter "A" does exist, we will have to live with it for ever.

The only possible "correction" would be to create a competing correct "A" 
registry or to turn to another "world" (this is propobly the main reason of 
the interest in ITU) and then to start an alt-root war, with most of the 
users using the wrong solution, due to the IANA prestige; and turning down 
the whole IANA if the recognise the problem.

If the matter is important, and if it concerns governments (the Internet 
R&D funders) the implications for the Internet stability and future can be 
totaly out of proportion with the "A" matter.

This is why I would propose that new IANA registries are accepted ad 
experimenda (for test) until confirmed by a standard: during that period a 
registry could have several "versions" (even opposing). And that best 
common practices document (except for the Internet standard process) give 
criteria to verify (when) they reached the "common" level.

The formula could be "roug consensus, running code, proven practice and 
used registry".

jfc

---

From: Ralph Droms <[email-masked]>
Date: Fri, 06 May 2005 17:28:22 -0400

Comments in line...

> people with different technical focuses?  You can't.

Giving ADs a private veto (private in the sense of not discussed in
public) seems to compare technical astuteness and assign more weight to
ADs.  I suggest public discussion avoids giving ADs' technical
astuteness undue weight.

> (particularly on Y but also a bit on X) participate in the discussion?

By making the discussion public...

> technically astute about that area than anyone in the WG producing a 

So might any number of other IETF members...

> follow-ups, outsiders opinions, etc.)?

I suggest that AD input on a doc be considered equally with other input.
Details are TBD.

> What about Informational/Experiemental WG submissions which so far 

If there is a Discuss on such docs, the Discuss input should be reviewed
in public...

- Ralph

---

From: Pekka Savola <[email-masked]>
Date: Sat, 7 May 2005 08:19:28 +0300 (EEST)

> By making the discussion public...
[...]

There is nothing to prevent making such discussions public.  Many 
groups already do so so; the comments are already recorded in a public 
database, and when the WG (chair or document editor) follows up on 
those, e.g., by discussing the Discuss, it can be done Cc'ing the 
mailing list.

Or do you have the problem that the discussion is held on the WG list 
(and not on a wider forum), and for individual submissions, probably 
in private? (individual submissions are problematic in this sense, but 
they _are_ individual..)

So I don't see what problem you're seeing.  To me, a Discuss is a flag 
an AD can raise "I think there may be a serious issue here, and we 
need to discuss it in case you disagree".

In principle, I consider that reasonable given the responsibily of the 
IESG for technical quality.

However, sometimes this has some issues.  Specifically, sometimes ADs 
seem to be very busy and do not have time to respond to attempts at 
initiating dialogue or the round-trip time between messages is high. 
(Also, there are cases where the WG drops the ball for months, and 
then gets back later -- by then the AD has probably already forgotten 
what (s)he said and has to re-read parts of the spec.)  This is a 
particular problem if the WG thinks AD's issue is not really relevant 
or is flat out wrong; it may take a while to convince the AD (though 
the responsible AD may be able to help in these scenarios).

Maybe _this_ is something where we should consider how to increase the 
timeliness and effectiveness of the dialogue.

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Sat, 07 May 2005 18:41:22 +0200

I am puzzled by this thread which only document the way the Internet 
standard process is _not_ respected and how to fix the problem in _not_ 
respecting it. The discussion should either lead to calls of order, or to 
formal amendments to the Internet standard process.

I never shared in an IETF F2F meeting because too expensive - and I do not 
see the interest if it is not universal. So, I only see the IETF-on-line. I 
am dismayed by some narrowness of its vision. I fully accept its structural 
quasi-impossibility to innovate (innovation calls for a step ahead, 
consensus takes time). I am impressed by the simplicity of the Internet 
standard process methodology. The only real problem is the NOMCOM issue - 
actually the number of IESG candidates: I submit that this problem could 
resolve by itself if the other problems were on the way to be addressed. 
This calls complete review of the logic.

I see all this as a standard press/publisher process (and this is why I am 
surprised that the IASA project was not thought as financially self 
sustaining).

It offers two possible tracks:

1. an author has an idea for an editorial series/article which can come 
from the continuation of a previous article, to make a "standard" of it or 
to keep reporting on a common practice.
2. he contacts a specialised editorial manager (like sports, economy, 
leisures, fashion, etc. in a magazine) and his partner.
3. if interested the editorial managers talk to the editorial committee
4. if interested the committee create an editorial staff and team and 
define the charter of the planned deliverables.
5. the editorial team starts delivering

a. a free-lance sends an article - often reporting on something existing - 
to the editorial committee

6. deliverables are reviewed by the editorial committe
7. deliverables are circulated to every journalist and freelance for final 
review and QA
8. deliverables are sent to the publisher for final massaging, formating 
and publication through the RFC track (magazine) or the IANA database 
(associated radio station).

At every step, everyone, through the editorial committee, can call on the 
advise/arbitration of the editorial board and use the in-house tools and 
procedure, using a few web sites and a large number of independent mailing 
lists. The publishing entreprise is a cooperative structure.  A selection 
committee is in charge of feeding the structre (managers, editorial 
committee and board) with appropriate members.

Am I wrong reading it this way?

If not, I see several flaws in the editorial governance policy  (the way it 
works, not the way it is built)

1. the editorial board does not maintain an editorial line everyone 
(readers alike) can refer to. A network architecture model.
2. there are constant track violations. Once the opord of an editorial team 
has been given, understood and possibly commented, it should be respected 
or updated. But not whimsically changed by editorial managers or the 
editorial committee. They should only consider if the deliverables fit the 
order. Or change the order. This is what this thread is mostly about.
3. authors/journalists are not paid enough. Their salary is recognition and 
usefullness. The magazine does not promote its editorial horde (hoard?). 
What is wrong because recognition means exposure and exposure leads to 
self-quality control. As a whole exposure and recognition also means public 
interest and sponsors.
4. audience is not monitored. People start being bored with the Dallas-like 
IPv6 saga and the NAT/noNAT love story.They would like new topics on their 
real daily life, sometimes totally unknown to the magazine 'experts' as I 
just experimented in the languages area.
5. the editorial culture is still very confuse and provincial. The magazine 
is still very rooted as a Californian local publication. It should open 
itself to the world and get multilingual (spirit, editorial concerns and 
culture, languages, structural organization and concerns) and multinational 
(local list should help people to know each other, outreach and start 
innovative projects)
6. the tools (internal and external) have become confuse and outdated The 
website is very low relational quality. With a Washingtown Times 1990 look. 
I used a better mailing list system in 25 years ago.
7. the Radio Station (IANA) has grown-up out of control, yet the Internet 
standard process is still only magazine oriented. Control over the IANA or 
its replacement should be organised and a IANA standard process defined. 
Same for web site.
8. other side services like innovation (IRTF?), training, catalog, 
statistics etc. exist in a way. They should be considered and 
organized/managed the same.
9. PRs are poor. Every new RFC should be promoted to be made understood and 
the authors exposed and recognized. There should be a press release 
explaining its interest, etc.
10. there should be a "year book" summarising the state of the art in the 
different area of interest of the magazine, readers could refer to.

Once this is solved, or on commitment to be solved, I suppose that people 
ready to accept nominations will me more numerous. There is no risk in 
publishing the list of the nominated persons (by an least n persons 
including x from IESG?) who have accepted to be candidate. There would 
already a strong recognition in it.

jfc

At 07:19 07/05/2005, Pekka Savola wrote:
>There is nothing to prevent making such discussions public.  Many groups 
>already do so so; the comments are already recorded in a public database, 
>and when the WG (chair or document editor) follows up on those, e.g., by 
>discussing the Discuss, it can be done Cc'ing the mailing list.
>
>Or do you have the problem that the discussion is held on the WG list (and 
>not on a wider forum), and for individual submissions, probably in 
>private? (individual submissions are problematic in this sense, but they 
>_are_ individual..)
>
>So I don't see what problem you're seeing.  To me, a Discuss is a flag an 
>AD can raise "I think there may be a serious issue here, and we need to 
>discuss it in case you disagree".
>
>However, sometimes this has some issues.  Specifically, sometimes ADs seem 
>to be very busy and do not have time to respond to attempts at initiating 
>dialogue or the round-trip time between messages is high. (Also, there are 
>cases where the WG drops the ball for months, and then gets back later -- 
>by then the AD has probably already forgotten what (s)he said and has to 
>re-read parts of the spec.)  This is a particular problem if the WG thinks 
>AD's issue is not really relevant or is flat out wrong; it may take a 
>while to convince the AD (though the responsible AD may be able to help in 
>these scenarios).
>

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 09 May 2005 16:26:04 +0200

Pekka Savola wrote:
> already do so so; the comments are already recorded in a public 
> (and not on a wider forum), and for individual submissions, probably in 
> an AD can raise "I think there may be a serious issue here, and we need 
> to discuss it in case you disagree".

Definitely, there aren't supposed to be any private vetos these days.
Even for individual submissions, the DISCUSS texts if any will be visible
in the tracker, at least.

> IESG for technical quality.

But the WGs are *primarily* responsible for quality. The better the
WG output, the less the IESG will be seen as a blockage.

> (Also, there are cases where the WG drops the ball for months, and then 
> gets back later -- by then the AD has probably already forgotten what 
> (s)he said and has to re-read parts of the spec.)  This is a particular 
> problem if the WG thinks AD's issue is not really relevant or is flat 
> out wrong; it may take a while to convince the AD (though the 
> responsible AD may be able to help in these scenarios).

When everyone involved is overloaded and impatient, this can happen,
and I don't see any general solution, except for the PROTO shepherd
to be on the job every day. Since the PROTO solution is new, it's too
soon to say whether this is working.

    Brian