Subject: Review of: draft-resnick-on-consensus-05

From: Dave Crocker <[email-masked]>
Date: Sun, 06 Oct 2013 17:03:48 -0700

Review

Title:        On Consensus and Humming in the IETF
               draft-resnick-on-consensus-05

Reviewer:     D. Crocker
Reivew data:  6 Oct 2013

Summary:

      The draft discusses IETF processes for making group decisions, 
especially in the face of disagreement.  As the draft notes, the IETF 
has no formal membership, which makes strict voting impossible.  It also 
notes the engineering reasons that strict voting can be inappropriate. 
Hence the IETF uses the term "rough consensus" to describe it's approach 
to decisions.  The draft offers the view that this is term represents a 
group journey, rather than only a decision-making moment.  It also 
stresses the need for diligently dealing with objections, rather than 
merely developing an overwhelming numerical preference.

      In terms of philosophy and desirable practice, the draft discusses 
an extremely appealing model and generally explains its nature and 
practice well.  However the draft tends to confuse what is (or has been) 
with what should be.  This includes confusing the IETF's historical 
sense of what 'rough consensus' means, with what the draft suggests it 
mean. As a consequence, it sometimes merely explains what is preferred, 
rather than the problems with what is common practice.  (Sometimes.  At 
other times, it nicely makes its case.)  Also it sometimes does not lay 
out the context for its assertions.

      In effect, the draft is offering fundamental enhancements to 
existing IETF practice, rather than merely clarifying the ways things 
are "supposed to be".  By way of properly setting context, the draft 
should cite RFC 1603/3934, Working Group Guidelines and Procedures.  Its 
section 3.3 has text that describes long-standing community views on the 
meaning of rough consensus, which don't quite align with what the draft 
puts forward:

>                In general, the dominant view of the
>    working group shall prevail.  (However, it must be noted that
>    "dominance" is not to be determined on the basis of volume or
>    persistence, but rather a more general sense of agreement.) Consensus
>    can be determined by a show of hands, humming, or any other means on
>    which the WG agrees (by rough consensus, of course).  Note that 51%
>    of the working group does not qualify as "rough consensus" and 99% is
>    better than rough.  It is up to the Chair to determine if rough
>    consensus has been reached.

      The disparity between that document's text and what this draft 
describes should not be a problem for what the draft is recommending, 
but merely requires changing the explanatory tone of the draft and how 
it makes its case.

      On reflection, it's a bit odd that the draft hasn't already cited 
any formal IETF documentation on rough consensus and its use.

      The draft needs to distinguish historical practice from proposed 
enhancement.  It also needs to explicate things more clearly, as 
indicated in the detailed comments.  In some cases, discussion conflates 
two or more topics together and needs to separate them.

      The changes being suggested to the draft are substantial, but will 
not affect the essential points for which the draft is lobbying.

Detailed comments:

> Abstract
>
>    The IETF has had a long tradition of doing its technical work through
>    a consensus process, taking into account the different views among
>    IETF participants and coming to (at least rough) consensus on
>    technical matters.  In particular, the IETF is supposed not to be run
>    by a "majority rule" philosophy.  This is why we engage in rituals

    a "majority rule-> a "simple majority rule"

Majorities come in different forms or degrees and the fact that 'rough 
consensus' is often taken to mean 67% or 75%, as a rule of thumb can 
make this confusing.

 From what you've written, your basic point seems to be that 51% isn't 
enough; it's worth making that explicit.

However...

>    like "humming" instead of voting.  However, more and more of our
>    actions are now indistinguishable from voting, and quite often we are
>    letting the majority win the day, without consideration of minority

This is a more important sentence than what came before, IMO, but it 
doesn't have context.  What came before doesn't obviously have anything 
to do with paying attention to minority concerns.

If your above "majority rule" statement is meant to say that no 
mathematical majority of any degree is automatically sufficient -- which 
is what you've been stressing in our discussions and later in the 
document -- then you haven't made that point at all, yet.

For example, "voting" can be done with models that give quite a bit of 
power to minorities.  So, you need an explicit, direct, early statement 
that asserts the nature and importance of "rough consensus" factoring in 
minority views.

>    concerns.  This document is a collection of thoughts on what rough
>    consensus is, how we have gotten away from it, and the things we can
>    do in order to really achieve rough consensus.
...

> 1.  Introduction
>
>    Almost every IETF participant knows the aphorism from Dave Clark's
>    1992 plenary presentation [Clark] regarding how we make decisions in
>    the IETF:
>
>       We reject: kings, presidents and voting.
>
>       We believe in: rough consensus and running code.
>
>    That is, our credo is that we don't let a single individual make the

As the draft phrases it, that isn't really true.  We let individuals 
make decisions all the time.  What I believe the aphorism is saying -- 
and what reflects our community belief -- is:  We do not assign to any 
one individual the authority to /dictate/ a decision.  ("make" is too 
soft a word for this point, IMO.)

>    decisions, nor do we let the majority dictate decisions, nor do we

I believe the model being pursued in this draft -- of the nature and 
essential importance of properly dealing with minority views -- is 
hugely powerful and appropriate.  But I believe it has never been a 
prevalent view in the community, including back when the aphorism was 
seeking to summarize the community view.

The language I remember from the 90s primarily distinguished "strongly 
dominant majority" from 51% or 67% or 75%.  That is, it was about 
imprecise nature of the required clear and strong majority, rather than 
the moderating role of the minority.  Very occasionally -- but only 
occasionally -- we would use language that was along the lines of 
"strongly dominant view, with no vigorous and coherent opposition". That 
latter part probably should be seen as a crude version of what the draft 
is seeking to discuss, but I believe it was never consistently voiced or 
understood.

In other words, I think the role of minority views that the draft is 
putting forward is not a mere clarification and refinement to the 
existing IETF model, but rather is a significant enhancement.   So its 
premise about the potential importance of minority views does have some 
historical hooks in the IETF philosophy, but not as much as the draft 
seems to assert.

>    allow decisions to be made in a vacuum without practical experience.

Again, this statement goes beyond actual practice.  The language here 
prohibits decisions without experience and of course, the real IETF has 
always indulged in some decision-making based on theory that lacked 
practice.  We /prefer/ experience.  We /value/ the input it provides. 
But we are perfectly capable of making decision in a vacuum.

>    Instead, decisions are made by (more or less) consent of all

"Consent of all"?   Hell no.  Not even close, as your following text 
already note.

>    participants, and the actual products of engineering trump

Evidently, Pete hasn't been tracking the vigorous complaints about the 
SPF use of TXT records, arguing that such an offensive misuse of the DNS 
/must not/ be standardized, no matter how popular and integral the 
mechanism has become.  This represents a serious view that products 
absolutely do /not/ trump theoretical designs.  (Not surprisingly, I 
disagree with that view...)

>    theoretical designs.  Our ideal is full consensus, but we don't
>    require it: Full consensus would allow a single intransigent person

      Full consensus would allow
      ->
      Requiring full consensus gives each person a veto, by allowing

>
>
>
> Resnick                  Expires April 07, 2014                 [Page 2]
> 
> Internet-Draft                On Consensus                  October 2013
>
>
>    who simply keeps saying "No!" to stop the process cold.  We only
>    require rough consensus: If the chair of a working group determines

Again, this seems to conflate two points.  Both are extremely important. 
  Both need to be discussed.  But not intermixed.

One is that we do not require everyone to agree, as long as there is a 
strongly dominant group that do agree.

The other is that all concerns must be diligently considered, but do not 
have to be thoroughly resolved.  'Diligently considered' is an 
imposition on the entire group but it is mandatory.  'Not thoroughly 
resolved' is acceptable when the strongly dominant group agree that it 
is acceptable.

>    that a technical issue brought forward by an objector has been truly
>    considered by the working group, and the working group has made an
>    informed decision that the objection has been answered or is not
>    enough of a technical problem to prevent moving forward, the chair
>    can declare that there is rough consensus to go forward, the
>    objection notwithstanding.  To reinforce that we do not vote, we have

"To reinforce" switches from discussing the handling of minority 
concerns about voting.  At a minimum, break the paragraph.

>    also adopted the tradition of "humming": When (for example) the chair
>    of the working group wants to get a "sense of the room", instead of a
>    show of hands, the chair asks for each side to hum for or against a
>    question.

The draft has historically drawn a strong distinction between hands and 
hums.  It's an interesting distinction, and well might be useful, but I 
believe it has not been a universally-shared distinction in the 
community.  It essentially relies on a theoretical analysis of 
differences in psychological effects, but there's no objective basis for 
validating the one really is superior to the other.

In effect, the draft is asserting rationale and precision about audience 
measurement that haven't been present. Of course, there is an objective 
truth that humming does not permit counting, while hand-raising does. 
However humming has it's own psychologically biasing factors and they 
aren't trivial.

Now, perhaps the draft is seeking to lobby for use of humming, as 
preferred over hand-raising.  That's fine.  Cast it as such and argue 
for it.  It's relatively anonymous, it's a bit easier and quicker, it 
doesn't permit a false sense of measurement precision, ...

>    However, in recent years we have seen participants (and even some
>    folks in IETF leadership) who do not understand some of the
>    subtleties of consensus-based decision making.  Participants ask,
>    "Why are we bothering with this 'humming' thing?  Wouldn't a show of
>    hands be easier?  That way we could really see how many people want
>    one thing over another."  Chairs, many of whom have little experience
>    in leading large volunteer groups like those in the IETF, let alone
>    experience in how to gather consensus, are faced with factious
>    working groups with polarized viewpoints and long-running unresolved
>    issues that return again and again to the agenda.  More and more
>    frequently, people walk away from working groups, thinking that
>    "consensus" has created a document with horrible compromises to
>    satisfy everyone's pet peeve instead of doing "the right thing".
>    None of these things are indicators of a rough consensus process
>    being used, and are likely due to some basic misperceptions.

Is the issue their perception, or the fact that horrible compromises are 
actually being done?  From the text, I can't tell what point is intended.

>    This document attempts to explain some features of rough consensus,
>    explain what is not rough consensus, and suggest ways that we might
>    achieve rough consensus and judge it in the IETF.  Though this
>    document describes some behaviors of working groups and chairs, it
>    does so in broad brushstrokes and it is not intended to dictate

      it is not intended to dictate ->  it does not dictate

>    specific procedures.  It is intended to foster understanding of the
>    underlying principles of IETF consensus processes.
>
> 2.  Lack of disagreement is more important than agreement

The substance of this section has an important focus and makes useful 
comments.

However the title could be taken to cover such things as failure to gain 
affirmative support.  WGs often suffer from a lack of active involvement 
by more than a few participants.  In such cases, lack of disagreement is 
apathy, which is much worse than (active) agreement.

>    A working group comes to a technical question of whether to use
>    format A or format B for a particular data structure.  The chair
>    notices that a number of experienced people think format A is a good
>    choice.  The chair asks on the mailing, "Is everyone OK with format
>    A?"  Inevitably, a number of people object to format A for one or
>    another technical reason.  The chair then says, "It sounds like we
>    don't have consensus to use format A. Is everyone OK with format B?"
>    This time even more people object to format B, on different technical
>    grounds.  The chair, not having agreement on either format A or
>
>
>
> Resnick                  Expires April 07, 2014                 [Page 3]
> 
> Internet-Draft                On Consensus                  October 2013
>
>
>    format B, is left perplexed, thinking the working group has
>    deadlocked.
>
>    The problem that the chair got themselves into in the above case was
>    thinking that what they were searching for was agreement.  "After
>    all", thinks the chair, "consensus is a matter of getting everyone to
>    agree, so asking whether everyone agrees is what the chair ought to
>    do.  And if lots of people disagree, there's no consensus."  But
>    _determining_ consensus and _coming to_ consensus are different
>    things than _having_ consensus.  Consensus is not when everyone is

par break at start of sentence.  And possibly move this and the 
remaining sentences higher up, since they voice the essential point of 
the section.  In other words, try to start with the premise so that 
folks are oriented, and then explain it.

>    happy and agrees that the chosen solution is the best one.  Consensus
>    is when everyone is sufficiently satisfied with the chosen solution,
>    such that they no longer have specific objections to it.The
>    distinction might be a bit subtle, but it's important.    It is almost certain that any
>    time engineering choices need to be made, there will be options that
>    appeal to some people that are not appealing to some others.  The key
>    is to separate those choices that are simply unappealing from those
>    that are truly problematic.

Here's a suggestion for some re-casting to this separate paragraph, to 
help the reader bit, with the draft's logic and meaning:

      Engineering always involves a set of tradeoffs.  The process of 
gaining agreement about tradeoffs is the essence of a good rough 
consensus effort, and requires distinguishing the group debate and 
compromise needed to _come to_ consensus, the chair's process of 
_determining_ (or assessing) consensus, and the final point of the 
group's _having_ consensus on it choices.

...

>    However, there is another sense of "compromise" that involves
>    compromising between people, not engineering principles.  For
>    example, a minority of a group might object to a particular proposal,
>    and even after discussion still think the proposal is deeply
>    problematic, but decide that they don't have the energy to argue for
>    it and say, "Forget it, do what you want".  That surely can be called
>    a compromise, but a chair might mistakenly take this to mean that
>    they agree, and have therefore come to consensus.  But really all
>    that they've done is conceded; they've simply given up by trying to
>    appease the others.  That's not coming to consensus; there still
>    exists an outstanding unaddressed objection.  Even worse is the
>    "horse-trading" sort of compromise: "I object to your proposal for
>    such-and-so reasons.  You object to my proposal for this-and-that
>    reason.  Neither of us agree.  If you stop objecting to my proposal,
>    I'll stop objecting to your proposal and we'll put them both in."
>    That again results in an "agreement" of sorts, but it also results in
>    two outstanding unaddressed issues, again ignoring them for the sake
>    of expedience.  These sorts of "giving up" or "horse-trading"
>    compromises have no place in consensus decision making.  In each
>    case, a chair who looks for "agreement" might find it in these
>    examples because it appears that people have "agreed".  But again,
>    answering technical disagreements is what is needed to achieve
>    consensus, sometimes even when the people who stated the
>    disagreements no longer wish to discuss them.

This seems to argue that political compromise is never acceptable. 
That's not realistic.  Folks often can "live with" unresolved concerns. 
  The meta-issue is distinguishing disagreements about key points versus 
disagreement about minor ones.  Political compromise about a flat vs. 
hierarchical namespace is a rather different kettle of importance than 
between specifying a syntactic separator of comma vs. semicolon.  (No 
matter how much prettier I might think commas are, and superior for good 
visual presentation...)

The expedient of political compromise about more minor preferences is a 
huge benefit for making progress.  Compromising on major concerns, of 
course, is more likely to undermine utility or adoption.

>    Coming to consensus is when everyone comes to the conclusion that
>    either the objections are valid (and therefore making a change to
>    address the objection) or that the objection was not really a matter
>    of importance, but merely a matter of taste.  Of course, coming to
>    full consensus like that does not always happen.  That's why in the
>    IETF, we talk about "rough consensus".
>
> 3.  Rough consensus is achieved when all issues are addressed, but not
>     necessarily accommodated
...

>    So, an objector might say, "The proposal to go with protocol X is
>    much more complicated than going with protocol Y. Protocol Y is a
>    much more elegant and clean solution, which I can code much more
>    easily, and protocol X is a hack."  The working group might consider
>    this input, and someone might respond, "But we have a great deal of
>    code already written that is similar to protocol X. While I agree
>    that protocol Y is more elegant, the risks to interoperability with
>    an untested solution is not worth it compared to the advantages of
>    going with the well-understood protocol X." If the chair finds, in
>    their technical judgement, that the issue has truly been considered,
>    and that the vast majority of the working group has come to the
>    conclusion that the tradeoff is worth making, even in the face of
>    continued objection from the person(s) who raised the issue, the
>    chair can declare that the group has come to rough consensus.
>
>    Note that this portrays rough consensus as an "exception".  In one

I don't understand this first sentence.  How is it an exception?  Even 
with the following text, I don't see the basis for claiming that it 
might/would be seen as exceptional.

>    sense, it is: As a working group does its work and makes its choices,
>    it behaves as if it is striving toward full consensus and tries to
>    get all issues addressed to the satisfaction of everyone in the
>    group, even those who originally held objections.  It treats rough
>    consensus as a sort of "exception processing", to deal with cases
>    where the person objecting still feels strongly that their objection
>    is valid and must be accommodated.  But it is certainly true that
>    more often than not in the IETF, at least someone in the group will
>    be unsatisfied with a particular decision.  In that sense, rough
>    consensus might be closer to the norm than the "exception".  However,
>    when a participant says, "That's not my favorite solution, but I can
>    live with it; I'm satisfied that we've made a reasonable choice",
>    that participant is not in the "rough" part of a rough consensus; the
>    group actually reached consensus if that person is satisfied with the
>    outcome.  It's when the chair has to declare that an unsatisfied
>    person still has an open issue, but that the group has truly answered
>    the objection, that the consensus is only rough.
...

>    Now, a conclusion of having only rough consensus relies heavily on
>    the good judgement of the consensus caller.  The group must truly
>    consider and weigh an issue before the objection can be dismissed as
>    being "in the rough".  The chair of the working group in one of these

In spite of being appealing to use here, I think that the phrase "in the 
rough" is actually distracting, and possibly inappropriate:

      http://idioms.thefreedictionary.com/in+the+rough

      google:

         1. in a natural state; without decoration or other treatment.
            "a diamond in the rough"

         2. in difficulties. "even before the recession hit, the project
            was in the rough"

>    cases is going to have to decide that not only has the working group
>    taken the objection seriously, but that it has fully examined the
>    ramifications of not making a change to accommodate it, and that the
>    outcome does constitute a failure to meet the technical requirements
>    of the work.  In order to do this, the chair will need to have a good
>    idea of the purpose and architecture of the work being done and use
>    their own technical judgement to make sure that the solution meets
>    those requirements.  What can't happen is that the chair bases their
>    decision solely on hearing a large number of voices simply saying,
>    "The objection isn't valid."  That would simply be to take a vote.  A
>    valid justification needs to me made.

This begs the question of who determines that a justification is valid. 
  I'll suggest that it mostly should be the working group, with the 
chair as fall-back or safety valve, but not default.

By the way, the draft seems to think that it's always the chair who 
leads discussion and debate; it often/mostly isn't.  Rather it's 
typically the authors/editors, with a chair serving to moderate impasses.

>    Any finding of rough consensus needs, at some level, to provide a
>    reasoned explanation to the person(s) raising the issue of why their
>    concern is not going to be accommodated.  A good outcome is for the
>    objector to understand the decision taken and accept the outcome,
>    even though their particular issue is not being accommodated in the
>    final product.  Remember, if the objector feels that the issue is so

par break at Remember.

>    essential that it must be attended to, they always have the option to
>    file an appeal.  A technical error is always a valid basis for an
>    appeal.  The chair (or whoever is responsible to hear an appeal) may
>    determine that the issue was addressed and understood, but they also
>    have the freedom and the responsibility to say, "The group did not
>    take this technical issue into proper account" when appropriate.
>    Simply having a large majority of people agreeing to dismiss an
>    objection is not enough to claim there is rough consensus; the group
>    must have honestly considered the objection and evaluated that other
>    issues weighed sufficiently against it.  Failure to do that reasoning
>    and evaluating means that there is no true consensus.
>
> 4.  Humming should be the start of a conversation, not the end
...

>    So, why do we hum?  The first reason is pragmatic.  Quite often, a
>    chair is faced with a room full of people who seem to be
>    diametrically opposed on some choice facing the group.  In order to
>    find a starting place for the conversation, it is often useful for
>    the chair to ask for a hum to see if one of the choices already has a
>    stronger base of support than the other.  If choice "foo" has a
>    significant amount more support than choice "bar", it is likely
>    easier to start the discussion by saying, "OK, 'foo' seems to have
>    quite a bit of support.  Let's have the people that think 'foo' is a
>    bad idea come up and tell us why it is problematic."  At that point,

This section reads nicely and sounds quite good and very possibly might 
be something that we /should/ do.  However it has little to do with 
common IETF practice.  The simple reality is that hums are almost only 
used for indicating whether the group has a 'strongly dominant' 
preference.  Currently, it is rarely used as a means of deciding who to 
ask for objections.  Should we form a wg? Will folk work on the spec? Do 
folks prefer alternative A?  Is mechanism X workable here?

Even better is that I think the draft has buried it's more important 
point, namely that the /form of the question/ asked when seeking a hum 
is the really hard part of the effort, and the most important part. 
Some forms invite constructive discussion and some close it off.  Some 
are fair and balanced (...) and some are highly biasing.

>    the group can start going through the issues and see if any of them
>    are showstoppers.  It may turn out that one of the objections is
>    instantly recognized by the entire group as a fatal flaw in "foo" and
>    the group will then turn to a discussion of the merits (and demerits)
>    of "bar" instead.  All that the hum does is give the chair a starting
>    point: There were likely to be less objections to "foo" than to "bar"
>    at the beginning of the discussion, so starting with whatever got the
>    most hums can shorten the discussion.
...

>    But couldn't the above could have been done with a show of hands

This paragraph goes back to the hum-vs-hand methodology issue.  It's 
important, but it's out of place here, since this section is about the 
/role/ of a hum, not its method.  Discuss methodology separately, 
possibly up where it is first introduced.

>    instead of a hum?  Sure, but there are more formal reasons for using
>    a hum instead of a show of hands: Another reason we hum is because it
>    actually gives the chair the opportunity to take the temperature of
>    the room.  A smaller bunch of loud hums for choice A and a larger
>    number of non-committal hums for choice B might indicate that some
>    people believe that there are serious problems with choice B, albeit
>    the more popular by sheer number of people.  The chair might decide
>    that starting with choice A and getting objections to it is the
>    easier path forward and more likely to result in consensus in the
>    end.  Remember, coming to consensus is a matter of eliminating
>    disagreements, so the chair wants to choose the path that gets to the
>    least objections fastest.  A bunch of people who are not strongly

"the chair wants to choose the path that gets to the least objections 
fastest" sounds like an extremely appealing expedient, by virtue of 
being almost mechanical, but numbers often don't matter.  A single 
objection that observes (and substantiates) that a choice will not or 
cannot be used in the real world will outweigh a much larger number of 
trivial objections.

Perhaps merely qualifying the statement a bit will suffice:  the chair 
wants to choose the path that gets to the least number of significant 
objections fastest".  Hmmm.

>    committed to B might have no real technical objection to A, even
>    though it is not their first preference.  There is always a chance
>    that this could be misleading, because some people are more willing
>    to hum loudly than others (just by dint of personality), but keep in
>    mind that taking the hum is to figure out how to start the
>    conversation.  The chair could always be surprised because the hum
>    turns out to be unanimous.  Otherwise, the hum begins the discussion,
>    it doesn't end it.
>
>    Of course, a chair could get the temperature of the room by doing a
>    show of hands too, and knowing who specifically feels one way or
>    another can help a good chair guide the subsequent conversation.

And possibly place the minority under group pressure.

...

>    This also points to another misuse of the hum: If the chair is
>    already convinced that the group has come to consensus, there is no
>    reason to take a hum.  In fact, taking the hum only serves to
>    discourage those who might be in the minority from voicing their
>    concerns to the group in the face of a large majority who want to
>    move forward.  The right thing for the chair to do if they already
>    sense consensus is to say, "It sounds to me like we have consensus
>    for choice A. Does anybody have any concerns or objections to going
>    with A?"  This allows folks to bring up issues to the group that the
>    chair might have mistakenly missed without having them feel that the
>    majority has "already spoken".

+1

...

>
> 5.  Consensus is the path, not the destination
...

>    We often hear chairs say that they are making a "consensus call".
>    Sometimes, they simply mean they are making a call _of_ the
>    consensus; that is, they are declaring the consensus that has, in
>    their view, been reached when the discussion has reached an end.
>    That's a fine thing and what chairs are supposed to do: They are
>    "calling" the consensus.  Sometimes, when a chair says that they are

Might be worth tossing in the word "confirm" or "confirming existing 
consensus" here, to leave no doubt about what "calling" means.

...

>
> 6.  One hundred people for and five people against might not be rough
>     consensus
...

>
>    So in a large working group with over 100 active participants and
>    broad agreement to go forward with a particular protocol, if a few
>    participants say, "This protocol is going to cause congestion on the
>    network, and it has no mechanism to back off when congestion occurs;
>    we object to going forward without such a mechanism in place", and
>    the objection is met with silence on the mailing list, there is no

The problem here is who is the authority to determine that the objection 
needs to hold sway.  It's easy to lay this on the chair, but that makes 
the mechanisms fragile; it's too easy for one person to get this sort of 
thing wrong.

>    consensus.  Even if the working group chair makes a working group
>    last call on the document, and 100 people actively reply and say,
>    "This document is ready to go forward", if the open issue hasn't been
>    addressed, there's still no consensus.  It's the existence of the
>    unaddressed open issue, not the number of people, which is
>    determinative in judging consensus.  As discussed earlier, you can
>    have rough consensus with issues that have been purposely dismissed,
>    but not ones that have been ignored.
...

>
> 7.  Five people for and one hundred people against might still be rough
>     consensus
...

>    Now the worst case scenario happens.  The objector, still unhappy
>    that their preferred solution was not chosen, recruits one hundred
>    people, maybe a few who were silent participants in the working group
>    already, but mostly people who work at the same company as the
>    objector who never participated before.  The objector gets them all
>    to post a message to the list saying, "I believe we should go with
>    the new elegant algorithm in section Z instead of the current one.
>    It is more elegant, and works better on our hardware."  The chair
>    sees these dozens of messages coming in and posts a query to each of
>    them: "We discussed this on the list, and we seemed to have consensus
>    that, given the inherent risk of a new algorithm, and the widespread
>    deployment of this current one, it's better to stick with the current
>    one.  Do you have further information that indicates something
>    different?"  And in reply the chair gets utter silence.  These
>    posters to the list (say some of whom were from the company sales and
>    marketing department) thought that they were simply voting and have
>    no answer to give.  At that point, it is within bounds for the chair
>    to say, "We have objections, but the objections have been
>    sufficiently answered, and the objectors seem uninterested in
>    participating in the discussion.  Albeit rough in the extreme, there
>    is rough consensus to go with the current solution."

There's an interesting methodological point buried quite deep in this 
example, and probably worth surfacing sooner and more directly:

      Besides being careful about how the question for a consensus check 
is phrased, it is sometimes advisable to seek to discover what the 
responses mean.

In the example here, the question is whether those responding were 
sufficiently well-informed to be able to justify their position.

>    There is no doubt that this is the degenerate case and a clear
>    indication of something pathological.  But this is precisely what
>    rough consensus is designed to guard against: vote stuffing.  In the

Sorry, no.  It's not 'designed' for that at all, at least not in typical 
practice in the IETF.  It probably /should/ be, and the draft does a 
pretty good job of explaining how it /can/ be, but there is nothing in 
the "design" of the IETF's consensus process that has a history of 
mitigating against the problem the draft discusses here, and the problem 
isn't mentioned in formal IETF documents.

>    presence of an objection, the chair can use their technical judgement
>    to decide that the objection has been answered by the group and that
>    rough consensus overrides the objection.  Now, the case described
>    here is probably the hardest call for the chair to make (how many of
>    us are willing to make the call that the vast majority of people in
>    the room are simply stonewalling, not trying to come to consensus?),
>    and if appealed it would be incredibly difficult for the appeals body
>    to sort out.  Indeed, it is likely that if a working group got this
>    dysfunctional, it would put the whole concept of coming to rough
>    consensus at risk.  But still, the correct outcome in this case is to
>    look at the very weak signal against the huge background noise in
>    order to find the rough consensus.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 7 Oct 2013 11:34:57 +1100

On 07/10/2013, at 11:03 AM, Dave Crocker <[email-masked]> wrote:

> In spite of being appealing to use here, I think that the phrase "in the rough" is actually distracting, and possibly inappropriate:
> 

I think he's using it in the sense that it's used on a golf course...

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Melinda Shore <[email-masked]>
Date: Sun, 06 Oct 2013 16:46:49 -0800

On 10/6/13 4:34 PM, Mark Nottingham wrote:
>> 1. in a natural state; without decoration or other treatment. "a
>> diamond in the rough"
>> 2. in difficulties. "even before the recession hit, the project was
>> in the rough"

> I think he's using it in the sense that it's used on a golf

I think that's pretty clearly the case, although it might
be confusing in juxtaposition with "rough consensus," where
"rough" means "approximate."  I'm not sure it's a big deal
and I don't want to lawyer the document to death.  We could
spin in circles for years trying to nail down "current practice."
Part of the problem that this document addresses is that some
current practice is anathema to consensus processes, not
because IETF process is wrong and needs modification but
because some chairs, I* folk, etc., aren't very skilled at
managing consensus.

Melinda

---

From: "Dearlove, Christopher (UK)" <[email-masked]>
Date: Mon, 7 Oct 2013 10:13:03 +0000

[email-masked]

> enough; it's worth making that explicit.

To add to the confusion, and to emphasise the point about making clear, British and American English differ here. If three proposals (not the most common case, I agree, but it can happen) have 45%, 35% and 20% of the votes, the first of these has a majority, sometimes emphasised as simple majority, in British English. (We do not - to our loss - use the word plurality. Just 51% is given the strong term absolute majority.) I haven't checked the context here, but saying not just a simple majority might suggest to a British English user that 51% is enough.

-- 
Christopher Dearlove
Senior Principal Engineer, Communications Group
Communications, Networks and Image Analysis Capability
BAE Systems Advanced Technology Centre
West Hanningfield Road, Great Baddow, Chelmsford, CM2 8HN, UK
Tel: +44 1245 242194 |  Fax: +44 1245 242124
[email-masked] | http://www.baesystems.com

BAE Systems (Operations) Limited
Registered Office: Warwick House, PO Box 87, Farnborough Aerospace Centre, Farnborough, Hants, GU14 6YU, UK
Registered in England & Wales No: 1996687

********************************************************************
This email and any attachments are confidential to the intended
recipient and may also be privileged. If you are not the intended
recipient please delete it from your system and notify the sender.
You should not copy it or use it for any purpose nor disclose or
distribute its contents to any other person.
********************************************************************

---

From: Jaap Akkerhuis <[email-masked]>
Date: Mon, 07 Oct 2013 21:12:34 +0200

To add to the confusion, and to emphasise the point about making
    clear, British and American English differ here.  If three
    proposals (not the most common case, I agree, but it can happen)
    have 45%, 35% and 20% of the votes, the first of these has a
    majority, sometimes emphasised as simple majority, in British
    English. (We do not - to our loss - use the word plurality.
    Just 51% is given the strong term absolute majority.) I haven't
    checked the context here, but saying not just a simple majority
    might suggest to a British English user that 51% is enough.

Let's avoid any dependency on either idioms. Given that the bulk
of the intended audience might not be aware of either specific idiom
but have english as their second of nth language, it is probably
better to explain what is meant.

	jaap

---

From: [email-masked] (Martin Rex)
Date: Mon, 7 Oct 2013 22:56:29 +0200 (CEST)

Dearlove, Christopher (UK) wrote:
> To add to the confusion, and to emphasise the point about making clear,
> British and American English differ here. If three proposals (not the
> most common case, I agree, but it can happen) have 45%, 35% and 20%
> of the votes, the first of these has a majority, sometimes emphasised
> as simple majority, in British English. (We do not - to our loss - use
> the word plurality. Just 51% is given the strong term absolute majority.)
> I haven't checked the context here, but saying not just a simple
> majority might suggest to a British English user that 51% is enough.

Voting as is done in elections for political parties is often going
to produce a political result, a personal preference of a few rather
than an engineering solution that adequately addresses the concerns
of the community at large.

What we could do in the IETF, is not just trying to pick the lesser evil,
but rather use the _engineering_ skills to modify and/or merge proposals
to increase the number of folks that support the result and reduce
the amount of folks that object the result.

With your example of three competing proposals: A, B and C, and by couting
"votes" the WG chair determine support of 45%, 35% and 20% respectively,
does this mean that A has signficant support?   Not at all.  It could be
that the folks who voted for B and C did so because they are both strongly
opposed to A.

A WG chair who wants to be neutral on the decision should probably not
just call:
   which of the three proposals do you prefer:  A, B or C

and perform political inferences on the result, but rather
_ask_ the engineers direclty:

   if we were to select A, would you "support it, are neutral, are opposed"
     if you're either neutral or opposed to A, what change(s) to A would
     make you supportive for A?

   if we were to select B, would you "support it, are neutral, are opposed"
     if you're either neutral or opposed to A, what change(s) to A would
     make you supportive for B?

   if we were to select C, would you "support it, are neutral, are opposed"
     if you're either neutral or opposed to A, what change(s) to A would
     make you supportive for C?

I've seen a few "WG consensus calls" which appeared somewhat skewed/biased
to me towards exclusing specific outcomes.  I do _not_ know whether this
was causes by malicous intent or just accidental / not sufficiently well
thought out.  I'm OK with a leadership decision, but leadership decisions
should not be exerted early in the process by preventing certain questions
to get asked at all.  

How well the process works can be seen by how objections are resolved.
Are objections handled by spin doctors, or by engineers that are open
to improvements of their work.

-Martin

---

From: "Dearlove, Christopher (UK)" <[email-masked]>
Date: Tue, 8 Oct 2013 10:27:20 +0000

I wasn't making any suggestion about what we should be doing. My sole point was that as the language differs, we should be aware of that and word accordingly, i.e. not use phrases like "simple majority" to mean 51%, as it may not.

-----Original Message-----
From: Martin Rex [mailto:[email-masked]] 
Sent: 07 October 2013 21:56
To: Dearlove, Christopher (UK)
Cc: [email-masked]; Pete Resnick; IETF Discussion
Subject: Re: Review of: draft-resnick-on-consensus-05

----------------------! WARNING ! ----------------------
This message originates from outside our organisation,
either from an external partner or from the internet.
Keep this in mind if you answer this message.
Follow the 'Report Suspicious Emails' link on IT matters
for instructions on reporting suspicious email messages.
--------------------------------------------------------

-Martin

---

From: Yoav Nir <[email-masked]>
Date: Tue, 8 Oct 2013 10:52:09 +0000

>     make you supportive for C?

I think in this case the WG chair should do something different. A better question would be:

  Which is worse? To decide on *any* of these 3 proposals, or to keep debating this for another few months?

At that point you might get something like
    - Just choose any of them - 90%
    - No, we need to choose the right one - 10%

But percentages don't really matter. You get the people who are in the 10% (1-3 in most working groups) and get them to the mic or mailing list and ask them what is so terrible about proposal A (or B or C, whichever one or two they really object to) that it's better to delay the process by some extra months. That gets the objections that are really worth discussing. 

If you can get those objections ironed out so that there's consensus (rough or not) for the assertion that "Any of them is better than not proceeding", the selection can proceed by any method that seems fair: majority, coin toss, 3-way Rechambeau ([1]), shortest document.

Yoav

[1] http://tools.ietf.org/html/draft-harkins-rochambeau-02

---

From: Pete Resnick <[email-masked]>
Date: Thu, 10 Oct 2013 17:48:09 -0500

Finally back to this original review.

On 10/6/13 7:03 PM, Dave Crocker wrote:
>      In terms of philosophy and desirable practice, the draft 
> discusses an extremely appealing model and generally explains its 
> nature and practice well.  However the draft tends to confuse what is 
> (or has been) with what should be.  This includes confusing the IETF's 
> historical sense of what 'rough consensus' means, with what the draft 
> suggests it mean. As a consequence, it sometimes merely explains what 
> is preferred, rather than the problems with what is common practice.  
> (Sometimes.  At other times, it nicely makes its case.)  Also it 
> sometimes does not lay out the context for its assertions.
>
> are "supposed to be".

I think this is a fair criticism and I will do a grand editing pass to 
try to tease the two apart: Where the draft is actually describing real 
historical practices vs. where it is describing a theory that does not 
comport with our practices. I know of several places beyond what you've 
mentioned that make the same conflation, and I think it's well worth 
making it clear. There are some instances where "what is (or has been)" 
is actually mushy: That is, people haven't really thought about it in 
depth. In those cases, I've definitely done a bit of saying what I think 
folks would mean if they were actually asked about it. (See more on this 
point in the discussion of the intro.) But with all of these, I think I 
should be more explicit about which is which.

> By way of properly setting context, the draft should cite RFC 1603/3934

You mean 2418. But yes, I think you're right.

> Working Group Guidelines and Procedures.  Its section 3.3 has text 
> that describes long-standing community views on the meaning of rough 
> consensus, which don't quite align with what the draft puts forward:
>

Yup. I think this document has to cite that statement in 2418, and it 
has to disagree with it, at least to the extent of claiming it is an 
over-simplification. I had intended to do so and simply forgot. That 
51%/99% statement actually leads down the path that the document wants 
to discourage.

> [...]
>      The changes being suggested to the draft are substantial, but 
> will not affect the essential points for which the draft is lobbying.

I agree, though I think the changes are actually pretty straightforward. 
Maybe substantial in number of actual edits, but not in essential content.

> enough; it's worth making that explicit.

Setting aside the possible confusion in British usage, I don't think you 
understood my intention here. And since you and Christopher and Jaap 
ended up in the same place, I'm happy to say that it's my fault for not 
being clear enough.

I intended that the abstract simply summarize the document's *purpose*, 
not its conclusions. I was using it like the abstracts that appear on 
scientific papers. Of course, the abstracts on our technical 
specifications usually give away the punch line, which is what we're 
used to seeing. I'll have a go at an edit, at least to clarify what I'm 
doing.

> power to minorities.  So, you need an explicit, direct, early 
> statement that asserts the nature and importance of "rough consensus" 
> factoring in minority views.

This is an abstract, not a place to put early statements. I think the 
kind of statement you're looking for goes in the intro, which I 
hopefully address below.

>> [...]

Fair enough.

> seeking to summarize the community view.

Yes, I think this all goes to your earlier point of cleanly 
distinguishing between historical view and new theory.

> imprecise nature of the required clear and strong majority, rather 
> than the moderating role of the minority.  Very occasionally -- but 
> only occasionally -- we would use language that was along the lines of 
> "strongly dominant view, with no vigorous and coherent opposition".

When I started having conversations about the draft with folks, 
especially people who've been around quite a while, I was struck by the 
number of times the discussion started with, "No, that's not what we 
meant", but ended in, "Oh, yes, in those extreme cases, that's exactly 
what we meant." That is, I think the "51% is not enough, but 99% is more 
than enough" view was certainly in the front of people's minds, but when 
probed about the implications of that, almost everyone came to the same 
conclusion as the document: 51/99 is often a good rule of thumb if one 
wanted to guess about the consensus based on numbers of people, but it's 
still reasonable to call certain outcomes (like the ones described in 
the document) "rough consensus" even if the views of a huge majority 
might not win the day.

But it is certainly fair to say that in the 90s, few of us would have 
independently articulated the idea that a 90% majority might not be 
rough consensus.

> That latter part probably should be seen as a crude version of what 
> the draft is seeking to discuss, but I believe it was never 
> consistently voiced or understood.
>
> premise about the potential importance of minority views does have 
> some historical hooks in the IETF philosophy, but not as much as the 
> draft seems to assert.

Yep. I hope I can edit to make that clearer.

> prohibits decisions without experience and of course, the real IETF 
> has always indulged in some decision-making based on theory that 
> lacked practice.  We /prefer/ experience.  We /value/ the input it 
> provides. But we are perfectly capable of making decision in a vacuum.

Remember that I'm saying what the "credo" says, not how tightly we 
adhere to it. How well we stick to it is talked about later. I think we 
do take as a principle that we want to base decisions on running code. 
We certainly don't always do so.

>>    theoretical designs.

Again, here it is stating what the credo says, not the specifics of what 
we do (or even mean to do) in practice.

[Compare: Person A: "The US Constitution says, 'Congress shall make no 
law...abridging the freedom of speech.' Our credo is that the government 
will not limit speech." Person B: "But that's not true. The government 
limits all sorts of speech." Person A: "Never said it didn't. I said 
what the credo was. I will talk at length in a moment about how the 
government does and does not adhere to the credo, and how some of that 
is fine and still inline with our overall principles, and how some of it 
is problematic, and how they can do things that are less problematic."]

This is the Introduction. It's setting up the bigger picture. It's not 
the place to say, "The details are complicated"; of course they are.

>>    Our ideal is full consensus, but we don't

Right. I'm inclined to skip the "veto" language, but I will definitely edit.

> Again, this seems to conflate two points.  Both are extremely 
> important.  Both need to be discussed.  But not intermixed.
>
> The other is that all concerns must be diligently considered, but do 
> not have to be thoroughly resolved.  'Diligently considered' is an 

All true, but that discussion is not appropriate to the Introduction. 
The distinction between the two points is discussed at length later.

> concerns about voting.  At a minimum, break the paragraph.

Done.

> The draft has historically drawn a strong distinction between hands 
> and hums.  It's an interesting distinction, and well might be useful, 
> but I believe it has not been a universally-shared distinction in the 
> differences in psychological effects, but there's no objective basis 
> for validating the one really is superior to the other.

And the draft says as much. See section 4, paragraphs 3-5. This is 
discussed at length. I don't see what's missing from here.

>>    More and more
> Is the issue their perception, or the fact that horrible compromises 
> are actually being done?  From the text, I can't tell what point is 
> intended.

Here, it's only about the perception. I talk about the real problems 
with compromises at the end of section 2.

>      it is not intended to dictate ->  it does not dictate

Will fix.

> However the title could be taken to cover such things as failure to 
> gain affirmative support.

I don't intend any of my section titles to be complete and precise. If 
folks really don't like that, I can go about changing them, but I'd 
really rather not.

>> ... Consensus is not when everyone is
>
> par break at start of sentence.

Sure.

> And possibly move this and the remaining sentences higher up, since 
> they voice the essential point of the section.  In other words, try to 
> start with the premise so that folks are oriented, and then explain it.

I know this is apropos of a conversation we're having offline, but I do 
believe that people can get through the example of the problem before we 
tell them what the solution is.

> group's _having_ consensus on it choices.

Did you intend for that to replace something or just to insert it? I 
think it works well inserted, but if you meant something to be deleted, 
please clarify.

>>    compromising between people, not engineering principles.
>
> That's not realistic.  Folks often can "live with" unresolved 
> concerns.  The meta-issue is distinguishing disagreements about key 
> points versus disagreement about minor ones.  Political compromise 
> about a flat vs. hierarchical namespace is a rather different kettle 
> of importance than between specifying a syntactic separator of comma 
> vs. semicolon.  (No matter how much prettier I might think commas are, 
> and superior for good visual presentation...)

So, the "live with" case is mentioned above, but I think it needs 
clarification in the previous paragraph. But I *do* want to argue that 
political compromise *is* never acceptable. It's one thing for me to 
conclude that I can live with semicolons over commas even though my 
preference is for commas (i.e., that it is a reasonable, though not 
preferred, engineering choice); that's just fine. It's quite different 
for me to conclude that the rest of these people are idiots for not 
choosing commas, but I'm going to put up with the semicolons because I 
want to make people happy so they'll do a hierarchical namespace, or 
because I've simply given up on trying to make this a good document. If 
it's a serious issue, giving up or trading is bad. If it's an "I can 
live with it" issue, then you're simply agreeing that the choice is 
acceptable.

And I'm not saying that the bad sort of compromise won't ever happen (we 
are humans, after all), or that there are no borderline cases (Is this 
me "living with it" or capitulating?), but I do want to say that 
participants have to decide whether they're making a reasonable 
engineering compromise or they are capitulating (or horse trading), and 
not do it if it's the latter.

And all that said, I need to clarify this.

>>    Note that this portrays rough consensus as an "exception".
>

Maybe I need a different word. I didn't mean "exceptional" in the sense 
of "extraordinary" or "atypical". I simply meant it in the "exception 
handling" sense: We work toward full agreement, but if there's still 
disagreement, we undertake an alternative path. I'm open to suggestions 
for text.

> In spite of being appealing to use here, I think that the phrase "in 
> the rough" is actually distracting

As I mentioned earlier, I think it is worth defining as the golf term, 
since it is used quite frequently among IETFers nowadays.

> This begs the question of who determines that a justification is 
> valid.  I'll suggest that it mostly should be the working group, with 
> the chair as fall-back or safety valve, but not default.

Remember, this is precisely the case where the working group claims to 
have considered the objection and found it to be wanting, but the 
objector still insists that the objection is valid. The rest of the 
working group can make the case that it has properly considered the 
objection, but it can't make the final call on whether it has; only the 
chair can do that.

> typically the authors/editors, with a chair serving to moderate impasses.

During my large edit, I'll have a look at this. My intention wasn't to 
say that the chair *leads* discussion. Indeed, beyond authors/editors, 
quite often it is someone else in the group who takes up the mantle of 
analyzing the discussion and proposing paths forward. But the chair in 
the end is responsible for making the final call as to the consensus of 
the group.

>>    A good outcome is for the

Sure.

>>    So, why do we hum?
>
> This section reads nicely and sounds quite good and very possibly 
> might be something that we /should/ do.  However it has little to do 
> with common IETF practice.

Yep. This goes back to your main critique. So really the above should 
say, "So why should we hum?", or something to that effect. Wilco.

> are fair and balanced (...) and some are highly biasing.

I'll try to expand on that.

> possibly up where it is first introduced.

I disagree with you on this one. The previous discussion was in the 
Introduction, which says, "Something's screwy here. What's up with 
that?" Here, I am trying to say, "Here's what we ought to be doing. Hums 
are good. Hands work too, but have their weaknesses." I think it works 
together well.

> objections fastest".  Hmmm.

Yeah, this gets talked two paragraphs previously, but I think it's fair 
to reiterate here. I think the better fix is to add to the below sentence:

>>    There is always a chance
>>    conversation.

Add after the parenthetical: "or that one of the quieter hums actually 
turn outs to be a showstopper that makes the original choice impossible,"

> And possibly place the minority under group pressure.

Sometimes a hum can do exactly the same.

> consensus" here, to leave no doubt about what "calling" means.

Yeah, or use the word "stating" or "expressing". I'll work on that.

> The problem here is who is the authority to determine that the 
> objection needs to hold sway.  It's easy to lay this on the chair, but 
> that makes the mechanisms fragile; it's too easy for one person to get 
> this sort of thing wrong.

The case we're talking about here is an objection being met with 
silence. In that case, the presumption (i.e., not the conclusion) is, 
"Real objection; needs to be dealt with." The only way that the chair 
can "get this sort of thing wrong" is to at that point silently decide 
that it's not an objection worth addressing. The chair can say to the 
group, "I've seen this objection, but I don't think it's valid. Can 
someone confirm my view that it's not valid, or can the objector or 
someone else please explain why it is?" But this is only about whether a 
completely dropped-on-the-floor objection is OK. It's not.

Did I miss your point?

>      Besides being careful about how the question for a consensus 
> check is phrased, it is sometimes advisable to seek to discover what 
> the responses mean.
>

I'll take an edit pass. Yes, that is exactly what I was trying to get at 
(among other things).

> Sorry, no.  It's not 'designed' for that at all, at least not in 
> typical practice in the IETF.  It probably /should/ be, and the draft 
> does a pretty good job of explaining how it /can/ be, but there is 
> nothing in the "design" of the IETF's consensus process that has a 
> history of mitigating against the problem the draft discusses here, 
> and the problem isn't mentioned in formal IETF documents.

Yes, fair criticism. "But this is precisely what rough consensus is 
ideally suited to guard against:"

Excellent comments. Thanks.

On to the next set....

pr

-- 
Pete Resnick<http://www.qualcomm.com/~presnick/>
Qualcomm Technologies, Inc. - +1 (858)651-4478

---

From: t.p. <[email-masked]>
Date: Fri, 11 Oct 2013 10:53:03 +0100

A minor point inline, rest snipped

Tom Petch

----- Original Message -----
From: "Pete Resnick" <[email-masked]>
To: <[email-masked]>
Cc: "IETF Discussion" <[email-masked]>
Sent: Thursday, October 10, 2013 11:48 PM

> On 10/6/13 7:03 PM, Dave Crocker wrote:
<snip>

> since it is used quite frequently among IETFers nowadays.

I just saw this term used by an AD in a DISCUSS for the first time that
I can recall, and I have not a clue what he means (yes, I understand
golf but do not see the relevance).  My guess is that the AD is flying a
kite, but would not expect that to be any more widely understood:-(

Tom Petch

<snip>