Subject: RE: Now there seems to be lack of communicaiton here...

From: John C Klensin <[email-masked]>
Date: Fri, 01 Sep 2006 11:44:55 -0400

--On Thursday, 31 August, 2006 17:30 -0400 Eastlake III
Donald-LDE008 <[email-masked]> wrote:

> John,
> 
> If the selection method is random, it makes no difference
> whatsoever how the list of nomcom volunteers is ordered. It
> only matters that the numbered list become fixed and be posted
> before the selection information is available. Alphabetic or
> the order they volunteered or any other order is perfectly
> fine.

Agreed, except that an alphabetic sort is not, by any stretch of
the imagination, random. Phillip's suggestion of using a
well-established hash that is known to give good distributions
would work; there are many other methods that would work.  But a
number of observable distribution issues make an alphabetic sort
on names unacceptably random if one is then going to use the
ordering for successive samplings/selections.

I want to stress that, given this mess has occurred, I would
find just about anything the Nomcom Chair decides to do
acceptable although I do not approve of his consulting the IETF
Chair (or IAB Chair) in the matter.  But, if we are going to
make sure this problem does not occur in the future, I think we
should make the procedure as gaming-proof as possible.  That, to
me, implies two requirements going forward:

(1) We get strong randomization of the selection process

(2) We do not redraw the entire Nomcom pool and _never_ do so
after anyone who has discretion has had an opportunity to see
the initial list of Nomcom members.  If someone is selected (or
volunteers) and then determined to be ineligible, the people who
have already been selected by the mechanism specified stay
selected.  Anything else just has a bad odor whether actual
improprieties are suspected or not.

In addition, I am extremely concerned by hints on this list that
the Secretariat's checking procedures ruled people ineligible to
volunteer who had, in fact, attended the correct number of
meetings.  That, it seems to me, is a much larger threat to the
integrity of the Nomcom model and perceptions of trust in it
than any issue that impacts a single volunteer or even, within
broad limits, the randomization and Nomcom member selection
processes.

      john

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "Eastlake III Donald-LDE008" <[email-masked]>
Date: Fri, 1 Sep 2006 16:05:08 -0400

See below at @@@

-----Original Message-----
From: John C Klensin [mailto:[email-masked]] 
Sent: Friday, September 01, 2006 11:45 AM
To: Eastlake III Donald-LDE008; IETF-Discussion
Subject: RE: Now there seems to be lack of communicaiton here...

--On Thursday, 31 August, 2006 17:30 -0400 Eastlake III
Donald-LDE008 <[email-masked]> wrote:

> John,
> 
> If the selection method is random, it makes no difference whatsoever 
> how the list of nomcom volunteers is ordered. It only matters that the

> numbered list become fixed and be posted before the selection 
> information is available. Alphabetic or the order they volunteered or 
> any other order is perfectly fine.

Agreed, except that an alphabetic sort is not, by any stretch of the
imagination, random. Phillip's suggestion of using a well-established
hash that is known to give good distributions would work; there are many
other methods that would work.  But a number of observable distribution
issues make an alphabetic sort on names unacceptably random if one is
then going to use the ordering for successive samplings/selections.

@@@ I'm not sure why you agree with me and then say the opposite. If it
doesn't matter what the ordering of the nomcom volunteer pool is then it
doesn't matter what the ordering of the nomcom volunteer pool is, and,
in particular, it doesn't matter how random or "biased" it is. The RFC
3797 algorithm takes random inputs and uses them to make successive
uniformly distributed non-repeating selections from a range of integers.
The sole purpose of the published ordered volunteer pool list is to
provide a pre-announced mapping from those integers to the people who
volunteered. If it mattered what that mapping was, then you are not
making uniformly distributed random selections.

@@@ Both RFC 3797 and Phil Hallam-Baker's suggestion use a
well-established hash. (I happen to personally not like the details of
Phil's specific suggestion because of questions related to email address
canonicalization and because it would require publishing email addresses
for all nomcom volunteers.)

I want to stress that, given this mess has occurred, I would find just
about anything the Nomcom Chair decides to do acceptable although I do
not approve of his consulting the IETF Chair (or IAB Chair) in the
matter.  But, if we are going to make sure this problem does not occur
in the future, I think we should make the procedure as gaming-proof as
possible.  That, to me, implies two requirements going forward:

@@@ We already have this. See RFC 3797.

(2) We do not redraw the entire Nomcom pool and _never_ do so after
anyone who has discretion has had an opportunity to see the initial list
of Nomcom members.  If someone is selected (or
volunteers) and then determined to be ineligible, the people who have
already been selected by the mechanism specified stay selected.
Anything else just has a bad odor whether actual improprieties are
suspected or not.

@@@ It may be possible, with sufficient care, to make vanishingly small
the chance that a nomcom chair discretionary decision would be needed
for this aspect of nomcom selection. But I am not sure that, in the real
world, it is possible to do this for all aspects of nomcom selection.
See Section 5.2 of RFC 3797.

In addition, I am extremely concerned by hints on this list that the
Secretariat's checking procedures ruled people ineligible to volunteer
who had, in fact, attended the correct number of meetings.  That, it
seems to me, is a much larger threat to the integrity of the Nomcom
model and perceptions of trust in it than any issue that impacts a
single volunteer or even, within broad limits, the randomization and
Nomcom member selection processes.

@@@ Well, you are welcome to be as concerned as you like, but we live in
an imperfect world. As far as I know, there are usually a few disputes
about the volunteer list, usually in connection with people whose
eligibility the Secretariat doubts. Sometimes they are determined to be
correct and sometimes the Secretariat is determined to be right.
Sometimes the volunteer is confused about the eligibility requirements
or about what their attendance actually was, or has variant versions of
their name, or has changed their name, or ... But this sort of thing
doesn't effect very many people on the volunteer list and is almost
always resolved between the volunteer and the Secretariat before the
list is published.

      john

@@@ Thanks,
@@@ Donald

---

From: Ned Freed <[email-masked]>
Date: Fri, 01 Sep 2006 15:07:50 -0700 (PDT)

> making uniformly distributed random selections.

John, I have to agree with Donald here. Nothing anyone has said has indicated
that there is a flaw in the algorithmic part of the selection process. If
something ain't broke don't fix it.

Indeed, not only is the algorithmic part of the selection process fine, the
non-algorithmic parts were well thought out and various contingencies were
covered, including the specific one where disqualified people found their way
onto the final qualified candidate list. The problem is quite simply that the
nomcom chair failed to follow these recommendations.

It also appears we have dodged the bullet this time around. But this was mostly
a matter of luck. We may not be so lucky the next time, and that means we need
to act now to fix the problem that the nomcom chair has the dangerous and
unnecessary ability to reset the selection process in cases where no reset
should be allowed. (I have already described why there is real danger here -
see my previous posting for specifics.) This absolutely must be attended to
before the next nomcom.

Now, various people have argued that the nomcom chair was within the rules to
reset the process when the error came to light. I agree with this assessment,
but this is really beside the point. The point is the nomcom chair should NOT
have the reset option available in this circumstance, and that our current
process is BROKEN in allowing this level of discretion.

> for all nomcom volunteers.)

Agreed.

> matter.

I'm sorry, but I'm not so forgiving. The nomcom chair made a major blunder
here, abetted by some exceptionally poor advice from the IETF chair. This
blunder could have caused major damage to the organization. The combination of
there being essentially no time before the second selection process and the
fairly drastic nature of an ISOC appeal is the only - repeat ONLY - reason I
for one didn't start the ball rolling.

I've seen no indication that there's been any direct fallout from this, such as
a lawsuit being filed. (Of course something along these lines could still
happen.) But this does not mean that what has happened hasn't weakened the
faith people have in the fairness of the process. I know for a fact that I'm
much less sanguine about the process than I used to be, and I suspect I'm not
alone in this.

I also share your discomfort with the nomcom chair's decision to
consult the IETF chair, although my discomfort falls short of wanting to
see some formal rule against it happening.

> But, if we are going to make sure this problem does not occur

> @@@ We already have this. See RFC 3797.

Yep. Again, if it ain't broke...

> See Section 5.2 of RFC 3797.

Section 5.2 talks about potential issues with using stock prices or
volumes. I don't see how this connects with the need to remove the
ability for the nomcom chair to reset the process for no good reason.

> always resolved between the volunteer and the Secretariat before the

Nevertheless, I believe John is correct in saying that even if we eliminate the
unnecessary and dangerous discretionary powers the nomcom chair has currently,
there's still an issue when the list we use is subsequently found not to
contain people it should. However, there doesn't appear to be a good way to
address this problem should it arise. If someone has a novel suggestion for how
to deal with this I'd love to hear it, but absent such a suggestion I'm at a
loss as to how to deal with this one other than to say "yes, it's a potential
issue, hopefully it won't happen".

				Ned

---

From: John C Klensin <[email-masked]>
Date: Sat, 02 Sep 2006 08:49:27 -0400

--On Friday, 01 September, 2006 15:07 -0700 Ned Freed
<[email-masked]> wrote:

>> > only matters that the
>...
 
>> @@@ I'm not sure why you agree with me and then say the
>> opposite. If it doesn't matter what the ordering of the
>> nomcom volunteer pool is then it doesn't matter what the
>> ordering of the nomcom volunteer pool is, and, in particular,
>> it doesn't matter how random or "biased" it is. The RFC 3797
>> algorithm takes random inputs and uses them to make successive
>> uniformly distributed non-repeating selections from a range
>> of integers. The sole purpose of the published ordered
>> volunteer pool list is to provide a pre-announced mapping
>> from those integers to the people who volunteered. If it
>> mattered what that mapping was, then you are not making
>> uniformly distributed random selections.
> 
> John, I have to agree with Donald here. Nothing anyone has
> said has indicated that there is a flaw in the algorithmic
> part of the selection process. If something ain't broke don't
> fix it.

In my original note to Brian, I tried to ask a question
involving a statistical nicety.  It has to do with preserving
the original randomness given a second draw from a rather small
population (the number of volunteers).  As you and Donald
certainly know at least as well as I do, the key question about
pseudo-random number generators and processes is whether they
are good enough, not whether they produce randomness by all
possible measures.  I agree (but unfortunately did not say so
explicitly) that the procedures defined in 3797 is more than
adequate if followed exactly.  I had and have some doubts about
some of the variations 3797 permits if someone performs a
partial reset (as distinct from a full reset, which would
involve reopening the call for volunteers and maybe the Nomcom
Chair appointment).

However, I do not believe now, and have never believed, that
quibbles about statistics and randomness are the issue here.  I
simply wanted to understand Brian's reasoning (and the reasoning
of anyone else who recommended re-drawing the Nomcom composition
after eliminating one name).

I see two, related, issues at the core of the actual problem
here:

(1) Whatever the reason, essentially eliminating the time
between when the volunteer pool was announced and the time the
Nomcom membership list was determined took a series of checks
out of the system, both with regard to eliminating individuals
who were not eligible and with regard permitting people who
volunteered and had been dropped to identify and question that
decision (or find lost email, etc.).    We will never know
whether the numbers of the latter were large enough to have an
impact on the selection model, and that lack of knowledge is a
threat to the perceived integrity of the system.

(2) Because the notion of a reset after the Nomcom membership
list is selected permits someone, in principle, to examine the
membership list and say "no, I don't like that one, let's do it
over", a reset is an extremely bad idea if there is any possible
other alternative.   To the extent to which a potential
candidate in this round, or anyone with very strong opinions
about a potential candidate, have an opportunity to examine the
first-selected list and then provide substantive input into
whether a reset should be done, we move beyond "extremely bad"
into something worse.   I assume that everything have been done
with good faith here but, to paraphrase an off-list discussion,
at least some of the community will always have a nagging fear
that the reset was motivated by some members of the community
believing that a second draw would yield a more favorable Nomcom.

> Indeed, not only is the algorithmic part of the selection
> process fine, the non-algorithmic parts were well thought out
> and various contingencies were covered, including the specific
> one where disqualified people found their way onto the final
> qualified candidate list. The problem is quite simply that the

yes
 
> It also appears we have dodged the bullet this time around.
> But this was mostly a matter of luck. We may not be so lucky
> the next time, and that means we need to act now to fix the
> problem that the nomcom chair has the dangerous and
> unnecessary ability to reset the selection process in cases
> where no reset should be allowed. (I have already described
> why there is real danger here - see my previous posting for
> specifics.) This absolutely must be attended to before the
> next nomcom.

I think we are in violent agreement.
 
> Now, various people have argued that the nomcom chair was
> within the rules to reset the process when the error came to
> light. I agree with this assessment, but this is really beside
> the point. The point is the nomcom chair should NOT have the
> reset option available in this circumstance, and that our
> current process is BROKEN in allowing this level of discretion.
>...
  
> I'm sorry, but I'm not so forgiving. The nomcom chair made a
> major blunder here, abetted by some exceptionally poor advice
> from the IETF chair. This blunder could have caused major
> damage to the organization. The combination of there being
> essentially no time before the second selection process and the
> fairly drastic nature of an ISOC appeal is the only - repeat
> ONLY - reason I for one didn't start the ball rolling.
> 
> I've seen no indication that there's been any direct fallout
> from this, such as a lawsuit being filed. (Of course something
> along these lines could still happen.) But this does not mean
> that what has happened hasn't weakened the faith people have
> in the fairness of the process. I know for a fact that I'm
> much less sanguine about the process than I used to be, and I
> suspect I'm not alone in this.

Agreed.  See above.  My level of willingness to forgive was
motivated by the situation you described as having dodged the
bullet and because the Nomcom chair clearly has this discretion
under the current rules, not because I'm happy with the
situation.

> I also share your discomfort with the nomcom chair's decision
> to consult the IETF chair, although my discomfort falls short
> of wanting to see some formal rule against it happening.

Given this demonstration, I think some formal guidance that it
is inappropriate for the nomcom chair to seek procedural advice
from anyone who might be expected to be a candidate for
selection or re-selection.  But I have no idea how to write a
formal rule that would work, even if it were desirable... and
I'm not convinced that it would be desirable.

>> in the future, I think we should make the procedure as
>> gaming-proof as possible.  That, to me, implies two
>> requirements going forward:
> 

I do not disagree.  Just pointing out that it needs to be, and
continue to be, part of the process.  

>...

>> In addition, I am extremely concerned by hints on this list
>> that the Secretariat's checking procedures ruled people
>> ineligible to volunteer who had, in fact, attended the
>> correct number of meetings.  That, it seems to me, is a much
>> larger threat to the integrity of the Nomcom model and
>> perceptions of trust in it than any issue that impacts a
>> single volunteer or even, within broad limits, the
>> randomization and Nomcom member selection processes.
> 
>> @@@ Well, you are welcome to be as concerned as you like, but
>> we live in an imperfect world. As far as I know, there are
>> usually a few disputes about the volunteer list, usually in
>> connection with people whose eligibility the Secretariat
>> doubts. Sometimes they are determined to be correct and
>> sometimes the Secretariat is determined to be right.
>> Sometimes the volunteer is confused about the eligibility
>> requirements or about what their attendance actually was, or
>> has variant versions of their name, or has changed their
>> name, or ... But this sort of thing doesn't effect very many
>> people on the volunteer list and is almost always resolved
>> between the volunteer and the Secretariat before the list is
>> published.
> 
> Nevertheless, I believe John is correct in saying that even if
> we eliminate the unnecessary and dangerous discretionary
> powers the nomcom chair has currently, there's still an issue
> when the list we use is subsequently found not to contain
> people it should. However, there doesn't appear to be a good
> way to address this problem should it arise. If someone has a
> novel suggestion for how to deal with this I'd love to hear
> it, but absent such a suggestion I'm at a loss as to how to
> deal with this one other than to say "yes, it's a potential

Actually, there is a perfectly good way to address this issue,
at least at the 95% level.  Suppose the list of volunteers were
made public significantly in advance of the selection of Nomcom
members so that people have the opportunity to detect and
protest inappropriate volunteers and so that people who are
dropped have an opportunity to protest.  Then these things can
be straightened out before the Nomcom is selected, eliminating
later second-guessing and debates about what actions various
problems should cause.   Hmm.  That is already specified in
3797: we don't need to fix anything, just follow our own,
existing, rules.

    john

---

From: Avri Doria <[email-masked]>
Date: Sat, 2 Sep 2006 19:10:19 +0200

On 2 sep 2006, at 14.49, John C Klensin wrote:

> I assume that everything have been done

I think i agree that it is safe to say that is due to human error,  
though the consultation with someone who is on the list of those to  
be considered for renewal does compound the error and put it all at  
further risk.  but i still think it is an error and not an  
intentional act to subvert the process.

On 2 sep 2006, at 17.42, Theodore Tso wrote:
> On Sat, Sep 02, 2006 at 06:59:36AM -0700, Hallam-Baker, Phillip wrote:
>> The bug here is that the process is insufficiently robust under
>> operator error.
>>
>> That is broke.
>>

>
> I think we are in violent agreement that it should be fixed before the
> next Nomcom.  There may be a difference of opinion of how exactly to
> fix the problem, but I don't think anyone has suggested that we should
> leave things the way that they are, at least for the next Nomcom.

i don't think the process is broken, but the implementation was.  and  
i don't believe this is sufficient reason to reopen the nomcom process.

i think the process is clear about what should happen once someone is  
committed to the 3797 process.  what is broken is that the process  
was, perhaps, not well enough  understood and, as is so often the  
case in the IETF and elsewhere, was followed loosely as opposed to  
strictly. i do wonder if the previous nomcom chair was consulted in  
making the decision to redraw - in fact perhaps previous chairs could  
be a useful resource for the new chair faced with such a dilemma as  
opposed to representatives of the bodies to be staffed.

however, in the sprit of accepting broken implementations (i.e.  
conservative in what you send, liberal in what you receive) i suggest  
we wish Andrew luck and let him get on with it.

a.

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 04 Sep 2006 12:27:17 +0200

...
>>of wanting to see some formal rule against it happening.

Well, let me observe that if there had been a formal rule,
it would have been observed. And as a matter of fact, it wasn't
Andrew who put me and the IAB Chair on copy - it was a person
who noticed the anomaly. I happened to be aware that the ISOC
President was on travel and Andrew turned out to be moving
house, and at the same time the secretariat was experiencing
sluggish mail queues (that problem is being addressed). So there
was quite a combination of circumstances this time.
> I'm not convinced that it would be desirable.

I believe there is another, related, gap that has bothered me
several times - there is nothing to indicate where the ISOC President
should look for advice and consultancy when needed. So although
it may take some debate, I think clarification in this area would be
desirable. I can't speak for other people, but it seems to me that
it's an obvious default for both the ISOC President and the Nomcom
chair to turn to the I* chairs when in doubt - we have suggested
nobody else except the previous Nomcom chair as a source
of advice. If we don't want that to be the default, we need to
provide an alternative.

     Brian

---

From: "Eastlake III Donald-LDE008" <[email-masked]>
Date: Tue, 5 Sep 2006 11:58:22 -0400

My theme here was that it is probably not humanly possible to absolutely
guarantee that no discretionary decision by the nomcom chair will ever
be required somewhere in the process of nomcom formation because it is
very hard to anticipate all possible real world events. The theme of
Section 5.2 of RFC 3797 is that this applies in the area of specifying
the source of future randomness. It is true that 5.2 probably harps too
much on using stocks as a bad idea. But problems have occurred with
them. It is easy to image things that could have happened this time,
such as a listed company merging or splitting, that would have required
a nomcom chair discretionary decision with regard to the random input.
However, the problem that occurred was not in applying the specification
of random input to the real world but a human error in the timing of
announcements.

Donald

-----Original Message-----
From: Ned Freed [mailto:[email-masked]] 
Sent: Friday, September 01, 2006 6:08 PM
To: Eastlake III Donald-LDE008
Cc: IETF-Discussion
Subject: RE: Now there seems to be lack of communicaiton here...

...

> anyone who has discretion has had an opportunity to see the initial 
> list of Nomcom members.  If someone is selected (or

> @@@ It may be possible, with sufficient care, to make vanishingly 
> small the chance that a nomcom chair discretionary decision would be 
> needed for this aspect of nomcom selection. But I am not sure that, in

> the real world, it is possible to do this for all aspects of nomcom
selection.

...

				Ned

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 05 Sep 2006 18:25:06 +0100

Eastlake III Donald-LDE008 wrote:

>very hard to anticipate all possible real world events.
>
Also some input are sent to the nomcom chair for anonymous transmission
to the rest of the nomcom. The chair needs to validate the authenticity of
these comments and the to transmit them to the nomcom. There is an element
of trust and discression in this process.

- Stewart

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 05 Sep 2006 18:25:06 +0100

- Stewart