Subject: RE: Now there seems to be lack of communicaiton here...

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Sat, 2 Sep 2006 06:59:36 -0700

If it ain't broke? How much more evidence of being broke do we need?

The bug here is that the process is insufficiently robust under operator error.

That is broke.

The underlying problem here is the lack of auditability in the process.

There is a simple fix here, eliminate the dependency on the list ordering and the system does not have such a critical dependence on the operator.

Again nobody is claiming anything dishonest has happened here. The concern is that the accident could be repeated on purpose in the future to exclude undesirable candidates. Having spent part of last month watching this attempted in Alabama it is a real concern.

When something is broke admit the fact. Prattling on about not fixing what aint broke only makes people angry.

Sent from my GoodLink Wireless Handheld (www.good.com)

 -----Original Message-----
From: 	John C Klensin [mailto:[email-masked]]
Sent:	Saturday, September 02, 2006 05:56 AM Pacific Standard Time
To:	Ned Freed; Eastlake III Donald-LDE008
Cc:	IETF-Discussion
Subject:	RE: Now there seems to be lack of communicaiton here...

--On Friday, 01 September, 2006 15:07 -0700 Ned Freed
<[email-masked]> wrote:

>> See below at @@@
> 
>> -----Original Message-----
>> From: John C Klensin [mailto:[email-masked]]
>> Sent: Friday, September 01, 2006 11:45 AM
>> To: Eastlake III Donald-LDE008; IETF-Discussion
>> Subject: RE: Now there seems to be lack of communicaiton
>> here...
> 
>> --On Thursday, 31 August, 2006 17:30 -0400 Eastlake III
>> Donald-LDE008 <[email-masked]> wrote:
> 
>> > John,
>> > 
>> > If the selection method is random, it makes no difference
>> > whatsoever how the list of nomcom volunteers is ordered. It
>> > only matters that the
>...
 
>> @@@ I'm not sure why you agree with me and then say the
>> opposite. If it doesn't matter what the ordering of the
>> nomcom volunteer pool is then it doesn't matter what the
>> ordering of the nomcom volunteer pool is, and, in particular,
>> it doesn't matter how random or "biased" it is. The RFC 3797
>> algorithm takes random inputs and uses them to make successive
>> uniformly distributed non-repeating selections from a range
>> of integers. The sole purpose of the published ordered
>> volunteer pool list is to provide a pre-announced mapping
>> from those integers to the people who volunteered. If it
>> mattered what that mapping was, then you are not making
>> uniformly distributed random selections.
> 
> John, I have to agree with Donald here. Nothing anyone has
> said has indicated that there is a flaw in the algorithmic
> part of the selection process. If something ain't broke don't
> fix it.

In my original note to Brian, I tried to ask a question
involving a statistical nicety.  It has to do with preserving
the original randomness given a second draw from a rather small
population (the number of volunteers).  As you and Donald
certainly know at least as well as I do, the key question about
pseudo-random number generators and processes is whether they
are good enough, not whether they produce randomness by all
possible measures.  I agree (but unfortunately did not say so
explicitly) that the procedures defined in 3797 is more than
adequate if followed exactly.  I had and have some doubts about
some of the variations 3797 permits if someone performs a
partial reset (as distinct from a full reset, which would
involve reopening the call for volunteers and maybe the Nomcom
Chair appointment).

However, I do not believe now, and have never believed, that
quibbles about statistics and randomness are the issue here.  I
simply wanted to understand Brian's reasoning (and the reasoning
of anyone else who recommended re-drawing the Nomcom composition
after eliminating one name).

I see two, related, issues at the core of the actual problem
here:

(1) Whatever the reason, essentially eliminating the time
between when the volunteer pool was announced and the time the
Nomcom membership list was determined took a series of checks
out of the system, both with regard to eliminating individuals
who were not eligible and with regard permitting people who
volunteered and had been dropped to identify and question that
decision (or find lost email, etc.).    We will never know
whether the numbers of the latter were large enough to have an
impact on the selection model, and that lack of knowledge is a
threat to the perceived integrity of the system.

(2) Because the notion of a reset after the Nomcom membership
list is selected permits someone, in principle, to examine the
membership list and say "no, I don't like that one, let's do it
over", a reset is an extremely bad idea if there is any possible
other alternative.   To the extent to which a potential
candidate in this round, or anyone with very strong opinions
about a potential candidate, have an opportunity to examine the
first-selected list and then provide substantive input into
whether a reset should be done, we move beyond "extremely bad"
into something worse.   I assume that everything have been done
with good faith here but, to paraphrase an off-list discussion,
at least some of the community will always have a nagging fear
that the reset was motivated by some members of the community
believing that a second draw would yield a more favorable Nomcom.

> Indeed, not only is the algorithmic part of the selection
> process fine, the non-algorithmic parts were well thought out
> and various contingencies were covered, including the specific
> one where disqualified people found their way onto the final
> qualified candidate list. The problem is quite simply that the
> nomcom chair failed to follow these recommendations.

yes
 
> It also appears we have dodged the bullet this time around.
> But this was mostly a matter of luck. We may not be so lucky
> the next time, and that means we need to act now to fix the
> problem that the nomcom chair has the dangerous and
> unnecessary ability to reset the selection process in cases
> where no reset should be allowed. (I have already described
> why there is real danger here - see my previous posting for
> specifics.) This absolutely must be attended to before the
> next nomcom.

I think we are in violent agreement.
 
> Now, various people have argued that the nomcom chair was
> within the rules to reset the process when the error came to
> light. I agree with this assessment, but this is really beside
> the point. The point is the nomcom chair should NOT have the
> reset option available in this circumstance, and that our
> current process is BROKEN in allowing this level of discretion.
>...
  
> I'm sorry, but I'm not so forgiving. The nomcom chair made a
> major blunder here, abetted by some exceptionally poor advice
> from the IETF chair. This blunder could have caused major
> damage to the organization. The combination of there being
> essentially no time before the second selection process and the
> fairly drastic nature of an ISOC appeal is the only - repeat
> ONLY - reason I for one didn't start the ball rolling.
> 
> I've seen no indication that there's been any direct fallout
> from this, such as a lawsuit being filed. (Of course something
> along these lines could still happen.) But this does not mean
> that what has happened hasn't weakened the faith people have
> in the fairness of the process. I know for a fact that I'm
> much less sanguine about the process than I used to be, and I
> suspect I'm not alone in this.

Agreed.  See above.  My level of willingness to forgive was
motivated by the situation you described as having dodged the
bullet and because the Nomcom chair clearly has this discretion
under the current rules, not because I'm happy with the
situation.

> I also share your discomfort with the nomcom chair's decision
> to consult the IETF chair, although my discomfort falls short
> of wanting to see some formal rule against it happening.

Given this demonstration, I think some formal guidance that it
is inappropriate for the nomcom chair to seek procedural advice
from anyone who might be expected to be a candidate for
selection or re-selection.  But I have no idea how to write a
formal rule that would work, even if it were desirable... and
I'm not convinced that it would be desirable.

>> But, if we are going to make sure this problem does not occur
>> in the future, I think we should make the procedure as
>> gaming-proof as possible.  That, to me, implies two
>> requirements going forward:
> 
>> (1) We get strong randomization of the selection process
> 
>> @@@ We already have this. See RFC 3797.
> 
> Yep. Again, if it ain't broke...

I do not disagree.  Just pointing out that it needs to be, and
continue to be, part of the process.  

>...

>> In addition, I am extremely concerned by hints on this list
>> that the Secretariat's checking procedures ruled people
>> ineligible to volunteer who had, in fact, attended the
>> correct number of meetings.  That, it seems to me, is a much
>> larger threat to the integrity of the Nomcom model and
>> perceptions of trust in it than any issue that impacts a
>> single volunteer or even, within broad limits, the
>> randomization and Nomcom member selection processes.
> 
>> @@@ Well, you are welcome to be as concerned as you like, but
>> we live in an imperfect world. As far as I know, there are
>> usually a few disputes about the volunteer list, usually in
>> connection with people whose eligibility the Secretariat
>> doubts. Sometimes they are determined to be correct and
>> sometimes the Secretariat is determined to be right.
>> Sometimes the volunteer is confused about the eligibility
>> requirements or about what their attendance actually was, or
>> has variant versions of their name, or has changed their
>> name, or ... But this sort of thing doesn't effect very many
>> people on the volunteer list and is almost always resolved
>> between the volunteer and the Secretariat before the list is
>> published.
> 
> Nevertheless, I believe John is correct in saying that even if
> we eliminate the unnecessary and dangerous discretionary
> powers the nomcom chair has currently, there's still an issue
> when the list we use is subsequently found not to contain
> people it should. However, there doesn't appear to be a good
> way to address this problem should it arise. If someone has a
> novel suggestion for how to deal with this I'd love to hear
> it, but absent such a suggestion I'm at a loss as to how to
> deal with this one other than to say "yes, it's a potential
> issue, hopefully it won't happen".

Actually, there is a perfectly good way to address this issue,
at least at the 95% level.  Suppose the list of volunteers were
made public significantly in advance of the selection of Nomcom
members so that people have the opportunity to detect and
protest inappropriate volunteers and so that people who are
dropped have an opportunity to protest.  Then these things can
be straightened out before the Nomcom is selected, eliminating
later second-guessing and debates about what actions various
problems should cause.   Hmm.  That is already specified in
3797: we don't need to fix anything, just follow our own,
existing, rules.

    john

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Theodore Tso <[email-masked]>
Date: Sat, 2 Sep 2006 11:42:32 -0400

> The bug here is that the process is insufficiently robust under

I think we are in violent agreement that it should be fixed before the
next Nomcom.  There may be a difference of opinion of how exactly to
fix the problem, but I don't think anyone has suggested that we should
leave things the way that they are, at least for the next Nomcom.

						- Ted

---

From: "Eastlake III Donald-LDE008" <[email-masked]>
Date: Sat, 2 Sep 2006 18:39:17 -0400

Depends what you mean by "it". The overall process may have broke in
this case but the "it" referred to in the message you were responding to
is the "cryptographic" part of the process. The one in RFC 3797 depends
on pre-announcement of the ordered list of volunteers. The one you
suggested depends on pre-announcement of the email address of every
volunteer. Neither is any more robust than the other against a failure
to make all the information necessary for public verification available
in advance, including the specification of the source of future
randomness.
 
Donald

________________________________

From: Hallam-Baker, Phillip [mailto:[email-masked]] 
Sent: Saturday, September 02, 2006 10:00 AM
To: John C Klensin; Ned Freed; Eastlake III Donald-LDE008
Cc: IETF-Discussion
Subject: RE: Now there seems to be lack of communicaiton here...

The bug here is that the process is insufficiently robust under operator
error.

That is broke.

There is a simple fix here, eliminate the dependency on the list
ordering and the system does not have such a critical dependence on the
operator.

Again nobody is claiming anything dishonest has happened here. The
concern is that the accident could be repeated on purpose in the future
to exclude undesirable candidates. Having spent part of last month
watching this attempted in Alabama it is a real concern.

When something is broke admit the fact. Prattling on about not fixing
what aint broke only makes people angry.

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 03 Sep 2006 04:15:31 +0200

Eastlake III Donald-LDE008 wrote:

> the "it" referred to in the message you were responding to
> is the "cryptographic" part of the process. The one in RFC
> 3797 depends on pre-announcement of the ordered list of
> volunteers.

Just for fun I've added RFC 3797 to an MD5 test suite, works
like a charme, <http://purl.net/xyzzy/src/md5.cmd> (REXX)

Reading RFC 3797 again, I have "float" wrong, I'll fix that.
Maybe a hypothetical 3797bis should get an example covering
also "float".  With a sanity check if there is enough entropy.
Decimal digits * 3 could be an approximation, the first digit
for traded stocks is not completely random.  The public source
for these input numbers should be given as one or more URLs.

> Neither is any more robust than the other against a failure
> to make all the information necessary for public verification
> available in advance, including the specification of the
> source of future randomness.

The "bug" (if any) is related to 3777, not to 3797:  If the
IETF secretariat is unable to determine the eligible population
in time the NomCom Chair is in trouble, (s)he must get a final
result (modulo appeals) two months before the 3rd IETF meeting.

Maybe a public list maintained by the secretariat and updated
after each IETF meeting (within two weeks) would help, just
the names and up to five numbers like 66 65 64 63 62.  Folks
can then see if they are eligible, and they can discuss errors
as soon as they see them (with the secretariat, and appeals to
ISOC, because IAB and IESG shouldn't interfere in such cases).

RFC 3777 4.15 is not 100% clear.  Obviously IAB or IESG members
cannot be NomCom members at the same time.  But what if they
volunteer because they know that the random numbers are drawn
after they have resigned from their IAB or IESG seat ?  And if
they manage to "forget" to resign they're not eligible, ready.

Under an "assume good faith" doctrine anything in this "tea-pot
tempest" (quoting Randy) makes sense, an IAB member volunteers
because the rules promise that this is a guarantee to get no
other job, several screw-ups between the secretariat and some
volunteers, an announcement sent too late caused by another
hiccup, an RO asking for advice how to fix it, and using the
(intuitively) most conservative approach "reset".  But RFC 3797
defines that "most conservative" would be "draw next number" -
which doesn't cover the case "announcement sent after input
random numbers are already determined".

Publishing the complete list two days after the random input
numbers are drawn is a hopeless case.  With "secret volunteers"
the result could be manipulated, with a reset an "undesirable"
result could be vetoed.  Whatever the RO does, (s)he loses. :-(

If you want to harden the algorithm you could use N, N-1, N-2,
etc. as prefix + suffix for the MD5 input, instead of 0, 1, 2,
etc.  Won't help if the list is published too late, with two
consenting "secret volunteers" Zacharias and Abel it would be
possible to silently remove Zacharias from the end of the list,
and add Abel at the begin.  Phil's proposal also doesn't help
in this case, it's hopeless.  This situation must not happen.

Frank