Subject: Re: Gen-art review of draft-hartman-mailinglist-experiment-01.txt

From: Sam Hartman <[email-masked]>
Date: Fri, 03 Mar 2006 14:40:59 -0500

>>>>> "Elwyn" == Elwyn Davies <[email-masked]> writes:

    Elwyn> I was selected as General Area Review Team reviewer for
    Elwyn> this specification (for background on Gen-ART, please see
    Elwyn> http://www.alvestrand.no/ietf/gen/art/gen-art-FAQ.html).

Hi.
I'm sorry it has taken me so long to get back to your comments.
I've been busy trying to clear documents before the Dallas IETF.

    Elwyn> Summary: [Note: This is the first time that I have done a
    Elwyn> Process Experiment review and I will have to stretch the
    Elwyn> usual review criteria a bit.  Basically I believe I should
    Elwyn> be looking for internal self-consistency, consistency with
    Elwyn> associated processes and likelihood of damage to the
    Elwyn> functioning of the IETF.]

That seems like a good approach.  I'm also doing this for the first
time so your cooperation in helping me is greatly appreciated.
    Elwyn> I think this draft is NOT currently in a suitable state to
    Elwyn> produce a well-defined experiment.  The main reason for
    Elwyn> this conclusion is that the experiment consists of enabling
    Elwyn> a meta-mechanism and suggesting something the IESG might
    Elwyn> possibly do with this power rather than explicitly stating
    Elwyn> that this is what the IESG should put in place.  

I'd like to focus on your general comments about the draft not being
ready.  I think we can come back to specific textual comments after
we've reached general agreement.  I'd like to first take your issue of
the IESG charter and then come back to the meta issue that all this
experiment does is enable the IESG to do things.

    Elwyn> My reading
    Elwyn> of s7.2 of the IESG Charter [RFC3710] is that the IESG has
    Elwyn> the power to do this sort of thing already anyway.  

Note that the IESG charter is an informational document.  It cannot
give the IESG power that it does not otherwise have.

It was not clear to the participants on the IETF list that the IESG
has the power to create mechanisms between 3683 and 3934 without a BCP
or experiment.  If the consensus of the IETF community is that the
IESG already has that power and that the IESG's power is already well
documented then I will withdraw my draft.  My personal opinion is that
the power is not well documented and thus is subject to higher
probability of successful appeal.

    Elwyn> Were
    Elwyn> the suggested mechanisms eventually adopted, I would have
    Elwyn> some qualms about the possibility of indefinite bans being
    Elwyn> possible without allowing a wider (possibly IETF as opposed
    Elwyn> to IESG) consensus, but that point is currently moot as the
    Elwyn> actual proposals that would be put in place are not
    Elwyn> specified by this document.

I don't think the point is moot.  If there are specific limits on the
IESG's power that should be put in place, here is the place to do it.
Alternatively when we evaluate the experiment we could decide
additional limits are needed.

But let's come back to the question of whether meta-experiments are a
good idea.  I think that in order for 3933 to be a valuable tool many
of the experiments are going to be meta-experiments.  So let me first
explain why I think that's the case and then discuss how to evaluate a
meta experiment.

The primary reason you want to encourage meta-experiments is that a
lot of the hypotheses you want to test involve delegation.  For
example I want to test the hypothesis that the right way to solve the
mailing list mess is to delegate it to the IESG.  I could delegate it
to the IESG as part of an experiment along with a initial procedure.
But if I do that I'm testing a different hypothesis: is delegating
something to the IESG with the whole IETF designing the initial
conditions a way to solve the problem.  As you might imagine that's a
different hypothesis.  Since I'm on the IESG I'm actually in a
reasonably good position to negotiate an initial procedure that the
IESG will be happy with and that would be similar to a procedure the
IESG would come up with on its own.  However we want 3933
experiments--even experiments delegating things to the IESG--to be
documents that anyone can write.  So we should require that authors of
3933 experiments demonstrate stakeholder buy-in for experiments but
not require that they take actions as if they were the stakeholders.

The second reason that you want to allow meta-experiments is that we
want to encourage RFC 3933 as the first step in process change.
Process change often results in BCPs.  You want the 3933 experiment to
be reasonably similar to a BCP so that when appropriate you can easily
convert a successful experiment into a BCP.  You would probably
replace any evaluation criteria with results of the evaluation,
replace the sunset clause with something else.  However you want the
operative language to remain the same.  A significant result of the
mailing list discussion is the concern that our BCPs are too specific
and encode operational details.  If you require that meta-experiments
are not allowed you strongly push us in the direction of
overly-specific BCPs.  I think that would be a very bad idea.

Finally, we want 3933 experiments to be easy to write.  One of my
personal goals with this particular experiment was to see how easy I
could make it to write the experiment.  I think we want to come away
from this process with the conclusion that writing the document is
easy.  The hard part of process change should be building consensus,
recruiting stakeholders, educating the community and actually trying
to use a process to do superior technical work.  We should not make it
hard for people who clearly know what they want to try to express it.
So I'd like to resist the temptation to raise the bar for experiments
beyond what is necessary.  Bars I'm asked to meet will probably be at
least as high as future experiments.

In conclusion, the hypothesis I'm testing is meta, so my experiment is
meta.  i think allowing this is desirable because it allows us to test
the hypothesis, begins to align with an eventual BCP if the test is
successful and supports the cultural engineering goal of making
experimentation the preferred direction for process change.

You are absolutely right that we need to evaluate this at the end.
I'll first point to RFC 3933 and remind you that specific criteria for
evaluation are not required.  In that case the basic evaluation will
be based on community reaction.  Let us take a look at how that might
work in this case.

One possibility is that the IESG adopts no procedures based on this
experiment.  That sounds like a failed experiment to me: why is the
power needed if never used.  There might be some arguments that having
a power prevents future problems, but these arguments would be
unlikely to be compelling.

Another case is that the IESG adopts some procedures and these
procedures never get used.  We'd actually learn a lot in this case.
The IESG would get a feel for whether this specification was clear
enough to describe what procedures could be adopted and what steps
need to be followed to adopt a procedure.  It's a strong
understatement to say that there tend to be rough edges around any
process document the first time it is run.  There are things I'd word
more clearly in 3683, 3934, 3933, 3932 and other process documents.
These tend to pop out the first time you try and use them.  We would
get a significant part of that feedback from just adopting procedures
under this experiment.  We'd need to have a community discussion about
whether we wanted to extend the experiment, codify it in a BCP or drop
it.  If the reason no procedures were used was that no problems popped
up on any mailing lists, I'd argue against dropping it.  If the
community felt comfortable, we could publish a BCP.  Otherwise we
could renew the experiment.  If the discussion proved contentious then
the next round of the experiment would probably need more evaluation
criteria.

[As an aside, RFC 3933 seems to have a rough edge that you cannot
renew an experiment without publishing a new RFC.  I'm not at all sure
that's desirable.  However if you end up needing to add evaluation
criteria most times you renew and experiment, perhaps that is OK.]

If procedures are adopted and used, I think we're in a reasonable
position to evaluate the results.  I know one criteria I'd apply: was
the situation less disruptive to the work of the IETF and the IETF
leadership than an RFC 3683 action
If so, then we made progress.  Others might have different criteria to
apply.  One possible outcome of the evaluation is that we need to
renew the experiment with explicit criteria because we cannot agree on
an evaluation.  Another is that the experiment should be the basis for
a BCP.  Another is of course that the experiment made things worse and
should terminate.

In conclusion I've proposed evaluation criteria that I believe cover
the possible outcomes.  These criteria are fuzzy.  I believe it is no
more fuzzy than if a specific procedure is included.  Clearly such
fuzzy criteria are allowed by RFC 3933.

All that said, if the community wants changes then the draft will
change.  We're looking for an IETF consensus here.  If there is
significant demand for specific evaluation criteria we can include
those.  If there is a community requirement for a specific initial
procedure, I can include one.  I think that particular change would
set a harmful precedent but would make the change if that is what the
community wants.  However, having explained that I'm explicitly trying
to experiment with delegation and having proposed possible ways of
thinking about evaluating the result,I would want to see a much
stronger call for change than one review.  If the experiment actually
isn't viable then one comment should be sufficient.  However I don't
think that is the case.

    Elwyn> Review:

    Elwyn> s1: I think that this section of the last paragraph of s1
    Elwyn> overstates/mis-states what this document actually does:
    >>  This memo is an RFC 3933[RFC3933] experiment to provide the
    >> community with additional mechanisms to manage its mailing
    >> lists while the community decides what mailing list guidelines
    >> are appropriate.  IN particular this experiment creates a level
    >> of sanction between RFC 3934 and RFC 3683 for working group
    >> lists and creates sanctions other than RFC 3683 for
    >> non-working-group lists.
    >> 
    Elwyn> In practice all that s4 mandates is:
    >>  During the experiment period, the IESG MAY approve other
    >> methods of mailing list control besides those outlined in RFC
    >> 3683 and RFC 3934 to be used on a specified set of IETF mailing
    >> lists.
    >> 
    Elwyn> i.e., it enables a meta-mechanism.  s4 then goes on to
    Elwyn> suggest some things the IESG *might* adopt (second half of
    Elwyn> para 1 and all of para 2 of s4) and the procedures that it
    Elwyn> must go through to get them adopted.  The result of this is
    Elwyn> (in the first instance) merely the provision to allow the
    Elwyn> IESG to decide to do something.  I don't think this
    Elwyn> constitutes a well-formed experiment.

Often, the community creates mechanisms by delegating the power to
someone.  I'd be interested in other comments on whether the last
sentence of S1 over states things?

I think I'd rather hear your responses to these general comments
before going through the rest of your detailed comments.

--Sam

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Elwyn Davies <[email-masked]>
Date: Tue, 07 Mar 2006 19:56:59 +0000

Hi.

I am going to take my gen-art hat off here because I want to suggest 
alternatives rather than just assessments.

I have no inherent problem with what we are calling meta-experiments, 
although there are issues regarding whether the community will feel 
comfortable with just granting the IESG some power to 'do something' 
without having much idea what it is.  Determining what constraints might 
placed on that power might take as long as actually specifying what 
should be done.  On the other hand if you constrain the powers too 
tightly you might as well not bother with the meta-experiment. I think 
that is the problem you have already identified.

The main difficulty I have with the draft as it stands is that  the last 
paragraph of s1 says that the experiment will do something definite, but
 s4 actually just gives the IESG a general power.  This is not 
consistent - it falls between the two stools mentioned above:  although 
s4 suggests what types of delegation are envisaged the IESG can do 
whatever it likes regarding mailing list control and what actually 
happens might bear no resemblance to the second half of s4 para 1 and s4 
para 2.  So I think the draft can go one of two ways:

1 (which I think Sam would prefer):
Redraft the memo as purely the meta-experiment and remove the 
suggestions of what the IETf might do, to avoid any possibility that 
people might be convinced that this was the only thing that could happen.
===============================================
Delete the last but one sentence of s1 - the modified document won't 
actually create any particular level of sanction.
Modify the previous sentence:
>
> This memo is an RFC 3933[RFC3933] experiment to provide
>    the community with additional mechanisms to manage its mailing lists
>    while the community decides what mailing list guidelines are
>    appropriate.
>
to:
This memo is an RFC 3933[RFC3933] experiment which will enable the IESG 
to provide the community with additional mechanisms to manage its 
mailing lists while the community decides what mailing list guidelines 
are appropriate.

Remove the 3rd and 4th sentences (from 'Such methods...') of para 1 and 
the whole of para 2 of s4.

Replace the last part of para 4 of s4:
>
> While such last calls are encouraged, they are not
>    required.  The reason that the last call is not required is that
>    under RFC 2418, no last call is required; there seems to be no reason
>    to have a procedure more strict than that proposed in RFC 2418.
>
Since RFC2418 does not require actions taken to be Last Called, it is 
explicitly permitted  that any procedure put in place under the powers 
granted by this experiment need not require a Last Call before it is 
implemented, as there seems to be no reason to have a procedure more 
strict than is permitted in RFC 2418.

If the resulting document is approved as an RFC3933 experiment, then the 
text removed from paras 1 and 2 can be put to the IESG as a proposal.

The main issue with this as an experiment is that there are two sets of 
evaluations that can be done:
- was the meta-experiement a success?
- were any procedures adopted under the extra powers succcessful?
It is not very obvious how to judge the original meta-experiment 
especially if the adopted procedures don't work out!
=====================================================
2.  Redraft the document to remove the general power given in s4 - then 
s1 is accurate, and the experiment would just cover the explicit 
procedures set up in the second half of para 1 and para 2 of s4.

======================================================

Thoughts?

Regards,
Elwyn

---

From: Sam Hartman <[email-masked]>
Date: Tue, 07 Mar 2006 16:06:57 -0500

I am happy to make a change similar to the one you propose in section
1.

I'm happy to split the parts of section 4 dealing with what the IESG
might do into their own section as an example.

I do not want to remove them completely.

Would that be OK

---

From: Elwyn Davies <[email-masked]>
Date: Tue, 07 Mar 2006 23:15:02 +0000

Hi.

> might do into their own section as an example.
That's fine by me.. it should make a self-consistent document.
> I do not want to remove them completely.

As regards gen-art I think it would be fine.

Ultimately I am only a small part of the consensus as regards the 
experiment proposed.
Personally, I would prefer that we didn't have to waste inordinate 
amounts of time on mailing list management.  Unfortunately knocking 
virtual heads together isn't very effective.

/Elwyn

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 22 Mar 2006 14:10:37 -0600

Sam,

in the gen-art meeting today, you asked me to read and reply to the long 
note you wrote to Elwyn.
I'm assuming you mean this one.

The discussion in the genarea meeting clarified quite a bit what you are 
trying to achieve with the draft, and was a very useful background to 
writing this note. Thank you!

I've taken the gen-art team off the CC list, since they are not 
otherwise involved.

In summary:

> different hypothesis.
I do not believe that is a correct statement.
An RFC 3933 experiment by its very nature involves three different 
sub-experiments:
- Whether you can get the IETF to agree to running this experiment
- Whether the bodies that administer the experiment can perform the 
experiment so that we have something to measure
- Whether the experiment produces useful results

I think what you are proposing will lead to failure of the *first* 
experiment, which means that you won't get to the other two.
If you say "give the IESG the power to produce procedures, we're not 
going to give you even a hint as to what these procedures might be", 
this is what I would characterize as selling a pig in a poke, and it's 
perfectly reasonable for the IETF community to ring up a "no sale" on 
the consensus register.

The theory that there is a different hypothesis being tested if the IESG 
sets the initial procedures than if the initial procedures are part of 
the experiment also seems doubtful to me; in both cases what I think 
will happen is that the IESG will start off with procedures designed by 
Sam Hartman, and the big difference is that the community will have seen 
the initial procedures before deciding to run the experiment.
The whole IETF has never designed anything and never will; all design is 
done by specific people, and (hopefully) reviewed by some large-enough 
fraction of the IETF. That's how EVERYTHING in the IETF works.
>   Since I'm on the IESG I'm actually in a
I do not believe this theory either.

RFC 3933 talks about the IESG and the community. The "stakeholder" 
concept is of your introduction, and I believe it serves no benefit.
An RFC 3933 proposal author HAS to do three things:

- Write up what he's proposing in enough detail that it can be evaluated
- Convince the IESG that the experiment has enough merit to warrant a 
Last Call
- Generate enough review and comment in the community that the IESG can 
confidently say that there's community consensus to run the experiment

Certainly, a community feedback that says "I'm one of the people 
required to do work under this experiment, and I'm not going to do it" 
is a strong statement that needs consideration when assessing whether 
there's IETF consensus to run the experiment. But sometimes that's just 
part of the "rough" in rough consensus, even when deciding to run the 
experiment will cause someone to resign his role. And sometimes it's not.
> overly-specific BCPs.  I think that would be a very bad idea.
Here I agree, but disagree that this document does an appropriate job of it.
I think the separation of "principle" and "process" is a good thing - 
the principle of this document being (I think) that the IESG sets 
mailing list management procedures, and the process being the process by 
which such procedures are set, and the third level being the procedures 
themselves.

However, just stating the principle alone at an abstract level is not 
enough to either start or evaluate the experiment.
> least as high as future experiments.
Here I also disagree. Making experiments easy to write is NOT a goal. 
Successful change that helps the IETF mission is a goal.
Making experiments easy to write, and approving experiments without 
reasonably careful review, CAN lead to the process turning into an 
ever-changing morass of conflicting and confusing experiments - this is 
not something that furthers the mission of the IETF.

I believe your document is not good enough for the process change 
experiment it wants to achieve - despite the fact that I think an 
experiment in this area is a very reasonable thing to do, and should be 
done.
> experimentation the preferred direction for process change.
And my answer is that I believe that your focus on the meta level is a 
non-useful distraction.

We need a change to the handling of mailing lists. One possible form of 
that change is to introduce an experiment that clearly separates 
principles from process from procedures, and gives the IESG the power to 
change the two latter during the course of the experiment.

I would support running such an experiment. But I cannot support 
approving the present document.

                              Harald