Subject: Re: improving WG operation

From: Margaret Wasserman <[email-masked]>
Date: Sun, 1 May 2005 10:09:44 -0400

>People can and do use powerpoint slides in many ways.  Some folks will rework
>text in real-time, based on interaction with the participants.  Some folks
>just talk their slides rather than actually engaging with the participants.
>
>A thing to keep in mind is that slides and the jabber activity can be
>incredibly helpful to folks for whom English not their native language.
>
>I think that, in fact, the issue is not powerpoint-vs-no-powerpoint.  I think
>it is exactly and only the concern you raise:  meetings need to be for working
>group interaction.  If that is the clear goal and if the meeting is run with
>that goal enforced, then none of the trappings matter.

I completely agree with this.  And, I've been to plenty of 
non-interactive lectures that didn't involve any slides.

Dave, do you have any thoughts about how we can change the IETF 
culture from presentation to interaction (with or without slides). 
This is something that the IESG has been talking about, among others, 
but I'm not sure that we've come up with any really concrete ways to 
provoke/encourage this change.

Margaret

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Dave Crocker <[email-masked]>
Date: Sun, 1 May 2005 09:04:12 -0700

>  Dave, do you have any thoughts about how we can change the IETF culture
>  from presentation to interaction (with or without slides). This is
>  something that the IESG has been talking about, among others, but I'm not
>  sure that we've come up with any really concrete ways to provoke/encourage
>  this change.

Margaret, in general I think the issue is stricter meeting planning and 
management, where the goals and process are more explicit.  Of course, good 
wgs already do that.

Off the top of my head:

1. Require that the meeting have a web-posted agenda -- and really enforce the 
requirement. So, this makes sure others have a chance to evaluate what is 
going to take place.  No agenda a week in advance; no meeting.

2. Require that the deliverables of the meeting be pre-stated and prioritized.

3. Have the agenda include statements about how the deliverables will be met.  
I guess this is something on the order of stating what the meeting process 
will be.  Hence, interaction with the participants is a component of that 
process.

 d/
 ---
 Dave Crocker
 Brandenburg InternetWorking
 +1.408.246.8253
 dcrocker  a t ...
 WE'VE MOVED to:  www.bbiw.net

---

From: Alex Zinin <[email-masked]>
Date: Mon, 2 May 2005 05:43:06 -0700

Margaret, Dave, et al-

Based on the discussion started in the IESG, one thing we are going to try
to do in Paris is have a couple of smaller rooms with a different chair
setup--herringbone instead of the theater style, a couple more radio
microphones, and appropriate-size (smaller than huge) projector screens.

I'm working with the secretariat on this.

--
Alex

---

From: Eliot Lear <[email-masked]>
Date: Mon, 02 May 2005 14:49:15 +0200

Margaret,

The words I hate most when I am in a WG meeting are these:

   "take it to the mailing list"

That is usually short for "we can't agree in person so we'll now 
continue to disagree by email".  Debate has been shut off, and usually 
prematurely because there is something else on the agenda.  I'd rather 
that never happen.  I think it's fair to specify the parameters for a 
decision and then go to the mailing list so that people could evaluate 
different solutions based on those parameters, but simply blowing off a 
topic because the group cannot agree is a failure of leadership.

So, in answer to the question you asked Dave, I would agree with him 
about [1] and [2] in his message.  I don't fully understand [3].  I 
would go a bit further to say that the agendas should be approved by an 
AD.  Why?  Because it forces the AD to pay attention to the group.  No 
group should run on auto-pilot.  Any AD that cannot do this with little 
or no effort, should spend more time with the WG in question.  The AD 
gets to approve the order.  If agenda bashing shows that the chair 
missed something, then there was a failure on the mailing list, and 
corrective action should be taken to fix the problem.

I would not penalize a WG for not getting to the end of its agenda. 
That, in fact, is a call for an interim meeting, perhaps.

I would add one more thing.  We need whiteboards, ones with erasers.  it 
used to be that we had them years and years ago.  Being able to draw out 
solutions and list and reorder problems is a good thing.

So, a not so fictitious example:

The ISMS WG is currently struggling to choose between one of three 
architectures for integrated SNMP security models.  A call for consensus 
has been issued, and thus far there is none.  The reason there is none 
is that people do not yet agree on the underlying requirements, IMHO. 
This is all good fodder for an in person meeting.  If neither mailing 
list nor in person meeting can solve the problem, then the AD needs to 
step in and do something.

Prior to the meeting there should be a short summary of the issues, pro 
and con for each alternative, as well as proposed evaluation criteria. 
The meeting may be a good venue to expand or contract those criteria.

Eliot

---

From: "Steve Silverman" <[email-masked]>
Date: Mon, 2 May 2005 09:56:58 -0400

It seems to me that the fundamental problem is that most of the
meeting
has not read most of the drafts let alone the latest version under
discussion.
There is a fundamental IETF tenet that nothing is explained but there
is a false assumption
that the people in the meetings have read the drafts.  Whenever I've
seen the chair ask how many hve read
the draft, it is usually < 5%.  I think this is a key issue but the
solution is not obvious.  Nobody can
read the number of drafts that are issued for a meeting. Not even for
the subset of attended WGs.

Other organizations have proponents explain what they are proposing.
IMO this leads to a better quality of discussion.
But this limits the number of topics that can be worked on in a week
to far less than the IETF tries to cover.

Steve Silverman

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]]On
> Behalf Of Margaret
> Wasserman
> Sent: Sunday, May 01, 2005 10:10 AM
> To: Dave Crocker
> Cc: [email-masked]
> Subject: Re: improving WG operation
>
>
>
> This message was passed through
> [email-masked], which is a sublist of
> [email-masked]. Not all messages are passed. Decisions on
> what to pass are made solely by IETF_CENSORED ML
> Administrator ([email-masked]).
>

---

From: Keith Moore <[email-masked]>
Date: Mon, 2 May 2005 10:54:50 -0400

> It seems to me that the fundamental problem is that most of the meeting has not read most of the drafts let alone the latest version under discussion.

I think that's a symptom; a more fundamental problem is that WGs are trying to do too many things at once.

I've lost track of how many times I've seen a WG 

a) take valuable meeting time to have a presentation about a draft that is only peripherally related to the WG's current task
b) get a show of hands "how many people think this draft should be a WG work item?"
c) accept the draft as a WG work item without any discussion of whether doing so will affect the WG's ability to get other work done, or the WG's ability to give adequate attention to the work already accepted

Now there are certainly cases in which a WG needs to generate lots of documents in order to fulfill its mission.  But to the extent that new work items are identified in the manner described above, it probably indicates a lack of planning.  It should be possible to identify early on which topics need to be addressed by WG documents and which ones are either peripheral to the WG's mission or need to wait until the primary deliverables are completed.  The initial charter is generally too early to do this, but it would be reasonable for such a work plan to be one of the first deliverables of the initial charter.  Once that work plan is established, WGs need to push back on taking on additional work.  And the push back probably needs to come from the chairs, or if the chairs won't do it, the ADs.

Keith

---

From: John C Klensin <[email-masked]>
Date: Mon, 02 May 2005 12:21:35 -0400

--On Monday, 02 May, 2005 09:56 -0400 Steve Silverman
<[email-masked]> wrote:

> has not read most of the drafts let alone the latest version
> under discussion.
> There is a fundamental IETF tenet that nothing is explained
> but there is a false assumption
> that the people in the meetings have read the drafts.
> Whenever I've seen the chair ask how many hve read
> the draft, it is usually < 5%.  I think this is a key issue
> but the solution is not obvious.  Nobody can
> read the number of drafts that are issued for a meeting. Not
> even for the subset of attended WGs.

If it were true that no one can read the drafts relevant to work
they are actually materially concerned with (a slightly
different definition than yours), and I suggest it is not, then
WGs are trying to do too much, and handle too many documents, at
once.  Others have made that point.

As far as the 5% is concerned, we have, it seems to me, a choice:

	* We can decide to focus on the people who are doing the
	work and making real contributions.  If they have read
	the drafts, fine.  If most of them have not, then it is
	time to cancel the meeting after that show of hands.
	Those who are not usefully contributing don't count at
	all.
	
	* We can decide that the people who haven't done the
	reading shouldn't be in the room and either evict them
	or impose admission requirements for participation in a
	WG.  Note that many of the other groups to which you
	refer have such admission requirements, whether they are
	taken seriously or not.

> Other organizations have proponents explain what they are
> proposing. IMO this leads to a better quality of discussion.
> But this limits the number of topics that can be worked on in
> a week to far less than the IETF tries to cover.

It also, often, leads to much more superficial evaluation of
what is being standardized than the IETF has traditionally been
willing to tolerate.  Note that we still expect most work to be
done on mailing lists and between meetings, not in face-to-face
"no one things about this in between, then we get together and
try to make standards" meetings.  I think either model can be
viable, but they are different... and there are still
significant contributors to the IETF who have no set foot in a
face-to-face IETF meeting in years (if ever).

    john

---

From: John C Klensin <[email-masked]>
Date: Mon, 02 May 2005 12:30:00 -0400

--On Monday, 02 May, 2005 05:43 -0700 Alex Zinin <[email-masked]>
wrote:

> Margaret, Dave, et al-
> 
> Based on the discussion started in the IESG, one thing we are
> going to try to do in Paris is have a couple of smaller rooms
> with a different chair setup--herringbone instead of the
> theater style, a couple more radio microphones, and
> appropriate-size (smaller than huge) projector screens.
> 

Alex,

Given other discussions on this list, might I suggest that those
rooms also be equipped, if possible, with some mechanism by
which interactions within the group/meeting can easily be
recorded: Any of overhead projectors with foils and markers,
flip charts with markers, or whiteboards would all do the job
and all have been mentioned.   As others have pointed out, these
things can, in theory, all be done in powerpoint, but I suggest
that few of us are efficient enough with it to be able to do
pictures and serious mark-up in real time (independent of
whether we can transcribe text or not).

Or, if that isn't feasible for Paris, could it be considered for
Vancouver?

thanks,
     john

---

From: Dave Crocker <[email-masked]>
Date: Mon, 2 May 2005 10:09:49 -0700

>  As others have pointed out, these
>  things can, in theory, all be done in powerpoint, but I suggest that few
>  of us are efficient enough with it to be able to do pictures and serious
>  mark-up in real time (independent of whether we can transcribe text or
>  not).

A simple, ad hoc, and useful approach is to make sure someone takes a picture 
as legacy-technology renditions -- paper, whiteboard, whatever -- are created, 
and then uploads it or emails it to an agreed-to location.

It is procedurally a bit awkward, but already fits into existing practise, 
albeit not for this purpose.

And it does not require any formal support.

 d/
 ---
 Dave Crocker

---

From: "Tom Petch" <[email-masked]>
Date: Mon, 2 May 2005 18:21:37 +0200

Margaret

Choose the right chairperson (or having chosen one, train them).  I have heard
you speak of the roles of the WG chair (and of the editors of I-Ds) and think
your ideas are absolutely right.  The WGs that I find less effective are those
where the chairs, for all their undoubted engineering skills, are less effective
at the role of chairmanship of meetings, a role which starts with the agenda,
ends with
the minutes, and involves making the most effective use of the 'face time' in
between.

Tom Petch

---

From: Alex Zinin <[email-masked]>
Date: Wed, 4 May 2005 10:44:06 -0700

John,

  I was thinking about whiteboards too. I'll check with the secretariat
  if smth like this would be possible.
  Thanks.

-- 
Alex
http://www.psg.com/~zinin

---

From: "Marshall Eubanks" <[email-masked]>
Date: Wed, 04 May 2005 14:23:14 -0400

On Wed, 4 May 2005 10:44:06 -0700
>   if smth like this would be possible.

There are, of course, a number of companies which make products for
group interaction. Some that I like (and I have no connection with these) are
eLLuminate, which makes a nice java based platform for sharing information on a desktop
(we tested this a while back on Windows, OS X and Linux with good results)
and Smart Technologies (which makes a set of really cool smart whiteboards).

Of course, there is a learning curve for anything, and none of this is free. Even if we used 
VRVS, which we could probably get from CalTech for free, IMHO it would be
a full time job for someone to get and keep all of this running for every WG at an IETF meeting.

It might be time to revist some of these tools for interactive collaboration. Smart is a
Canadian company; maybe we should get them as a sponosor for IETF 64.

Regards
Marshall Eubanks

---

From: Joe Touch <[email-masked]>
Date: Thu, 05 May 2005 09:12:19 -0700

Keith Moore wrote:
> a) take valuable meeting time to have a presentation about a draft
> that is only peripherally related to the WG's current task

How many of those have been at the suggestion, or _insistance_, that an
individual or other WG's work be 'checked' in that WG?

> b) get a show of hands "how many people think this draft should be a WG work item?"

The art of asking a survey question is key; there have been a number of
recent docs accepted as standards-track simply because issues of "should
this even be a WG issue", "is this the best doc to lead that issue", and
"should this doc be informational, BCP, or standards-track" were all
asked at once.

> c) accept the draft as a WG work item without any discussion of
> whether doing so will affect the WG's ability to get other work done, or
> the WG's ability to give adequate attention to the work already accepted

Or whether it is the WG or the IESG that has the real interest in the
area of work. When a doc hasn't even been read by a handful of people -
even after _multiple_ requests _at_ repeated WG meetings, it's amazing
when the result is a call for decision on what to do.

I've said it before: sometimes silence speaks for itself.

Joe

---

From: Keith Moore <[email-masked]>
Date: Thu, 5 May 2005 12:57:51 -0400

> individual or other WG's work be 'checked' in that WG?

I've never seen an AD insist that a WG devote valuable face-to-face meeting
time to "checking" work that was peripheral to the WG's interest.  OTOH,
I have seen WGs saddled with trying to make some other group's work into
something sane - it's a thankless job, but sometimes a necessary one.  
(mDNS comes to mind most readily here).  

> when the result is a call for decision on what to do.

Why is that amazing?  Yes, sometimes silence speaks for itself, but 
there's nothing wrong with asking the question - so long as lack of
response isn't taken for "yes".

Keith

---

From: Joe Touch <[email-masked]>
Date: Thu, 05 May 2005 10:26:29 -0700

Keith Moore wrote:
> time to "checking" work that was peripheral to the WG's interest.

Check again, please. I personally have been asked to take items to WGs
that I've already presented them to repeatedly - even at the meeting
adjacent to a Last Call.

> OTOH,

Sure - and that's where the process works. Since it's hard to know that
ahead of time, why are you complaining about the check?

> response isn't taken for "yes".

That's the amazing part - the question was something akin to "any
objection to taking this as a WG item?"

Joe

---

From: Keith Moore <[email-masked]>
Date: Thu, 5 May 2005 14:04:35 -0400

> that I've already presented them to repeatedly - even at the meeting

Okay, so maybe that was a botch.  But surely you can find a quicker  
and more effective way to remedy that botch than by whining about it  
endlessly here?  And if you couldn't figure out how to do that by  
yourself, why couldn't you ask some people with more experience  
working in and/or with IESG?

(and did the AD really insist that you bring this up in a _face-to- 
face_ WG meeting, or is that just how you and/or the WG chair chose  
to interpret it?)

Why is this one botch evidence of such a fundamental problem with the  
IETF process that it needs to be altered in a way that there's plenty  
of reason to believe will work far worse than what we have?

Keith

---

From: Joe Touch <[email-masked]>
Date: Thu, 05 May 2005 11:14:33 -0700

Keith Moore wrote:
>>> I've never seen an AD insist that a WG devote valuable face-to- face
>>> meeting
> Okay, so maybe that was a botch.  But surely you can find a quicker  and
> more effective way to remedy that botch than by whining about it 
> yourself, why couldn't you ask some people with more experience  working
> in and/or with IESG?
> 
> face_ WG meeting, or is that just how you and/or the WG chair chose  to
> interpret it?)

What's the difference if it eats time you perceive as wasted post-facto?

> of reason to believe will work far worse than what we have?

It isn't - the point is that wasting valuable face-to-face time at WGs
doing cross-area checks is one of the points of the face-to-face
meetings. Whether time is wasted is easy to assert post-facto, but short
of avoiding cross-area review and entrusting it solely to the mythical
"omniscient, wise, and prudent AD", what's the alternative to erring on
the side of wasting time?

Joe

---

From: John C Klensin <[email-masked]>
Date: Sat, 07 May 2005 09:54:18 -0400

--On Wednesday, 04 May, 2005 14:23 -0400 Marshall Eubanks
<[email-masked]> wrote:

>...
> It might be time to revist some of these tools for interactive
> collaboration. Smart is a Canadian company; maybe we should
> get them as a sponosor for IETF 64.

Now _that_ would be way cool.

    john

---

From: Dave Crocker <[email-masked]>
Date: Sat, 7 May 2005 12:08:50 -0700

>  Now _that_ would be way cool.

maybe.

however,  rather than our starting with an interesting in some set of tools 
that fall into an interesting, albeit large and vague -- area, perhaps we 
should figure out what we need to do that we can't reasonably do now.

tools have their own expense, and we already have quite a few.  adding useful 
ones would be... useful.  adding ones that are not highly useful would be more 
hassle that they are worth.

given email, jabber and multi-casting voice/video, what additional 
collaborative processes might be helpful for getting IETF face-to-face 
meetings more productive? 

i believe this sub-thread was triggered by an interest in something like a 
real whiteboard that could be digitally captured and shared.  Is that the 
function we need a tool for?  are there others?

  d/
  ---
  Dave Crocker

---

From: John C Klensin <[email-masked]>
Date: Sat, 07 May 2005 15:24:41 -0400

--On Saturday, 07 May, 2005 12:08 -0700 Dave Crocker
<[email-masked]> wrote:

>...
> given email, jabber and multi-casting voice/video, what
> additional  collaborative processes might be helpful for
> getting IETF face-to-face  meetings more productive? 
> 
> i believe this sub-thread was triggered by an interest in
> something like a  real whiteboard that could be digitally
> captured and shared.  Is that the  function we need a tool
> for?  are there others?

Yes, real whiteboards (or equivalent) were what started the
subthread.   And yes, ones that could be captured and shared
would, IMO, enhance the process, especially for remote
participants, and especially since we have largely dropped video
(which might permit remote participants to look at on-screen
materials).

I could, of course, be wrong, but I don't know how to find out
without trying it.  And I responded strongly to Marshall's
suggestion because those digital-capable whiteboards are not
cheap and, unlike VGA projectors, are hard or impossible to
obtain from the local rent-a-computer or hotel AV facility.

Whether it is worth the experiment --and I acknowledge that
experiments, like tools, have their own costs-- I'm happy to
leave to whatever community consensus emerges.

    john

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 09 May 2005 17:09:28 +0200

(catching up on old stuff)

Eliot Lear wrote:
> continue to disagree by email".

Sometimes it's short for "We are out of time." In fact, I think that's
the common case. Other times it's "We here today can't form a consensus,
but in the IETF consensus is formed on the list anyway, so let's
try that way when we've all had time to think."

> Debate has been shut off, and usually 
> that never happen.

Yes, but it is bound to happen, unfortunately.

> I think it's fair to specify the parameters for a 
> decision

Even that may be non-trivial. But I agree, if it can be done in the
time available, it should be.

> and then go to the mailing list so that people could evaluate 

I think it's rare; do you think it's frequent?

    Brian

---

From: Pekka Savola <[email-masked]>
Date: Mon, 9 May 2005 22:27:18 +0300 (EEST)

On Mon, 9 May 2005, Brian E Carpenter wrote:
>> That is usually short for "we can't agree in person so we'll now continue 
>> to disagree by email".
>

Sometimes I have come to treat "take it to the list" as:

  "we don't seem to have enough people in the room that actually care 
about this, as nobody comes to say anything on the mike.  We'll just 
put the document (or the issue) on the pending list until anyone gets 
interested on the list, or until the next meeting when we'll have this 
discussion again."

Yeah, that's a problem, but it's not clear which is better: the chairs 
more aggressively pursuing a timeliness (even though sufficiently many 
people don't seem to be paying attention or voicing strong opinions) 
so that issues or documents can be finished, or intentionally delaying 
work which hasn't garnered sufficient momentum (or the momentum has 
been lost).

My personal take is that it's very important to finish the work that 
has been started on a timely fashion, but we might be more strict on 
accepting new work if we see a loss of interest.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Aaron Falk <[email-masked]>
Date: Tue, 17 May 2005 17:27:22 -0700

--On May 1, 2005 9:04:12 AM -0700 Dave Crocker <[email-masked]> wrote:
>
> in general I think the issue is stricter meeting planning and
> management, where the goals and process are more explicit.

Sorry for coming very late to this discussion.

I might suggest scheduling a (preferably voice) conversation between the AD 
and the chairs of each working group before an IETF meeting to discuss the 
issues and objectives surrounding the next wg meeting.  I'm finding that 
voice chats with IRTF RG chairs have been very useful for me to sync up on 
the RG activities and to give some nudges on moving forward.

--aaron