Subject: RE: New root cause problems?

From: "Hollenbeck, Scott" <[email-masked]>
Date: Tue, 24 May 2005 09:18:57 -0400

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Bruce Lilly
> Sent: Tuesday, May 24, 2005 8:59 AM
> To: [email-masked]
> Cc: Margaret Wasserman
> Subject: Re: New root cause problems?
> 
> On Tue May 24 2005 08:35, Margaret Wasserman wrote:
> > 
> > 
> > 
> > Hi Bruce,
> > 
> > At 5:57 PM -0400 5/10/05, Bruce Lilly wrote:
> > >OK, I'll bite -- where are the statistics (I know of one 
> WG that has been
> > >active for more than 8 years and has set to produce a 
> single document for
> > >IESG review; that's got to skew the statistics a bit)?
> > 
> > What WG is that?
> 
> http://www.ietf.org/html.charters/usefor-charter.html
> 
> N.B. the "charter rewrite/update is underway" has been there
> for years -- it is NOT related to the missed Apr 05 milestone.
> 
> I'm still interested in seeing those statistics...

I can't talk to whatever went on in USEFOR prior to 2004, but I have
appointed two new co-chairs to help the group evaluate and meet it's
goals.  Something (either document production or a working group action)
should happen within the next few months.

-Scott-

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Bruce Lilly <[email-masked]>
Date: Tue, 24 May 2005 10:22:20 -0400

> should happen within the next few months.

I should clarify a few points:
1. sorry for the typo in the original "set" should be "yet"
2. the intent was not to cast aspersions on the cognizant ADs (past
   or present) or WG Chairs (past or present), or to discuss possible
   causes for the situation -- indeed, obviously I didn't even name
   the WG until asked -- but merely to observe that the history of at
   least one WG seemed to be out of whack with what was asserted was
   shown by some statistics.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 25 May 2005 14:05:24 +0200

>    shown by some statistics.

I'd like statistics too. To be slightly provocative, if the average
times are forced upwards by a long tail of WGs/drafts/RFCs that
take extremely long times to get done due to one-of-a-kind reasons,
it would seem fair to remove thoses cases from consideration. We should
probably concentrate on causes of delay that affect the mode of the
distribution, not the tail.

     Brian

---

From: Dave Crocker <[email-masked]>
Date: Wed, 25 May 2005 15:39:51 +0200

>  o be slightly provocative, if the average
>  it would seem fair to remove thoses cases from consideration.

use the median, rather than the mean.  

  d/
  ---
  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

---

From: Steve Crocker <[email-masked]>
Date: Wed, 25 May 2005 10:05:27 -0400

In addition to choosing the right way to look at the distribution of 
total action times, I strongly recommend breaking down the transactions 
into component parts and looking at the details.  The exchange I had 
with Sam Hartman was a good example.  On 5/23. Sam wrote:

> I'd think two months would be doing good for IESG processing.  That
> includes AD review, IETf last call, telechat delays and a bit of slop
> for interaction with the authors.  However that's time to travel
> through the queue, not actually time spent on that document.
> 
> Because of the delays involved with telechats and last call, getting a
> WG document done in less than a month of wall time or an individual
> document done in less than 1.5 months is very unlikely.

What catches my attention -- and brings shudders as I remember my time 
as an AD -- is that he paints a picture that looks like

T[IESG] = T[AD] + T[last call] + T[telechat delay] + T[authors]

Now each of these terms has quite different characteristics.  The last 
call and telechat delays add up to a few weeks, to the overall time is 
at least a couple of months.  But those terms probably have moderately 
low variance.  On the other hand, the T[AD] and T[authors] terms have 
very low minimums but extremely high variance, and it's within those 
terms that the real issues emerge.

I suspect if you break those terms down even further, interesting and 
useful dynamics will emerge.

I'm certainly not picking on Sam but merely using his responses as a 
good clue for untangling the separate effects.

In a subsequent note, in regard to T[AD], Sam said

> It depends a lot on a document.  I can often do a document in in two
> hours if it is reasonably short and I understand the technology and
> the document quality is good.  I have one document languishing
> somewhat in my queue because I need to block out an entire day for it
> and finding a full day to work on one document is hard.
> 
> 
> Keep in mind that AD review can easily have round trips with the
> authors.
> 
> Also, as you are well aware, finding the time among all the other
> things is difficult.

To me, this suggests it would be useful to try to identify the issues 
that make it easy and quick to process some documents, but harder and 
longer to process other documents.

There may be some issues related to just the quantity of work, but I 
suspect the bigger issues are qualitative and we may learn something if 
we probe more deeply and not just study the distributions of the overall 
processing times.

Steve

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 25 May 2005 16:31:41 +0200

> use the median, rather than the mean.  

I'm actually more interested in the mode, I think, but first,
some actual data would be good.

    Brian

---

From: Ned Freed <[email-masked]>
Date: Wed, 25 May 2005 09:56:36 -0700 (PDT)

> T[IESG] = T[AD] + T[last call] + T[telechat delay] + T[authors]

I see two problems with this formulation:

(1) T[AD] is in fact composed of several separate things, and the
    differences between them are almost certainly critical in understanding
    process delays. In particular, AD time is spent in several different ways:

    (a) Initial document review (done by all ADs)
    (b) AD followups (done by the sheparding AD)
    (c) AD rereview and discussion to clear discussses (done by all ADs)

    There are timeouts associated with (a) - once a document is on the
    IESG's agenda, ADs are required to have their review done by the next
    telechat. They can get an extension, but only once. This sohuld therefore
    end up being similar to telechat timinig.

    There are, however, no timeouts associated with (b) and (c). And more
    to the point, the pressures on sheparding AD are quite different than
    those on the other ADs to act in a timely fashion.

    My experience has been that the distribution for (b) is bimodal (or
    trimodal if you count the case where the document just passes without
    any discusses). In some cases the sheparding AD only has to 
    pass on the discuss comments to the authors, done. In others, however,
    the AD has to spend a bunch of time in effect figuring out what
    amounts to a compromise. This can take a lot of time.

    In my experience (c) is the truly problematic case. In my experience
    it is both highly variable and the average time taken seems to be much
    longer than it should be. Additionally, as sheparding AD I often found
    it was only possible to clear discusses by explicitly bringing the
    document up on a subsequent telechat, so there's a periodicity here as
    well.

    I have in the past advocated putting a timeout on (c), and I continue to
    think this is a really good idea.

(2) When I was an AD I went to a good deal of trouble to overlap the
    various phases of the process whenever possible in order to save time.
    Nothing prevents ADs from entering comments early. Nothing prevents
    documents from being revised during last call. I don't think I was
    terribly successful in this, but much of my time as AD was spent without
    the tools we have now. I therefore think the additive nature of your
    formula has to be characterized as a worst case, and it needs to be
    noted that substantial gains can be had by overlapping terms.

> terms that the real issues emerge.

Agreed.

> useful dynamics will emerge.

Exactly so. Breaking them down further is IMO essential to getting
anything useful out of the analysis.

				Ned

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Thu, 26 May 2005 02:50:12 +0200

Brian,
you asked about non documented root case problems. The main one I see is 
the lack of common identification of the ultimate purpose of the network 
itself. Most in the IETF target a good "end to end" interoperability. Some 
also target good "technology to technology" interconnection. But the idea 
that the Internet technology is only another way to help the people in 
their daily relations seems to be missing. And the idea that languages are 
human communications protocols to be supported the same as HTTP or SMTP is 
a few parsec away.

I will not dwell on the resulting lack of users participation (those who 
will use the IETF deliverable): there always is someone to say the IETF is 
not here to address users needs but that the Internet standard process is 
only to document a consistent set of propositions of its Members. And that 
the market will freely decide to use or not. This is addressed by RFC 3774.

But I will mention the technical architecture limitations it leads to. When 
compared with other network technology implementations, the Internet 
current system is often characterized by architectural parameters defaulted 
to one. This certainly works well as KISS. If all the users are equal and 
the same. But it appeals only standardizers who wants to default to that 
single conception. Who have a limited or no interest  and no culture about 
diversity. With the consequence that they do not appeal on people with a 
diversity of ideas: they go for grassroots processes.

This translates into a single IANA, a single root, a single IPv4 tree, a 
single IPv6 plan, a single language, a single network culture. And an 
experimented singular difficulty to address real life issues.

As you know, today there is the support of the ccTLDs specific issues, 
there will be soon the language equal opportunity support, then the 
reference center distribution, then the middle box and network intelligence 
integration, etc. Each time there are a few members to say "the system can 
do it if you set the parameters to two, or even to 256", like for TLDs, but 
nothing moves, nothing is tested, nothing becomes a proven standard.

A typical example is the ICANN ICP-3 Part 5, It calls for DNS 
experimentation. Experimentation of classes (current parameter is 2, the 
second class being "CHAOS") and of the possible evolution towards a non 
single authoritative root file. It quotes the IETF as a possible technical 
leader. IETF was never interested (what makes ccTLDs to ask now where they 
should go for the results).

Another example is the technology monolingualism. It limits contributions 
from non-English speaking people, who develop their solutions by their own. 
This worrying because it may affect the network stability. It aslo limits 
the conceptual and sometimes the operational scalability of the technology. 
Mechanisms making the technology independent from languages are missing. 
The logic, libraries, experience of their deployment are missing. This will 
make an evolution towards the Multilingual Global NGN more complex.

This root cause "monolithic" problem is very well documented, starting by 
the mere fact it was not identified.
jfc

---

From: Bill Fenner <[email-masked]>
Date: Wed, 25 May 2005 21:09:39 -0700

My historical data on document progress in the I-D tracker, collected
since the beginning of 2003, is available at
https://rtg.ietf.org/phpmyadmin username ietf password ietf; pick
database trackerdata and table dochistory.  (Ignore the permission
denied error; go straight for the table list on the left or the SQL
tab on the top of the right pane)

A starter query to somewhat understand the table relationships is

SELECT docname, changedate, statename, substatename
FROM dochistory, docs
LEFT JOIN states ON stateid = newstate
LEFT JOIN substates ON substateid = newsubstate
WHERE docs.docid = dochistory.docid
ORDER BY dochistory.docid, histid;

(This will return some NULL transitions where something other than the
substate or substate transitioned, like the version number, etc.)

I don't have any great ideas on how to do the analysis, but at least I
have some data to supply.

  Bill