Subject: Moving forward on IETF problems

From: Bruce Lilly <[email-masked]>
Date: Mon, 2 May 2005 09:04:21 -0400

This has been an interesting discussion, but somewhat one-sided.
Authors, editors, and WG Chairs and participants tend to believe
that they produce perfect output, but that is demonstrably not
always true.  Although the IESG supposedly provides a quality
control, it's clearly not working.  Rather than vague assertions,
I'll mention a couple of specifics: RFC 4021 and draft-fax-esmtp-conneg.

RFC 4021 has been published as Standards Track. In accordance with
RFC 2026, it contains a section referencing RFC 2119 and the latter's
keywords which indicate levels of requirements.  However, none of
those keywords appear anywhere in the document but the section stating
that 2119 in fact defines such keywords!  Moreover, the document
does not impose any protocol or format requirements and does not lend
itself to any implementations (its purpose is to document items being
included in an IANA registry); the Standards Track is the wrong
category for such a document -- there is no way that it can advance
to Draft Standard.  This is a spectacular and surprising instance of
complete failure of the authors, the IESG, and the RFC Editor to
control quality.  It is surprising and spectacular because the
document structure is so simple and its purpose evident.  There is
no way that the IESG and RFC Editor should have categorized that RFC
as Standards Track.

There are many issues with the draft mentioned above; despite having
ABNF expertise among the authors, there are at least a half-dozen
specific problems with the ABNF.  The document lacks discussion of
interaction with directly-related protocols (ESMTP submission, e.g.),
has no BCP 90 registration forms for message header fields defined,
no discussion of internationalization issues, and an inadequate
IANA considerations section.  The draft has been through at least
one Last Call (version -09 and possibly also version -02), and has
been approved as a Proposed Standard (-13) despite the known technical
omissions, the vagueness which precludes the specification from
being well-understood, etc.  This is another failure of quality
control; the document does not meet the minimal requirements for
Proposed Standard (RFC 2026 section 4.1.1).

There are clear requirements for the Standards Track; apparently
the authors, WGs, the IESG and the RFC Editor aren't taking the
necessary time to review documents for compliance with those
requirements.  I have separately (NEWTRK mailing list) suggested
that those requirements should be distilled into a checklist, and
have further suggested that some tolls might be capable of
highlighting specific problems such as the RFC 4021/2119 issue.

Clearly, I'm bemoaning the lack of quality of recent work; others
have complained about timeliness.  Still others have mentioned
relevance and comprehensibility.  There have been some allusions
to cost (in the form of increased manpower for review).  There is
a well-known engineering issue:

   o Quality
   o Timeliness
   o Cost
   Pick any two.

Part of the problem with time/quality tradeoff is that many drafts
are in such a poor editorial state that it impedes review.  One
possible way to expedite the technical review would be to ensure
that documents are edited for readability and comprehensibility
prior to IESG review and Last Call.  That may involve cost (somebody
has to provide editorial advice).  Or the IESG could simply push
back documents of poor editorial quality (ideally with a list of
resources -- currently scattered among RFCs, presentations, semi-
official documents (ID-guidelines, ID-Checklist, ID-nits), etc.).

RFC 3774 has been mentioned in this discussion; that document
and RFC 3844 discuss some problems and potential solutions, but
do not address the generally poor quality of documents entering
the review stage.  Some of the issues identified have been partially
addressed (e.g. RFC 3935 provides the mission statement mentioned
as lacking in section 2.1 of RFC 3774 -- but the IETF web site
makes no mention of the IETF mission and has no direct link to the
mission statement).  3774 also bemoans a perceived lack of assessment
based on practical experimentation, ignoring the fact that most
RFC categories specifically provide for such practical experimentation
(i.e. the Standards Track plus Experimental (and Historic for failed
experiments)).  3774 also makes the extraordinary claim that the
quality bar has been raised.

As mentioned above, a part of the quality problem can be attributed
to lack of organization of relevant material in a single place. The
NEWTRK "ISD" effort might be able to help -- but as noted above, many
of the relevant information is in documents that are not RFCs, and
which may have no clear official status, and moreover, which may
change frequently.  However, there is a danger that the benefit of
consolidating pointers to related information may be diluted by
incorporation of unrelated agendas (specific document format
issues), and the danger of causing a further deterioration in
quality (by weakening the iterative improvement process embodied
in progression along the Standards Track).  And it's not at all clear
that making information more readily accessible will improve the
quality of documents; the examples of quality control failures
given above cannot be attributed to ignorance of relevant requirements.

Some relevant information could be more widely disseminated with clear
benefit; for example, some boilerplate pointing to the RFC Editor's
errata page could be included in every published RFC, informing
readers of the mechanism for documenting relevant updates.

One problem is that the IESG routinely sabotages development along
the Standards Track by disbanding WGs as soon as a PS is produced,
leaving nobody to do the work necessary for advancement to Draft.
Charters should probably explicitly provide for WG activity leading
at least to Draft Standard (or to Historic if the necessary two
independent implementations fail to develop within a reasonable time).

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 04 May 2005 17:18:59 +0200

Bruce Lilly wrote:

(Interesting thoughts read and deleted)
...
> One problem is that the IESG routinely sabotages development along

That is, I think, an inappropriate choice of word.

> leaving nobody to do the work necessary for advancement to Draft.

Er, we don't terminate the people. And we have always been advised
by the community to avoid eternal WGs.

> independent implementations fail to develop within a reasonable time).

That assumes we care about moving stuff to DS. That wasn't at all
obvious during the discussion of 2774 and is by no means obvious
in the newtrk discussions.

    Brian

---

From: [email-masked]
Date: Wed, 04 May 2005 09:48:25 -0700 (PDT)

> Er, we don't terminate the people.

No, but WG termination often terminates people's interest in further work. And
even if the people continue to be interested, the folks they work for often
balk at continuing to support the work in the absence of a formal standards
activity.

> And we have always been advised

Some members of the community, myself included, have always had doubts about
wisdom of apply this principle blindly and unconditionally.

> in the newtrk discussions.

The often-heard plaint that "nobody cares about moving to DS or S" has always
struck me as confusing cause with effect. Like it or not, the message we send
by doing things like close WGs the moment their docs come out at proposed, by
setting the bar for rechartering groups to deal with DS issues quite high, and
in many others ways says to people that "worrying about DS is a waste of time". 
Change the message we send and people will start to care; don't change it and
they won't. Pretty simple, really.

				Ned

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 4 May 2005 13:40:23 -0400

> That is, I think, an inappropriate choice of word.

No offense intended, and it should not be construed as indicating volition,
but I can't immediately think of another word that adequately describes
 
> Er, we don't terminate the people.

Of course not.  But RFC 2026 assigns the responsibility for documentation
of the two fully interoperable, independent implementations required for
advancement to Draft to the Working Group Chair (sect. 4.1.2, 3rd paragraph).
It's unclear who would do that, along with the associated testing and
production of a protocol action request after the Working Group has been
disbanded.  Moreover, Proposed Standards rarely include a concise, well-
defined set of conformance criteria that would facilitate testing and
production of the requisite documentation in the absence of a panel of
experts such as the WG that devised the specification.  It seems to me
that if the IESG is going to disband WGs working on Standards Track
documents prior to production of Draft (or movement to Historic), that
it ought to ensure that there is such a set of criteria in Proposed
Standards so that the testing and documentation can proceed in the
absence of a panel of experts; but it has not done so.  Otherwise, how
can the IESG expect that there will be an advance?  How will the IESG
fulfill its obligations for RFC 2026 section 6.2 (last paragraph)
obligations?  If the answer to either question is "it cannot", then how
would you describe the practice of disbanding WGs with neither a Draft
Standard nor an objectively testable set of conformance criteria?

It is a fact that after 24 months, former WG chairs move on to other
responsibilities and/or interests, and Document Editors may be
unwilling to approve errata and/or to provide clarification or other
updates after a WG has been disbanded.  Terminating the people isn't
necessary; these facts of life have the same result, at least in some
specific observed cases, and the 2418 provision for a dormant WG explicitly
provides a mechanism to address those issues.  When the IESG drags out
the review schedule beyond the 24 month deadline, former WG Chairs and/or
Editors may no longer recall details of WG discussions and decisions,
exacerbating the problems.
 
> by the community to avoid eternal WGs.

I see no reason why a time limit could not be stated.  RFC 2418 (section
4, first paragraph) explicitly provides for a WG to go dormant, rather
than be disbanded, pending advancement along the Standards Track.  As
2026 provides for explicit review at 24 months after Proposed, and at
regular intervals thereafter, those would seem to be appropriate values
for charter timeline milestones, with progress review and a decision
re. rechartering as a means of implementing the 2026 review schedule
(which hasn't been followed in practice).  That's hardly "eternal"; at
some point either full Standard status is reached and the WG can be
disbanded as the ultimate goal of the Standards Track has been reached,
or the WG can be rechartered to amend the specification in such as way
as to permit advancement (e.g. if there is some unimplemented provision
as noted in 2026 sect. 4.1.2 2nd paragraph) or for a brief period to
  
> in the newtrk discussions.

We ought to care about the issues mentioned in 2026 sect. 4.1.2 for DS:
demonstration of community interest and utility (two independent fully
interoperable implementations), removal of unused or unimplemented
features (cruft; as noted in the NEWTRK discussion, that removes potential
security and interoperability loopholes), and clarification of the
specification based on field experience.  If we don't care about utility
and community interest, it means that we're likely wasting effort on dead
ends.  If we don't care about security and interoperability, we're headed
for serious trouble.  And if we don't care about clear specifications,
there's not much point in being an SDO.   "The goal of the IETF is to
make the Internet work better" -- the IETF Mission Statement, RFC 3935 --
certainly relates to utility and interest, as does its definition of
"Relevant".  The definition of "Quality" in the Mission Statement directly
relates to the other points.

3774 notes (Introduction) that the comments started on the wgchairs
mailing list.  It is somewhat to be expected that some of those closely
involved with the work believe that they have produced perfection (we
certainly wouldn't want them to knowingly produce shoddy work); see
slide 59, first bullet point of
ftp://ftp.rfc-editor.org/in-notes/rfc-editor/tutorial62.pdf
for example.  Now if some of those closely involved in work believe that
they produce perfection in one go, it's hardly surprising that they do
not see value in the multi-step standards track.   The history of
specifications which have advanced to full Standard status clearly
shows that there is in fact progressive modification as a specification
progresses.  The fact that there are errors in published Standards Track
specifications (http://www.rfc-editor.org/errata.html) is another data
point that indicates that specifications rarely emerge fully-baked and
perfect.  From such a wider perspective it seems clear that progression
is valuable; maybe 3 stages isn't the right number (but I suspect that
it's about right -- if anything, the DRUMS experience tells us that there
is sometimes a need for a 4th stage (consolidation of extensions and
amendments).  In that specific case, the relevant Standards have been
rewritten and published back at entry level (Proposed Standard), the
WG has been disbanded, and after more than 4 years -- double the 24-month
review period specified in RFC 2026 -- there is no Draft Standard for the
core application protocols addressed.  There are known errors in the
Proposed Standard specifications, some documented; others remain
unpublished and known to a small group (lower-case 'g').  Meanwhile,
there have been still more extensions and amendments, with more apparently
to come (depending on what happens with various drafts).  Perhaps in
another decade or two, these core protocols will have advanced to
Standard again, by which time there will again be a large number of
amendments and extensions, resulting in yet another cycle of consolidation
and progression.  I submit that the practices of prematurely disbanding WGs
and then dragging out the review schedule are the primary causes of the
lack of timeliness of production of Standards (and as a side note, there
is no reason -- with a relevant issue and an active WG to do the work --
that progress cannot be made much faster; progression to Draft can occur
in as little as 6 months, and to full Standard in another 4 months per
2026 section 6.2).

There is also a degree of inconsistency in the way that revisited Standards
are addressed; the rfc-index states that 821, a.k.a. STD 10, has been
obsoleted by 2821 (a Proposed Standard); likewise for 822 (STD 11) and 2822.
The std-index, however, still points to the earlier, supposedly obsoleted,
RFCs.  In other cases where a full Standard is being reformulated, the
std-index (e.g. for STD 12) has a reserved placeholder.  It's unclear why
there is such an inconsistency, but there does seem to be some confusion
about the effective status of 821, 822, 2821, and 2822.

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 4 May 2005 13:47:28 -0400

> That is, I think, an inappropriate choice of word.

OK, s/sabotages/undermines/

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 09 May 2005 15:41:56 +0200

[email-masked] wrote:
...
> in many others ways says to people that "worrying about DS is a waste of 
> time". Change the message we send and people will start to care; don't 
> change it and

Oh, that's true enough. So we need newtrk to reach a consensus on
what the message should be. I have a radical personal view on that,
but that discussion belongs on the newtrk list.

      Brian

---

From: Allison Mankin <[email-masked]>
Date: Mon, 09 May 2005 06:58:23 -0700

Brian, and others,

I do have experience of WGs that care about DS.

Preparing implementation reports has been viewed as
important for the protocol.

TSV has several in that process now.

Allison

---

From: [email-masked]
Date: Mon, 09 May 2005 07:00:50 -0700 (PDT)

> I do have experience of WGs that care about DS.

So do I. But this has not been the norm in my experience.

				Ned

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 10 May 2005 13:44:08 +0200

> So do I. But this has not been the norm in my experience.

If you think back to when we were discussing a one-stage
standards track in newtrk, I at least was arguing for the
importance of interoperability reports even in a one-stage
system. The value in the DS label is surely the knowledge
that an interop report has been presented and accepted. And that's
why I conceded that I could live with a two-stage process
(although I still think one stage plus interop reports would
be just fine).

    Brian (speaking only for myself)