Subject: RANT: posting IDs more often -- more is better -- why are we so shy?

From: Michael Richardson <[email-masked]>
Date: Wed, 01 Mar 2017 11:04:26 -0500

For the third time in two days I find myself, when asking others for opinions
about some text, pointing at github commit logs.  With the beautiful
makefiles we often have, one can't even depend upon having a formatted .txt
version there!

This is not a rant for or against git or github, but rather about what I
perceive as a shyness about posting intermediate versions of Internet Drafts
to the datatracker.

I understand that in academia, they never like letting half-baked ideas out,
and so the -00 that we see from academics are often overdue and overly
polished.  I know I can't fight that, but at least the -01, etc. could be
issued faster?

I've even heard some push back from people along the lines of, "wow, that ID
has 27 revisions, is it really stable?", and my feelings have often been more
along the lines of, "wow, that revision has 27 revisions, the authors are
really keen and responsive".

I appreciate for some reviewers that having more revisions implies that they
think they have to look at the text more often.

But given the diff utilities, it shouldn't matter how many revisions there
were before times you look, as you can just skip the intermediate versions.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Kathleen Moriarty <[email-masked]>
Date: Wed, 1 Mar 2017 11:09:45 -0500

> were before times you look, as you can just skip the intermediate versions.

+1 and I'd rather save subsequent reviewers from picking up on the
same issues that have already been identified.  I do have WGs post
revisions at several stages once it gets to AD review & IESG
processing for that reason and versions are cheap ;-)

-- 

Best regards,
Kathleen

---

From: Patrick McManus <[email-masked]>
Date: Wed, 1 Mar 2017 12:07:11 -0500

> makefiles we often have, one can't even depend upon having a formatted .txt

You can fix this with better tooling within github and travis integration
fwiw. Checkout the httpbis github https://github.com/httpwg/http-extensions
- you'll see the for each adopted draft the README.md has links to the
"editor's draft" in both txt and html as well as the working group draft.
The former is auto-generated on every push to the repo, the latter is a
pointer to datatracker.

I'm not arguing against updating data tracker more often - just saying this
'editor's draft' convention can work very well between official revisions
no matter the cadence a WG chooses.

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 2 Mar 2017 08:24:41 +1300

> no matter the cadence a WG chooses.

The details of that discussion probably belong on [email-masked],
but I must point out that this way of working *excludes* from the 
discussion WG participants who don't grok github. Substantial issues
need to be discussed on the mailing list and substantial (non-typo)
revisions need to be posted as I-Ds.

    Brian

---

From: [email-masked]
Date: Wed, 01 Mar 2017 12:05:34 -0800 (PST)

> > > For the third time in two days I find myself, when asking others for
> > > opinionsabout some text, pointing at github commit logs.  With the beautiful

...

> revisions need to be posted as I-Ds.

+1 on all points. Additionally, I've observed a seeming reluctance to update
drafts for small stuff, independent of the use or non-use of github.

Cases where it's actually best to withhold an update are quite rare, so having
this practice as a default really needs to go away.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Wed, 01 Mar 2017 15:22:17 -0500

--On Thursday, March 2, 2017 08:24 +1300 Brian E Carpenter
<[email-masked]> wrote:

>...
>> I'm not arguing against updating data tracker more often -
>> just saying this 'editor's draft' convention can work very
>> well between official revisions no matter the cadence a WG
>> chooses.
> 
> The details of that discussion probably belong on
> [email-masked], but I must point out that this way
> of working *excludes* from the  discussion WG participants who
> don't grok github. Substantial issues need to be discussed on
> the mailing list and substantial (non-typo) revisions need to
> be posted as I-Ds.

Brian,

It is a bit more than "not grokking".  For any given WG, or
task/document within a WG, there are three plausible ways to
review an emerging document:

(1) Very active participation, including tracking and
understanding all changes, more or less in real time.  

(2) Intermittent review of snapshots, ones that are generally
believed by their editors to be coherent and self-consistent.
Github revision tracking often makes that approach very
burdensome; change logs in documents and diffs between versions
are often more helpful.

(3) Simply deciding it is all or nothing and waiting until IETF
Last Call.

Only the first really makes effective use of the github style of
doing things.

The recent discussion about comments at Last Call rather than
while the WG was actively developing a document, including the
"don't have time" subthread obviously interact with this.

Also, IANAL, but if a change were contemplated, it might be
useful to ask appropriate ones whether, if questions ever arose
about when particular things happened, it would be easier or
more effective to explain an I-D and related WG consensus
discussions to a judge or jury or whether github editing traces
were easier.  

    john

---

From: Patrick McManus <[email-masked]>
Date: Wed, 1 Mar 2017 16:41:41 -0500

Hi All,

This thread started with a comment that github versions that lived
in-between ID versions did not have plain text representations - and
hopefully that specific misconception has been put to bed with the
counterexample of reasonable tooling mentioned upthread. Apparently this
has unearthed a spirited defense of the old way against a new way. So let
me espouse the virtues of the new way using John's outline for paradigms of
participation in a wg.

> understanding all changes, more or less in real time.
what tends to happen for this scenario in a github world is that a lot of
the native-wg-email moves into github-issues-email (you can still subscribe
and reply to these directly from your mailer or via any number of
alternative github interfaces) - with much more disciplined threading than
happens traditionally. A busy document benefits greatly from the tagging
these issues can have - an extra level of metadata and state tracking that
email alone can't give you. Multiple suggestions can be attached as pull
requests to the issues and each can be discussed independently and in
context - again, that discussion can all be gatewayed to email. I promise
you will still send and receive too much email - I have 18 github gatewayed
comments re QUIC in my inbox in the last 24 hours. Each one very specific
and substantive - Its just a better structured wg discussion in my inbox.
But the unstructured list persists for topics that aren't ready a github
issue yet. TBH if I were to only read one of them it would be the github
issue gateway - the signal to noise ratio is better.

have a look at
https://github.com/quicwg/base-drafts/issues?utf8=%E2%9C%93&q=is%3Aissue ..
what I see is essentially a very well threaded mailing list archive plus
some useful metadata (classifications, open-vs-resolved, tags about stuff
that needs more discussion, issues that are basically decided but an editor
needs to propose some text, etc..). And you can consume/contribute with a
mailer if you are happy with the old way.

> burdensome; change logs in documents and diffs between versions

This seems to suggest that github based drafts go from -00 to WGLC with
interim changes only happening on github. That's not the way it works in my
experience - to me it feels like the same number of drafts are produced
(some data here would be interesting.. http/2 had around 19 revisions at
the end iirc even though we used github extensively.. I believe tls 1.3 is
on draft 19 or 20 and it is also github centric) and they continue to have
changelogs. Git pushes that are not also uploaded to datatracker are simply
a better (more visible) version of the locally saved version an editor has
always done in between updates. Periodically, new datatracker releases are
made to garner wider visibility, test interop, and as a signal that things
are, as you say, internally consistent.

Reviewing these new datatracker revs can be done the same as always (diff
between -N and -N+1, changelogs, etc..), but it can also be *so much more*..
you can git blame your way back to a changeset and probably an issue # from
the commit message.. from there you can read the discussion that went into
the change if you weren't able to tune in in real time. If you're not cool
with the way things stand, you just open a new issue (or send an email to
the list and the editor or chair will open an issue).

If you want to review two non-data tracker revisions (e.g. you normally
track things in real time but you were out of office for a week) you can
use the same tools but with git revisions. version control software is
pretty good at that :)

> Only the first really makes effective use of the github style of
I disagree with this conclusion - the accumulated repo history is most
valuable near the end of the process. We finally have something more than a
messy email database as a record of the process.

This is especially powerful during the last calls, when issues from
some-time-ago can be re-raised (hopefully with new information!).. rather
than everyone spelunking through the archives to remember how it ended up
like that (or simply retorting 'read the archive') there is some structure
to the history of the document - the 'git blame' and the closed issue
threads intersect powerfully for understanding how consensus was reached.

There is also the added bonus of having to close all the open issues before
a document can be published. Stuff doesn't get lost in the shuffle (though
it may be over taken by other changes and closed without action - but at
least it is done with deliberate review). Its better project management.

I freely admit I am discussing what are emerging as best practices and I
hope [email-masked] helps codify some of this. This system could
certainly be run poorly - but that's not a reason for avoiding well run
ones. The old email and upload XML snapshot approach doesn't live in the
Internet ecosystem that we seek to serve for good reasons.

---

From: Joe Touch <[email-masked]>
Date: Wed, 1 Mar 2017 14:47:34 -0800

On 3/1/2017 8:04 AM, Michael Richardson wrote:
> polished. 
FWIW, industry often doesn't let out ideas until filing preliminary
patents, which has a similar effect.

AFAICT, the speed of an -00 is more tightly correlated to the motivation
for the document.

When a WG has a known need for a particular doc, the -00 comes out
earlier (pull model).

When an idea is driven by the doc itself (push model), it's more useful
to have enough in place to have a useful discussion about.

That said, I really wish that -00s that aren't pull-driven would try to
stay under 5 pages of content. Dumping dozens of pages and asking for
feedback isn't a good way to *_start _*a discussion.

Joe

---

From: Michael Richardson <[email-masked]>
Date: Wed, 01 Mar 2017 21:05:43 -0500

> AFAICT, the speed of an -00 is more tightly correlated to the
    > motivation for the document.

Good point, but tell me why -01 and -09 and -17 don't come out faster?
If it's because the authors are busy, and aren't working on that document,
that's fine.  But I'm seeing some kind of shyness to putting new versions out.

Is there a distrust of documents which have "too many" revisions?
I consider that often a good sign!

Should the beautiful history bar in the datatracker, have some measure of
changes?  Would graph of lines changed (in the XML!) per revision be
interesting?   Would that help know how close a document is to being ready?

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 2 Mar 2017 03:08:56 +0100

> Good point, but tell me why -01 and -09 and -17 don't come out faster?

One contributing factor might be that the submission interface is a pain.

(I still don’t have a Makefile entry for that.)

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Thu, 02 Mar 2017 09:19:19 -0500

> One contributing factor might be that the submission interface is a pain.

I would like to avoid the confirmation email if I'm logged in, and I'm logged
in with the email which is an author.  Aside from that, I don't find it a
pain at all.

    > (I still don’t have a Makefile entry for that.)

I agree it would be nice to have a wput script for that. That doesn't
seem that hard to me.

---

From: Joe Touch <[email-masked]>
Date: Thu, 2 Mar 2017 08:02:15 -0800

Hi, Michael,

> that's fine.  But I'm seeing some kind of shyness to putting new versions out.

I'm not sure it's easy to determine except on a case-by-case basis.

Other reasons: the community wasn't ready for the doc yet, other docs
took priority (for the author or WG), there might be lots of email list
activity to resolve an issue, etc.

> Is there a distrust of documents which have "too many" revisions?
I'm not sure I'd call it "distrust". But it is hard to keep up with a
constant stream of revisions, esp. for very large docs.

> interesting?   Would that help know how close a document is to being ready?
Not necessarily. First, moving sections around can give a false sense of
a major change. And a doc that is only wordsmithed could either be close
to being ready or be an indication of apathy.

There's no substitute for actually tracking what's going on with a
document, and no replacement for a person investing time to figure that out.

Joe

---

From: Warren Kumari <[email-masked]>
Date: Thu, 2 Mar 2017 14:40:10 -0500

I've worked with a number of different authors, and there does seem to
be a wide range of preferences.

I personally like to publish early, and often; I'll often publish a
new version integrating just one persons comments (or, of they are
nits, batch up a few sets into one version). If there are lots of
comments going back and forth, I don't think it is easy for WG
participants to mentally keep track of all the different comments, how
they interrelate, and what the final text will look like. I also think
that it is politer to respond to feedback by integrating and
publishing a new version, instead of just saying "Thanks, I'll get to
them sometime....".
Committing to GitHub kinda accomplishes this, but a: it's harder for
participants to find, and b: the github version is a second class
citizen.

But, other authors seem to have a different view -- they'd much rather
get everything fully squared away, all comments addressed, all 't's
crossed and 'i's dotted.

One of the stickier points is what to do during WGLC -- unfortunately,
in many groups this is where the majority of the review and feedback
happens, and it is often viewed as poor form to revise during WGLC.
It's often hard for the *authors* to keep track of what the consensus
is when there are lots of comments, what the new text would look like,
etc -- expecting random WG participants to do so is (IMO) unreasonable
and leads to frustration and overlapping comments. I'd personally
rather publish new versions *during WGLC* saying "This is what this
looks like now, does this address your issues?" than trying to explain
that we will move text from Section 3.2.5.4.3 bullet 9 to Section
1.7.4 to address Mike's comment, but that will mean that Billy's
comments no longer apply because it removes the text that he's
commenting on, other than the nits about the case of the acronym,
which we agreed to change globally, except in section 8, because it is
quoting from another RFC. Confused yet?

W

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

---

From: tom p. <[email-masked]>
Date: Fri, 3 Mar 2017 12:36:03 +0000

----- Original Message -----
From: "Warren Kumari" <[email-masked]>
To: "Michael Richardson" <[email-masked]>
Cc: <[email-masked]>
Sent: Thursday, March 02, 2017 7:40 PM

> get everything fully squared away, all comments addressed, all 't's

How sensible other authors are!

A number of I-Ds I have seen in the recent past got a comment from one
experienced in the field which led the author to issue a revised
version, which prompted a comment from another experienced in the field
contradicting the first comment which led the author to ..

I give up and await IETF Last Call, or if it is an I-D where my interest
is not major, give up altogether; life's too short to track a series of
somersaults.

The IETF proceeds at a leisurely pace, taking a week or two or more for
comments to appear, so updating more often than that is
counter-productive.

Tom Petch

> > For the third time in two days I find myself, when asking others for
opinions

---

From: Julian Reschke <[email-masked]>
Date: Fri, 3 Mar 2017 13:51:38 +0100

> participants to find, and b: the github version is a second class

And because of that, the boilerplate of an internet draft always should 
point out where to send feedback, look for issue trackers, etc.

Best regards, Julian

---

From: John C Klensin <[email-masked]>
Date: Fri, 03 Mar 2017 09:23:10 -0500

An observation on part of this thread, stimulated by some recent
experiences...

As people think about what should go where and how rapidly, it
is easy to forget two things about working group documents.  

First, the people we call authors are supposed to be reflecting
WG decisions and preferences, not their own, and are accountable
to WG chairs in the latter's capacity as interpreters of WG
consensus.  Situations and preferences differ as do WG styles of
operation, but, for some situations, that responsibility may
justify semi-private "did I get this right?" checking and/or
wanting a draft to reflect all of the conclusions from a given
WG meeting or email thread rather than having those changes
posted incrementally and without context.

Second, we often end up with co-authors (or co-editors) on a
given document and do so for a wide variety of different
reasons.  For example, when a document is a revision of an
earlier RFC, authors of the previous version may be listed as
authors on the new one as a courtesy even if they are not
actively participating editing the revision or even in the WG
(whether that is appropriate is another discussion).  However,
there are also cases in which two or more author-editors are
active at the same time, passing the virtual pen back and forth
or otherwise collaborating in a fairly dynamic way.  For those
cases, "are these changes ok with you?" and "did I miss anything
important?" checks are appropriate and often result in document
improvements and saved WG time.

Either of those situations can justify a delay of some days in
getting an I-D posted -- longer than pushing micro-changes to
github or equivalent.  They also provide further justification
for  a snapshot-based model.   I would suggest that, if delays
for these reasons are weeks (or longer) rather then days, the WG
may have a management problem, but that, too, is a almost
entirely a separate issue.

    john

---

From: Michael Richardson <[email-masked]>
Date: Mon, 06 Mar 2017 22:33:36 -0500

Lloyd Wood <[email-masked]> wrote:
    > Those advocating 'early and often' might consider the case of
    > WhatWG, hybi, Ian Hickson, and his working style, as shown by
    > https://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-76

    > where it became impossible to figure out what had changed,
    > whether it was important, or really, anything other than the
    > way Hixie wanted to work, and that the WG really didn't like
    > how they were expected to run.

    > Diffs aren't useful unless they're of source.

Agreed: too many revisions is also a problem.
I think that the WG can push back.

Was it useful to diff version 60 against version 76?

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 6 Mar 2017 20:01:38 -0800

On Wed, Mar 1, 2017 at 1:41 PM, Patrick McManus <[email-masked]>
wrote:
> changelogs.

Just to reinforce a couple of points mcmanus is making here:

1. I'm not seeing much evidence that people produce significantly fewer
draft
revisions when Github is in use. As patrick says, TLS is currently on -18
and
I'm about to do -19. To take three comparison points of roughly comparable
size:

- TLS 1.2 (RFC5246) was published after -09
- IKEv2 (RFC4306) was published after -17
- ICE (RFC 5245) was published after -19

My impression is that with Github, much like with non-Github, the draft
issuance
cadence is largely driven by the IETF meeting schedule. Certainly we've
found
that to be true for many of the drafts I am working on (JSEP and TLS in
particular), even though we use Github.

I'd certainly be interested in seeing some evidence that Github drafts come
out less frequently than ones that didn't use Github.

2. Changelog quaity varies, of course, and AFAIK there's no actual
requirement
to do a Changelog, but my experience is that the Github drafts actually do a
somewhat better job of Changelogs, probably because the tooling encourages
you to package things up in a way that makes it easy to do a ChangeLog. This
is especially true if you use PRs because then you can easily go back and
look at every merged PR and that becomes the input to the ChangeLog.
Here, for example, is the TLS ChangeLog.
https://tlswg.github.io/tls13-spec/#rfc.section.1.2

I think what *is* fair to say is that Github enables a far richer
interaction model
(Patrick alluded to this as well) with the result that it highlights the
relative
poverty of the old occasional draft model, but that's not because that model
has suddenly gotten absolutely worse, just that there are now better tools
available and people are taking advantage of them.

-Ekr

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 6 Mar 2017 20:06:53 -0800

> revisions need to be posted as I-Ds.

Well, it's hard to know what to make of this without knowing what you
mean by "substantial" but an active draft takes literally hundreds of PRs
in its lifetime with perhaps half of those being non-typos. We could of
course gateway every PR merge to an IETF draft push. Is that what you're
looking for?

It seems to me that some of the benefit of having discrete draft revisions
is that they represent coherent checkpoints, but you totally lose that if
you treat each commit or merged PR as a co-equal revision.

-Ekr

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 7 Mar 2017 17:20:15 +1300

> course gateway every PR merge to an IETF draft push. Is that what you're

No. In fact (countering Michael's point slightly) I get quite annoyed
by draft versions that turn out only to fix few typos or grammatical errors;
those can wait. As for what constitutes "substantial", that's very subjective.
Anything that causes an on-the-wire protocol change would certainly be
substantial. Clarifying ambiguous text might be substantial. But YMMV.

> you treat each commit or merged PR as a co-equal revision.

Yes, I agree.

    Brian

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 6 Mar 2017 20:28:20 -0800

On Mon, Mar 6, 2017 at 8:20 PM, Brian E Carpenter <
[email-masked]> wrote:

> [email-masked]
> >> ,

But this would still lead to a colossal number of drafts. It's easy to see
this by
looking at recent versions of TLS, where I have handily noted all
on-the-wire
changes in the changelog (or at least tried to). Here are the numbers from
recent drafts:

Draft               Wire changes
-19 (unpublished)   5
-18                 1
-17                 9
-16                 6
-15                 5
-14                 5

As you can see, we have an average of 5 wire-level changes per revision,
so given that we are at draft-19, we would be at around draft-100 by now.
Probably more, because we were more aggressive earlier, and I just
started noting these because people were fielding implementations and
I wanted to make it easier to update. I find it hard to believe that people
would in fact be happier if TLS 1.3 were currently at -100 and to the
best of my knowledge, next to no WG has every produced drafts
at anything like that frequency.

Note that I'm not saying that TLS is especially active (though I suspect
it is more than average). Rather, people (both with and without Github)
tend to batch up a bunch of changes and submit them in a single draft.
It's just that because Github makes the revision history so public and
explicit that people are aware of these as discrete events where you
could publish as individual drafts, rather than as a pile of things that
go into a single draft.

-Ekr

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 8 Mar 2017 08:33:14 +1300

On 07/03/2017 17:28, Eric Rescorla wrote:
> But this would still lead to a colossal number of drafts. 

I realised when reading your reply that it depends a lot
on what stage the draft has reached. In the early phases,
you're right of course. There could be many protocol changes
in quick succession, and they should be bundled up. My comment
was over-influenced by a draft I'm currently editing, where
IETF Last Call has led to a couple of minor on-the-wire changes.
At this stage, they are definitely significant and require a
new version (but even so, they will be bundled with numerous
editorial updates).

<snip>

> As you can see, we have an average of 5 wire-level changes per revision,

I have data for draft-ietf-anima-grasp, since I started logging
protocol changes at version 02. The average is about 3 per revision.
So maybe we are in fact agreeing violently.

    Brian