Subject: Long review tail

From: "Fred Baker (fred)" <[email-masked]>
Date: Thu, 2 May 2013 14:54:32 +0000

I your blog, you wrote:

> Having been involved in the process for many years, often the bigger changes at this stage relate to cross-area issues, or the fact that the careful reviews from the IETF last call, directorates, and 15 ADs often represents a significant increase in the number of non-WG people looking at the document.

If I may make a suggestion, I suspect that the problem statement suggests its own solution: get cross-area reviewers involved in the working group process earlier. There is a trade-off in that, of course, which is that if the cross-area review happens too early it may happen on a document that isn't really ready for cross-area review. But bottom line, that's what you need to have.

As a working group chair, what I have done on a number of occasions is ask one directorate or another (usually the security directorate, which is not to pick on them but to be frank) to review a document that I think has need of their expertise. This has not, in my history, been fruitful; I have gotten a few responses, but not many. If there were a way to request those that would in fact result in reviews, I would include it in my standard WGLC process, and I suspect that would have a salubrious effect.

The other thing that I might suggest is a comment I have made before: if in the opinion of the IESG a document needs a lot of work (e.g., non-trivial changes), return it to the working group. I know of, due to some spam I received a few months ago, an editorial service in London that will for a small fee (a common IETF document would be perhaps 100 pounds) review and update a document; pointing the option out to a working group chair might be as useful a response as any. Corporate technical writing services might also be called in. Whatever the technical issues a document has, the place for them to *not* be worked out is with an AD simply holding the document hostage; it is to say it needs work and send it back for the work.

Both of those would be likely to simplify the life of an AD as well.

---

From: Stephen Farrell <[email-masked]>
Date: Thu, 02 May 2013 16:12:38 +0100

> As a working group chair, what I have done on a number of occasions is ask one directorate or another (usually the security directorate, which is not to pick on them but to be frank) to review a do…

As a data point, Sean and I discussed this with folks at the secdir
lunch in Orlando. We currently get about 80% of IETF LC documents
that actually get reviews by secdir, and many of those are very
useful to the ADs, though it is late in the process for the authors
and WGs. So we've a 20% dropped-review rate and that's been stable
for quite a while even with secdir membership changes.

When asked if more could be done, (without any specific proposal
for what to do) the response was that increasing the workload
would maybe lead to a significant drop in that 80% figure since
secdir folks are also busy with their day-jobs. So increasing
the workload for reviewers is tricky.

Trying to shift the workload to earlier in the process is certainly
worth a look, but also has risks. It would increase the reviewer
workload since the best reviews generally involve a bunch of follow
up mails, and extending that over months takes more of the
reviewers time. I suspect the rate of dropped-reviews might also
increase if documents are reviewed too early. OTOH, for the
authors and WG, this does seem like it'd be better.

I guess if this were easy, we'd have solved it already;-)

> The other thing that I might suggest is a comment I have made before: if in the opinion of the IESG a document needs a lot of work (e.g., non-trivial changes), return it to the working group. I kno…

I fully agree with that.

S

PS: If a WG chair ask Sean or I for an early review we do usually
stick that into the secdir rotation. At the moment that's ok since
its not that common, but if we got lots of them it might become an
issue.

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Thu, 2 May 2013 15:19:43 +0000

> the workload for reviewers is tricky.

Sounds like we need more security reviewers. Easier said than done, I know. I might suggest polling people who have done documents in the security area.

---

From: Stephen Farrell <[email-masked]>
Date: Thu, 02 May 2013 16:31:21 +0100

> Sounds like we need more security reviewers. Easier said than done, I know. I might suggest polling people who have done documents in the security area.

Worth a look. We already have quite a few though. [1]

S.

[1] http://trac.tools.ietf.org/area/sec/trac/wiki/SecDirReviewers