Subject: Re: Last Call: draft-dusseault-impl-reports (Guidance on Interoperation and Implementation Reports) to BCP

From: Alexey Melnikov <[email-masked]>
Date: Sun, 24 May 2009 00:11:12 +0100

The IESG wrote:

>The IESG has received a request from an individual submitter to consider
>the following document:
>
>- 'Guidance on Interoperation and Implementation Reports '
>   <draft-dusseault-impl-reports-02.txt> as a BCP
>
>The IESG plans to make a decision in the next few weeks, and solicits
>final comments on this action.  Please send substantive comments to the
>[email-masked] mailing lists by 2009-06-18. Exceptionally, 
>comments may be sent to [email-masked] instead. In either case, please 
>retain the beginning of the Subject line to allow automated sorting.
>
>The file can be obtained via
>http://www.ietf.org/internet-drafts/draft-dusseault-impl-reports-02.txt
>  
>
This is a fine document and I support its publication.

---

From: [email-masked]
Date: Sun, 24 May 2009 09:31:08 -0700 (PDT)

> This is a fine document and I support its publication.

+1

				Ned

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 26 May 2009 10:35:26 +0200

On Thu, May 21, 2009 at 11:09:01AM -0700,
 The IESG <[email-masked]> wrote 

>    <draft-dusseault-impl-reports-02.txt> as a BCP

It's a fine and useful document. But something is missing: how to
select the implementations tested when there are "many". For RFC 4234
(mentioned in the I-D), some implementations were tested
<http://www.ietf.org/IESG/Implementations/RFC4234_implem.txt> and some
were not. On what criteria?

---

From: Lisa Dusseault <[email-masked]>
Date: Tue, 26 May 2009 09:50:52 -0700

That's an excellent question, but I think like so many others it has
to fall under the judgement of the person writing the implementation
report.  Is it OK to just test 2 implementations or is it important to
test 2 servers and 2 clients?  It might be possible to go to an
interoperability forum and test 15 different implementations, yet if
that's a protocol for which there's a sizable *other* community that
doesn't implement a required feature, that ought to be noted in the
implementation report.

I'm hoping that by putting the onus on the writer of the report to
carefully characterize interoperability, that we can encompass many
such judgement questions.  On the flip side, if we tried to address
every such judgement question, we couldn't possibly foresee every
corner case.

Do you have any suggestions for criteria that could be broadly
applicable and useful?

Thanks,
Lisa

On Tue, May 26, 2009 at 1:35 AM, Stephane Bortzmeyer <[email-masked]> wrote:
> _______________________________________________
> Ietf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ietf
>

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 26 May 2009 11:47:15 -0700

>Do you have any suggestions for criteria that could be broadly

One possible criterion would be "if you considered multiple implementations but focused on a subset for the report, at least discuss the wider set somewhere in the report". That could include a simple list of all the implementations considered, but also might include reasons for subsetting (some of the ones left out had difficult admin UIs, had licensing issues, and so on).

--Paul Hoffman, Director
--VPN Consortium

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Tue, 26 May 2009 21:38:33 +0200

On Tue, May 26, 2009 at 09:50:52AM -0700,
 Lisa Dusseault <[email-masked]> wrote 

> Do you have any suggestions for criteria that could be broadly

Following Paul Hoffman, you could put in the I-D something like: 

All known (by the author of the report) implementations SHOULD be
listed in the report and, if only a subset is tested, the reasons for
selecting this subset SHOULD be written down. For instance, "We tested
Foobar and Example because they are widely regarded as the two most
common implementations." or "We tested Foobar and Example because they
are the only ones available under a free software licence."

(Paul's message contains other good examples.)

---

From: Lisa Dusseault <[email-masked]>
Date: Tue, 26 May 2009 14:45:21 -0700

Good suggestions.  I do agree with explaining why a subset was tested
and how that subset was chosen.  In some cases, however, listing all
known implementations would be onerous, ambiguous (an implementation
that shipped (or went beta) with partial (or untested) possibly
unannounced standards support) or even contrary to privacy agreements.

Lisa

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 26 May 2009 15:58:44 -0700

>unannounced standards support) or even contrary to privacy agreements.

Fully agree. That's why I listed them as suggestions, and I hope that this document does so as well.

--Paul Hoffman, Director
--VPN Consortium

---

From: Lisa Dusseault <[email-masked]>
Date: Tue, 26 May 2009 16:28:32 -0700

Will do, assuming my co-author agrees.

---

From: Sean Turner <[email-masked]>
Date: Tue, 26 May 2009 19:45:27 -0400

>> This is a fine document and I support its publication.

I used an earlier version during the writing of the CMS [RFC3852] 
Interoperability I submitted in March.  I too found this a fine document 
and support its publication.

spt

---

From: Thomas Narten <[email-masked]>
Date: Wed, 27 May 2009 17:15:42 -0400

Overall, I like this document and support it going forward.

One thing it doesn't mention (and did come up when I was an AD) is the
following.

The goal of interoperability testing is to demonstrate that our
specifications are good. I.e., that the text in the final RFC is
complete and can be implemented.

We sometimes have WGs where folk are implementing while the
specification is being developed. What they end up implementing is not
(strictly speaking) based on the text in the final RFC, but may be
implemented "from the mailing list". In such cases, implementation
issues that come up may be resolved on the mailing list (or during
interoperability tests where the implementors just work it out), but
somehow the final spec is still not clear on the point at issue.

It would be good to point out in the document that the intention is
that implementations be based on the final spec. In cases where there
is reason to suspect this isn't the case, and that the implementation
is not actually based on the RFC itself, some caution is advised is
using those implementations to demonstrate readiness to advance to
draft. I.e., insist on multiple implementations, rather than just
using 2 (the absolute minimum required to go to draft).  Clearly some
judgement is needed here, as we don't want to make it too hard to go
to draft either.

Thomas

---

From: Lisa Dusseault <[email-masked]>
Date: Wed, 27 May 2009 18:16:31 -0700

I think this is also a good idea and will try to come up with
something for the next draft.

thanks,
Lisa