Subject: What's an experiment?

From: Brian E Carpenter <[email-masked]>
Date: Wed, 15 Feb 2006 16:06:35 +0100

When considering some recent appeals, the IESG discovered that
we have very little guidance about the meaning of "experiments"
in relation to Experimental RFCs. RFC 2026 refers to work which
is "part of some research or development effort" and the IESG
has adopted some guidelines to discriminate between Experimental
and Informational documents (see
http://www.ietf.org/u/ietfchair/draft-iesg-info-exp-01.html ).
But beyond that, we do not know what constitutes an acceptable
experiment on the Internet.

The IESG notes that the community could establish a variety of
guidelines describing what is and is not acceptable in experiments.
Historically, the IESG has made decisions based on its perception
that there is a strong desire in the community to publish technology
that is being deployed experimentally.  We encourage community discussion
and development of more specific guidelines on operational conflicts
caused by experiments and how this should affect what we choose to
publish.  (However we recommend that such discussion
focus on the general issue rather than the specifics of any case.)

   Brian Carpenter
   for the IESG

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Eliot Lear <[email-masked]>
Date: Wed, 15 Feb 2006 08:05:00 -0800

Brian,

It would seem to me that the purpose of an experimental RFC is to let
people perform and participate in (rather public) experiments on the
Internet.  A reasonable standard for "experimental" is that there be a
thesis and a procedure so that the experiment can be repeated,
observations can be made, and conclusions can be drawn.  The thesis
should be clear as to what is being tested, and it should at least pass
the laugh test.  Further, the standard for design of the experiment
should provide for the least practicable interference with other ongoing
operations (experimental or not).

So, is that simple enough or too simplistic?

Eliot

---

From: Robert Sayre <[email-masked]>
Date: Wed, 15 Feb 2006 12:16:39 -0500

On 2/15/06, Brian E Carpenter <[email-masked]> wrote:
> in relation to Experimental RFCs.

<http://www.ietf.org/rfc/rfc3160.txt>

What's wrong with the definition that appears in RFC3160, section 6.5? Quoting:

>>  Experimental RFCs are for specifications that may be interesting, but
>>  for which it is unclear if there will be much interest in
>>  implementing them.  That is, a specification might solve a problem,
>>  but if it is not clear many people think that the problem is
>>  important, or think that they will bother fixing the problem with the
>>  specification, the specification might be labeled an Experimental
>>  RFC.

This definition allows publication of specifications that some people
don't like or don't care about. It also implies that there might be
other RFCs attempting to solve the same problem.

> We encourage community discussion and development of more specific
> guidelines on operational conflicts caused by experiments and how this
> should affect what we choose to publish.

The overview of the IETF[1] highlights two areas of concern:

  a. The evolution of the Internet archicture
  b. The smooth operation of the Internet

So, it would seem to me that experiments must not damage the Internet.

[1] http://ietf.org/overview.html

--

Robert Sayre

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 Feb 2006 09:19:55 -0800

There are two different potential intentions to 'Experimental':

1. to conduct an experiment, as Eliot notes below, i.e.,
   to gain experience that a protocol 'does good' 'in the wild'

2. to gain experience that a protocol does no harm 'in the wild'

I think of IETF Experimental track as being more about #2 than #1.  #1
may be the responsibility of the proponents, but given the IETF doesn't
even conduct conformance testing for standards-track, it it seems
overzealous to expect it to verify a scientific procedure for
experimental-track.

The document should definitely state why experimental is chosen - what
the concerns are, and what to look out for (as to #1). But I would stop
well short of specifying theses, procedures, etc. (#2) for this track.

Joe

---

From: "Brian Rosen" <[email-masked]>
Date: Wed, 15 Feb 2006 14:37:13 -0500

I believe if the community does not have confidence that the protocol will
actually work on the Internet, then we are experimenting.  I think this
definition would cover a number of protocols we would now consider for
Proposed Standard (rather than Informational), and pushes us back towards
"Running Code".

A protocol that has no implementations and is significantly new and
different that we cannot inspect the document to have confidence it will
work as intended would be Experimental.   This is both a complexity and an
exposition test as well as a measure of how similar a protocol is to other
protocols we know work as intended.

I think that when an author, or an entire work group, dreams up a
significant new protocol and brings forth a complex document with no code,
then we have a research and development effort underway, which should be
allowed to complete; that is, test it.  We have a product (a real standard),
when we show that our research worked.  As with most research and
experimentation, it is not necessary to have market acceptance to have a
successful research project.  It is nice, and a good indicator that the
protocol works.

Brian

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Brian E Carpenter
Sent: Wednesday, February 15, 2006 10:07 AM
To: IETF discussion list
Subject: What's an experiment?

   Brian Carpenter
   for the IESG

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 15 Feb 2006 21:20:37 +0100

Brian E Carpenter wrote:

> we do not know what constitutes an acceptable experiment on
> the Internet.

Maybe something that doesn't harm folks who don't participate.
For whatever reasons, maybe because they don't know about the
experiment.

> focus on the general issue rather than the specifics of any
> case.

"Voluntary" and "opt-in" appear to be keywords.  A potentially
harmful experiment with non-consenting participants is dubious.

Intentionally disrupting other running experiments or breaking
standards is also dubious.  

Maybe the latter is acceptable if it's a documented purpose of
the voluntary experiment, and causes no "real harm" for others.

For some values of "real harm" - one possible outcome of such
experiments could be "the whole world adopted it, what next ?"

                          Bye, Frank

---

From: "william(at)elan.net" <[email-masked]>
Date: Wed, 15 Feb 2006 13:35:50 -0800 (PST)

> the voluntary experiment, and causes no "real harm" for others.

Breaking standards would be ok if it was agreed upon by IETF
with last-call (which is in fact available for experimental
documents) but without it you cant issue such a document as
an RFC and say its ok when original STD was agreed upon by
entire IETF (that of course does not mean the experimental
RFC should meet same high criteria as when you have RFC to
actually depreciate or replace a standard).

Just by itself without last call experiment is probably ok when
you have some new concept that needs to be tested and documented
and its use should would cause any significant problems for anything
else. In such a case experiment is there to basically confirm if there
are enough interest in the concept to go further and it does what
is expected and then further standardization maybe pursued.

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 15 Feb 2006 15:56:21 -0600

> is expected and then further standardization maybe pursued.

Hi, William,

We have had discussions a number of times in Problem/Solutions/Newtrk about 
topics like "working group snapshots" or "stable snapshots", that did not 
require last call and/or IESG approval to publish. We have had proposals for 
both ephemeral and non-ephemeral snapshots.

The goal of WGS/SSS was to provide a "stable draft" that implementers could 
develop to, without worry that there would be (well, in one case I can think 
of) another eight or ten draft revisions after some people started 
implementing.

Do you think that an RFC is required for the use of Experimental you are 
describing? Do you think an archival specification is required?

Thanks for your thoughts,

Spencer

---

From: "william(at)elan.net" <[email-masked]>
Date: Wed, 15 Feb 2006 14:47:09 -0800 (PST)

On Wed, 15 Feb 2006, william(at)elan.net wrote:

> else.

This was supposed to be:

  Just by itself without last call experiment is probably ok when
  you have some new concept that needs to be tested and documented
  and its use would not cause any significant problems for anything
  else.

---
William Leibzon
Elan Networks
[email-masked]

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Thu, 16 Feb 2006 03:25:20 +0100

Dear Brian,
ICANN ICP-3 document called for a DNS test-bed to carry experiments 
in a given framework (to test various DNS evolutions including the 
end of the root). The document lists interesting criteria/conditions. 
Some are related to the DNS (non profit, ultimate agreement by the 
community). Of the head two are important: reversibility and no harm 
to the current operations. The "non profit" can be generailised: if a 
community effort is carried to commonly consider an evolution, every 
option should be considered and equally supported. Experiments must 
not be a way to impose personnal or affinity group doctrines and DoE 
(Denial of Evolution). Reversibility would also mean the result 
cannot be published as BCP. It may reflect the practice of a group. 
But it would not be acceptable to impose it to non participants as 
there is no proof it would scale - before the experience convers the 
whole network. This means that experience may be a way to deploy or 
to transition. Should the IETF has started a large scale IPv6 
experimentation, may be would we have IPv6 by competition to the 
RIRs. This has been considered.
jfc

---

From: Joe Baptista <[email-masked]>
Date: Thu, 16 Feb 2006 08:16:19 -0500

JFC (Jefsey) Morfin wrote:

> Dear Brian,
> ICANN ICP-3 document called for a DNS test-bed to carry experiments in 
> a given framework (to test various DNS evolutions including the end of 
> the root). The document lists interesting criteria/conditions. Some 
> are related to the DNS (non profit, ultimate agreement by the 
> (Denial of Evolution). Reversibility would also mean the result cannot 
> be published as BCP. It may reflect the practice of a group. But it 
> would not be acceptable to impose it to non participants as there is 
> no proof it would scale - before the experience convers the whole 
> network. This means that experience may be a way to deploy or to 
> transition. Should the IETF has started a large scale IPv6 
> experimentation, may be would we have IPv6 by competition to the RIRs. 
> This has been considered.
> jfc

Thats happening regardless of the IETF - www.public-root.com, 
www.inaic.com and www.unifiedroot.com.  Failed experiments result in 
successful evolution.

regards
joe

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 16 Feb 2006 08:21:04 -0500

> 2. to gain experience that a protocol does no harm 'in the wild'

There is a third option, which is "we are not sure how this will work, or
if it will even work at all really, but we are confident enough in its
design stability to release it in limited form for further study"

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: Peter Dambier <[email-masked]>
Date: Thu, 16 Feb 2006 15:53:36 +0100

> www.inaic.com and www.unifiedroot.com.  Failed experiments result in 

Dont forget

http://www.opennic.unrated.net/
http://dot-root.com/
http://root.5wc.ch/

Those who are not commercial are difficult to reach sometimes.
E.g. http://www.opennic.unrated.net/ sometimes can be reached
only as http://www.opennic.glue and you have to add

host_name("131.161.247.68","www.opennic.glue").

to your /etc/hosts to find them.

Still they have nameservers and they happily communicate with
each other without ICANN even nowing about their existence.

Cheers

Peter and Karin

-- 
Peter and Karin Dambier
The Public-Root Consortium
Graeffstrasse 14
D-64646 Heppenheim
+49(6252)671-788 (Telekom)
+49(179)108-3978 (O2 Genion)
+49(6252)750-308 (VoIP: sipgate.de)
mail: [email-masked]
mail: [email-masked]
http://iason.site.voila.fr/
https://sourceforge.net/projects/iason/

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Thu, 16 Feb 2006 15:45:41 +0100

>www.inaic.com and www.unifiedroot.com. Failed experiments result in 

Dear Joe,
these are individual experiments we all carried. And this gave us 
some experience. It is like publishing an RFC for experimentation. I 
think Brian is asking for more. A generalised concept. This is what 
ICANN called for. This is what we tried to achieve with 
http://dot-root.com : a real test bed, for these kind of applications 
to work together ad experimenda. It told us much. But we closed it. 
This was good experience towards the "unitry" project I introduced in 
Luxembourg, at the ccTLD Meeting.

If I take your example, I would say experiment should be about how 
public-root can help unifiedroot, how can they mutually monitor and 
inform inaic if they have been hacked. Also, the applications you 
quote are here to stay. By nature experimentation is temporary - but 
a test-bed can be permanent. I know the mutual help spirit between 
open roots, but experimentation is about joint reporting, result 
sharing, common best eventual practices and generic code being produced.

I think if the IETF considers this, everyone should be more than 
welcome. The problem is to organise a conceptual framework. Then to 
build and maintain a test environment. May be now the IASA could 
help? Also users should clearly know that a few things may be 
different. Or if standard conditions are simulated to benefit from 
real traffic and test other issues. For example the Klensin's IDN 
test project: we should first know about it.

I think that the first target of experimentation is to convince 
everyone. Open and alternative roots for example are everywhere and 
this will increase. But they have not convinced the (RFC 3774) 
"affinity group". It shows that their first requirement to convince 
would be a total transparency and progressive innovation (risk 
containment). Open/alternative roots have always be seens as 
"commercials" and therefore as opaque (even when private or 
non-profit), and the vector of a radical change (even if it is more 
and more on us).

As a general remark, one the problems of the IETF is there are too 
many people who want to win, and only a few who are looking for being 
convinced ....
jfc

---

From: Joe Touch <[email-masked]>
Date: Thu, 16 Feb 2006 11:01:36 -0800

> design stability to release it in limited form for further study"

That's #1 - if it doesn't provide some gain (utility to someone),
there's no point in the experiment ;-)

Joe

---

From: "Christian Huitema" <[email-masked]>
Date: Thu, 16 Feb 2006 13:31:50 -0800

> >>2. to gain experience that a protocol does no harm 'in the wild'

There are also many ways to conduct an experiment on the Internet,
including:

1) Just release some software, without telling anyone about how it works

2) Set up a web page describing the experiment without involving the
IETF

3) Publish an Internet Draft and use it as the experiment's
documentation

4) Get an ID reviewed by the IETF and published as experimental RFC

For the IETF, there is a tension between two goals: protocols that are
reviewed and documented, so there can be multiple implementations; and
protocols that have a high quality so they run very well on the
Internet. If the IETF focuses a lot on quality and makes it too hard to
do #4, experimenters will end up doing #2 or #3, and there will be less
review and less open documentation.

So, there has to be a balance.

-- Christian Huitema

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 16 Feb 2006 17:30:28 -0500

> there's no point in the experiment ;-)

I think #1 is closer to what we use Proposed Standard for--to "gain
experience that a protocol 'does good'".

The scenario I proposed is for when you know that there are upsides and
downsides but you want to see which one comes out on top by exposing it to
scale/weirdness that is only found in the real world

---

From: Eliot Lear <[email-masked]>
Date: Fri, 17 Feb 2006 16:06:21 +0100

> So, there has to be a balance.

I really don't think either (1) or (2) in Joe's message calls for all
that much effort, with one exception: some investigation should be done
to ensure no harm.  That can take work in some cases.  Also, end host
experiments are probably easier to claim no harm is occurring than say,
routers or mail servers.

Eliot

---

From: Joe Baptista <[email-masked]>
Date: Thu, 16 Feb 2006 17:55:43 -0500

> each other without ICANN even nowing about their existence.
Out of touch with reality.

regards
joe