Subject: RFC/Protocol Quantification

From: Sreevathsa <[email-masked]>
Date: Thu, 21 Sep 2006 22:32:07 -0700

Hi Group,
I was thinking about methods to quantify the conformity of a
protocol's implementation to its RFC. What could be the possible
metrics that I could consider while quantifying the conformity of any
protocol's implementation to its RFC?

Let me make myself more clear with an example: Lets say I am
validating a particular TCP implementation on a server. I need to see
how much of the TCP implementation is in accord with RFC 793. After
the analysis of implementaion, once I know what parts of the RFC is
implemented and what part of it is not, I need to come up with a
number and say
something like, 93% of TCP implementation on this server is in
agreement with RFC 793 and 7% of it is not!

So to quantify like this, I should have some metrics to weigh a given
RFC. Does anyone have any idea as to how any given RFC could be
divided into various components and then assigned a weightage which
could then be used as a reference to check other implemenations??

Some of the components that I was thinking about are:

1. Packet structure
2. Query format
3. State transitions

If you have any ideas, I will be very much willing to know about it.

---------
Thanks,
Sreevathsa

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 22 Sep 2006 17:20:01 +0200

Sreevathsa wrote:

> something like, 93% of TCP implementation on this server is
> in agreement with RFC 793 and 7% of it is not!

Ticky, who cares about 80% if it works, or 99% if it doesn't ?

> components that I was thinking about are:
 
> 3. State transitions

For modern RFCs you'd also have a bunch of 2119-keywords.

> If you have any ideas, I will be very much willing to know

A hard part is to get an agreed upon conformance test suite.

And then do something with "inconclusive" outcome.  One case
where "93%" results were not plain nonsense was the "good
newsreader seal of approval" (or similar), but that wasn't
strictly based on RFCs, it's input for a draft.

Frank