Subject: running code vs. formal testing methods

From: Reinhard Scholl <[email-masked]>
Date: Fri, 30 Jul 1999 16:18:30 +0200

Maybe not that surprisingly, searching RFCs for SDL (Specification 
and Description Language), MSC (Message Sequence Charts) and 
TTCN (Tree and Tabular Combined Notation) doesn't give a single hit. 
Whereas the IETF/Internet is based on "running code", requiring 
different interworking implementations rather early in the standards 
process, the traditional telecoms standards process relies on formal 
testing methods which come rather late in the standards process 
(conformance testing; yes, there is also a lot of non-publicized 
testing among friendly competitors). Going a step further, in ITU-T's 
INAP CS-3/CS-4 the text has only informative status; it is the 
SDL-version that is normative.

If the above is correct, what is really the reason for the difference in 
testing philosophy? Is it historical? Cultural? Is it the business model 
(a few big players vs. everyone can set up shop in the Internet)? Is it 
openness in the Internet vs. a rather closed environment in the 
telecom world? Is it the complexity / robustness of telecom systems? 
Is it because telcos have very stringent requirements, with rather 
large penalties on equipment manufacturers if things don't work?

And will the testing philosophy change? Will bake-offs a la IETF find 
their way into the telco world, or/and will formalized testing methods 
find their way into the Internet world?

Trying to get a discussion going.

Reinhard Scholl
ETSI

---

From: Reinhard Scholl <[email-masked]>
Date: Sat, 31 Jul 1999 18:51:42 +0200

> -----Original Message-----
> From: vinton g. cerf [mailto:[email-masked]]
> Sent: samedi 31 juillet 1999 07:57
> To: Reinhard Scholl; [email-masked]'
> Subject: Re: running code vs. formal testing methods
> 
> 
> SDL's and other formal methods have not yielded results
> commensurate with the effort to use them. They are
> overblown, verbose, hard to use, hard to understand.
> What they might gain in precision they lose in 
> comprehensibility.

I am not sure at which point precision can be sacrificed.
The reason why ITU made SDLs normative for INAP CS-3/CS-4
is because several serious errors were detected in CS-2
that would have gone unnoticed if SDLs had not been used.
Or, to change the arena slightly: how could the Pentium bug 
have been prevented? Or the Ariane rocket blow-up?

It would be nice if s.o. could give numbers on the cost / 
benefit of formal methods.

> 
> Plain English specs and multiple implementations
> in <your language of choice> followed by interoperability
> bakeoffs continue to be a powerful means of achieving
> interworking implementations and confidence in the
> specifications.
> 
> Vint Cerf
> 
> 
> > 
> =================================================================
> "INTERNET IS FOR EVERYONE!" 
> Join the Internet Society and help to make it so.
> See you at INET2000, Yokohama, Japan July 18-21, 2000
> http://www.isoc.org/inet2000
> 
>

---

From: Reinhard Scholl <[email-masked]>
Date: Sat, 31 Jul 1999 18:51:42 +0200

---

From: "Marshall Rose" <[email-masked]>
Date: Sat, 31 Jul 1999 17:42:44 -0600

> have been prevented? Or the Ariane rocket blow-up?

testing is not a substitute for competence.

i'm certainly willing to believe that some specs are so poorly written that
reasonable implementors can produce implementations that do not
interoperate.

> benefit of formal methods.

look at the number of standards-track rfcs that have progressed past the
proposed level. i think you would be hard-pressed to find a comperable
number of real-world protocols specified and validated using formal methods.

history is pretty clear that the way to get success in the protocol arena is
to focus on well-written natural-language specifications iteratively
developed with multiple interoperable inplementations.

/mtr

---

From: "Marshall Rose" <[email-masked]>
Date: Sat, 31 Jul 1999 17:42:44 -0600

/mtr