Extended Db C Approach to Specification and Conformance

Extended Db. C Approach to Specification and Conformance Testing of Distributed Software Victor Kuliamin kuliamin@ispras. ru Nickolay Pakoulin Alexander Petrenko ISP RAS, Moscow

Outline Introduction n Some Theory n Application 1 : Standards Clean-up n Application 2 : Conformance Testing n

Distributed Software n Is urgent for economic and social development ¨ n Gives people more services very complex Is very complex Unsuitable for human mind, which operates mostly sequentially ¨ Involves lots of possibly heterogeneous components ¨ Errors are getting much more subtle and hard to observe ¨ n tested Need to be checked and tested thoroughly

Outline Introduction n Some Theory n Application 1 : Standards Clean-up n Application 2 : Conformance Testing n

Testing Fundamentals How is testing performed? n We act upon the system under test n We watch its reaction n We check whether that reaction is what should be n We repeat this until all the reasonable situations are exhausted

Testing of Distributed Software n Organize distributed action – harder then in non-distributed case, but possible n Watch distributed reaction – harder then in non-distributed case, but possible n Check whether it is expected – how we know what is expected? n Ensure reasonable exhaustiveness – what does this mean at all? Main sources of complexity

How to Describe Requirements? n The description should be ¨ Sufficiently expressive ¨ As clear as possible ¨ Scalable to rather complex systems – preferably, component-wise ¨ Suitable to distributed systems – include several sides n How do people describe mutual obligations in complex cases including several sides? – By means of contracts!

Contract Specifications n Pre- and postconditions (Hoare, 1969) – means for reasoning about program behavior – augment code elements to enforce rigorous development n Design by Contract (Meyer, 1992) – software is considered as a set of components interacting trough their interfaces – pre- and postconditions are defined for interface operations – constraints on data integrity are stated in invariants – together they form software contract between a component and its environment

Design by Contract Pro and Contra n Advantages Component-wise consideration of software – scalability – support for reuse, incremental and parallel development ¨ Possibility to use for different aspects and on different abstraction levels ¨ n Drawbacks ¨ PU 2 Insufficient for distributed systems – does not consider concurrency and asynchronous interaction – does not consider callbacks PU 1 PL 1

Interaction in Distributed Systems

Event Contracts Obligations of the environment Obligations of the system Precondition Pre-state System Input event says in what states such an event is possible Environment Post-state Obligations of the system Output event Environment Postcondition says what poststates can follow such an event in such a pre-state Pre-state System Post-state Obligations of the system

Concurrency ~

Event Contracts Implementation n Software Contracts Pre- and postconditions of events, invariants ¨ Possibility to specify constraints in form of predicates on the results, not the algorithm ¨ Component-wise consideration of software ¨ Asynchronous events and callbacks included ¨ n n Specifications in extensions of widely-used programming languages (C, Java, C#) Simplifications where possible ¨ Joint description of call and return if intermediate states do not matter

Barrier Example init(3) Threads wait() Height = 3

Barrier Example in Event Contracts public specification class Barrier { int awaited. Threads = 0; int waiting. Threads = 0; invariant Counters. Are. Nonnegative() { return awaited. Threads >= 0 && waiting. Threads >= 0; } public specification void Init(int n) { post { if(n < 0 || waiting. Threads > 0) { branch No. Changes; return awaited. Threads == pre awaited. Threads && waiting. Threads == pre waiting. Threads; } else { branch New. Height. Set; return awaited. Threads == n && waiting. Threads == 0; } } } public specification void Wait() { post { if(awaited. Threads <= 1) { branch Immediate; return awaited. Threads == 0 && waiting. Threads == pre waiting. Threads; } else { deferred branch Waiting; return awaited. Threads == pre awaited. Threads - 1 && waiting. Threads == pre waiting. Threads + 1; } } deferred return { pre { return awaited. Threads == 0 && waiting. Threads > 0; } post { return waiting. Threads == pre waiting. Threads - 1; } }

Testing Adequacy n n How can we define ‘reasonable exhaustiveness’? Possibilities Fault-based approaches n Percent of faults detected by tests to all faults reported n Mutants: percent of mutants detected ¨ Source code coverage-based approaches ¨ Requirements coverage-based approaches ¨ n Uni. Tes. K method : requirements-based Try first to cover structure of postconditions

Testing Goals post { if ( a <= 0 || c. is. Active() ) … else if( a > 3 & !b. closed() ) … else … a > 0 && !c. is. Active() && a <= 3 || a > 0 && !c. is. Active() && b. closed() }

Outline Introduction n Some Theory n Application 1 : Standards Clean-up n Application 2 : Conformance Testing n

Case Studies I n IPv 6 ¨ Parts considered n n ¨ Sending datagrams / receiving packets Neighbor discovery Multicast Listener Discovery UPD over IPv 6 Results n n n 2001 Minor defects found in RFC 2460 Conformance test suite developed (further) IPMP-2 ¨ 2004 Results n n n Several contradictions between standard parts found Interoperability flaws detected in Mutual Authentication protocol 2 accepted submissions on elaboration of the standard

Approach Used Formalization Standards Contract Specifications Inconsistencies, ambiguities, interoperability flaws Testing goals Test Suites Conformance testing Certification Interoperability testing Early debugging Requirements traceability Software

Outline Introduction n Some Theory n Application 1 : Standards Clean-up n Application 2 : Conformance Testing n

Testing Scheme System under Test ✕ Behavior Model Testing Model Coverage Model ✕ Single Event Oracles Composite Event Oracle On-the-fly Test Sequence Generation

Checking Composite Event Interleaving semantics ✕

Uni. Tes. K Test Construction Tools n n n C / Visual Studio 6. 0, gcc Java / Net. Beans C++ / Net. Beans + MS Visual Studio specifications in Java extension Specialized tool for compiler testing and complex data generation C# / Visual Studio. NET 7. 1 Java / Eclipse 2002 2003 ~ 2005

Tool Demonstration

Case Studies n n ISP RAS – Nortel Networks functional test suite development for Switch Operating System kernel IPv 6 implementations ¨ ¨ ¨ n n n 2001 -2003 Microsoft Research Mobile IPv 6 (in Windows CE 4. 1) Oktet Intel compiler optimization units IPSec Pilot projects ¨ ¨ ¨ 1994 -1997 2001 -2003 2004 -… Components of Tiny. OS Web-based banking client management system (Luxoft) Components of billing system (Vympelkom) Data Integration Framework (Vympelkom) Components of Java 1. 4 SE (parts of JDK, JVMTI) Real-Time POSIX-based Operating System http: //www. unitesk. com 2003 2004 2005

References 1. 2. 3. 4. 5. 6. 7. V. Kuliamin, A. Petrenko, I. Bourdonov, and A. Kossatchev. Uni. Tes. K Test Suite Architecture. Proc. of FME 2002. LNCS 2391, pp. 77 -88, Springer-Verlag, 2002. V. Kuliamin, A. Petrenko, N. Pakoulin, I. Bourdonov, and A. Kossatchev. Integration of Functional and Timed Testing of Real-time and Concurrent Systems. Proc. of PSI 2003. LNCS 2890, pp. 450 -461, Springer-Verlag, 2003. V. Kuliamin, A. Petrenko. Applying Model Based Testing in Different Contexts. Proceedings of seminar on Perspectives of Model Based Testing, Dagstuhl, Germany, September 2004. A. Kossatchev, A. Petrenko, S. Zelenova. Using Model-Based Approach for Automated Testing of Optimizing Compilers. Proc. Intl. Workshop on Program Undestanding, Gorno-Altaisk, 2003. V. Kuliamin, A. Petrenko, A. Kossatchev, and I. Burdonov. The Uni. Tes. K Approach to Designing Test Suites. Programming and Computer Software, Vol. 29, No. 6 , 2003, pp. 310 -322. (Translation from Russian) S. Zelenov, S. Zelenova, A. Kossatchev, A. Petrenko. Test Generation for Compilers and Other Formal Text Processors. Programming and Computer Software, Vol. 29, No. 2 , 2003, pp. 104 -111. (Translation from Russian) V. Kuliamin, N. Pakoulin, A. Petrenko. Practical Approach to Specification and Conformance Testing of Distributed Network Applications. Proc. of 2 -nd International Service Availability Symposium, April 2005, to be published.

Contacts Red. Verst group web site http: //www. ispras. ru/groups/rv/rv. html n Uni. Tes. K projects web site http: //www. unitesk. com n Group leader n Alexander Petrenko petrenko@ispras. ru

Thank You!