Model-Based Testing with Labelled Transition Systems Jan Tretmans

Model-Based Testing with Labelled Transition Systems Jan Tretmans Embedded Systems Institute Eindhoven, NL Radboud University Nijmegen, NL jan. tretmans@esi. nl

Testing: checking or measuring some quality characteristics of an executing object by performing experiments in a controlled way w. r. t. a specification © Jan Tretmans tester IUT 2

Types of Testing Level of detail system integration module unit portability maintainability efficiency usability reliability functionality Accessibility white box black box Characteristics © Jan Tretmans 3

Automated Model-Based Testing test TTCN generation cases tool model IUT confto model test execution tool IUT passes tests © Jan Tretmans pass fail 4

Towards Model Based Testing F Increase in complexity, and quest for higher quality software ¨ testing effort grows exponentially with complexity ¨ testing cannot keep pace with development F More abstraction ¨ less detail ¨ model-based development F Checking quality ¨ practice: testing - ad hoc, too late, expensive, lot of time ¨ research: formal verification - proofs, model checking, . . with disappointing practical impact © Jan Tretmans 5

Towards Model Based Testing F Model based testing has potential to combine ¨ practice - testing ¨ theory - formal methods F Model Based Testing : ¨ testing with respect to a (formal) model / specification state model, pre/post, CSP, Promela, UML, Spec#, . . ¨ promises better, faster, cheaper testing: · algorithmic generation of tests and test oracles : tools · formal and unambiguous basis for testing · measuring the completeness of tests · maintenance of tests through model modification © Jan Tretmans 6

A Model-Based Development Process informal requirements validation informal world formalizable specification formal verification design modelbased testing code world of models real world realization © Jan Tretmans 7

Formal Verification specification s m modcheck s model checker m modcheck s sat of i formal world implementation i Jan Tretmans complete m modcheck s real world © sound sat m sat s model m assumption: m is valid model of i 8

Formal Testing correctness of specification s test generation ? confto formal world IUT test suite TS real world implementation IUT ii © Jan Tretmans IUT test execution confto s test generation passes Ts IUT fails Ts 9

Approaches to Model-Based Testing Several modelling paradigms: F Finite State Machine F Pre/post-conditions F Labelled. Transition Systems Labelled Transition Systems F Programs as Functions F Abstract Data Type testing F. . . . © Jan Tretmans 10

Model-Based Testing for LTS Involves: • model / specification model test generation confto tests IUT test execution © Jan Tretmans • implementation IUT + models of IUTs • correctness • test generation • test execution • test result analysis pass / fail 11

Models of Specifications: Labelled Transition Systems Labelled Transition System S, L, T, s 0 initial states actions transitions s 0 S T S (L { }) S !coffee ? coin !alarm ? button © Jan Tretmans 12

Example Models (Input-Enabled) Transition Systems ? kwart ? dub !coffee !tea ? dub !coffee © Jan Tretmans ? dub ? kwart ? dub !choc !coffee !tea !choc 13

Correctness Implementation Relation ioco i ioco s =def Straces (s) : out (i after ) out (s after ) Intuition: i ioco-conforms to s, iff • if i produces output x after trace , then s can produce x after • if i cannot produce any output after trace , then s cannot produce any output after ( quiescence ) © Jan Tretmans 14

Correctness ioco Implementation Relation i ioco s =def Straces (s) : out (i after ) out (s after ) !x LU { }. p !x Straces ( s ) = { (L { })* | s p after = { p’ | p out ( P ) = { !x LU | p !x p © Jan Tretmans p = } p’ } , p P } { | p p, p P } 15

Implementation Relation ? kwart ? dub ioc o !tea ? dub !coffee Jan Tretmans !choc ? dub ? kwart ? dub !coffee !tea © ? dub !coffee ? dub ? kwart ? dub ioco s i oco ioc o !tea !choc 16

Test Cases !dub Model of a test case = transition system : !kwart ¨ ‘quiescence’ label ¨ tree-structured ? coffee ¨ finite, deterministic ¨ final states pass and fail ¨ from each state pass, fail : · either one input !a · or all outputs ? x and pass © Jan Tretmans fail !dub ? tea fail ? coffee pass fail 17

ioco Test Generation Algorithm To generate a test case from transition system specification s 0 compute T(S), with S a set of states, and initially S = s 0 after ; For T(S), apply the following recursively, non-deterministically: 1 end test case pass 2 supply input !a T( S after ? a ) © Jan Tretmans 3 observe output forbidden outputs ? y allowed outputs ? x fail T ( S after !x ) allowed outputs or : !x out ( S ) forbidden outputs or : !y out ( S ) 18

Test Generation Example s test ? dub !dub ? coffee !coffee ? tea ? coffee ? tea fail © Jan Tretmans fail pass 19

Test Execution Example i test ? dub !dub ? coffee !coffee ? tea ? dub ? coffee ? tea Two test runs : t i dub coffee © Jan Tretmans pass i' fail pass i passes t 20

Test Result Analysis Completeness of ioco Test Generation For every test t generated with algorithm we have: F Soundness : t will never fail with correct implementation i ioco s implies i passes t F Exhaustiveness : each incorrect implementation can be detected with a generated test t i ioco s © Jan Tretmans implies t : i fails t 21

Formal Testing with Transition Systems Test hypothesis : s LTS ioco = gen : LTS (TTS) Proof soundness and exhaustiveness: Ts TTS i. IUT IOTS IUT IMPS passes : exec : IOTS TTS TESTS IMPS {pass, fail} (OBS) © Jan Tretmans IUT IMP. i. IUT IOTS. t TTS. IUT passes t i. IUT passes t i IOTS. ( t gen(s). i passes t ) i ioco s pass / fail 22

Variations on a Theme i ioco s Straces(s) : out ( i after ) out ( s after ) ior s ( L { } )* : out ( i after ) out ( s after ) i ioconf s traces(s) : out ( i after ) out ( s after ) i ioco. F s i mioco s F : i out ( i after ) multi-channel ioco i uioco s universal ioco i wioco s non-input-enabled ioco i sioco s symbolic ioco i (r)tioco s (real) timed tioco i iocor s i hioco s © out ( s after ) refinement ioco . . . Jan Tretmans (Aalborg, Twente, Grenoble, Bordeaux, . . ) hybrid ioco 23

Testing Transition Systems: Extensions Status model with data and time and hybrid and action refinement test case ? coin 1 ? coin 2 ? money n: int ! money ? coin 3 ? [ n 35 ] -> [ n 50 ] -> ? button 1 ? button 2 c : = 0 c t : = 0 Vc : = 0 V : = 0 c dt d Vt </10 = 3 [Vt = 15 ] -> ! tea ! button 2 c / dt d Vc< 15 = 2 [[Vc = 10 -> -> c 5] ] ! coffee ? tea pass © Jan Tretmans fai l 24

but Component Based Testing but ? but !err ? but ok err ? but © ? err Jan Tretmans X y !y i 2 ioco s 2 ? ok ? err ioco ? ok ? err !x i 1||i 2 ? but ok err ? ok ? err !ok ? but ? ok ? but !y i 1 ioco s 1 !x !x X y s 1||s 2 25

Compositional Testing Component Based Testing i 1 ioco s 1 i 2 ioco s 2 i 1 || i 2 ioco s 1 || s 2 If s 1, s 2 input enabled - s 1, s 2 IOTS - then ioco is preserved ! © Jan Tretmans 26

Variations on a Theme: uioco i ioco s s 0 ? a Straces(s) : out ( i after ) out ( s 0 after ) ? a s 1 ? a ? b !x !y s 2 out ( s 0 after ? b ) = but ? b Straces(s) : under-specification : anything allowed after ? b !z out ( s 0 after ? a ) = { !x } and ? a Straces(s) but from s 2 , ? a is under-specified : anything allowed after ? a ? © Jan Tretmans 27

Variations on a Theme: uioco i uioco s Utraces(s) : out ( i after ) out ( s 0 after ) Utraces(s) = s 0 ? a s 1 ? a ? b s 2 ? b { Straces (s) | 1 ? a 2 = , s': s Now !x !y !z Jan Tretmans s' ? a } s is under-specified in s 2 for ? a : anything is allowed. ioco © 1 uioco 28

Variations on a Theme: uioco i uioco s Utraces(s) : out ( i after ) out ( s 0 after ) s 0 ? a s 1 ? a !x © Jan Tretmans ? b !y s 2 ? b LI L I L U ? a !z Alternatively, via chaos process for under-specified inputs 29

Testing Components method invocation IUT component method call method return IUT method invocations || IUT methods invoked component method called returned component © Jan Tretmans 30

Testing Components method call specification s LTS(LI, LU) IUT component method tester call LI © method return = offered methods calls used methods returns LU = offered methods returns used methods calls Jan Tretmans 31

Testing Components method call method return IUT component method call tester © Jan Tretmans Input-enabledness: s of IUT, ? a LI : s ? a ? No ! ? method return 32

Correctness Implementation Relation wioco i uioco s =def Utraces (s) : out (i after ) out (s after ) i wioco s =def Utraces (s) : out (i after ) out (s after ) and in (s after ) s after © Jan Tretmans in (i after ) in (s after ) = { a? LI | s after must a? } must a? = s’ ( s s’ a? ) 33

Formal Testing with Transition Systems Test hypothesis : s LTS ioco = gen : LTS (TTS) Proof soundness and exhaustiveness: Ts TTS i. IUT IOTS IUT IMPS passes : exec : IOTS TTS TESTS IMPS {pass, fail} (OBS) © Jan Tretmans IUT IMP. i. IUT IOTS. t TTS. IUT passes t i. IUT passes t i IOTS. ( t gen(s). i passes t ) i ioco s pass / fail 34

Test Assumption input a? LI IUT output x? LU quiescence Sequencing of inputs, outputs, and : IUT Input-enabledness: s of IUT, ? a LI : s ? a IUT behaves as an IOTS (input-enabled LTS) © Jan Tretmans 35

Comparing Transition Systems Testing Equivalences S 1 S 2 environment F Suppose an environment interacts with the systems: ¨ the environment tests the system as black box by observing and actively controlling it; ¨ the environment acts as a tester; F Two systems are equivalent if they pass the same tests. © Jan Tretmans 36

Formal Testing : Test Assumption Test assumption : IUT. i. IUT MOD. t TEST. IUT passes t i. IUT passes t IUT test t © Jan Tretmans i. IUT test t 37

Completeness of Formal Testing IUT passes Ts ? IUT confto s IUT passes Ts t Ts. IUT passes t def IUT passes Ts Test. i. IUT passes t Ts hypothesis : t © i. IUT Jan Tretmans t Ts. IUT passes t Proof. TEST. IUT : passes t i. IUT passes t t obligation imp s i MOD. Definition confto s ( t Ts. i passes t: ) IUT confto ss i imp 38

Test Assumption ? dub !coffee !tea ? dub ioco !tea !choc s More tests may be needed, starting in initial state: © Jan Tretmans meta-assumption: reliable restart 39

Alternative Test Assumption ioco ? dub !tea !choc Test: 1. 2. 3. 4. ioco ? dub !tea !choc do ? dub make core dump make many copies of core dump continue test with each copy An “Ambramsky”-test can distinguish them © Jan Tretmans 40

Alternative Test Assumption ? dub ? kwart !tea ? kwart !choc ioco ? dub ? kwart !tea ? dub ? kwart !choc With test ? dub. ? kwart. undo you can distinguish them © Jan Tretmans 41

Concluding F Testing can be formal, too (M. -C. Gaudel, TACAS'95) ¨ Testing shall be formal, too F A test generation algorithm is not just another algorithm : ¨ Proof of soundness and exhaustiveness ¨ Definition of test assumption and implementation relation F For labelled transition systems : ¨ ioco for expressing conformance between imp and spec ¨ a sound and exhaustive test generation algorithm ¨ tools generating and executing tests: TGV, Test. Gen, Agedis, Tor. X, . . © Jan Tretmans 42

Perspectives Model based formal testing can improve the testing process : J model is precise and unambiguous basis for testing ¨ design errors found during validation of model J longer, cheaper, more flexible, and provably correct tests ¨ easier test maintenance and regression testing J automatic test generation and execution ¨ full automation : test generation + execution + analysis J extra effort of modelling compensated by better tests © Jan Tretmans 43