Скачать презентацию Model-Based Testing with Labelled Transition Systems Jan Tretmans Скачать презентацию Model-Based Testing with Labelled Transition Systems Jan Tretmans

08acf2743998dcb80dc14a62c3a15c03.ppt

  • Количество слайдов: 44

Model-Based Testing with Labelled Transition Systems Jan Tretmans Embedded Systems Institute Eindhoven, NL Radboud Model-Based Testing with Labelled Transition Systems Jan Tretmans Embedded Systems Institute Eindhoven, NL Radboud University Nijmegen, NL jan. [email protected] nl

Testing: checking or measuring some quality characteristics of an executing object by performing experiments Testing: checking or measuring some quality characteristics of an executing object by performing experiments in a controlled way w. r. t. a specification © Jan Tretmans tester IUT 2

Types of Testing Level of detail system integration module unit portability maintainability efficiency usability Types of Testing Level of detail system integration module unit portability maintainability efficiency usability reliability functionality Accessibility white box black box Characteristics © Jan Tretmans 3

Automated Model-Based Testing test TTCN generation cases tool model IUT confto model test execution Automated Model-Based Testing test TTCN generation cases tool model IUT confto model test execution tool IUT passes tests © Jan Tretmans pass fail 4

Towards Model Based Testing F Increase in complexity, and quest for higher quality software Towards Model Based Testing F Increase in complexity, and quest for higher quality software ¨ testing effort grows exponentially with complexity ¨ testing cannot keep pace with development F More abstraction ¨ less detail ¨ model-based development F Checking quality ¨ practice: testing - ad hoc, too late, expensive, lot of time ¨ research: formal verification - proofs, model checking, . . with disappointing practical impact © Jan Tretmans 5

Towards Model Based Testing F Model based testing has potential to combine ¨ practice Towards Model Based Testing F Model based testing has potential to combine ¨ practice - testing ¨ theory - formal methods F Model Based Testing : ¨ testing with respect to a (formal) model / specification state model, pre/post, CSP, Promela, UML, Spec#, . . ¨ promises better, faster, cheaper testing: · algorithmic generation of tests and test oracles : tools · formal and unambiguous basis for testing · measuring the completeness of tests · maintenance of tests through model modification © Jan Tretmans 6

A Model-Based Development Process informal requirements validation informal world formalizable specification formal verification design A Model-Based Development Process informal requirements validation informal world formalizable specification formal verification design modelbased testing code world of models real world realization © Jan Tretmans 7

Formal Verification specification s m modcheck s model checker m modcheck s sat of Formal Verification specification s m modcheck s model checker m modcheck s sat of i formal world implementation i Jan Tretmans complete m modcheck s real world © sound sat m sat s model m assumption: m is valid model of i 8

Formal Testing correctness of specification s test generation ? confto formal world IUT test Formal Testing correctness of specification s test generation ? confto formal world IUT test suite TS real world implementation IUT ii © Jan Tretmans IUT test execution confto s test generation passes Ts IUT fails Ts 9

Approaches to Model-Based Testing Several modelling paradigms: F Finite State Machine F Pre/post-conditions F Approaches to Model-Based Testing Several modelling paradigms: F Finite State Machine F Pre/post-conditions F Labelled. Transition Systems Labelled Transition Systems F Programs as Functions F Abstract Data Type testing F. . . . © Jan Tretmans 10

Model-Based Testing for LTS Involves: • model / specification model test generation confto tests Model-Based Testing for LTS Involves: • model / specification model test generation confto tests IUT test execution © Jan Tretmans • implementation IUT + models of IUTs • correctness • test generation • test execution • test result analysis pass / fail 11

Models of Specifications: Labelled Transition Systems Labelled Transition System S, L, T, s 0 Models of Specifications: Labelled Transition Systems Labelled Transition System S, L, T, s 0 initial states actions transitions s 0 S T S (L { }) S !coffee ? coin !alarm ? button © Jan Tretmans 12

Example Models (Input-Enabled) Transition Systems ? kwart ? dub !coffee !tea ? dub !coffee Example Models (Input-Enabled) Transition Systems ? kwart ? dub !coffee !tea ? dub !coffee © Jan Tretmans ? dub ? kwart ? dub !choc !coffee !tea !choc 13

Correctness Implementation Relation ioco i ioco s =def Straces (s) : out (i after Correctness Implementation Relation ioco i ioco s =def Straces (s) : out (i after ) out (s after ) Intuition: i ioco-conforms to s, iff • if i produces output x after trace , then s can produce x after • if i cannot produce any output after trace , then s cannot produce any output after ( quiescence ) © Jan Tretmans 14

Correctness ioco Implementation Relation i ioco s =def Straces (s) : out (i after Correctness ioco Implementation Relation i ioco s =def Straces (s) : out (i after ) out (s after ) !x LU { }. p !x Straces ( s ) = { (L { })* | s p after = { p’ | p out ( P ) = { !x LU | p !x p © Jan Tretmans p = } p’ } , p P } { | p p, p P } 15

Implementation Relation ? kwart ? dub ioc o !tea ? dub !coffee Jan Tretmans Implementation Relation ? kwart ? dub ioc o !tea ? dub !coffee Jan Tretmans !choc ? dub ? kwart ? dub !coffee !tea © ? dub !coffee ? dub ? kwart ? dub ioco s i oco ioc o !tea !choc 16

Test Cases !dub Model of a test case = transition system : !kwart ¨ Test Cases !dub Model of a test case = transition system : !kwart ¨ ‘quiescence’ label ¨ tree-structured ? coffee ¨ finite, deterministic ¨ final states pass and fail ¨ from each state pass, fail : · either one input !a · or all outputs ? x and pass © Jan Tretmans fail !dub ? tea fail ? coffee pass fail 17

ioco Test Generation Algorithm To generate a test case from transition system specification s ioco Test Generation Algorithm To generate a test case from transition system specification s 0 compute T(S), with S a set of states, and initially S = s 0 after ; For T(S), apply the following recursively, non-deterministically: 1 end test case pass 2 supply input !a T( S after ? a ) © Jan Tretmans 3 observe output forbidden outputs ? y allowed outputs ? x fail T ( S after !x ) allowed outputs or : !x out ( S ) forbidden outputs or : !y out ( S ) 18

Test Generation Example s test ? dub !dub ? coffee !coffee ? tea ? Test Generation Example s test ? dub !dub ? coffee !coffee ? tea ? coffee ? tea fail © Jan Tretmans fail pass 19

Test Execution Example i test ? dub !dub ? coffee !coffee ? tea ? Test Execution Example i test ? dub !dub ? coffee !coffee ? tea ? dub ? coffee ? tea Two test runs : t i dub coffee © Jan Tretmans pass i' fail pass i passes t 20

Test Result Analysis Completeness of ioco Test Generation For every test t generated with Test Result Analysis Completeness of ioco Test Generation For every test t generated with algorithm we have: F Soundness : t will never fail with correct implementation i ioco s implies i passes t F Exhaustiveness : each incorrect implementation can be detected with a generated test t i ioco s © Jan Tretmans implies t : i fails t 21

Formal Testing with Transition Systems Test hypothesis : s LTS ioco = gen : Formal Testing with Transition Systems Test hypothesis : s LTS ioco = gen : LTS (TTS) Proof soundness and exhaustiveness: Ts TTS i. IUT IOTS IUT IMPS passes : exec : IOTS TTS TESTS IMPS {pass, fail} (OBS) © Jan Tretmans IUT IMP. i. IUT IOTS. t TTS. IUT passes t i. IUT passes t i IOTS. ( t gen(s). i passes t ) i ioco s pass / fail 22

Variations on a Theme i ioco s Straces(s) : out ( i after ) Variations on a Theme i ioco s Straces(s) : out ( i after ) out ( s after ) ior s ( L { } )* : out ( i after ) out ( s after ) i ioconf s traces(s) : out ( i after ) out ( s after ) i ioco. F s i mioco s F : i out ( i after ) multi-channel ioco i uioco s universal ioco i wioco s non-input-enabled ioco i sioco s symbolic ioco i (r)tioco s (real) timed tioco i iocor s i hioco s © out ( s after ) refinement ioco . . . Jan Tretmans (Aalborg, Twente, Grenoble, Bordeaux, . . ) hybrid ioco 23

Testing Transition Systems: Extensions Status model with data and time and hybrid and action Testing Transition Systems: Extensions Status model with data and time and hybrid and action refinement test case ? coin 1 ? coin 2 ? money n: int ! money ? coin 3 ? [ n 35 ] -> [ n 50 ] -> ? button 1 ? button 2 c : = 0 c t : = 0 Vc : = 0 V : = 0 c dt d Vt ! tea ! button 2 c / dt d Vc< 15 = 2 [[Vc = 10 -> -> c 5] ] ! coffee ? tea pass © Jan Tretmans fai l 24

but Component Based Testing but ? but !err ? but ok err ? but but Component Based Testing but ? but !err ? but ok err ? but © ? err Jan Tretmans X y !y i 2 ioco s 2 ? ok ? err ioco ? ok ? err !x i 1||i 2 ? but ok err ? ok ? err !ok ? but ? ok ? but !y i 1 ioco s 1 !x !x X y s 1||s 2 25

Compositional Testing Component Based Testing i 1 ioco s 1 i 2 ioco s Compositional Testing Component Based Testing i 1 ioco s 1 i 2 ioco s 2 i 1 || i 2 ioco s 1 || s 2 If s 1, s 2 input enabled - s 1, s 2 IOTS - then ioco is preserved ! © Jan Tretmans 26

Variations on a Theme: uioco i ioco s s 0 ? a Straces(s) : Variations on a Theme: uioco i ioco s s 0 ? a Straces(s) : out ( i after ) out ( s 0 after ) ? a s 1 ? a ? b !x !y s 2 out ( s 0 after ? b ) = but ? b Straces(s) : under-specification : anything allowed after ? b !z out ( s 0 after ? a ) = { !x } and ? a Straces(s) but from s 2 , ? a is under-specified : anything allowed after ? a ? © Jan Tretmans 27

Variations on a Theme: uioco i uioco s Utraces(s) : out ( i after Variations on a Theme: uioco i uioco s Utraces(s) : out ( i after ) out ( s 0 after ) Utraces(s) = s 0 ? a s 1 ? a ? b s 2 ? b { Straces (s) | 1 ? a 2 = , s': s Now !x !y !z Jan Tretmans s' ? a } s is under-specified in s 2 for ? a : anything is allowed. ioco © 1 uioco 28

Variations on a Theme: uioco i uioco s Utraces(s) : out ( i after Variations on a Theme: uioco i uioco s Utraces(s) : out ( i after ) out ( s 0 after ) s 0 ? a s 1 ? a !x © Jan Tretmans ? b !y s 2 ? b LI L I L U ? a !z Alternatively, via chaos process for under-specified inputs 29

Testing Components method invocation IUT component method call method return IUT method invocations || Testing Components method invocation IUT component method call method return IUT method invocations || IUT methods invoked component method called returned component © Jan Tretmans 30

Testing Components method call specification s LTS(LI, LU) IUT component method tester call LI Testing Components method call specification s LTS(LI, LU) IUT component method tester call LI © method return = offered methods calls used methods returns LU = offered methods returns used methods calls Jan Tretmans 31

Testing Components method call method return IUT component method call tester © Jan Tretmans Testing Components method call method return IUT component method call tester © Jan Tretmans Input-enabledness: s of IUT, ? a LI : s ? a ? No ! ? method return 32

Correctness Implementation Relation wioco i uioco s =def Utraces (s) : out (i after Correctness Implementation Relation wioco i uioco s =def Utraces (s) : out (i after ) out (s after ) i wioco s =def Utraces (s) : out (i after ) out (s after ) and in (s after ) s after © Jan Tretmans in (i after ) in (s after ) = { a? LI | s after must a? } must a? = s’ ( s s’ a? ) 33

Formal Testing with Transition Systems Test hypothesis : s LTS ioco = gen : Formal Testing with Transition Systems Test hypothesis : s LTS ioco = gen : LTS (TTS) Proof soundness and exhaustiveness: Ts TTS i. IUT IOTS IUT IMPS passes : exec : IOTS TTS TESTS IMPS {pass, fail} (OBS) © Jan Tretmans IUT IMP. i. IUT IOTS. t TTS. IUT passes t i. IUT passes t i IOTS. ( t gen(s). i passes t ) i ioco s pass / fail 34

Test Assumption input a? LI IUT output x? LU quiescence Sequencing of inputs, outputs, Test Assumption input a? LI IUT output x? LU quiescence Sequencing of inputs, outputs, and : IUT Input-enabledness: s of IUT, ? a LI : s ? a IUT behaves as an IOTS (input-enabled LTS) © Jan Tretmans 35

Comparing Transition Systems Testing Equivalences S 1 S 2 environment F Suppose an environment Comparing Transition Systems Testing Equivalences S 1 S 2 environment F Suppose an environment interacts with the systems: ¨ the environment tests the system as black box by observing and actively controlling it; ¨ the environment acts as a tester; F Two systems are equivalent if they pass the same tests. © Jan Tretmans 36

Formal Testing : Test Assumption Test assumption : IUT. i. IUT MOD. t TEST. Formal Testing : Test Assumption Test assumption : IUT. i. IUT MOD. t TEST. IUT passes t i. IUT passes t IUT test t © Jan Tretmans i. IUT test t 37

Completeness of Formal Testing IUT passes Ts ? IUT confto s IUT passes Ts Completeness of Formal Testing IUT passes Ts ? IUT confto s IUT passes Ts t Ts. IUT passes t def IUT passes Ts Test. i. IUT passes t Ts hypothesis : t © i. IUT Jan Tretmans t Ts. IUT passes t Proof. TEST. IUT : passes t i. IUT passes t t obligation imp s i MOD. Definition confto s ( t Ts. i passes t: ) IUT confto ss i imp 38

Test Assumption ? dub !coffee !tea ? dub ioco !tea !choc s More tests Test Assumption ? dub !coffee !tea ? dub ioco !tea !choc s More tests may be needed, starting in initial state: © Jan Tretmans meta-assumption: reliable restart 39

Alternative Test Assumption ioco ? dub !tea !choc Test: 1. 2. 3. 4. ioco Alternative Test Assumption ioco ? dub !tea !choc Test: 1. 2. 3. 4. ioco ? dub !tea !choc do ? dub make core dump make many copies of core dump continue test with each copy An “Ambramsky”-test can distinguish them © Jan Tretmans 40

Alternative Test Assumption ? dub ? kwart !tea ? kwart !choc ioco ? dub Alternative Test Assumption ? dub ? kwart !tea ? kwart !choc ioco ? dub ? kwart !tea ? dub ? kwart !choc With test ? dub. ? kwart. undo you can distinguish them © Jan Tretmans 41

Concluding F Testing can be formal, too (M. -C. Gaudel, TACAS'95) ¨ Testing shall Concluding F Testing can be formal, too (M. -C. Gaudel, TACAS'95) ¨ Testing shall be formal, too F A test generation algorithm is not just another algorithm : ¨ Proof of soundness and exhaustiveness ¨ Definition of test assumption and implementation relation F For labelled transition systems : ¨ ioco for expressing conformance between imp and spec ¨ a sound and exhaustive test generation algorithm ¨ tools generating and executing tests: TGV, Test. Gen, Agedis, Tor. X, . . © Jan Tretmans 42

Perspectives Model based formal testing can improve the testing process : J model is Perspectives Model based formal testing can improve the testing process : J model is precise and unambiguous basis for testing ¨ design errors found during validation of model J longer, cheaper, more flexible, and provably correct tests ¨ easier test maintenance and regression testing J automatic test generation and execution ¨ full automation : test generation + execution + analysis J extra effort of modelling compensated by better tests © Jan Tretmans 43

Thank You © Jan Tretmans 44 Thank You © Jan Tretmans 44