Model-Based Testing Ed Brinksma University of Twente Dept

Model-Based Testing Ed Brinksma University of Twente Dept. of Computer Science Formal Methods & Tools group Enschede The Netherlands ARTIST 2 Summer School October 1, 2005 WTRTES, Pisa Nässlingen

Contents l introduction & background l testing pre-orders l input/output & quiescence l ioco implementation relation l test generation l Tor. X l test case study l real-time testing October 1, 2005 ARTIST 2 Summer School 2

Contents l introduction & background l testing pre-orders l input/output & quiescence l ioco implementation relation l test generation l Tor. X l test case study l real-time testing October 1, 2005 ARTIST 2 Summer School 3

Practical problems of testing Testing is: But also: l important l ad-hoc, manual, error-prone l much practiced l hardly theory / research l 30% - 50% of project effort l no attention in curricula l expensive l not cool : “if you’re a bad programmer you might be a tester” l time critical l not constructive (but sadistic? ) sible pos s ! ents thod ovem al me mpr orm I ith f w October 1, 2005 ? Attitude is changing: l more awareness l more professional ARTIST 2 Summer School 4

Types of Testing Level system integration unit Accessibility robustness performance white box black box usability reliability functional behaviour Aspect October 1, 2005 ARTIST 2 Summer School 5

Test Automation Traditional test automation Why not = tools to execute and manage test cases generate test automatically? ! specification test TTCN cases pass implementation under test tool fail October 1, 2005 ARTIST 2 Summer School 6

Verification and Testing Verification : l formal manipulation l prove properties l performed on model Testing : l experimentation l show error l concrete system formal world concrete world Verification is only as good as the validity of the model on which it is based October 1, 2005 Testing can only show the presence of errors, not their absence ARTIST 2 Summer School 7

Testing with Formal Methods l Testing with respect to a formal specification l Precise, formal definition of correctness : good and unambiguous basis for testing l Formal validation of tests l Algorithmic derivation of tests : tools for automatic test generation l Allows to define measures expressing coverage and quality of testing October 1, 2005 ARTIST 2 Summer School 8

Challenges of Testing Theory l Infinity of testing: n too many possible input combinations -- infinite breadth n too many possible input sequences -- infinite depth n too many invalid and unexpected inputs l Exhaustive testing never possible: n when to stop testing ? n how to invent effective and efficient test cases with high probability of detecting errors ? l Optimization problem of testing yield and invested effort n usually stop when time is over. . . October 1, 2005 ARTIST 2 Summer School 9

Formal Testing S correctness criterion implementation relation imp exhaustive test generation i imps specification sound i passes Ts test suite TS implementatio i n test execution October 1, 2005 ARTIST 2 Summer School pass / fail 10

Formal Testing : Conformance specification S IUT conforms-to s correctness criterion implementatio n IUT s SPECS Specification IUT Implementation under Test IUT is concrete, physical object Model the physical world But IUT is black box ! ? Assume that model i. IUT exists October 1, 2005 ARTIST 2 Summer School 11

Contents l introduction & background l testing pre-orders l input/output & quiescence l ioco implementation relation l test generation l Tor. X l test case study l real-time testing October 1, 2005 ARTIST 2 Summer School 12

Testing Preorders on Transition Systems implementation i specification s environment e For all environments e all observations of an implementation i in e i s e Env. obs ( e, i ) obs (e, s ) should be explained by observations of the specification in e. s ? October 1, 2005 ? ARTIST 2 Summer School ? 13

Classical Testing Preorder implementation i te specification s environment e i te s environment e e E. obs ( e, i ) obs (e, s ) LTS(L) Deadlocks(e||s) October 1, 2005 ARTIST 2 Summer School 14

Classical Testing Preorder implementation i te environment e i te s environment e e LTS(L). L*. e||i |deadlocks after } e deadlocks after { e||i deadlocks ||s { | e||s deadlocks after } FP (i) FP (s) FP (p) October 1, 2005 specification s = { , A | p afer refuses A } ARTIST 2 Summer School 15

Quirky Coffee Machine [Langerak] Can we distinguish between these machines? coin tea coin bang coffee coin coffee te tea ARTIST 2 Summer School bang coffee tea They are testing equivalent! October 1, 2005 coin 16

Refusal Preorder implementation i rf specification s Deadlocks δ(e||i) = { (L {δ})* | environment e e||i deadlocks after } environment e i rf s e E. obs ( e, i ) obs (e, s ) e observes with δ deadlock on all alternative actions October 1, 2005 LTS(L {δ}) Deadlocks δ(e||i) ARTIST 2 Summer School 17

Quirky Coffee Machine Revisited coin tea coin bang coffee tea coin te tea rf tea coin coffee bang October 1, 2005 δ only enabled if coffee is not tester ARTIST 2 Summer School coffee bang coffee coin 18

Contents l introduction & background l testing pre-orders l input/output & quiescence l ioco implementation relation l test generation l Tor. X l test case study l real-time testing October 1, 2005 ARTIST 2 Summer School 19

I/O Transition Systems l testing actions are usually directed, i. e. there are inputs and outputs L=Lin Lout with Lin Lout= l systems can always accept all inputs (input enabledness) for all states s, for all a Lin s a l testers are I/O systems n output (stimulus) is input for the SUT n input (response) is output of the SUT October 1, 2005 ARTIST 2 Summer School 20

Quiescence l Because of input enabledness S||T deadlocks iff T produces no stimuli and S no responses. This is known as quiescence l Observing quiescence leads to two implementation relations for I/O systems I and S: 1. I iote S iff for all I/O testers T: n Deadlocks(I||T) Deadlocks(S||T) (quiescence) 2. I iorf S iff for all I/O testers T: n Deadlocksδ(I||T) Deadlocksδ(S||T) (repetitive quiescence) October 1, 2005 ARTIST 2 Summer School 21

Input-Output QCM states must be saturated with input coin? coffee? loops for input tea? enabledness bang? coin! coin? bang? tea? coffee? coffee! tea ! coffee? coffee! October 1, 2005 quiescen t states iote tea? iorf ARTIST 2 Summer School 22 bang! coffee ! tea ! coffee?

Contents l introduction & background l testing pre-orders l input/output & quiescence l ioco implementation relation l test generation l Tor. X l test case study l real-time testing October 1, 2005 ARTIST 2 Summer School 23

Implementation Relation ioco δ By adding a transition p p to every quiescent state of a system we treat quiescence as an observable (synchronizable) action: i iorf s I/O tests T: Deadlocksδ(i||T) Deadlocksδ(s||T) ( L { } )*: out ( i after ) out ( s after ) To allow under-specification we restrict the set of traces: i ioco s Tracesδ( s ) : out ( i after ) out ( s after ) October 1, 2005 ARTIST 2 Summer School 24

Implementation Relation ioco Correctness expressed by implementation relation ioco: i ioco s =def Tracesδ( s ) : out (i after ) out (s after ) Intuition: i ioco-conforms to s, iff if i produces output x after trace , then s can produce x after if i cannot produce any output after trace , then s cannot produce any output after ( quiescence ) October 1, 2005 ARTIST 2 Summer School 25

Implementation Relation ioco i ioco s =def Straces (s) : out (i after ) out (s after ) i out ( i after e ) ? dub ? kwart out ( i after ? dub ) = { !coffee } out ( i after ? dub ) ? kwart = { } = { !coffee } out ( i after ? dub. !coffee) = { } October 1, 2005 = { } = out ( i after ? dub. !tea ) = out ( i after ) ? dub ? kwart out ( i after ? kwart ) out ( i after !coffee ) !coffee = { } ARTIST 2 Summer School 26

Implementation Relation ioco i ioco s =def Straces (s) : out (i after ) out (s after ) i s ? dub !tea !coffee ioco ? dub out (i after ? dub) = { !coffee } October 1, 2005 !coffee out (s after ? dub) = { !coffee, !tea } ARTIST 2 Summer School 27

Implementation Relation ioco i ioco s =def Straces (s) : out (i after ) out (s after ) i s ? dub !tea ? dub !coffee ioco ? dub out (i after ? dub) = { !coffee, !tea } October 1, 2005 !coffee ARTIST 2 Summer School out (s after ? dub) = { !coffee} 28

Implementation Relation ioco i ioco s =def Straces (s) : out (i after ) out (s after ) i ? dub s ? kwart ? dub ? kwart !coffee out (i after ? dub) !tea = { !coffee } out (i after ? kwart) = { !tea } !coffee ioco out (s after ? dub) = { !coffee } out (s after ? kwart) = But ? kwart Tracesδ( s ) October 1, 2005 ARTIST 2 Summer School 29

Contents l introduction & background l testing pre-orders l input/output & quiescence l ioco implementation relation l test generation l Tor. X l test case study l real-time testing October 1, 2005 ARTIST 2 Summer School 30

Formal Testing i iocos specification S correctness criterion implementation relation ioco ? test generation i passes Ts test suite TS implementatio i n test execution October 1, 2005 ARTIST 2 Summer School pass / fail 31

Test Cases Test case t TTS coin? TTS - Test Transition System : coin ? n labels in L { } n tree-structured n finite, deterministic coffee! n final states pass and fail n from each state pass, fail l or all outputs o! and tea! pass October 1, 2005 ARTIST 2 Summer School fail coin? l either one input i? 32 tea! coffee! pass fail

Test Cases coin? test case t !coin ? !coin ; Start timer 1 ? tea fail ? timer 1 fail coffee! tea! ? coffee !coin ; Start timer 2 ? tea pass ? coffee coin? pass ? timer 2 fail tea! pass October 1, 2005 ARTIST 2 Summer School fail 33 coffee! pass fail

Test Generation Algorithm To generate a test case t(S) from a transition system specification with S set of states ( initially S = {s 0} ) Apply the following steps recursively, non-deterministically 1 end test case 3 PASS 2 observe output o! supply input allowed outputs o! FAIL forbidden outputs t(S after o!) supply i? t(S after i? ) October 1, 2005 ARTIST 2 Summer School 34

Test Generation Example Equation solver for y 2=x specification test !9 ? x (x < 0) ? x (x >= 0) ! x otherwise ! - x ? 3 FAIL PASS otherwise To cope with non-deterministic behaviour, tests are not linear traces, but trees October 1, 2005 ARTIST 2 Summer School FAIL 35 ? -3 !4 ? 2 PASS ? -2 PASS

Validity of Test Generation For every test t generated with algorithm : l Soundness : t will never fail with correct implementation i ioco s implies i passes t l Exhaustiveness : each incorrect implementation can be detected with a generated test t i ioco s October 1, 2005 implies t : i fails t ARTIST 2 Summer School 36

Contents l introduction & background l testing pre-orders l input/output & quiescence l ioco implementation relation l test generation l Tor. X l test case study l real-time testing October 1, 2005 ARTIST 2 Summer School 37

Formal Testing with Transition Systems s LTS ioco der : LTS (TTS) Ts TTS pass i. IUT IOTS obs : TTS IOTS (traces) October 1, 2005 ARTIST 2 Summer School traces 38 t : (traces) {fail, pass} fail

Test Generation Tools for ioco l TVEDA (CNET - France Telecom) n derives TTCN tests from single process SDL specification n developed from practical experiences n implementation relation R 1 ioco l TGV (IRISA - Rennes) n derives tests in TTCN from LOTOS or SDL n uses test purposes to guide test derivation n implementation relation: unfair extension of ioco l Test. Composer n Combination of TVEDA and TGV in Object. Geode l Test. Gen (Stirling) n Test generation for hardware validation l Tor. X (Côte de Resyste) October 1, 2005 ARTIST 2 Summer School 39

A Test Tool : Tor. X l On-the-fly test generation and test execution l Implementation relation: ioco l Specification languages: LOTOS, Promela, FSP, Automata user: manual automatic next input specification check output offer input Tor. X observe output pass fail inconclusive October 1, 2005 ARTIST 2 Summer School 40 IUT

Tor. X Tool Architecture spec. explorer specification text October 1, 2005 primer states transitions Tor. X driver abstract actions ARTIST 2 Summer School adapter abstract actions 41 IUT concrete actions IUT

On-the-Fly Batch Testing on the fly spec. explorer batch test generation October 1, 2005 primer driver adapter TTCN test taal ARTIST 2 Summer School IUT batch test execution 42

spec On-the-Fly Testing New menu Concrete action Action Choice !! x x(x << 0) Action Choice Check Abstract action Concrete action Check<0) !x (x (x 0) Abstract action Concrete action ! 00001001 ! -1 !! x x(x 3 >=0) (quiescence) ? -13 9 ? ? 1111 00000011 ? (x>= 0) ? (x >= 0) !x (quiescence) ? ? ! 3 ! ? 9 (quiescence) ! ! (timeout) explorer states transition s primer transitio n driver t abstrac actions adapter specification IUT ? x (x < 0) ? x (x >= 0) October 1, 2005 bytes implementation ? x (x < 0) ! x bits ! x ! - x ARTIST 2 Summer School ? x 43

Tor. X October 1, 2005 ARTIST 2 Summer School 44

Contents l introduction & background l testing pre-orders l input/output & quiescence l ioco implementation relation l test generation l Tor. X l test case study l real-time testing October 1, 2005 ARTIST 2 Summer School 45

Tor. X Case Studies academic l Conference Protocol Philips l Easy. Link TV-VCR protocol CMG l Cell Broadcast Centre component Interpay l Road Toll Payment Box protocol Lucent l V 5. 1 Access Network protocol CMG l Easy Mail Melder academic l FTP Client l “Oosterschelde” storm surge barrier-control l TANGRAM: testing VLSI lithography machine October 1, 2005 ARTIST 2 Summer School 46 CMG ASML

Interpay Highway Tolling System October 1, 2005 ARTIST 2 Summer School 47

Highway Tolling Protocol Characteristics : l Simple protocol l Parallellism : many cars at the same time l Encryption l System passed traditional testing phase October 1, 2005 ARTIST 2 Summer School 48

Highway Tolling System Payment Box Onboard Unit (PB) Road Side Equipment UDP/IP Wireless October 1, 2005 ARTIST 2 Summer School 49

Highway Tolling: Test Architecture spec PB + Obu. Sim + TCP/IP + UDP/IP Test Context Tor. X Obu. Sim PCO TCP/IP UDP/IP Payment Box IAP SUT October 1, 2005 ARTIST 2 Summer School 50

Highway Tolling: Results l Test results : n 1 error during validation (design error) n 1 error during testing (coding error) l Automated testing : n beneficial: high volume and reliability n many and long tests executed ( > 50, 000 test events ) n very flexible: adaptation and many configurations l Real-time : n interference computation time on-the-fly testing n interference quiescence and time-outs l Step ahead in formal testing of realistic systems October 1, 2005 ARTIST 2 Summer School 51

Contents l introduction & background l testing pre-orders l input/output & quiescence l ioco implementation relation l test generation l Tor. X l test case study l real-time testing October 1, 2005 ARTIST 2 Summer School 52

RT Tor. X Hacking Approaches 1. Ignore RT functionality: 1. test pure functional behaviour 2. analyse timing requirements using Tor. X log files & assumed timing constraints 2. Add timestamps to observations l adapter adds timestamps to observations when they are made and passed on to the driver l timestamps are used to analyse Tor. X log files 3. Add timestamps to stimuli & observations l adapter add timestamps to observations when they are made and passed on to the driver 1. adapter adds timestamps to stimuli when they are applied and returned to the driver 2. analysis: a. b. October 1, 2005 timing error logging: observed errors are written to Tor. X log file timing error failure: observed errors cause fail verdict of test case ARTIST 2 Summer School 53

Real-time Testing and I/O Systems l can the notion of repetitive quiescence be combined with real-time testing? l is there a well-defined and useful conformance relation that allows sound and (relative) complete test derivation? l can the Tor. X test tool be adapted to support Realtimed conformance testing? October 1, 2005 ARTIST 2 Summer School 54

Do We Still Need Quiescence? Yes! the example processes should also be distinct in a real-time context October 1, 2005 coin? coffee? tea? bang? coin? bang? tea? coffee! tea ! coffee? tea? coffee! ARTIST 2 Summer School tea ! 55

Real-Time and Quiescence l s is quiescent iff: a(d) for no output action a and delay d: s l special transitions: δ s s for every quiescent system state s l testers observing quiescence take time: Test. M: set of test processes having only δ(M)-actions to observe quiescence l assume that implementations are M-quiescent: for all reachable states s and s’: (M) if s s’ then s’ is quiescent October 1, 2005 ARTIST 2 Summer School 56

Real-Time and Quiescence M i tiorf s T Test. M: Deadlocksδ(i||T) Deadlocksδ(s||T) ( L { (M) } )*: out. M ( i after ) out. M ( s after ) i tioco. M s Tracesδ(M)( s ) : out. M ( i after ) out. M ( s after ) October 1, 2005 ARTIST 2 Summer School 57

Properties 1. for all M 1 M 2: M M 2 1 i tiorf s implies i tiorf s 2. for all time-independent i, s and M 1, M 2 0 i October 1, 2005 M 1 tiorf s iff i M 2 tiorf ARTIST 2 Summer School s iff i iorf s 58

A limitation states are saturated with input loops that reset the clocks coin? tea? x=k tea ! coin? coffee? tea? x x x k x<M bang? this process cannot be distinguished from the next from the previous x M bang? coffee? x k x=k x<M coffee! bang? x k x=k coffee! October 1, 2005 ARTIST 2 Summer School x=k tea ! 59

Test Cases x: = 0 Test case t TTA x k TTA – Test Timed Automata : n labels in L { }, G(d) off! fail n tree-structured off! n finite, deterministic x=5 on? x: =0 x M off! n final states pass and fail x: =0 n from each state pass, fail x<5 x M fail l choose an input i? and a time k and wait for the time k accepting all outputs o! and after k time unit provide input i? pass l or wait for time M accepting all outputs o! and October 1, 2005 ARTIST 2 Summer School 60 off! fail x=M fail

Timed test Generation Algorithm can transition system To generate a test case t(S) from a timedbe calculated specification with S set of states ( effectively 0} ) for initially S = {sonly subclasses of timed transition systems! apply the following steps recursively, non-deterministically PASS 1. end test case 2. choose k (0, M) and input μ 3. wait for observing possible output x: =0 forbidden outputs oi! after d’ time-units o 1! x=d’ 1 on’! x=d’n’ FAIL October 1, 2005 x: =0 x k μ? x=k tμ allowed outputs oj! after d time-units o 1! on! x=d 1 x=dn t 1 forbidden outputs oi! after d’ time-units x=d’ 1 tn o 1! on’! x=d’n’ FAIL ARTIST 2 Summer School x M x=M tδ 61 allowed outputs oj! after d time-units o 1! x=d 1 on! t 1 x=dn tn

Example spec: δ t! m? c? t? m? b? t? c! x 1 c! fail c? c! c? impl: M=k t! t! m? b? October 1, 2005 t? c! c? fail c! c? x=1 x: =0 fail t! fail x M δ x=M x: =0 t! fail x<k ARTIST 2 Summer School b? x=1 x: =0 t! fail x 1 c! c? x<k t! x 1 : test t! x 1 t? x<k pass c! m? c? t? c! t? m? x=1 x: =0 fail c? x=1 x: =0 t! fail x M c! pass 62 δ x=M fail t! fail

Soundness & Completeness l the non-timed generation algorithm can be shown to generate only non-spurious sound real-time test cases errors l test generation is complete = for every erroneous trace it can generate a errors with a positive test that exposes it probability of l test generation is not limit complete occurring because of continuous time there are uncountably many timed error traces and only countably many test are generated by repeated runs l test generation is almost limit complete repeated test geration runs will eventually generate a test case that will expose one of the non-spurious errors of a non-conforming implementation October 1, 2005 ARTIST 2 Summer School 63

Current Work l Extension of the framework n M as a function of the specification state/output channel n integration with symbolic data generation n test action refinement n robustness & tolerance in real-time testing l Extending Tor. X environment using CORBA IDL n generate abstract Tor. X actions n generate TTCN-3 signatures n generate adapter code l Practical application n TANGRAM project: testing control software for VLSI lithography machines (ASML) n smooth transition between timed & untimed testing October 1, 2005 ARTIST 2 Summer School 64

Future Work l stochastic systems l quality of service l hybrid systems l coverage measures l integration white/black box spectrum l. . . October 1, 2005 ARTIST 2 Summer School 65

For more information fmt. cs. utwente. nl/research/testing October 1, 2005 ARTIST 2 Summer School 66