13 Automated Model-Based Testing of Hybrid Systems Michiel

13 Automated Model-Based Testing of Hybrid Systems Michiel van Osch PROSE January 25, 2007

Motivation • Hybrid Systems • Testing might be expensive, dangerous, or resources might be limited • Discrete and real-time model-based testing does not test the continuous aspects of the system 2

Content • Part I: Theory – – – Model-based Testing Input-Output Conformance & Discrete Tests Hybrid Systems Hybrid Input-output Conformance Hybrid Tests Results • Part II: Tool – – – – Test Architecture Specification Tester The Connection with the Implementation Under Test Adapter Limitations and Future Work Case Study: Vacuum Control 3

Model-Based Testing test generation tool model IUT conforms to model IUT conf model exhaustive sound IUT passes test execution tool SUT pass fail 4

Input-output Conformance for Discrete Systems If there is an output action from state s then out(s) = {o in O| s →} else out(s) = {δ} Furthermore, out(S) = Us S out(s) Impl. ioco Spec. iff for all traces α: out(Impl. after α) out(Spec. after α) 5

Test-case Generation and Execution 1. Terminate with verdict pass 2. Select an input from the specification and apply it to the implementation 3. Observe an output or a timeout from the implementation and check if it is allowed according to the specification 6

Example t 0 ? Activate s 2 !Coffee t 1 ? Button 1 !XLCoffee s 0 ? Activate !XLCoffee s 1 Fail !δ t 2 !Coffee Fail ? Button 2 s 3 t 3 !δ Fail !XLCoffee Pass !Coffee Fail 7

Hybrid Systems • In Practice: – Discrete behavior plus continuous behavior – Continuous behavior can be input observed through sensors or output generated by actuators • In Theory: – Discrete actions plus flow of continuous variables (trajectories) – Variables can be input variables and output variables – Hybrid Transition Systems 8

Hybrid Systems (Output Only) s 2 Coffee’ = 3 cl/sec. Δt = 5 sec. Coffee’ = 0 Δt = 1 sec. s 0 ? Button 1 ? Activate Coffee’ = 4 cl/sec. Δt = 8 sec. Coffee’ = 0 Δt = 1 sec. . ? Button 2 s 3 Coffee 0 ? Button 1 s 1 Time ? Button 2 9

Hybrid Systems (Including Input) Water’ = 3 cl/sec. Coffee’ = Water’ Δt = 5 sec. Water’ = 0 Coffee’ = 0 Δt = 1 sec. s 0 Water’ = 4 cl/sec. Coffee’ = Water’ Δt = 8 sec. s 2 Water ? Button 1 ? Activate s 1 Water’ = 0 Coffee’ = 0 Δt = 1 sec. . ? Button 2 s 3 10

Hybrid Conformance • For every reachable state, the set of output actions possible by the implementation is a subset of the set of output actions possible by the specification • For every reachable state, the set of trajectories possible by the implementation is a subset of the set of trajectories possible by the specification • In contrast to ioco, no quiescence action because there is always continuous output. 11

Continuous Output Only s 2 Coffee’ = 3 cl/sec. Δt = 5 sec. Coffee’ = 0 Δt = 1 sec. s 0 ? Button 1 ? Activate Coffee’ = 4 cl/sec. Δt = 8 sec. s 1 Coffee’ = 0 Δt = 1 sec. . ? Button 2 s 3 Impl. is input-output conform a Spec. iff for all traces α: out(Impl. after α) out(Spec. after α) and traj(Impl. after α) traj(Spec. after α) 12

With Continuous Input • The implementation is input enabled (for both discrete behavior and continuous behavior). • We do not require the specification to be input complete. Impl. is input-output conform a Spec. iff for all traces α: out(Impl. after α) out(Spec. after α) and traj(Impl. after α) traj(Spec. after α) Does not work!! Solution: Look at the trajectories of the Implementation with respect to the trajectories of input variables of the Specification 13

Hybrid Conformance (Continuous Input plus Output) infilter(traj(Impl. after α), traj(Spec. after α)) traj(Spec. after α) Still does not work because … s 1 u 1 Water’ = 0 Coffee’ = 0 Δt = 1 sec. . ? Button 2 s 3 Water’ = 0 Coffee’ = 0 Δt = 3 sec. . Water’ = 0 Coffee’ = 0 Δt = 1 sec. . u 3 s 4 Water’ = 0 Coffee’ = 0 Δt = 1 sec. . s 5 !Out of Cups u 4 u 5 !Out of Cups Water’ = 0 Coffee’ = 0 Δt = 1 sec. . Specification Implementation 14

Hybrid Conformance (continuous input plus output) If there is a trajectory from state s then out(s) = {o in O| s →} {ξ} else out(s) = {o in O| s →} Impl. hioco Spec. iff for all traces α: out(Impl. after α) out(Spec. after α) and infilter(traj(Impl. after α), traj(Spec. after α)) traj(Spec. after α) 15

Hybrid Tests A Special kind of Hybrid Transition Systems: • Tree like structure • Two terminal states: pass and fail • Deterministic for actions • Strongly time deterministic for trajectories 16

Hybrid Tests 1. Terminate with verdict pass 2. Select an input from the specification and apply it to the implementation Water’ = 0 Coffee’ = 0 s 0 ? Activate s 1 t 0 ? Activate t 1 Specification Test 17

Hybrid Test-case Generation 3. If an output action has to happen immediately according to the specification then observe an output action and check if it is allowed according to the specification or let time pass by selecting and applying and observing a trajectory s 4 !”Out of Cups” s 5 Coffee’ = 0 Water’ = 0 Δt = 1 Specification t 4 Coffee’ = 0 Water’ = 0 Δt = 1 !”Out of Cups” Pass Fail Test-Case 18

Hybrid Test-case Generation 4. Select an input trajectory from the specification, apply it to the implementation and observe the output trajectory simultaneous, possibly interrupted by an output action. t 1 Water’ = 0 Coffee’ = 0 Δt = 1 s 1 Specification Water’ = 0 Coffee’ = 0 Δt = 1 !”out of cups” Fail t 2 Fail Water’ = 0 Coffee’ = 3 Δt = 1 Fail Test 19

Results • A hybrid conformance theory • Proven Sound and exhaustive • A Natural extension of discrete and timed conformance theories 20

Content • Part I: Theory – – – Model-based Testing Input-Output Conformance & Discrete Tests Hybrid Systems Hybrid Input-output Conformance Hybrid Tests Results • Part II: Tool – – – – Test Architecture Specification Tester The Connection with the Implementation Under Test Adapter Limitations and Future Work Case Study: Vacuum Control 21

Tester Architecture Spec Tester Adapter Medium Libraries IUT • Specification: The Model from which Tests are Generated • Tester: Implements The Test Algorithm and Gives the Verdict • Adapter: Translated Input/Output from Model to a format suitable for the Implementation Under Test and vice versa • Medium: The Interface between Tester and Implementation • IUT: The Implementation Under Test 22

Specification Spec Tester Adapter Medium Libraries IUT Needs to: • Model Discrete behavior and Continuous Behavior • Make Distinction between Input Actions, Output Actions, and Internal Actions • Make Distinction between Input Variables, Output Variables and Internal Variables • Model in an Intuitive way 23

Specification Spec Tester Adapter proc Control(cont V: real, chan h, out: real)= |[ *(V <= 2 -> h!!1. 0; out!!1. 0 ; V >= 10 -> h!!0. 0; out!!0. 0) ]| Medium Libraries IUT model Spec()= |[ cont V: real = 10. 0, chan h, out: real : : Control(V, h, out)|| Env(V, h) ]| proc Env(cont V: real, chan h: real)= |[ var n: real = 0. 0 : : V’=3. 0*n - 1. 0 | *(h? n) ]| 24

Tester Spec Tester Adapter Medium Libraries IUT Implements: • On the Fly Test Generation – – Select Input from Specification Apply Input Observe Output Compare the Observed Output with the Output allowed by the Specification – Give a Verdict or Continue Test 25

On the Fly Testing proc Control. S(cont V: real, chan h: real)= |[ var n: real = 0. 0 : : V’=3. 0*n - 1. 0 | *(V <= 2 -> n: =1. 0; h!!1. 0 ; V >= 10 -> n: =0. 0; h!!0. 0) ]| Select Input (χ)(Manually/ Automatic) V’=3. 0*0. 0 -1. 0 Δt = 8 sec. V=2 model Spec()= |[ cont V: real = 10. 0, chan h: real : : Control. S(V, h) ]| Pressure’= -1. 0 mbar/sec Δt = 8 sec. Continue V=2 h!!0. 0 Give Verdict (with trace) V=10 fail Apply (Via adapter) h!!1. 0 pass IUT Compare Observe (Via adapter) Values (χ , Maple) Pump OFF IUT Pass 26

Additional Libraries Spec Tester Adapter Medium Libraries IUT • χ –stepper for computing sets of allowed transitions and current state of the specification • E. g. Maple for comparing observed continuous output (samples) with specified trajectories and comparing observed discrete output values with specified send actions 27

The Connection Spec Tester Adapter Medium Libraries IUT Jabber χ Model TCP/IP Labview Controller Wires Electronics Buttons/ Sensors Robot Arm 28

The Adapter Spec Tester Adapter Medium Libraries IUT • Implements – Mapping of Variables/Actions of Specification to a Implementation and vice versa (e. g. channels to function calls , or variables to wires) – Translating Input/Output of Specification to Implementation and vice versa (e. g. functions to samples, or signals) 29

Limitations and Future Work • This is just a prototype, there are shortcomings! – Real Time Testing is Not Possible Yet – The complexity of Continuous behavior is limited by the Hybrid χ –stepper implementation. E. g. currently only standard differential equations. – Models are not ‘ideal’ for testing. E. g. in case of identifying input and output – For performance reasons we only deal with deterministic specifications. – We assume that the communication medium is reliable • Adaptation of theory for Sampling and Inaccuracy • Case Studies 30

Real Time • Generating and applying input (e. g. samples) • Observing output and Time at which output Occurred in the Implementation 31

Limitations and Future Work • This is just a prototype, there are shortcomings! – Real Time Testing is Not Possible Yet – The complexity of Continuous behavior is limited by the Hybrid χ –stepper implementation. – Models are not ‘ideal’ for testing. – For performance reasons we only deal with deterministic specifications. – We assume that the communication medium is reliable • Adaptation of theory for Sampling and Inaccuracy • Case Studies 32

The Vacuum Case • Lithography Process takes place in vacuum • Waferstepper has Five Chambers • Chambers are kept in Vacuum by a system of Pumps and Valves • Pumps and Valves are Controlled by Software (discrete) • Software observes Pressure in Chambers through Sensors (continuous) 33

Activities • Modeling Hardware in Hybrid χ and Stand Alone Simulation • Modeling (translating) Hardware in discrete (timed) χ and Integration with Software Controller • Modeling (translating) in Uppaal for Model Checking • Testing Models and Software Controller with the Hybrid Tester 34

Questions? 35