Скачать презентацию Qualitätssicherung von Software SWQS Prof Dr Holger Schlingloff Скачать презентацию Qualitätssicherung von Software SWQS Prof Dr Holger Schlingloff

512a03fa93a6e524472f85bacf921f80.ppt

  • Количество слайдов: 45

Qualitätssicherung von Software (SWQS) Prof. Dr. Holger Schlingloff Humboldt-Universität zu Berlin und Fraunhofer FOKUS Qualitätssicherung von Software (SWQS) Prof. Dr. Holger Schlingloff Humboldt-Universität zu Berlin und Fraunhofer FOKUS 14. 5. 2013: Modellbasierter Test (Doppelstunde)

Fragen zur Wiederholung • • Vorgehensweisen beim Integrationstest? Was ist ein Testorakel? Ziele des Fragen zur Wiederholung • • Vorgehensweisen beim Integrationstest? Was ist ein Testorakel? Ziele des Systemtests? Relevanz des Abnahmetests? H. Schlingloff, Software-Qualitätssicherung Folie 2

Wo stehen wir? Kapitel 1: Einleitung, Begriffe, Software-Qualitätskriterien Kapitel 2. Softwaretest 2. 1 Testen Wo stehen wir? Kapitel 1: Einleitung, Begriffe, Software-Qualitätskriterien Kapitel 2. Softwaretest 2. 1 Testen im SW-Lebenszyklus 2. 2 Funktionale Tests, Modultests, Testfallauswahl 2. 3 Strukturelle Tests, Integrationstests 2. 4 Modellbasierte Tests H. Schlingloff, Software-Qualitätssicherung Folie 3

Achtung, Werbung! H. Schlingloff, Software-Qualitätssicherung Folie 4 Achtung, Werbung! H. Schlingloff, Software-Qualitätssicherung Folie 4

Modellbasierter Test • Folien aus verschiedenen internationalen Sommerschulen • Buchkapitel in Vorbereitung H. Schlingloff, Modellbasierter Test • Folien aus verschiedenen internationalen Sommerschulen • Buchkapitel in Vorbereitung H. Schlingloff, Software-Qualitätssicherung Folie 5

Specification-based Testing • Constructing the test suite from the specification § as opposed to Specification-based Testing • Constructing the test suite from the specification § as opposed to constructing it from the implementation (= code-based testing) specified behaviour implemented behaviour tested behaviour § code-based testing cannot detect missing requirements § specification-based testing cannot detect additional (unspecified) features H. Schlingloff, Software-Qualitätssicherung Folie 6

The Validation Triangle Te st de Sy Re f in st em em en The Validation Triangle Te st de Sy Re f in st em em en de t ve lo pm en t Specification System Test execution H. Schlingloff, Software-Qualitätssicherung ve ge ne lo pm ra tio en t n Testsuite 7 Folie 7

Model-based Design and Model-based Testing Requirements System Spec Test Spec System Model Test Model Model-based Design and Model-based Testing Requirements System Spec Test Spec System Model Test Model System Impl. Test Cases • Often: same syntax, different pragmatics § e. g. test cases can be formulated in Java § e. g. system spec can be formulated with LTL H. Schlingloff, Software-Qualitätssicherung Folie 8

An embedded-systems example A video camcorder § owner's manual almost incomprehensible § can be An embedded-systems example A video camcorder § owner's manual almost incomprehensible § can be found in the internet § typical for such devices • Multifunctional on-off switch: § up: off § down: cycles through "tape", "memory" and "play/edit" mode • Intuitively, tape mode is for video, memory mode for pictures and play mode for viewing recorded material H. Schlingloff, Software-Qualitätssicherung Folie 9

Transition system • How can we formally describe the behaviour of this switch? (Natural Transition system • How can we formally describe the behaviour of this switch? (Natural language description is ambivalent: What does "cycle" mean? What if I push dn-dn-up-dn? ) • Modelling by finite transition system: off dn tape up dn up memory up dn play dn • States: {off, tape, memory, play} • Transition relations: {up, dn} H. Schlingloff, Software-Qualitätssicherung Folie 10

 • Every model abstracts from details (e. g. , there is a little • Every model abstracts from details (e. g. , there is a little green button within the switch which arrests it in the "off" position) • A model is a means of communication between humans • (designers, users, . . . ) Structuring the model as parallel hierarchical transition system gives a statechart / state machine diagram camera switch but_hi dn on off up but_lo dn, pwr_res on dn tape up, pwr_fail dn memory dn play dn H. Schlingloff, Software-Qualitätssicherung Folie 11

UML State Machines • UML state machines are basically (extended) LTSs with • parallelism UML State Machines • UML state machines are basically (extended) LTSs with • parallelism and hierarchy Each state machine consists of at least one region that contains states and transitions H. Schlingloff, Software-Qualitätssicherung Folie 12

UML State Machines • • • UML state machine: set of regions Region: contains UML State Machines • • • UML state machine: set of regions Region: contains vertices and transitions Vertex: state, pseudostate, connection point State: simple or composite (containing regions) Pseudostate: initial state, fork state Transition: connects source and target vertex Trigger: event (msg rec, op exec) Guard: OCL condition Effect: assignment, triggering an event, condition H. Schlingloff, Software-Qualitätssicherung Folie 13

UML State Machines – States • A state models a situation during which (usually UML State Machines – States • A state models a situation during which (usually implicit) invariant conditions hold - e. g. waiting for an event to occur - e. g. performing some behavior • Associated with each state may be § entry, do and exit actions § constraints (=state invariants) • Pseudostates - initial, history, fork, join, junction, choice, entry, exit, terminate, (final) H. Schlingloff, Software-Qualitätssicherung Folie 14

UML State Machines – Transitions • A transition is a directed relationship between a UML State Machines – Transitions • A transition is a directed relationship between a source vertex • and a target vertex Labels consist of Trigger [Guard] / Action where § trigger is a transition label (i/o-event) § guard is a logical formula on internal variables § action is an update of the variables • “Run-to-completion” semantics § if an action is also a trigger, it will be processed before the next external trigger is taken into consideration § maintaining an “event pool” during execution H. Schlingloff, Software-Qualitätssicherung Folie 15

State Machine Meta Model H. Schlingloff, Software-Qualitätssicherung Folie 16 State Machine Meta Model H. Schlingloff, Software-Qualitätssicherung Folie 16

VCR Switch as a State Machine H. Schlingloff, Software-Qualitätssicherung 17 Folie 17 VCR Switch as a State Machine H. Schlingloff, Software-Qualitätssicherung 17 Folie 17

System models vs. Test models • Such models can help in the development of System models vs. Test models • Such models can help in the development of • complex systems The more concrete the formalism, the closer it is to an implementation § executable code may be generated from state diagrams § We might additional information such as timing, communication, variables and such. • Test model as opposed to implementation model describes properties of the targeted system § not aiming at a complete description of the system § not aiming at the generation of executable code H. Schlingloff, Software-Qualitätssicherung Folie 18

The State of an Object • Technical systems convert or relocate physical objects (matter The State of an Object • Technical systems convert or relocate physical objects (matter and/or energy) • Physical objects are characterized by their state § State = observable appearance of an object in space and time § „a complete description of a system in terms of parameters such as positions and momentums at a particular moment in time” (wiki) § shape, size, position, movement, temperature, pressure, voltage, … • Observation of physical state by sensors § camera, folding rule, light sensor, tachometer, thermometer, … • Modification of physical state by actuators § motor, valve, relais, transducer, heater, … H. Schlingloff, Software-Qualitätssicherung Folie 19

Technical Systems and Processes • Technical system: perform technical process • Technical process: reshaping Technical Systems and Processes • Technical system: perform technical process • Technical process: reshaping or transporting physical objects • Description of states by state variables § formally, a state is a mapping of variables to values • Description of processes by state changes § discrete state changes are called events § continuously changing state constituents are sometimes called signals H. Schlingloff, Software-Qualitätssicherung Folie 20

Example: A Kitchen Toaster • A toaster § what is the technical process? § Example: A Kitchen Toaster • A toaster § what is the technical process? § what are the states, events and signals of the (technical) process? § what are the boundaries of the system? § which information processing is to be done? § what are the interfaces between technical system and information processing component? H. Schlingloff, Software-Qualitätssicherung Folie 21

Modeling the Toaster • User Interfaces: turning knob, • • side lever, stop button Modeling the Toaster • User Interfaces: turning knob, • • side lever, stop button Internals: heating element, retainer latch Extra: defrost button • First approach: timing is neglected (timer event) • Advanced approach: timing depends on various parameters H. Schlingloff, Software-Qualitätssicherung Folie 22

Toaster – Simple State Machine H. Schlingloff, Software-Qualitätssicherung Folie 23 Toaster – Simple State Machine H. Schlingloff, Software-Qualitätssicherung Folie 23

Toaster – Hierarchical Design H. Schlingloff, Software-Qualitätssicherung Folie 24 Toaster – Hierarchical Design H. Schlingloff, Software-Qualitätssicherung Folie 24

Toaster – with Variables H. Schlingloff, Software-Qualitätssicherung Folie 25 Toaster – with Variables H. Schlingloff, Software-Qualitätssicherung Folie 25

Test Generation • Graph traversal yields abstract test cases § Dijkstra shortest path algorithm Test Generation • Graph traversal yields abstract test cases § Dijkstra shortest path algorithm § Depth-first and breadth-first search • Static analysis for input parameters § § Category partition table Classification tree method Boundary value analysis Abstract interpretation H. Schlingloff, Software-Qualitätssicherung Folie 26

All-States with Dijkstra and DFS H. Schlingloff, Software-Qualitätssicherung Folie 27 All-States with Dijkstra and DFS H. Schlingloff, Software-Qualitätssicherung Folie 27

Exercise • Construct some test cases for the toaster with variables! H. Schlingloff, Software-Qualitätssicherung Exercise • Construct some test cases for the toaster with variables! H. Schlingloff, Software-Qualitätssicherung Folie 28

Automated Test Generation Tools • More than a dozen commercial and • • • Automated Test Generation Tools • More than a dozen commercial and • • • experimental research tools available Usually quite costly (>10 K€ per license) Mostly from UML State Machines Comparison e. g. by mutation analysis H. Schlingloff, Software-Qualitätssicherung Folie 29

Par. Te. G – Partition Test Generator • Since we didn’t like the pricing, Par. Te. G – Partition Test Generator • Since we didn’t like the pricing, and wanted to experiment with different technologies, a Ph. D. student built his own… § http: //parteg. sourceforge. net/ • UML class & state transition diagrams, connected by OCL • Plugin for Eclipse, supports XMI import / export H. Schlingloff, Software-Qualitätssicherung Folie 30

H. Schlingloff, Software-Qualitätssicherung 31 Folie 31 H. Schlingloff, Software-Qualitätssicherung 31 Folie 31

Test Generation from State Machines • Define a test case to be any execution Test Generation from State Machines • Define a test case to be any execution path • How to generate such paths? How many paths to generate? When to stop testing? Coverage criteria e. g. § § all-states all-transition-pairs all-n-paths • Test goal: one particular item to be covered H. Schlingloff, Software-Qualitätssicherung Folie 32

Goal-oriented search • Forward search § depth first - random - strategy? § breadth-first Goal-oriented search • Forward search § depth first - random - strategy? § breadth-first 2 1 7 6 3 5 • Backward search § depth/breadth first § model checking 4 2 4 1 7 6 3 5 H. Schlingloff, Software-Qualitätssicherung Folie 33

Tests for State Machines • For a state machine, a test case is just Tests for State Machines • For a state machine, a test case is just a finite • • sequence of external triggers and actions A test goal is a particular entity of the state machine (region, pseudostate, transition, n-path, …; for each test and goal it is defined whether the test reaches this goal The coverage of a test suite is the percentage of reached test goals § the coverage can either be measured during the execution of a test suite, or statically before execution H. Schlingloff, Software-Qualitätssicherung Folie 34

Coverage for State Machines • common coverage criteria for UML state machines § all-states Coverage for State Machines • common coverage criteria for UML state machines § all-states § all-transitions - subsumes statement coverage § all configurations: all combinations of parallel substates § n-transition-coverage means all reachable transition sequences of length n are covered (esp. : pairs) § All-loop-free-paths § All-n-loop-paths § decision coverage, condition coverage, MC/DC, . . . • all-requirements H. Schlingloff, Software-Qualitätssicherung Folie 35

Transition Tour • Nait 81: For a given LTS S, a transition tour is Transition Tour • Nait 81: For a given LTS S, a transition tour is a sequence • which takes S from the initial state s 0, traverses every transition at least once, and returns to s 0. Example: http: //www. protocol-engineering. tu-cottbus. de/vorlesung/ppts/Kapitel 8_2. ppt detects output faults in the SUT H. Schlingloff, Software-Qualitätssicherung Folie 36

How to Construct a Transition Tour • Chinese postman algorithm (named after Mei Ko How to Construct a Transition Tour • Chinese postman algorithm (named after Mei Ko Kwan [1962]) § postman must deliver letters alongside roads of a district (NP-complete optimization problem) • Euler’s “Königsberger Brückenproblem” § solvable iff strongly connected and for all states, in-degree=out-degree § In order to make the LTS eulerian, we may have to traverse certain transitions several times • Hierholzer’s algorithm § use depth-first-search until you hit a cycle § extend the cycle at the first junction H. Schlingloff, Software-Qualitätssicherung Folie 37

Example H. Schlingloff, Software-Qualitätssicherung Folie 38 Example H. Schlingloff, Software-Qualitätssicherung Folie 38

Par. Te. G‘s Algorithm • Ideas: § include the boundary values of linear ordered Par. Te. G‘s Algorithm • Ideas: § include the boundary values of linear ordered type variables as defined by the transition guard expressions § Deduce the value of system attributes from input parameters according to preconditions • Coverage criteria-oriented approach: § Define test goals § Search backwards § Create and adapt abstract domains for input parameters H. Schlingloff, Software-Qualitätssicherung Folie 39

Par. Te. G – Start from Test Goals • Test goal: § Coverage criterion Par. Te. G – Start from Test Goals • Test goal: § Coverage criterion applied to a concrete model § Example: one state for All-States • Generate abstract test case § Find a path • Generate concrete test case § Find concrete input values H. Schlingloff, Software-Qualitätssicherung Folie 40

Par. Te. G – Find Paths • Generation of test cases: § Path from Par. Te. G – Find Paths • Generation of test cases: § Path from initial node to test goal contains conditions (e. g. OCL) § Due to conditions not each found path is feasible § Consequence: include conditions into search algorithm § Deal with the relations between OCL conditions along the path H. Schlingloff, Software-Qualitätssicherung Folie 41

Par. Te. G – Interpret Logical Expressions • Generation of Test Cases: § Classify Par. Te. G – Interpret Logical Expressions • Generation of Test Cases: § Classify all variables used in the expressions - Which variables can change? § Algorithm - for each guard: - try to find postconditions that influence the result of the guard - Combine guard and postcondition to a new condition - If there are changeable variables in the condition: continue search § Basic Idea: - Transform conditions on system attributes into conditions on input parameters - Use them as input partitions H. Schlingloff, Software-Qualitätssicherung Folie 42

Testing and Fault Models • Problem: What is a “good” test suite? § Much Testing and Fault Models • Problem: What is a “good” test suite? § Much work has been done in trying to give a formal answer • Compare two LTSs A and B which are only “slightly” different: Test suite T is “good” if it can discover (the existence of) a difference § B is called a mutant of A, the difference is called a fault • Several methods for generation of test suites which are good in comparing LTSs exist § transition tour algorithm § W method § unique I/O method H. Schlingloff, Software-Qualitätssicherung Folie 43

Mutation Analysis • Testing to prove equivalence (or, in fact, to prove any preorder Mutation Analysis • Testing to prove equivalence (or, in fact, to prove any preorder relation between models) has not been very successful (personal opinion). § Even a “complete” test suite may miss errors in the SUT • However, we can use these ideas to assess the effectiveness of test generation algorithms § How to convince your boss that you are using a “good” test case generator Te. G? § Assume we are given a model M § Apply Te. G to M and obtain test suite T(M) § Inject a fault into M to get a slightly different model M’ § if T(M) fails on M’ this is an indication that Te. G is effective H. Schlingloff, Software-Qualitätssicherung Folie 44

 • In order to make this argument sound, we need to § repeat • In order to make this argument sound, we need to § repeat this process with several models M 1, …, Mn § select mutation operators which model “real” faults, i. e. M i’ could be a mistake made by an implementer § make sure that the effect of a mutation is not masked, i. e. it must be visible to the outside • Resulting statements are of the type: “For the • models M 1, …, Mn and mutation operators op 1, …, opk, G 1 can detect an average of 90% of all mutants, whereas G 2 can detect 93%. ” Such statements have proven to be useful! H. Schlingloff, Software-Qualitätssicherung Folie 45