Скачать презентацию 1 A Comprehensive Framework for Testing Graphical User Скачать презентацию 1 A Comprehensive Framework for Testing Graphical User

130a435089bf1fad28fdc66fc5ad4617.ppt

  • Количество слайдов: 59

1 A Comprehensive Framework for Testing Graphical User Interfaces Atif M. Memon atif@cs. pitt. 1 A Comprehensive Framework for Testing Graphical User Interfaces Atif M. Memon atif@cs. pitt. edu Dept. of Computer Science University of Pittsburgh Advisors: Mary Lou Soffa & Martha E. Pollack

Research focus 2 50% of code GUI Interactions between the GUI and the underlying Research focus 2 50% of code GUI Interactions between the GUI and the underlying code Underlying Code

Challenges of GUI Testing • What is the test input to the GUI? – Challenges of GUI Testing • What is the test input to the GUI? – (Test case generation) • How much testing is enough? – (Test coverage) • Is the GUI executing correctly during testing? – (Test oracles) • What test results can be salvaged from the previous test runs to test a new version? – (Regression testing) • How to represent the GUI to handle all the above? – (Representation) 3

Current Techniques • Capture/replay tools • GUI test case generation – FSM [Esmelioglu and Current Techniques • Capture/replay tools • GUI test case generation – FSM [Esmelioglu and Apfelbaum] and VFSM [Shahady and Siewiorek] models. – Genetic algorithm technique [Kasik and George] – Visual TDE for GUIs [Foster, Goradia, Ostrand, and Szermer] 4

Goals of My Research • Develop a comprehensive environment that integrates techniques (tools) to Goals of My Research • Develop a comprehensive environment that integrates techniques (tools) to address various aspects of GUI testing • Goals of developing the techniques – Automated: avoid manual repetition of tasks – General: apply to a wide variety of GUIs – Efficient: to meet tight testing deadlines – Robust: to recognize a GUI error as soon as it occurs and report the problem – Portable: apply to multiple platforms 5

Framework * * * • • • 6 New representation for GUIs Test case Framework * * * • • • 6 New representation for GUIs Test case generation based on AI planning New event-based coverage criteria Unique test oracles New automated regression testing algorithms • Automated test executor • A comprehensive framework that integrates all the above components *

Overview of the Framework GUI Specifications GUI Implementation: Tools (Languages/Toolkits) GUI Implementer GUI Representation Overview of the Framework GUI Specifications GUI Implementation: Tools (Languages/Toolkits) GUI Implementer GUI Representation Executing GUI Regression Tester Test Oracle Test Case Generator Test Executor Test Coverage Evaluator 7

GUI Representation GUI Specifications Test Designer 8 GUI Implementation: Tools (Languages/Toolkits) GUI Implementer GUI GUI Representation GUI Specifications Test Designer 8 GUI Implementation: Tools (Languages/Toolkits) GUI Implementer GUI Model Executing GUI Regression Tester Test Oracle Test Case Generator Test Executor Test Coverage Evaluator

Creating the GUI Model • Modeling – GUI’s state in terms of objects & Creating the GUI Model • Modeling – GUI’s state in terms of objects & their properties – Events as state transducers – GUI’s hierarchical structure • Classifying events 9

Modeling the GUI’s State 10 • A GUI at time T is modeled using: Modeling the GUI’s State 10 • A GUI at time T is modeled using: – Objects O = {o 1, o 2, o 3, …, om} – Properties P = {p 1, p 2, p 3, …, pl}, where pi is an ni-ary (ni >= 1) Boolean relation of the form: Property(oj, oa, ob, …, ox, value) Property Name True/False Optional Objects Caption(Button, “Cancel”) Object Value of Property GUI’s state: S = p 1^p 2^p 3^… ^pn

Example: Modeling the GUI’s State 11 Form 1 WState(Form 1, ws. Normal) Width(Form 1, Example: Modeling the GUI’s State 11 Form 1 WState(Form 1, ws. Normal) Width(Form 1, 1088) Scroll(Form 1, TRUE) Label 1 Align(Label 1, al. None) Caption(Label 1, “Files of type: ”) Color(Label 1, cl. Btn. Face) Font(Label 1, (tfont)) Button 1 Caption(Button 1, Cancel) Enabled(Button 1, TRUE) Visible(Button 1, TRUE) Height(Button 1, 65)

All Properties of Button 1 12 Caption Cancel Tab. Order 0 All Properties of Button 1 12 Caption Cancel Tab. Order 0

Determining Objects & Properties • Specifications (reduced set) – GUI being tested • Toolkit/language Determining Objects & Properties • Specifications (reduced set) – GUI being tested • Toolkit/language (complete set) – All available properties Now we know how to represent the GUI’s state 13

GUI Events • GUI’s state is not static • Events change the GUI’s state GUI Events • GUI’s state is not static • Events change the GUI’s state • Events E = {e 1, e 2, e 3, …, en}, associated with a GUI are functions from one GUI state Si to another state Sj • Notation: Sj = e 1(Si) 14

Example: An Event State: Si This is the text. Select. Text (“This”) Event: e Example: An Event State: Si This is the text. Select. Text (“This”) Event: e Sj = e(Si) This is the text. State: Sj 15

Representing Events • Infeasible to give exhaustive specifications of the state mapping for each Representing Events • Infeasible to give exhaustive specifications of the state mapping for each event • No set limit to the number of objects a GUI can contain at any point in time • There can be infinitely many states of the GUI • Model the GUI events using operators, which specify their preconditions and effects 16

Operators • An Operator is a 3 -tuple – – <Name, Precondition, Effects> Name Operators • An Operator is a 3 -tuple – – Name identifies an event and its parameters Precondition is a boolean expression Effects is a sequence of ADD and DEL commands on properties • Operator Op is applicable in any state Si in which: – The Precondition(Op) holds • The resulting state Sj is determined by using Effects(Op) – ADD commands add their properties, and – DEL commands delete properties, – in the specified order 17

Operator Example File Edit View Ins Menu 1 Cut Menu 2 Operator : : Operator Example File Edit View Ins Menu 1 Cut Menu 2 Operator : : CUT Precondition: is. Current(Menu 2). Effects: FORALL Obj in Objects Selected(Obj) ADD in. Clipboard(Obj) DEL on. Screen(Obj) DEL Selected(Obj) ADD is. Current(Menu 1) DEL is. Current(Menu 2). Primitive Operators Now we know how to represent the events of the GUI 18

Model GUI Hierarchically • Hierarchy – GUIs are designed as a hierarchy of objects Model GUI Hierarchically • Hierarchy – GUIs are designed as a hierarchy of objects – Objects are reused from libraries – Hierarchical model makes testing efficient • Classification – A new classification of events aids in creating the hierarchical model of the GUI 19

Classifying Events • Opening menus – Menu-open events • Opening modal windows – Restricted-focus Classifying Events • Opening menus – Menu-open events • Opening modal windows – Restricted-focus events • Opening modeless windows – Unrestricted-focus events • Interacting with underlying software – System-interaction events 20

GUI Modeling Steps • From the GUI’s specifications (formal/informal), – Identify objects and properties GUI Modeling Steps • From the GUI’s specifications (formal/informal), – Identify objects and properties – Create operators for GUI events – Using the event classification, create hierarchical and system-interaction operators • Details later 21

Test Case Generator GUI Specifications Test Designer GUI Implementation: Tools (Languages/Toolkits) GUI Implementer GUI Test Case Generator GUI Specifications Test Designer GUI Implementation: Tools (Languages/Toolkits) GUI Implementer GUI Model Executing GUI Test Oracle Tasks Regression Tester Test-Case Generator Test Executor Test Case Test Coverage Evaluator 22

GUI Test Cases • Individual user events – Not enough ! – Sequences of GUI Test Cases • Individual user events – Not enough ! – Sequences of user events lead to different states • Test case: sequence of user events 23

Definition: GUI Test Case • Event is defined as: – Sj = e(Si) • Definition: GUI Test Case • Event is defined as: – Sj = e(Si) • legal event sequence – e 1; e 2; e 3; …; en is a legal event sequence – for state S 0 – iff Exists S 0; S 1; S 2; …; Sn such that Si = ei(Si-1), 1 <= i <= n • A GUI test case is a pair – (S 0, e 1; e 2; e 3; …; en) • S 0 is any state, and • e 1; e 2; e 3; …; en is a legal event sequence 24

A Test Case for Word. Pad S 0 This is the text. Select. Text A Test Case for Word. Pad S 0 This is the text. Select. Text (“This”) Format Font Underline 18 OK OK This is the text. Select. Text (“text”) 25

Selecting Test Sequences • • Infinitely many Randomly choose sequences Expert chooses sequences Automatically Selecting Test Sequences • • Infinitely many Randomly choose sequences Expert chooses sequences Automatically generate events for COMMONLY USED TASKS This is the text. Initial State Multiple Event Sequences This is the text. Goal State 26

A Plan for a GUI Task Select. Text(“This”) 27 Set. Font. Size(18) This is A Plan for a GUI Task Select. Text(“This”) 27 Set. Font. Size(18) This is the text. Initial State Goal State This is the text. Select. Text(“text”) This is the text. Mouse. Click(U)

Overview of Test Generation 28 Overview of Test Generation 28

Using Primitive Operators • One operator for each event File Edit View Ins Menu Using Primitive Operators • One operator for each event File Edit View Ins Menu 1 Cut Menu 2 Operator : : CUT Precondition: is. Current(Menu 2). Effects: FORALL Obj in Objects Selected(Obj) ADD in. Clipboard(Obj) DEL on. Screen(Obj) DEL Selected(Obj) ADD is. Current(Menu 1) DEL is. Current(Menu 2). 29

Exploit the GUI’s Structure • Reduce the number of operators – System more efficient Exploit the GUI’s Structure • Reduce the number of operators – System more efficient – Easier for the test designer 30

Operator Abstractions Two types of abstractions – Combine buttons create systeminteraction operators – Decompose Operator Abstractions Two types of abstractions – Combine buttons create systeminteraction operators – Decompose GUI hierarchically create hierarchical operators 31

Create System-Interaction Operators 32 File Sys-Interaction Operator: File_Send. To_Mail. Recipient = <File + Send. Create System-Interaction Operators 32 File Sys-Interaction Operator: File_Send. To_Mail. Recipient = Mail Recipient Send To

Create Hierarchical Operators Set Language Using Primitive Operators Only Main GUI’s Operator Set … Create Hierarchical Operators Set Language Using Primitive Operators Only Main GUI’s Operator Set … Set Language Select. From. List() Default OK Cancel. . . English (United States) OK Cancel Default. . . Using Abstraction Main GUI’s Operator Set … Set Language. . . Language Window’s Operator Set Select. From. List() Default OK Cancel 33

Using Hierarchical Operators Language Window’s Operator Set Select. From. List() Default OK Cancel High Using Hierarchical Operators Language Window’s Operator Set Select. From. List() Default OK Cancel High Level Plan . . . Set. Language() . . . Planner Define Abstraction Set. Language() Hierarchical Operator Sub Plan Select. From. List (“English(US)”) OK 34

Effects of Exploiting the GUI’s Structure • Reduction in planning operators – 362 operators Effects of Exploiting the GUI’s Structure • Reduction in planning operators – 362 operators 32 operators – Ratio 10: 1 for MS Wordpad – 20: 1 for MS Word • Efficiency 35

36 Initial State This is the text. Goal State This is the text. 36 Initial State This is the text. Goal State This is the text.

Test Case 37 Primitive Hierarchical I Primitive G Operator N O I Select. Text Test Case 37 Primitive Hierarchical I Primitive G Operator N O I Select. Text Format. Font Select. Text A T (“This”) (“text”, Underline) L (“This”, 18 pt) (“text”) I Planner A Format. Font Underline OK OK Format. Font 18 L Mapping Format Select. Text (“This”) Format Font Underline OK 18 OK Select. Text (“text”)

Different from HTN Planning 38 Primitive Hierarchical I Primitive G Operator N O I Different from HTN Planning 38 Primitive Hierarchical I Primitive G Operator N O I Select. Text Format. Font Select. Text A T (“This”) (“text”, Underline) L (“This”, 18 pt) (“text”) I Planner A Format. Font Underline OK OK Format. Font 18 L No Interactions

Alternative Test Case I Primitive N Operator I Select. Text T (“This”) I A Alternative Test Case I Primitive N Operator I Select. Text T (“This”) I A L Primitive Operator Select. Text (“text”) Hierarchical Operator Format. Font (“This”, 18 pt) Primitive Operator Select. Text (“text”) Operator Select. Text (“This”) Primitive Operator Select. From. List (18) Select. Text (“This”) Select. From. List (18) Format Font Hierarchical Operator Format. Font (“text”, Underline) Select. Text (“text”) Underline OK 39 G O A L

Methods to Generate Alternative Test Cases 40 • Different results from planner • Hierarchical Methods to Generate Alternative Test Cases 40 • Different results from planner • Hierarchical operator decompositions • Linearizations of the partial-order plan

Experiments • Purpose – To determine whether planning is a feasible approach for GUI Experiments • Purpose – To determine whether planning is a feasible approach for GUI test case generation • Execution time • Human effort • Experimental design – GUI: our version of MS Wordpad (36 modal windows, 362 events) – Tasks: 50 tasks (initial & goal states) – Test cases: generated 290 test cases (6 -56 events) using the IPP AI planner – Hardware platform: 350 MHz Pentium based machine, 256 MB RAM 41

Test Case Generation 42 Test Case Generation 42

Automated Execution 43 Automated Execution 43

Test Oracles GUI Specifications Test Designer Oracle Expected-state Generator Verdict GUI Implementation: Tools (Languages/Toolkits) Test Oracles GUI Specifications Test Designer Oracle Expected-state Generator Verdict GUI Implementation: Tools (Languages/Toolkits) GUI Implementer GUI Model Executing GUI State (run-time) Tasks Expected-state sequence Verifier 44 Execution Monitor Test-Case Generator Test Case Actual State Information Regression Tester Test Executor Test Coverage Evaluator

What Is Correct Behavior ? ! This is the text. Select. Text (“This”) Format What Is Correct Behavior ? ! This is the text. Select. Text (“This”) Format Select. Text (“text”) Format Font 18 OK Underline This is the text. OK Check State, not only Output !! This is the text. 45

Determine Correct Behavior 46 • To check the GUI’s state after each event • Determine Correct Behavior 46 • To check the GUI’s state after each event • Approaches – Manual – Automated Actual State • Challenges Expected State Automated Verifier – Generating expected state Verdict – Extracting actual state – Comparing expected & actual states

Overview of GUI Oracle Test Case GUI Model Expected-state Generator Run-time information from executing Overview of GUI Oracle Test Case GUI Model Expected-state Generator Run-time information from executing GUI Expected State Verifier Oracle Verdict Actual State Execution Monitor 47

Expected State • Obtaining Next State 48 – Given a test case T with Expected State • Obtaining Next State 48 – Given a test case T with S 0, the initial state, and – A sequence of events e 1 e 2 e 3 en – The next state Sy is obtained from the current state Sx and operator Op • Delete property P from Sx if Effects(Op) contains the command “DEL P” • Add property P in Sy if Effects(Op) contains the command “ADD P”

Deriving Expected State 49 • Obtain S 1 = e 1(S 0) • And Deriving Expected State 49 • Obtain S 1 = e 1(S 0) • And Si = ei(Si-1) S 0 e 1 S 1 e 2 S 2 e 3 S 3 en • Expected state is a conjunction of properties of the form – Property(Objects…, Value) Sn

Obtaining Actual GUI’s State • Execution monitor – Compatible with expected state – Returns Obtaining Actual GUI’s State • Execution monitor – Compatible with expected state – Returns property(objects…, value) e. g. , Caption(button 1, “cancel”) – Actual state can be obtained by • Screen scraping • Queries 50

Automated Execution Test Executor GUI Under Test Cases Execution Monitor Verifier Expected State ACTUAL Automated Execution Test Executor GUI Under Test Cases Execution Monitor Verifier Expected State ACTUAL STATE: (is. Current ROOT) (Contains ROOT D 51

e 4 Select. File (“f 1. doc”) e 6 e 5 Open S 4 e 4 Select. File (“f 1. doc”) e 6 e 5 Open S 4 contains(Latex Samples) contains(private Latex) contains(root private) containsfile(private f 1. doc) containsfile(Samples f 4. doc) current. Font(Times Normal 12 pt) font("be modeled" Times Normal 12 pt) font("is the" Times Normal 12 pt) font("needs to" Times Normal 12 pt) font("text" Times Normal 12 pt) font("that" Times Normal 12 pt) font("This" Times Normal 12 pt) in(f 1. doc "be modeled") in(f 1. doc "is the") in(f 1. doc "needs to") in(f 1. doc "text") in(f 1. doc "that") in(f 1. doc "This") is. Current(private) selected. File(f 1. doc) S 5 Select. Text (“This”) contains(Latex Samples) contains(private Latex) contains(root private) containsfile(private f 1. doc) containsfile(Samples f 4. doc) current. File(f 1. doc) current. Font(Times Normal 12 pt) font("be modeled" Times Normal 12 pt) font("is the" Times Normal 12 pt) font("needs to" Times Normal 12 pt) font("text" Times Normal 12 pt) font("that" Times Normal 12 pt) font("This" Times Normal 12 pt) in(f 1. doc "be modeled") in(f 1. doc "is the") in(f 1. doc "needs to") in(f 1. doc "text") in(f 1. doc "that") in(f 1. doc "This") is. Current(private) on. Screen("be modeled") on. Screen("is the") on. Screen("needs to") on. Screen("text") on. Screen("that") on. Screen("This") selected. File(f 1. doc) 52

Comparing Actual and Expected States • Verifier • Three levels of checking – Changed Comparing Actual and Expected States • Verifier • Three levels of checking – Changed property set (operators) – GUI relevant property set (specifications) – Complete property set (toolkit/language) • Hybrid approach – Use all 3 53

Experiments • Purpose: determine – Time to derive expected state – Time to execute Experiments • Purpose: determine – Time to derive expected state – Time to execute monitor and verifier • Experimental design – GUI: our version of MS Wordpad (36 modal windows, 362 events) – Test cases: generated 290 test cases (6 -56 events) using an AI planner – Hardware platform: 350 mhz pentium based machine, 256 MB RAM – Properties: reduced set – Level of checking: GUI relevant property set 54

Deriving Expected State Total CPU time (all 290 test cases and expected states) 75. Deriving Expected State Total CPU time (all 290 test cases and expected states) 75. 84 sec. 55

Automated Execution GUI Relevant-properties checking Total running time < 10 minutes 56 Automated Execution GUI Relevant-properties checking Total running time < 10 minutes 56

Putting it All Together GUI Specifications Test Designer Oracle Expected-state Generator Verdict GUI Implementation: Putting it All Together GUI Specifications Test Designer Oracle Expected-state Generator Verdict GUI Implementation: Tools (Languages/Toolkits) GUI Implementer GUI Model Executing GUI State (run-time) Tasks Expected-state sequence Verifier 57 Execution Monitor Test-Case Generator Test Case Actual State Information Test Executor Regression Testing Algorithms Coverage Evaluation Algorithms Coverage Criteria Coverage Report

Future Work • Extensions – Web testing – Interface & Underlying code – GUI Future Work • Extensions – Web testing – Interface & Underlying code – GUI Specifications & Testing • Other Software • New Directions – Object-oriented & Component-based – Reactive – Execution Profiles – Testing Optimized Code 58

Further Reading: Publications 59 http: //www. cs. pitt. edu/~atif ü Atif M. Memon, Martha Further Reading: Publications 59 http: //www. cs. pitt. edu/~atif ü Atif M. Memon, Martha E. Pollack and Mary Lou Soffa, Hierarchical GUI Test Case Generation Using Automated Planning , IEEE Transactions on Software Engineering, vol. 27, no. 2, Feb. 2001. ü Atif M. Memon, Martha E. Pollack and Mary Lou Soffa, Automated Test Oracles for GUIs, Eighth International Symposium on the Foundations of Software Engineering (FSE-8), San Diego, CA, Nov. 6 -10, 2000. ü Atif M. Memon, Martha E. Pollack and Mary Lou Soffa, A Planning-Based Approach to GUI Testing, 13 th International Software/Internet Quality Week (QW 2000), San Francisco, CA, USA, 30 May - 2 June, 2000. ü Atif M. Memon, Martha E. Pollack and Mary Lou Soffa, Plan Generation for GUI Testing, The Fifth International Conference on Artificial Intelligence Planning and Scheduling (AIPS 2000), Breckenridge, CO, USA, pp. 226 -235, April 15 -17, 2000. ü Atif M. Memon, Martha E. Pollack and Mary Lou Soffa, Using a Goal-driven Approach to Generate Test Cases for GUIs , The 21 st International Conference on Software Engineering (ICSE 1999), Los Angeles, USA, pp. 257 -266, May 16 -22, 1999. (Selected as a best paper)