Hierarchical GUI Test Case Generation Using Automated Planning

Hierarchical GUI Test Case Generation Using Automated Planning Atif M. Memon, Student Member, IEEE, Martha E. Pollack, and Mary Lou Soffa, Member, IEEE

Introduction Graphical User Interfaces – important in interacting with today’s software Makes software easier to use Makes software testing more difficult than conventional software - must find a PATH from some initial state to a solution

What is a GUI? Consists of labels, buttons, menus, and pop-up lists User interacts with components to create events Object Window GUI Window Microsoft Word. Pad

Challenges in Testing GUI Applications GUI’s becoming more complex Complexity in testing correctness and underlying software Determining coverage of Test Cases Enormous amounts of possible interaction Regression testing Goal: Automatic generation of test cases Solution: Planning Assisted Tester for grap. Hical User interface Systems

State of the Art Pros Cons Finite state machine models Automated test case generation Not scalable Mimic novice users Scalable Test case quality Semi-automated depends on user expertise A new technique both scalable and partially automated that reduces dependence on user expertise

35, 000 foot View of the Mountain PATHS… …easier to specify goals than all the possible PATHS toward the goal Performs hierarchical structure of GUI create hierarchical operators Test designers describe operators(preconditions, effects) Takes goals as input and generates multiple sequences of events (“plans”) automatically Plans becomes the test cases for GUI

PATHS: AI Planning with Operators Operator (preconditions, effects) Given to the AI planning algorithm Guaranteed to result in the goal state when executed in the initial state Initial State Goal State

The PATHS Test Case Generation Process Setup Phase – PATHS creates a hierarchical model of the GUI and returns list of operators from the model to test designer Design preconditions and effects in simple, planning system language Plan Generation Phase – Test designer describes scenarios by defining a set of initial and goal states for test case generation PATHS generates a test suite for test case generation

Phase 1: Setup PATHS partitions GUI into several classes using structural properties. Menu-open Events – expand set of GUI events available to user Unrestricted-focus Events – open GUI windows that do not restrict user’s focus, expand set of GUI events available to user (ex. Toolbar) Restricted-focus Events – open GUI windows that have special property once invoked, monopolize GUI interaction, restrict user view of GUI to specific range until window is terminate (ex. Pop-up window) System-interaction Events – interact with underlying software to perform some action

Phase 1: Operator Types System-Interaction Operators – derived from those GUI events that generate interactions with the underlying software Ex. Prefix_Suffix File_Open = Abstract Operators – represents a sequence of GUI events that invoke a window which monopolizes the GUI interaction Treat this restricted-focus window as a separate planning problem and decompose

Phase 1: Setup Example GUI Events Open. Up File+ Open. Select Edit+ New Open. Cancel Open* Open Save. As. Up Save As* Save. As. Select Cut Save. As. Cancel Copy Save. As. Save Paste Planning Operators File_New File_Open* File_Save. As* Edit_Cut Edit_ Copy Edit_ Paste

Phase 1: Operator Event Mapping Transforms GUI menu events into planning operators These results Returned as Planning operators to the Designer …who then specifies preconditions and effects for each planning operator

Phase 2: Test Case Generation Test case designer inputs defined operators into PATHS and identifies a task (initial state, goal state) Defined Task Planning Operators PATHS Tests Cases Achieving Goal

Abstract Operator Decomposition Operators from the higher level need to be decomposed into lower level operations by Hierarchical mechanism aids in regression testing since Inserting expansion from one changes made to operator-event mappings, or component to do not necessarily invalidate all planner Making an additional call to the test cases.

Plan Generation: AI Planning Problem – P(Λ, D, I G) Solution – P(Operator Set, Finite Set of Objects, Initial State, Goal State) Si < Sj – Step i must occur before Step j (not necessarily right before) Causal Links -- Ensure no step threatens a required link

Partial Order Plans… …aids in deriving total order plan Add ordering constraints Want to learn more about how the Authors implemented AI Planning…? We invite you to read about it in the i. e. Linearization of partial-order planpaper. (IPP and HTN) … are solutions to a planning problem iff every consistent linearization of the partial-order plan meets the solution conditions

Evaluation: Empirical Approach A PATHS prototype was developed based Microsoft Word. Pad 325 GUI events Reduced to 32 system-interaction and abstract operators 10 -fold reduction of operators streamline plan generation (phase 2 of PATHS)

Evaluation: Run Two Experiments Two experiments were run using the PATHS prototype Experiment 1: Generate test cases for multiple tasks Is PATHS practical and efficient? Experiment 2: Compare hierarchical vs. singlelevel test case generation Is the hierarchical modeling approach more efficient than single-level?

Experiment 1: Generating Test Cases for Multiple Tasks Use PATHS prototype to generate a suite of test cases for 9 Word. Pad tasks Deploy experiment on Pentium-based computer running Linux on 200 Mb RAM Track times required to generate high-level plans and subplans PATHS is practical and efficient

Experiment 2: Hierarchical vs. Single-Level Test Case Generation Use PATHS prototype to generate test cases for 6 Word. Pad tasks Deploy experiment on same computer (Pentium -based running Linux on 200 Mb RAM) Track plan lengths Track times required to generate a single-level and a hierarchical test case Hierarchical test case generation is more efficient

Conclusions Applying AI planning to the problem is a good match Hierarchy of the model is efficient in generating test cases 4 - to 6 -fold saving in plan length Test case generated in seconds rather than hours New technique is scalable when software releases new version, operators can be recycled New technique is highly automated

PATHS: Contributions Use well-known and used AI planning technique capable of solving large-space problems Exploits structural features present in GUIs to reduce model size & complexity, and efficiently generate test cases Hierarchical Planning makes regression testing easier Increased portability – platform specific details incorporated last in test case generation Reuse of commonly-appearing operator definitions

Open Questions PATHS is practical, but is it useful? Preliminary exploration achieves 20 -fold reduction in GUI events How does one measure usefulness? How well does PATHS scale? Manual analysis of operators will vary from application to application How good are the test cases generated by PATHS? Coverage Negative test cases

Related / Future Work Use library of operators to minimize human intervention How to automatically generate preconditions and effects of operators