PISATEL An Automated Test Strategy Based on UML

PISATEL An Automated Test Strategy Based on UML Diagrams F. Basanieri *, A. Bertolino *, E. Marchetti *, A. Ribolini *, G. Lombardi **, G. Nucera ** *PISATEL LAB, IEI-CNR, Pisa, Italy ** ERICSSON LAB ITALY

PISATEL_LAB PISATEL Pisa Initiative on Software Architectures for Telecommunications The joint conduction by the academia in Pisa and ERI of strategic research projects on software for telecommunications, selected on an annual basis 2

PISATEL Pisatel Objectives • Main areas of interest are: – software engineering methods and tools, – software architectures, distibuted systems, networks, real time systems, and protocols. • Currently the active projects are: – Integrating Wireless, Wireline and Internet Networks – Analysis and Development of Real Time Software – Quality And Validation of Software Architecture More details: http: //www. iei. pi. cnr. it/ERI/ 3

PISATEL An Automated Test Strategy Based on UML Diagrams

PISATEL Objectives • Use for test planning the same UML diagrams developed (e. g. , by Rational Rose) for design and analysis • Tool-supported generation and maintenance of an updated and strategic test plan, as the design evolves. 5

PISATEL (Realistic) Preconditions It is plausible that …. • in early design phases many of the UCs are defined at a high level and only few of them are properly refined. • there is not yet the specification of every possible scenario, but only few SDs already exist that realize the defined UCs. • different people or teams take part to the development, everyone familiar with a different component or functionality. 6

PISATEL Fundamental Guidelines • FG 1: use the UML diagrams developed during specification and design, without imposing to the UML designers any additional formalism, or adhoc effort specifically for testing purposes. • FG 2: test planning for integration testing must cope with high level diagrams, even though not yet completely specified, or refined 7

PISATEL Trade-off • UML is a notation and does not impose a specific process. • A test strategy needs to refer to a systematic and documented design process We want to impose minimal requirements: – Use Cases (UCs) organized into a hierarchy – Express relevant scenarios as Sequence Diagrams (SDs) Development Process 8

PISATEL Requirements • The reference documentation is derived by collecting together and automatically organizing the diagrams developed during the design phase without additional manual effort of formalization. • Inconsistencies and incompleteness are highlighted (Design weaknesses) Designer Argo (from Logical View) Check. List Argo. UML vers. 0. 7 Sequence Diagram: Check. List. . . Libraries Argo. UML GEF UML Meta-Model Kernel To Do List Sequence User Model Diagram: Kernel Design Critics / Sequence Kernel_Sequ. . . Diagram: To. . . Sequence Diagra. . . Diagram: Desi. . . 9

PISATEL Cow_Suite approach Cow_Suite consists in the junction of two main components: v UIT (Use Interaction Test) and v Co. WTe. St (Cost Weighted Test Strategy) Cow_Suite =Cowtest plu. S UIT Environment The processing of the Cow_Suite approach starts by analysing the project description*, and proceeds in parallel with the design and analysis refinement steps. [*: the internal description of the project model, given by Rational Rose in a. mdl file] 10

PISATEL Cow_Suite tool The Cow_Suite tool consists of three elements working in combination: 1. the Main Tree: a systematic support to decide a. how to distribute the available test effort between the many potential test cases b. how to prioritise among the test cases when a functional coverage is fixed. 2. the UIT Tree: automated generation of abstract test cases from the UML diagrams. 3. the Test specification: a systematic support to generate the test procedures and subsequently the test scripts. 11

Cow_Suite Scheme Se le ct io n St rat eg y PISATEL Cow_Suite Obj 1 Obj 2 SDs selection according to the strategy chosen Obj 3 Method 1() Method 2() Automatic UIT application for Test cases derivation Test case Method 3() UI Test case T Interaction with the user for deriving executable test procedures Test case Test Procedures Test case ion cat Test case ifi pec ts Test case Test case list 12

PISATEL Main steps • • • Derive a Reference Tree Deduce the Critical Profile Select the Test Strategy Derive the Test Cases Implement the Test Procedures 13

PISATEL 1. 1 Tree Derivation Use Cases (UCs) and Sequence Diagrams (SDs) of the. mdl project design realized with Rational Rose are automatically organized by Cow_Suite tool in a sort of hierarchical (pseudo-)tree, starting from the Main Use Case Diagram, UCD. Hierarchical Tree The case study is Argo. UML, a tool supporting OO design with UML 14

PISATEL 1. 2 Tree Derivation Hierarchical Tree • Looking at the tree the Cow_Suite users can get a complete view of the status of the specification of the various functionalities; • The tree represents an ordered and organized documentation, that is continuously updated with the latest changes and progressively completed 15

PISATEL 2. 1 Deduce the Critical Profile • Systems are naturally composed of different parts realizing one or more functionalities, each with different importance for the overall system performance. • The effort devoted to the test of different functionalities must be planned and scheduled considering their relative “importance” (risk-based testing, reliability guided testing, etc…). • Following level by level the tree structure of the UCs and SDs, the strategy allows the people in charge of the development of one or more functionalities to simply annotate the corresponding node in the tree with a number (weight). 0. 75? 0. 60? To Do List 0. 65!!! Sequence Diagram: To. . . 16

PISATEL 2. 2 Weighted tree derivation Hierarchical Tree The user annotates every node with a value, leaf weight, representing the relative “importance” of this node (be it a UC or a SD) with respect to the other nodes at the same level in the tree 17

PISATEL 2. 3 Weights management The weight value belongs to the [0, 1] interval and must be assigned so that the sum of the weights associated to all the children of one node is equal to 1. The weights are user modifiable as the development proceeds. 18

PISATEL Integration Testing vs Integration Stage • To realize a complete system functionality, it is generally necessary to execute several actions realizing lower level functionalities • A system functionality is realized by the different components interaction. • Integration Testing: testing interactions between system components (processes, packages, subsystems… ) at an appropriate level of abstraction. • Integration Stage: is useful because generally the functionalities are not specified at the same level of detail and with the same number of UCs and SDs 19

PISATEL 2. 4 Integration Stage Selection • The first integration stage is represented by the main UC and the SDs (if any), which are children of this node (hence they are at level 2 of the tree). • The i-th integration stage is represented by the UCs positioned in at i-th level of the tree and every SDs, children of these nodes, situated at i+1 -th level. NOTE: the SDs at level i+1 represent the interaction between the different components that realize the functionalities described in the UCs at i-th level. 20

PISATEL Example: 4 th Integration Stage The Cow_Suite user selects the integration stage (4) at which the test is conducted. The nodes that are considered by Cow_Suite tool for the subsequent evaluations are all and only those belonging to the 4 th integration stage plus all the tree leaves existing at higher levels (3 rd, 2 nd, 1 st). SDs at 5 th tree level 21

PISATEL 2. 5 Final weights assignment • The weights assigned to each node are used to define a relative importance factor, in terms of how risky is that node and how much effort should be put to test it. • Considering every node, from the root down to the elements belonging to the integration stage considered, its final weight is computed as the product of all the nodes weights on the complete path from the root to this node. • The final resulting weight is an element of discrimination for choosing amongst the tests to execute. Designer (from Logical View) Argo. UML vers. 0. 7 0. 4 0. 3 0. 6 Libraries Argo. UML FW=0. 6 FW=0. 12 GEF UML Meta-Model 22

PISATEL 3. Cow_Test Strategy Two different situations are considered in test cases planning: 1: Cowtest_ing with fixed number of tests • A certain budget is available, which is translated in terms of number of tests, NT, to execute. Ca s e Te st se ase Ca t st Testsease Ca t Ca C s Teest Case se T Te se Te st NT Ca st C Tes t Caset Ca Tes se Ca Te st Te Test Case Case Case Case Case Test Test Test Test Test Case Case Case Case Case Test Test Test Test se • In this case the tool select the most adequate selection of NT tests from among the (many) various test cases that could be potentially conceived. 23

PISATEL 3. Cow_Test Strategy • • 2: Cowtest_ing with fixed functional coverage A certain percentage of functionalities must be covered. In this case by the tool is possible to define in advance which are the functionalities to be covered and the minimum number of tests to execute. Functional Coverage = 90% 10 2 2 Minimum Number of tests to execute = 16 1 1 24

PISATEL 3. 1 Select the Test Strategy Cowtest_ing with fixed number of tests. The selection of the most suitable distribution of Test Procedures is developed on the basis of leaves weights. 25

PISATEL Cowtest_ing with fixed number of tests Using the resulting final weight at the chosen integration stage, nw, for each SD the number of Test Procedures, NTtest, can be easily derived 26

PISATEL 3. 2 Select the Test Strategy Cowtest_ing with fixed functional coverage. The most critical system functionalities are selected for reaching the fixed coverage. Test Procedures are distributed accordingly. 27

PISATEL Cowtest_ing with fixed functional coverage Considering the final weights, nw, of every node the tool can: • select the functional test cases to be run, by ordering in decreasing manner the nw*100 values and summing them, starting from the heaviest one, until the fixed coverage, C, is reached. • Calculate the minimum number of tests with respect to the fixed coverage percentage. 28

PISATEL Notes • To make a more practical prediction in terms of man/hours, or required budget for testing, it would be necessary to be able to estimate the cost of the different test cases. • An alternative application of Cowtest is considering a certain budget, B, planned for the testing instead of the fixed number of test NT. In this case is possible to evaluate the “relative cost”, b, for the testing of every leaf of the fixed integration stage. 29

St rat eg y PISATEL Se le ct io n Cow_Suite Scheme Cow_Suite Obj 1 Obj 2 SDs selection according to the strategy chosen Obj 3 Method 1() Method 2() Automatic UIT application for Test cases derivation Test case Method 3() UI Test case Test case Test case Test case 30

PISATEL 3. 3 Sequence Diagrams Selection For each fixed integration stage, a weighted subtree is derived according to the choosen test criterion. Only the relative SDs are considered for Test Case generation. SDs selection Final weight N. Test Procedures 31

PISATEL 4. Test Case Derivation SDs Selection The tool uses the UIT method, Use Interaction Test, to derive the Test Cases. The method is mainly based on the SDs (particularly objects, messages and parameters) 32

PISATEL 4. 1 Use Interaction Test methodology • A method to systematically derive Integration Test suites from UML diagrams. • Inspired at large by the Category Partition method: it defines classes of functional equivalence by selecting significant values (choices) of relevant categories (parameters or data) • Incremental test strategy (over the Use Case organizational structure) 33

PISATEL 4. 1. 1 Sequence Diagram analysis Horizontal axis shows a set of objects, Test Units, that interact through messages. Those are system units separately testable (class testing) for exercising a possible use of system. Example: Decision. Model, Decision, Goal. Model, Goal. 34

PISATEL 4. 1. 2 Sequence Diagram analysis Vertical axis shows: • the exchanged messages, Interactions Categories, involved in object interactions; • their parameters and relevant inputs, Settings Categories. Decision. Model Settings: _decisions Interaction: Decision. Model() get. Decisions() Decision Interactions: Decision(name: String, priority: int) Get. Name(), Get. Priority(). . . 35

PISATEL Additional Information: Class Diagram Complementary analysis of the Class Diagram description (mainly used for searching the Settings Categories). These are attributes (or a subset of them) of a class (and the corresponding SD’s object) that affect object interactions and are represented by input parameters used in messages or data structures. 36

PISATEL 4. 1. 3 Search of Message Sequences Message. Sequences: set of Messages (in temporal order), involved in a SD, used by objects to define and elaborate specific functionalities. Example: get. Decisions() get. Priority() 37

PISATEL 4. 2 Test Case Derivation For each traceable Message Sequence a Test Case is generated. It contains the list of all its Settings and Interactions Categories and their values. Detailed Test Case Description 38

PISATEL Conditions A Test Case (TC) can be associated to some feasibility conditions formally expressed (notes or specifications tab). In this case TC is divided in different subcases. 39

PISATEL Number of Test Procedures The final weight of every SDs is used to automatically derive the number of Test Procedures associated to each Test Case. Number of associated Test Procedures 40

Cow_Suite Scheme Se le ct io n St rat eg y PISATEL Cow_Suite Obj 1 Obj 2 Interaction with the user for deriving executable test procedures Obj 3 Method 1() Method 2() Test case Method 3() UI Test case Test Procedures Test case ion cat Test case ifi pec ts Test case Test case list 41

PISATEL 5. 1 Test Procedures Specification Test Specification: For each identified category, we consider all its possible values and constraints (“choices”). The user interacts with the tool for inserting Choices values. 42

PISATEL 5. 1. 1 Interactions. Categories Specification Interactions Categories: objects interactions, exchanged messages. The tool implements two possible choices for the user: the default values, the user values 43

PISATEL State Diagram If a State Diagram is linked to a specific Test Unit (class), further information can be found about the relevant configurations of Interaction. Categories State diagram of the “Critic” class 44

PISATEL 5. 1. 2 Specification of Settings. Categories Settings Categories: parameters and inputs relevant for the Test Unit. The tool implements two possible choices for the user: the default values, the user values. 45

PISATEL 5. 2 Test Procedures Implementation A Test Procedure is automatically generated from a Test Specification, for each possible combination of choices of every category involved in a Messages Sequence. 46

Test Procedures: an example Test Specification: Decision. Model: Settings: _decisions Naming Storage Modularity Inheritance Stereotypes Relationships……. . Interactions: get. Decisions Opening a new file Opening a saved file After a modification and before saving After a modification and after saving Decision. Model() Class constructor Test Procedure get. Decisions() get. Name() Opening a saved file _decisions Naming Note: Such a Test Case must be repeated for all possible values of _decisions 47

PISATEL Constraints on Choices Some choices combinations could result contradictory or meaningless. Following the Category Partition methodology, constraints must be added to the Setting. Categories choices. Constraint are specified using: • Properties, that can be assigned to certain choices and testing for by other choices. • IF Selector, is a conjunction of properties that were previously assigned to other choices. 48

PISATEL Example Settings. Categories: pattern size: empty single character many character quoting pattern is quoted pattern is not quoted pattern is improperly quoted [Property Empty] [Property Not Empty] [Property quoted] [if Not Empty] 49

PISATEL Settings. Categories: Specification Constraints Test Specification: Decision(_name, _priority): _name Naming [if priority<5] Class Selection [if priority=0] Storage Inheritance Modularity…. . _priority 0 [Property priority=0] 1 2 3 7 [Property not priority<5] 50

Settings. Categories: Specification Constraints Test Procedure X Decision. Model() Opening an existing document Decision( _name, _priority) _name Naming [if priority<5] _priority 7 [Property not priority<5] Test Procedure Decision. Model() Opening an existing document Decision( _name, _priority) _name Naming [if priority<5] _priority 4 51

PISATEL 5. 2 Test Suite The set of all generated Test Procedures is named Test Suite 52

Note • The Test Procedures, as derived by the tool, are not directly executable ( Test Scripts). The derived Test Procedures have to be executed and logged by a test driver. Cow. Suite goal is: – giving automatic and systematic support to construct a meaningful Test Suite. – It is not to directly execute the tests. Cow_Suite Rational Rose REI. Tlb libr ary Test Driver. mdl Files MDL Analizer Test Tree Generator Weights Manager Test Case Generator Test Case Builder Test Case Checker 53

Remarks • The degree of detail of the Test Suite depends directly on: – the design detail level – granularity of the inserted values (selection of the constraints and choices of the UIT categories). • Currently: the final result is a text document containing the Test Suite. • In future: different format (e. g. XML), that can be accepted in input by the adopted test driver. 54

PISATEL Features under Construction Two different situations can occur during the tree construction: Designer Argo (from Logical View) Disconnected tree Check. List Argo. UML vers. 0. 7 Sequence Diagram: Check. List. . . Libraries Argo. UML GEF UML Meta-Model Reuse of subsystems Kernel To Do List Sequence Diagram: Kernel User Model Design Critics / Sequence Kernel_Sequ. . . Diagram: To. . . Sequence Diagra. . . Diagram: Desi. . . 55

PISATEL Disconnected tree • Analysing the. mdl file, the structure derived by the Cow_Suite tool is not a connected tree. Some “holes” in the design could be highlighted if existing pieces of a system have not a logical connection. More than one disconnected subtrees are derived by the Cow_Suite tool and visualized to the user. • Possible solution: inserting dummy nodes and arcs where necessary (according to the logical relation between the disconnected trees). Designer (from Logical View) Kernel Argo. UML vers. 0. 7 Libraries Argo. UML User Model GEF UML Meta-Model Design Critics Sequence Diagra. . . Diagram: Desi. . . 56

PISATEL Subsystems Reuse The reuse of a same subsystem in different parts of the system design causes anomalous connections, i. e. cycles, in the derived structure. 57

PISATEL Subsystems Reuse Possible solution: • the first time a subsystem is encountered: associate to it a node and derive for it a corresponding subtree, as usual; • for the following occurrences of the same subsystem, add to the pseudo-tree a “dummy node” referring back to the subtree already derived. 58

PISATEL Conclusions Systematic and automated support for test planning and test case generation, based on the same UML diagrams developed for design and analysis Tool integrated with the design tool (e. g. Rational Rose) 59

PISATEL Future Work Test case derivation (UIT) to be validated on ERI case studies Practicality of a test strategy based on manual weights (COW_test)? How to assign the weights? How to use Cow_Suite within a multi-phase test process (different test stages)? 60

PISATEL 61

PISATEL Cow_Suite Architecture Rational Rose REI. Tlb library Cow_Suite. mdl Files MDL Analizer Test Tree Generator Weights Manager Test Case Generator Test Case Builder Test Case Checker 62