Black-Box Test Reduction Using Input-Output Analysis ISSTA

Black-Box Test Reduction Using Input-Output Analysis ISSTA ‘ 00 Patrick J. Schroeder, Bogdan Korel Department of Computer Science Illinois Institute of Technology Chicago, IL USA PJS ISSTA '00 1

Presentation Overview 1) Motivation 2) Black-Box Testing 3) Input-Output Analysis 4) Determining Input-Output Relationships 5) Conclusion PJS ISSTA '00 2

Motivation for Research · Experience in Industry as a black-box tester of large, complex, telecommunications systems · Root cause analysis of customer-found faults often discovered ineffective or redundant black-box tests · Additional information from the software implementation often improves black-box testing (after the initial release!) PJS ISSTA '00 3

Black-Box Testing · Black-box testing is testing from a functional or behavioral perspective to ensure a program meets its specification · Testing usually conducted without knowledge of software implementation - system treated as a “black box” · Black-box test design techniques include: equivalence partitioning, boundary value analysis, cause-effect graphing, random testing PJS ISSTA '00 4

Observations on Black-Box Testing · Testing from the black-box perspective generates a large number of tests (e. g. AT&T Definity PBX > 3 million tests) · All testers are faced with resource limitations; the number of tests must be reduced · Test reduction done randomly or based on “engineering judgement” may lead to ineffective or redundant tests PJS ISSTA '00 5

Combinatorial Testing · Combinatorial testing: For programs with multiple inputs, one must decide how to test different combinations of selected test data values · The most thorough approach is to test all combinations - this is quite often impossible due to the large number of tests · However, not to consider testing different input combinations could lead to an unacceptable number of undetected software faults PJS ISSTA '00 6

Combinatorial Explosion X 1 X 2 X 3 . . . Xn Program P If n = 10 and 10 test data values are selected for each variable, the total number of combinatorial test cases is 1010 PJS ISSTA '00 7

Current Approaches · Testers may try to identify “important” combinations to test using specifications, historical fault data, and engineering judgement · Orthogonal arrays, or other combinatorial design techniques, may be used to tests pair-wise or n-way combinations of inputs · Difficult to determine impact on fault-detection capability of the test set using these techniques PJS ISSTA '00 8

Definition of the Problem · In combinatorial testing, black-box testers face the situation where the number of possible test cases far exceeds the time and resources available to execute them · How can we reduce the number of combinatorial tests while maintaining the fault-detection capability of combinatorial testing? PJS ISSTA '00 9

Proposed Technique · Observations: -Current techniques do not consider program output variables -For a class of programs, not all program inputs affect all program outputs · We propose the use of Input-Output (IO) Analysis to determine which program inputs affect program outputs · This information can be used to reduce the number of combinatorial tests while “maintaining” the fault-detection capability of the test set PJS ISSTA '00 10

Simple Example using IO Analysis Input data selected using equivalence partitioning: A={1, 3} B={N, S, E, W} C={TDC, BDM} W Z Total Number of Combinatorial Tests: 2 * 4 * 2 = 16 PJS ISSTA '00 11

IO Analysis Applied A={1, 3} B={N, S, E, W} W PJS C={TDC, BDM} Z ISSTA '00 12

IO Analysis Applied A={1, 3} B={N, S, E, W} C={TDC, BDM} W T(W) (1, N, TDC) (1, N, BDM) (3, N, TDC) (3, N, BDM) PJS Z T(Z) (1, N, TDC) (1, S, TDC) (1, E, TDC) (1, W, TDC) ISSTA '00 T(W) T(Z) (1, N, TDC) (1, N, BDM) (3, N, TDC) (3, N, BDM) (1, S, TDC) (1, E, TDC) (1, W, TDC) 13

Optimization Problem • Create the smallest possible test set which covers all data combinations in the individual output test sets • Test set optimized using “Don’t Care” (DC) conditions T(W) (1, DC, TDC) (1, DC, BDM) (3, DC, TDC) (3, DC, BDM) PJS T(Z) (DC, N, DC) (DC, S, DC) (DC, E, DC) (DC, W, DC) ISSTA '00 Toptimized (1, N, TDC) (1, S, BDM) (3, E, TDC) (3, W, BDM) 14

Determining Input-Output Relationships · Information may be gathered in a variety of ways (e. g. from specifications, code) · · Manual collection of information difficult and time consuming Automated collection possible through: -Static Analysis -Dynamic Analysis -Execution-oriented Analysis PJS ISSTA '00 15

Automated Analysis Techniques · Static techniques analyze a program’s source code - static program dependencies used to determine IO relationships · Dynamic techniques analyze a program’s execution trace across a set of test data - dynamic dependencies used to determine IO relationships · Execution-oriented techniques manipulate individual inputs across multiple program executions in a test harness to determine IO relationships PJS ISSTA '00 16

Technique Advantages/Disadvantages · Static techniques reliably determine the absence of relationships between inputs and outputs; however, they may identify relationships that are not actually present · Dynamic analysis and execution-oriented analysis reliably determine the presence of input-output relationships; however, they cannot guarantee that all relationships will be detected · Dynamic analysis and execution-oriented analysis may be used when static analysis, perhaps in over-estimation, indicates that all program inputs affect all program outputs PJS ISSTA '00 17

Experimental Study Results 1) Warehouse Management System Combinatorial Tests: Using IO Analysis 3, 125 40 2) DACS Configuration System Combinatorial Tests Using IO Analysis 22, 500 158 3) Liquidity Ratio Spreadsheet Combinatorial Tests Using IO Analysis PJS 1. 95 Million 625 ISSTA '00 18

Conclusion · For the class of program and type of problem defined, inputoutput analysis can reduce the number of combinatorial tests while “maintaining” the fault-detection capability of the test set · Research continues on identifying program types and testing situations where our approach is applicable · Additional experimentation with execution-oriented analysis to provide program dependencies, and other information useful in black-box testing, is underway PJS ISSTA '00 19