Chapter 14 Testing Tactics 1 V-Cycle 软件任务

Chapter 14 Testing Tactics 1

V-Cycle 软件任务交付软件验证与确认系统需求系统测试验证测试后的软件验证与确认软件需求合格性测试验证集成的软件验证概要设计集成测试验证详细设计测试后的单元验证验证单元测试编译后的单元编码 2

Three Examples 防不胜防的软件错误 —— 例1： 1963年, 美国, 飞往火星的火箭爆炸, 损失$ 10 million. 原因: FORTRAN循环 DO 5 I = 1, 3 误写为 DO 5 I = 1. 3 3

例2： 1996年，美国 Washington Post 报导 Dallas, Aug. 23 — 美国航空公司一架直升飞机撞上山崖，造成 163人死亡。原因：飞行目的地代码输入错误。驾驶Boeing 757的机长认为他输入了目的地Cali的单字符代码，而在南美的航空资料中， Cali的单字符代码和Bogota的一样，所以系统将目的地处理成了Bogota。但在大多数计算机数据库中Cali和Bogota的代码是不同的。 4

例3：[摘自Pfleeger 书中的“REAL-TIME EXAMPLE”] European Space Agency (ESA) 的Ariane-5航空运载火箭在June 4, 1996她第一次飞行中远程失控，造成火箭本身和4棵卫星损失，$500 million 原因：其前身Ariane-4声誉良好，在开发Ariane-5时，忽略了一个需求细节， Ariane-5与Ariane-4的轨道不同。 5

Outline The Concepts of Testing Software Testing Objectives Testing Principles Testability Test Case Design Whiter-box Testing Basis Path Testing Control Structure Testing Black-box Testing GUI Testing 6

The Concepts of Testing 7

The Relationship Between Testing and Development 8

What is Software Testing? Testing is the process of exercising a program with the specific intent of finding errors prior to delivery to the end user. Software testing is a critical element of software quality assurance and represents the ultimate review of specification, design, and code generation. 9

The Flow of Software Testing 10

What Testing Shows errors requirements conformance performance an indication of quality 11

What is a “good” test? It has a high probability of finding an error. It is not redundant. It should be “best of breed”. It should be neither too simple, nor too complex. 12

Who Tests the Software? developer Understands the system but, will test "gently" and, is driven by "delivery" independent tester Must learn about the system, but, will attempt to break it and, is driven by quality 13

Testing Objectives Testing is a process of executing a program with the intent of finding an error. A good test case is one that has a high probability of finding an as-yet undiscovered error. A successful test is one that uncovers an as-yet undiscovered error. 14

Testing Principles All tests should be traceable to customer requirements. Tests should be planned long before testing begins. The Pare-to (80% to 20%) principle applies to software testing. Testing should begin “ in the small” and progress toward testing “ in the large”. Exhaustive testing is not possible. To be most effective, testing should be conducted by an independent third party. 15

Testability Operability—it operates cleanly Observables—the results of each test case are readily observed Controllability—the degree to which testing can be automated and optimized Decomposability—testing can be targeted Simplicity—reduce complex architecture and logic to simplify tests Stability—few changes are requested during testing Understandability—of the design 16

Exhaustive Testing loop < 20 X 14 There are 10 possible paths! If we execute one test per millisecond, it would take 3, 170 years to test this program!! 17

Selective Testing Selected path loop < 20 X 18

Software Testing black-box methods white-box methods Methods Strategies 19

Methods of Testing 静态测试静态分析代码审查(Code Reading Review) 代码走查(Walkthroughs) 技术评审(Technical Review) 桌面检查(Desk Checking) 动态测试白盒测试基本路径测试控制结构测试黑盒测试功能分解等价类划分边值分析因果图随机测试猜错法 20

静态测试代码审查：小组集体阅读讨论检查代码代码走查：小组集体用“脑”执行并检查代码桌面检查：由程序员阅读自己编写的程序技术评审：会议形式讨论检查代码静态分析：对代码的机械性、程式化的特性分析方法，包括控制流分析、数据流分析、接口分析、表达式分析 21

Test Case Design 22

Test Case Design "Bugs lurk in corners and congregate at boundaries. . . " Boris Beizer OBJECTIVE to uncover errors CRITERIA in a complete manner CONSTRAINT with a minimum of effort and time 23

Whiter-Box Testing 24

What is White-Box Testing? white-box , or open-box, clear-box testing: Use the structure of the program to test. —— Structural testing . . . our goal is to ensure that all statements and conditions have been executed at least once. . . 25

Why Cover? logic errors and incorrect assumptions are inversely proportional to a path's execution probability we often believe that a path is not likely to be executed; in fact, reality is often counter intuitive typographical errors are random; it's likely that untested paths will contain some 26

Basis Path Testing 27

What is Basis Path Testing? It is a white-box testing technique first proposed by Tom Mc. Cabe. It enables the test case designer to derive a logical complexity measure of a procedural design and use this measure as a guide for defining a basis set of execution paths. Test cases derived to exercise the basis set are guaranteed to execute every statement in the program at least one time during testing. 28

Flow Graph Notation The structured constructs in flow graph form: Sequence; If; While; Until; Case; Where each circle represents one or more non branching PDL or source code statements. 29

Flow Graph 30

Flow Chart and Flow Graph 31

An Example of Flow Graph 32

Cyclomatic Complexity （秩复杂性） Cyclomatic complexity is a software metric that provides a quantitative measure of the logical complexity of a program. An independent path is any path through the program that introduces at least one new set of processing statements or a new condition. 33

Computing Complexity There are three ways: The number of regions of the flow graph correspond to the cyclomatic complexity; Cyclomatic complexity, V(g), for a flow graph G is defined as: – V(G) = E - N + 2 where E is the number of flow graph edges, N is the number of flow graph nodes; Cyclomatic complexity, V(G), for a flow graph G is also defined as: – V (G) = P + 1 where P is the number of predicate nodes contained in the flow graph G. 34

An Example First, we compute the cyclomatic complexity: V(G) = 11 edges - 9 nodes + 2 = 4 or V(G) = 3 predicate nodes + 1 = 4 In this case, V(G) = 4 35

A Conclusion A number of industry studies have indicated that the higher V(G), the higher the probability or errors. modules V(G) modules in this range are more error prone 36

Deriving Test Cases The basis path testing method can be applied to a procedural design or to source code. A series of steps: 1. Using the design or code as a foundation, draw a corresponding flow graph. 2. Determine the cyclomatic complexity of the resultant flow graph. 3. Determine a basis set of linearly independent paths. 4. Prepare test cases that will force execution of each path in the basis set. 37

An Example Next, we derive the independent paths: 1 Since V(G) = 4, there are four paths 2 3 4 5 7 8 6 Path 1: Path 2: Path 3: Path 4: 1, 2, 3, 6, 7, 8 1, 2, 3, 5, 7, 8 1, 2, 4, 7, 2, 4, . . . 7, 8 Finally, we derive test cases to exercise these paths. 38

Basis Path Testing Notes you don't need a flow chart, but the picture will help when you trace program paths count each simple logical test, compound tests count as 2 or more basis path testing should be applied to critical modules 39

Graph Matrices It is a square matrix whose size is equal to the number of nodes on the flow graph. 1 1 a 5 4 d c g 2 4 5 1 -1=0 2 b f 3 a 1 3 e 2 3 4 c 5 g 2 -1=1 b d f e 2 -1=1 3+1=4 Cyclomatic complexity 40

Control Structure Testing 41

Control Structure Testing There are other broaden testing coverage and improve quality of white-box testing: Condition testing; Data flow testing; Loop testing; 42

Condition Testing Condition testing is a test case design method that exercises the logical conditions contained in a program module. Condition testing focuses on testing each condition in the program. The errors include the following: Boolean operator error; Boolean variable error; Boolean parenthesis error; relational operator error; arithmetic expression error; 43

Condition Testing Strategies Branch testing; Domain testing; Branch and Relational Operator (BRO); 44

Data Flow Testing The data flow testing method selects test paths of a program according to the locations of definitions and uses of variables in the program. Data flow testing strategies are useful for selecting test paths of a program containing nested if and loop statements. 45

Loop Testing Simple loop Nested Loops Concatenated Loops Unstructured Loops 46

Loop Testing: Simple Loops Minimum conditions—Simple Loops 1. skip the loop entirely 2. only one pass through the loop 3. two passes through the loop 4. m passes through the loop m < n 5. (n-1), n, and (n+1) passes through the loop where n is the maximum number of allowable passes 47

Loop Testing: Nested Loops Start at the innermost loop. Set all outer loops to their minimum iteration parameter values. Test the min+1, typical, max-1 and max for the innermost loop, while holding the outer loops at their minimum values. Move out one loop and set it up as in step 2, holding all other loops at typical values. Continue this step until the outermost loop has been tested. Concatenated Loops If the loops are independent of one another then treat each as a simple loop else* treat as nested loops endif* for example, the final loop counter value of loop 1 is used to initialize loop 2. 48

Example func(int A, B, X) { if((A>1)&&(B=0)) {X: =X/A}; if((A=2)||(X>1)) {X: =X+1}; } a A>1 and B=0 YES c NO X=X/A b A=2 or x>1 NO YES e X=X+1 d 49

Sentences Coverage a A>1 and B=0 取A=2，B=0，X=3 YES c NO X=X/A b A=2 or x>1 NO YES e X=X+1 d 50

Branch Coverage a A>1 and B=0 A=3，B=0，X=1 沿路径acbd执行 YES c NO A=2，B=1，X=3 沿路径abed执行 X=X/A b A=2 or x>1 NO YES e X=X+1 d 51

Condition Coverage 共有四个条件：A>1，B=0， A=2，X>1 A=2，B=0，X=4 a A>1 and B=0 YES c NO X=X/A 沿路径acbe执行 A=1，B=1，X=1 沿路径abd执行 b A=2 or x>1 NO YES e X=X+1 d 52

Condition Compound Coverage a A>1 and B=0 YES c NO X=X/A b A=2 or x>1 NO YES e 共有8种条件： ①A>1, B=0②A>1, B≠ 0 ③A≤ 1, B=0④A ≤ 1, B≠ 0 ⑤A=2, X>1⑥A=2, X≤ 1 ⑦A≠ 2, X>1⑧A≠ 2, X≤ 1 A=2, B=0, X=4(①⑤) A=2, B=1, X=1(②⑥) A=1, B=0, X=2(③⑦) A=1, B=1, X=1(④⑧) X=X+1 d 53

Path Coverage a A>1 and B=0 YES c NO X=X/A b A>1 and B=0 NO 四条路径： acbe, abd, abed, acbd A=2, B=0, X=3(acbe) A=1, B=0, X=1(abd) A=2, B=1, X=1(abed) A=3, B=0, X=1(acbd) YES e X=X/A d 54

Black-Box Testing 55

requirements output input events 56

What is Black-Box Testing? black-box , or closed-box testing: Make sure that every kind of input is submitted, and the output observed matches the output expected. —— Functional testing 57

Black-Box Testing Black-box testing can find following errors: incorrect or missing functions; interface errors; errors in data structures or external database access; behaviour or performance errors; initialization and termination errors; The methods of black-box testing: Graph-based testing methods; Equivalence partitioning; Boundary value analysis; Comparison testing; Orthogonal array testing; 58

The Steps of Black-Box Testing The steps of black-box testing : to understand the objects and their relationships; to define a series of tests; to find the errors; 59

Graph-Based Testing Methods Transaction flow modeling The nodes represent steps in some transaction, and the links represent the logical connection between steps. (data flow diagram) Finite state modeling The nodes represent different user observable states of the software, and the links represent the transaction that occur to move from state to state. (state transition diagram) Data flow modeling the nodes are data object and the links are the transformations that occur to translate one data object into another. Timing modeling The nodes are program objects and the links are the sequential connections between those objects. 60

Equivalence Partitioning user queries FK output input mouse formats prompts picks data 61

对输入数据的规定等价类划分数值范围（如1 ---999）一个合理类（1 ≤ x ≤ 999）二个不合理类（x ＜ 1, x＞ 999) 数据个数一个合理类（个数合要求）二个不合理类（个数太少，个数太多) 一组可能的值每个可能值一个合理等价类其他值为一个不合理等价类规定输入值必须满足的条件满足条件者为一个合理等价类（如串的首字符为字母）不满足条件者为一个不合理等价类若一个等价类的各值在程序把此等价类划分为更小的等价类中处理方法不同 62

Sample Equivalence Classes Valid data user supplied commands responses to system prompts file names computational data physical parameters bounding values initiation values output data formatting responses to error messages graphical data (e. g. , mouse picks) Invalid data outside bounds of the program physically impossible data proper value supplied in wrong place 63

例：考察一个把数字串转变成整数的函数。用二进制补码表示整数，机器字长 16位，即整数范围最小为 32768，最大为 32767。函数及参数的PASCAL说明如下： function Str. To. Int (dstr : shortstr) : integer; type shortstr = array [1. . 6] of char; 要求被处理的数字串是右对齐的，即在少于6个字符的串左边补空格。负号在最高位数字左边一位。试用等价划分法设计测试方案。 64

解：首先根据规格说明划分等价类。考虑到PASCAL编译器的固有检错功能，测试时不需要使用长度不等于6的数组，也不需要用非字符数组类型的参数。有效输入类： ① 1~6个数字字符组成的数字串（最高位非 0）； ②最高位为 0的数字串； ③最高位左邻负号的数字串；无效输入类： ④空字符串（6位空格）； ⑤左边补位的既非 0亦非空格； ⑥最高位右边含有空格； ⑦最高位右边含有其它非数字字符； ⑧负号与最高位间有空格； 65

有效输出类：无效输出类： ⑨ 在合法范围内的负整数； 12 小于 - 32768的负整数； 10 0 ； 13 大于 32767正整数。 11 在合法范围内的正整数；下面根据等价划分，设计出一套测试方案： ① 1~6个数字字符组成的数字串，最高位非 0；输出为合法正整数。 1 预期输出： 1 输入： ②最高位为 0的数字串，输出为合法正整数。输入： 0 0 0 1 预期输出： 1 66

③负号与最高位数字相临；输出合法负整数。输入：预期输出：-1 - 0 0 1 ④最高位为 0；输出 0。输入： 0 0 0 预期输出： 0 ⑤太小的负整数。输入： - 3 2 7 6 9 预期输出：“错误，无效输入” ⑥太大的正整数。输入： 3 2 7 6 8 预期输出：“错误，无效输入” 67

⑦空字符串。输入：预期输出：“错误：没有数字” ⑧左边补位的非 0也非空格。输入： a a a 1 预期输出：“错误：非法填充” ⑨最高位右边也含空格。输入： 1 2 预期输出：“错误：无效输入” ⑩最高位右边含其它非数字字符。输入： 0 0 1 x x 2 预期输出：“错误：无效输入” 负号与最高位间有空格。输入： - 1 2 预期输出：“错误：负号位置非法” 68 11

Boundary Value Analysis user queries FK output input mouse formats prompts picks input domain data output domain 69

规定了数据大小选正好等于边界值的为合理的测试用例，刚好越过边界的为不合理的测试用例规定了数据个数个数刚好为边界值，少一个，多一个对输出数据也使用以上两条数据是有序集合注意集合的第一个和最后一个元素规定了数据满足的条件如（a+b>c) 合理： a=3 b=4 c=5 不合理：a=3 b=4 c=8 边界值，a=3 b=4 c=7 70

An Example 1: Rem Create a 10 element integer array 2: Rem Initialize each element to -1 3: Dim data(10) As Integer 4: Dim I As Integer 5: For I=1 To 10 6: data(I)=-1 7: Next I 8: End 71

Orthogonal Array （正交阵列） Testing It enables you to design test cases that provide maximum test coverage with a reasonable number of test cases; Used when the number of input parameters is small and the values that each of the parameters may take are clearly bounded 72

Other Black-Box Techniques error guessing methods decision table techniques cause effect graphing 73

白盒测试与黑盒测试对比黑盒测试白盒测试优点适用于各测试阶段从产品功能角度测试容易入手生成测试数据可以构成测试数据使特定程序部分得到测试有一定的充分性度量手段可获得较多具支持缺点某些代码段得不到测试如果规格说明有误则无法发现不易进行充分性度量不易生成测试数据无法对未实现规格说明的部分测试作量大，通常只用于单元测试，有引用局限性质是一种确认技术，回答“我们在构造一个正确的系统吗？” 是一种验证技术，回答“我们在正确地构造一个系统吗？” 74

Comparison Testing Used only in situations in which the reliability of software is absolutely critical (e. g. , humanrated systems) Separate software engineering teams develop independent versions of an application using the same specification Each version can be tested with the same test data to ensure that all provide identical output Then all versions are executed in parallel with realtime comparison of results to ensure consistency 75

OO Testing 76

OOT—Test Case Design (1) Berard [BER 93] proposes the following approach: 1. Each test case should be uniquely identified; 2. Each test case should be explicitly associated with the class to be tested; 3. The purpose of the test should be stated; 4. A list of testing steps should be developed for each test. 77

OOT—Test Case Design (2) A list of testing steps should contain [BER 94]: a. a list of specified states for the object that is to be tested; b. a list of messages and operations that will be exercised as a consequence of the test; c. a list of exceptions that may occur as the object is tested; d. a list of external conditions (i. e. , changes in the environment external to the software that must exist in order to properly conduct the test); e. supplementary information that will aid in understanding or implementing the test. 78

Testing Methods (1) Fault-based testing 1. The strategy for fault-based testing is to hypothesize a set of plausible faults and then derive tests to prove each hypothesis; 2. The objective of fault-based testing within an OO system is to design tests that have a high likelihood of uncovering plausible faults; 3. What types of faults are encountered in operation calls and message connections? Unexpected result; Wrong operation/message used; Incorrect invocation; 79

Testing Methods (2) Test Cases and Class Hierarchy 1. Even though a base class has been thoroughly tested, you will still have to test all classes derived from it. 2. Example: Class Base: inherited(), redefined() Subclass Derived: redefined() Do we need to retest Derived: : inherited()? The Base: : redefined() tests are applied to objects of class Derived; Test inputs may be appropriate for both base and derived classes, but the expected results may differ in the derived class. 80

Testing Methods (3) Scenario-Based Test Design Scenario-based testing will uncover errors that occur when any actor interacts with the software; Fault-based testing misses two main types of errors: incorrect specifications; interactions among subsystems Scenario-based testing concentrates on what the user does, not what the product does; Scenario-based testing tends to exercise multiple subsystem in a single test. 81

Testing Methods (4) Testing surface structure and deep structure Testing surface structure is analogous to black-box testing; Deep structure testing is similar to white-box testing; 82

Testing at the Class level (1) Random testing for OO Classes The number of possible permutations for random testing can grow quite large; A strategy similar to orthogonal array testing can be used to improve testing efficiency; identify operations applicable to a class; define constraints on their use identify a miminum test sequence an operation sequence that defines the minimum life history of the class (object) generate a variety of random (but valid) test sequences exercise other (more complex) class instance life histories Example: Account 83

Testing at the Class level (2) Partition Testing reduces the number of test cases required to test a class in much the same way as equivalence partitioning for conventional software state-based partitioning categorize and test operations based on their ability to change the state of a class attribute-based partitioning categorize and test operations based on the attributes that they use category-based partitioning categorize and test operations based on the generic function each performs 84

Inter-Class Test Case Design (1) Multiple Class Testing For each client class, use the list of class operators to generate a series of random test sequences. For each message that is generated, determine the collaborator class and the corresponding operator in the server object. For each operation in the server object (that has been invoked by messages sent from the client object), determine the messages that it transmits. For each of the messages, determine the next level of operators that are invoked and incorporate these into the test sequence 85

Tests Derived from Behavior Models The tests to be designed should achieve all state coverage [KIR 94]. That is, the operation sequences should cause the Account class to make transition through allowable states 86

Testing for Specialized Environments, Architectures, and Applications 87

GUI Testing A testing strategy similar to random or partition testing can be used to design UI tests; Because many modern GUIs have the same look and feel, a series of standard tests can be derived; Testing should be approached using automated tools; 88

Testing of Client/Server Architectures Testing become more difficulty, which deal with: distribution system; transaction process; different hardware platform; network communication; multiple clients; coordination requirements; 89

C/S Testing Approaches Application function tests; Server tests; Database tests; Transaction tests; Network communication tests; 90

Testing for Real-Time System Task testing Behavioral testing Inter task testing System testing 91

Testing Patterns (1) Benefits: Provide a vocabulary for problem-solvers; Focus attention on the forces behind a problem, that allows designers to better understand when and why a solution applies; encourage iterative thinking, each solution creates a new context in which new problems can be solved. Testing patterns can help a software team communicate more effectively about testing and better understand the forces that lead to a specific testing approach. 92

Testing Patterns (2) Pattern name: pair testing Abstract: A process-oriented pattern, pair testing describes a technique that is analogous to pair programming (Chapter 4) in which two testers work together to design and execute a series of tests that can be applied to unit, integration or validation testing activities. Pattern name: separate test interface Abstract: There is a need to test every class in an object-oriented system, including “internal classes” (i. e. , classes that do not expose any interface outside of the component that used them). The separate test interface pattern describes how to create “a test interface that can be used to describe specific tests on classes that are visible only internally to a component. ” [LAN 01] Pattern name: scenario testing Abstract: Once unit and integration tests have been conducted, there is a need to determine whether the software will perform in a manner that satisfies users. The scenario testing pattern describes a technique for exercising the software from the user’s point of view. A failure at this level indicates that the software has failed to meet a user visible requirement. [KAN 01] 93