Testing and Inspecting to Ensure High Quality Part

Testing and Inspecting to Ensure High Quality Part 1: Basic Definitions Effective and Efficient Testing Lots of Personal Notes

Overview Testing and Inspections – two very important concepts / activities in software development. We often have ‘roles’ of testers; individual developers; for different ‘kinds’ of testing; independent test groups; quality assurance groups, etc. Quality Control; Editors; Various sizes and shapes…. Very unique to the environment within which the software is developed. We have many different types of tests too. Exist at many different levels for many different stakeholders. Some are automated; many are hands-on… Some tests test specific parts of system; others whole; Some at different stages of development; …. Many names!!!! Heuristic: a successful test is one that identifies a flaw. We will present a lot of very important material in these lectures. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 2

Tests – just a few types of tests…. White box testing; Black Box Testing Verification and validation (V&V) Coverage Testing: Branch testing, Path testing, Statement testing Alpha testing, Beta testing Acceptance Testing Unit test, Subsystem test, System test, Integrated tests Environmental System Testing; Operational Field Testing Running in parallel; Running live; Implement in pieces. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 3

Organizational Considerations Deficiency Report (DIREP) Burroughs Hardware / UNISYS Field Assistance Branch; Customer Service Tracked, prioritized, resources allocated. . . Reported to senior levels of management Incident Report – Honeywell Problem Ticket – Fix Tickets (LPS) Office of Primary Responsibility (OPR); Office of Collateral Responsibility… (OCR) Product Manager; Account Rep… © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 4

Basic Definitions • A failure is an unacceptable behavior exhibited by a system —The frequency of failures measures the reliability —Important design objective: achieve a - very low failure rate and hence - a high reliability. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 5

Basic definitions: Failure Causes: • A failure can result from a violation of an explicit or implicit requirement • Specific —functional requirement; —non-functional requirements… © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 6

Basic definitions: MTBF MTTR Metrics • We clearly want to increase the Mean Time Between Failures (MTBF) and reduce the Mean Time to Repair (MTTR): • and damage caused by failures. • MTBF and MTTR (old engineering metrics): are considered measures of reliability. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 7

Basic Definitions - continued • A defect is a flaw in any aspect of the system that contributes, or may potentially contribute, to the occurrence of one or more failures —Might take several defects to cause a particular failure —Defects can occur anywhere – requirements, design, implementation, testing, etc. —They can also occur at any time! • True ‘War’ Story: “Mr. Smith said this couldn’t happen!” • An error is a slip-up or inappropriate decision by a software developer that leads to the introduction of a defect © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 8

Effective and Efficient Testing To test effectively, you must use a strategy that uncovers as many defects as possible. Testing really attempts to find defects! Pure & simple! © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 9

Effective and Efficient Testing • To test efficiently, find the • largest possible number of defects, using the • fewest possible tests. • Testing costs time, money and other resources! • Testing is not cheap! It is quite expensive! • This is where the problems come in. What is the expected return on investment? • There are many kinds of tests designed to uncover different kinds of defects. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 10

Effective and Efficient Testing – more… Testing must be efficient. So what do we test? ? We cannot test everything. (Exhaustive Testing is not possible – proven mathematically) Rather, we design different kinds of tests for ‘sufficient coverage. ’ All kinds of tests: Many Categories Some are lumped into Coverage Tests. statement coverage branch coverage path coverage…. . Others tests focus on creating appropriate test values (Equivalence Testing) © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 11

Black-Box Testing Testers provide the system with inputs; observe outputs • They can see none of: — Source code — Internal data (no algorithms or data structures, …) — Design documentation describing the system’s internals • They simply do inputting based on requirements and observe outputs. • Testing Charge: Often depends on the kind and level of testing and what the ‘charge’ of the testing group (organization) really is. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 12

Black-Box Testing - more • Kind of testing end-users most frequently undertake – for obvious reasons. • Sometimes called • validation testing / • acceptance testing - IF done when system is declared ‘finished. ’ • Black box testing may sometimes be called other things (alpha testing; beta testing…) but these tests (alpha and beta) usually have other connotations associated with them… • Black Box Testing is designed to show functionality and often (sometimes) satisfaction of non-functional requirements. (such as load considerations; simultaneously users, distribution, more) © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 13

Verification and Validation (V&V) • Developers usually do their own testing first (verification) —Usually unit testing; testing within development team. —May involve subsystem testing, integrated testing. … —But, in general, the developers are testing their products. —Generally their testing is white box testing, but can be black box testing. • End users test system against requirements (validation) —This is essentially black box testing.

Glass-Box Testing – White-Box Testing Also called ‘structural’ testing Developers and Testers have access to the system design • They can —Examine the Design and these Documents —View the code for Standardization / conformance! —Observe at run time the steps taken by algorithms and their internal data • Individual programmers often informally employ glass-box testing to verify their own code and functionality • Developers (using tools) can tell what parts of their programs are executed the most (most time spent in them) and then these may be candidates for optimization…. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 15

Glass-Box Testing - White Box Testing Unit Testing Developers are always responsible for their own ‘modules’. modules (units) = programs, classes, subsystems, etc. Developers ‘verify’ their products. Developer programs implement the design Programs satisfy functional requirements, etc… A lot of work to undertake ‘unit testing. ’ Hazardous though, since they may be required to simulate inputs, outputs, files passed, parameter values passed, etc…. Many kinds of unit testing… © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 16

Glass-Box - White Box Testing - More Path Testing – exhaustive; impossible; there is an infinite number of paths in a non-trivial program. Branch testing – Edge testing – the most feasible Design a test so that all of the edges of a node are executed. Often shown via flowgraphs. (ahead) What do we mean by ‘edge? ’ (all ‘else’ branches taken; all ‘then’ branches executed…) Consider a flowgraph of an algorithm (two slides up) …. Emphasis will be on control and coverage. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 17

White Box: Branch Testing - Flow Graphs Want minimum number of tests assuring high degree of reliability. Know we cannot test all paths in a non-trivial program. Want a ‘representative set. ’ Given a flow graph (next slides), there a couple of formulas that will provide the minimum number of tests. They are: • (number of nodes – number of edges + 1) • Number of Regions + 1 …. (cyclomatic complexity) Number of Regions is considered to be a measure of complexity. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 18

http: //www. scism. sbu. ac. uk/law/Section 5/chap 3/s 5 c 3 p 22. html Consider: The number of possible paths through a subprogram is equal to the number of regions in the subprogram's flowgraph. The flowgraph below is for a Square. Root function and has regions numbered, as shown. Can see there are four regions, as numbered, plus the outer region, or 5 in total. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 19

http: //www. scism. sbu. ac. uk/law/Section 5/chap 3/s 5 c 3 p 22. html This indicates that there are five possible paths through the flowgraph which can be described by listing the sequence in which the nodes must are visited. The five paths in the Square. Root flowgraph are as follows. etc… © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 20

White Box: Equivalence Class Testing • This is a Bigee! • It is inappropriate to test by brute force, using every possible input value —Discuss; Takes a huge amount of time —Is impractical and is pointless! • Far better: divide the possible inputs into groups which you believe will be treated similarly by all algorithms. —Such groups are called equivalence classes. —Remember Computational Structures? ? (COT 3100? ? ) © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 21

Equivalence Class Testing • Recall ramdomizing in File Structures (hashing…). • If we used an ‘Equivalence mod 4’, we essentially divided the set of all nonnegative integers by four and only take their ‘remainders’ 0, 1, 2, and 3. • If a numeric key was used as a primary or relative key, this approach was often used to arrive at a disk address or an address in a hash table. • Thus ‘Equivalence mod 4’ effectively partitions the set of all integers into one of five disjoint equivalent classes. • ALL divisions result in one of these remainders – and those having the same remainder (members of the same equivalence class) need only require one of its members to be tested to represent ALL values that fall into this equivalence class. . • We applied this equivalence relation to the set of integers and obtained a set of equivalence classes the Union of which constitutes the original set of Integers. —EC 1 U EC 2 U … ECn = set of I+ (non-negative) © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 22

Example of Equivalence Classes —A tester need only to test one member of an equivalence class rather than ALL numbers of that class. —The tester has to - understand the required input, - appreciate how the software may have been designed • Example: —Valid integer input is a month number (1 -12) —Equivalence classes are: [-∞. . 0], [1. . 12], [13. . ∞] • These are three equivalence classes. • In Equivalence Class Testing, we select an input from each of the equivalence classes as inputs to testing. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 23

More on Equivalence Class Testing • So, to test for valid integers ranging from 1 through 12, we have three equivalence classes as indicated: —Integers less than 1; —Integers between 1 and 12 inclusive —Integers greater than 12. • Is this enough? • Equivalence Class Testing cannot handle all kinds of testing: • What if there is more than a single value we are testing? © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 24

Combinations of Equivalence Classes • Combinatorial explosion means that you may not be able to realistically test every possible system-wide equivalence class. —Let’s say there are 4 inputs each of which have 5 possible values. This means that are 54 (i. e. 625) possible system-wide equivalence classes. • While you should first make sure that at least one test is run with every equivalence class of every individual input, you should also test all combinations where one input is likely to affect the interpretation of another. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 25

Compatibility Tests; Associativity Tests • Compatibility tests; Associativity tests. —Example of Associativity Test: Example from a Manpower Project in the USAF. - Field/attribute: Rated Performance Indicator (rpi); - Rule: If the rpi input value for an individual had a value of non-zero (individual had to be on an aircrew), then the ‘rank’ attribute for this individual (s/he must be an officer), must be in the range between O 1 -O 10 (and could NOT be in the range E 1 -E 9). —Example of Compatibility Test: - Range tests: If rank = ‘O 4’, salary must fall between $nnnn and $nnnnnn. Many others and many different types of tests. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 26

Testing at Boundaries of Equivalence Classes Boundary Value Testing • More errors in software occur at the boundaries of equivalence classes than at any other place!! AMEN!!! • The idea of equivalence class testing should be expanded to specifically test values at the extremes of each equivalence class —E. g. The number 0 often causes problems • E. g. : If the valid input is a month number (1 -12) —Test equivalence classes as before - Test -24; 6, +24, (representative values) —Test boundaries: 0, 1, 12 and 13 as well as very large positive and negative values —Are only integers allowed? (perhaps another test) © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 27

Coming Defects in ordinary algorithms…. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 28