Software testing techniques Introduction Prof Rimantas Šeinauskas Rimantas

Software testing techniques Introduction Prof. Rimantas Šeinauskas Rimantas. Seinauskas@ktu. lt Kaunas University of Technology

Objectives 1. To introduce software testing and to explain its importance 2. To introduce structure of teaching content 3. To explain software testing definition Introduction to Software testing techniques 2

SWEBOK project • • • Software Configuration Management Software Construction Software Design Software Engineering Infrastructure Software Engineering Management Software Engineering Process Software Evolution and Maintenance Software Quality Analysis Software Requirements Analysis Software Testing !!!!!! Introduction to Software testing techniques 3

Curricula • Module place on the curricula

Content 1. Introduction 2. Verification & Validation ( Repeated from software engineering according I Sommerville textbook) 3. Software testing ( Repeated from software engineering according I. Sommerville textbook) 4. Written quiz Topics of seminars: 5. Unit testing 6. Integration testing 7. Interface testing Introduction to Software testing techniques 5

Content 8. Drivers and stubs 9. Sandwich strategy 10. Testing objectives 11. Regression testing 12. Equivalence partitioning 13. Testing based on specifications 14. Testing based on error guessing 15. Input Output analyses Introduction to Software testing techniques 6

Content 16. Testing criteria based on control flow 17. Testing criteria based on data flow 18. Cycle testing 19. Mutation testing 20. Object-oriented software testing 21. Graphical user interface testing 22. Testing maturity model 23. Testing tools Introduction to Software testing techniques 7

Content 24. Automated software testing 25. Software testability 26. Statistical testing 27. State testing of automata Introduction to Software testing techniques 8

Labs Introduction to Software testing techniques 9

The “software crisis” • By the 1980’s, “quality” in software became a goal; SEI was born • “software engineering” became popular • the life cycle was studied • software developers and testers began to work together • by the 1990’s, testing tools became available Introduction to Software testing techniques 10

First: a riddle about testing • A mathematician, a physicist, and an engineer are told: “All odd numbers are prime. ” – The mathematician says, “That’s silly, nine is a nonprime odd number. – The physicist says, “Let’s see, 3 is prime, 5 is prime, 7 is prime -- looks like it’s true. ” – The engineer says, “let’s see, 3 is prime, 5 is prime, 7 is prime, 9 is prime, 11 is prime -- looks like it’s true. ” Introduction to Software testing techniques 11

Testing: our experience Test Case Software to be tested Output Introduction to Software testing techniques 12

When to stop? Test Case Generation Test Case Software to be tested Verification Output No Test Coverage Enough? Yes Introduction to Software testing techniques 13

Testing example Test Cases {1, 3, 2} {1, 2, 3} {3, 2, 3} {} {-1, -2} Just a list. SPECS: A sorted list. Repeated entry. Takes a list of numbers; Empty list. returns a Sorting numbers. Negative sorted list. Program {2, 2, 3, {1, {} 3} {-2, -1} Output Introduction to Software testing techniques 14

Manual testing Test Case Generation Test Case Software to be tested Verification Output No Enough? Yes Introduction to Software testing techniques Test Coverage 15

Automated testing Test Case Gener ator Test Specs Test Case Software to be tested Output Coverage Evaluator Introduction to Software testing techniques Verifie r OR Test Oracle 16

Software testing? • “The process of executing computer software in order to determine whether the results it produces are correct”, Glass • “The process of executing a program with the intent of finding errors”, Myers • “Program testing can be used to show the presence of bugs, but never their absence”, Dijkstra Introduction to Software testing techniques 17

Software testing? • “The aim is not to discover errors but to provide convincing evidence that there are none, or to show that particular classes of faults are not present”, Hennell • “Testing is the measure of software quality”, Hetzel • The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component. ” IEEE/ANSI, Introduction to Software testing techniques 18

The point of view • “If our goal is to show the absence of errors, we will find very few of them” • “If our goal is to show the presence of errors, we will discover a large number of them” Introduction to Software testing techniques 19

Testing efforts • 50% Brooks/Myers, • 80% Arthur Andersons’ Director of testing in North America Introduction to Software testing techniques 20

Testers/programmers • 1: 5 -10 Mainframes i. e. , 1 tester for every 5 to 10 developers • 2: 3 Microsoft • 2: 1 Lotus (for 1 -2 -3 Windows) • 1: 2 Average of 4 major companies, Microsoft, Borland, Word. Perfect, Novell Introduction to Software testing techniques 21

Software testing problems • • poorly expressed requirements informal design techniques nothing executable until coding stage Huge input set: consider testing software that categorises an exam grade: 101 inputs Introduction to Software testing techniques 22

Exhaustive software testing • Exhaustive software testing is intractable • Even if all possible inputs could be identified, the problem of identifying expected results is undecidable • Weyuker has shown that there is no algorithm that can determine if a given statement, branch or path will be exercised! Introduction to Software testing techniques 23

Software testing definition • Software testing consists of the dynamic verification of the behavior of a program on a finite set of test cases, suitably selected from the usually infinite executions domain, against the specified expected behavior. Introduction to Software testing techniques 24

Dynamic • This term means testing always implies executing the program on valued inputs. Static analysis techniques, such as peer review and inspection are not considered; nor is program execution on symbolic inputs, or symbolic evaluation; Introduction to Software testing techniques 25

Finite • Clearly the tests are not selected literally from an infinite set of possible tests, but a very large finite one (the set of all bit strings shorter than some length). Nonetheless, for even simple programs, so many test cases are theoretically possible that exhaustive testing could require even years to execute. This is why in practice the number of tests can generally be considered infinite. However, the number of executions which can realistically be observed in testing must obviously be finite, and -more than this- it must be manageable. Indeed, testing always implies a trade-off between limited resources and schedules, and inherently unlimited test requirements: this conflict points to well known problems of testing, both technical in nature (criteria for deciding test adequacy) and managerial in nature (estimating the effort to put in testing); Introduction to Software testing techniques 26

Selected • The many proposed test techniques essentially differ in how they select the (finite) test set, and testers must be aware that different selection criteria may yield largely different effectiveness. The problem of identifying the most suitable selection criterion under given conditions is still under research. Introduction to Software testing techniques 27

Expected • It must be possible to decide whether the observed outcomes of program execution are acceptable or not, otherwise the testing effort would be useless. The observed behavior may be checked against user's expectations (commonly referred to as testing for validation) or against a functional specification (testing for verification). The test pass/fail decision is referred to as the oracle problem, which can be addressed with different approaches. Introduction to Software testing techniques 28

Static techniques • Software testing is a means for quality evaluation. In the SQA chapter of the SWEBOK, activities and techniques for quality analysis are categorized into: static techniques (no code execution), and dynamic techniques (code execution). Both categories are useful. We would like to emphasize that static techniques are equally important for the purpose of building quality in a software product. Static techniques are covered into the SQA KA description Introduction to Software testing techniques 29

Testing activity • In the years, the view of software testing has evolved towards a more constructive attitude. Testing is no longer seen as an activity that starts only after the coding phase is complete, with the limited purpose of finding mistakes. Software testing is nowadays seen as an activity that encompasses the whole development process, and is an important part itself of the actual construction. Indeed, planning for testing starts since the early stages of requirement analysis, and test plans and procedures are systematically and continuously refined as the development proceeds. These activities of planning and designing tests constitute themselves a useful input to designers for highlighting potential weaknesses. Introduction to Software testing techniques 30

Error prevention • As more extensively discussed in the SQA chapter, the right attitude towards quality is one of prevention: it is better to avoid problems, rather than repairing them. Testing must be seen as a means primarily for demonstrating that the prevention has been effective, but also for identifying anomalies in those cases in which, for some reason, it has been not. Finally, it is worth recognizing that even after a good testing, the system could still contain mistakes. The remedy to system failures after delivery is provided by (corrective) maintenance actions. The maintenance issues of software engineering are covered into the Software Evolution and Maintenance chapter of the SWEBOK Introduction to Software testing techniques 31

General Content • • • A. Testing Basic Concepts and Definitions B. Test Levels C. Test Techniques D. Test-related Measures E. Managing the Test Process F. Test Tools. Introduction to Software testing techniques 32

A. Testing Basic Concepts and Definitions • When the outcome of a test execution diverges from the expected, a failure is observed, whose cause is a fault. These and other basic definitions within software testing area, as well as an introduction to the terminology, will be provided in next two chapters Introduction to Software testing techniques 33

B. Testing levels • Software testing is usually performed at different levels along the development process. That is to say, the object of the test can vary: a whole program, part of it (functionally or structurally related), a single module. The latter subject is expanded in Part B Test Levels. This consists of two (orthogonal) subsections: B. 1 lists the phases in which the testing of large software systems is traditionally subdivided. In B. 2 testing for specific conditions or properties is instead considered, and is referred to as "Objectives of testing". Clearly not all listed types of testing apply to every system, nor has every possible type been listed, but those most generally applied. Introduction to Software testing techniques 34

Test objective • Testing is conducted in view of a specific purpose (test objective), which is stated more or less explicitly, and with varying degrees of precision. Stating the objective in precise, quantitative terms allows for establishing control over the test process. Introduction to Software testing techniques 35

Testing aims • One of testing aims is to expose failures (as many as possible), and many popular test techniques have been developed for this objective. These techniques variously attempt to "break" the program, by running identified classes of (deemed equivalent) executions: the leading principle underlying such techniques is being as much systematic as possible in identifying a representative set of program behaviors (generally in the form of subclasses of the input domain). However, a comprehensive view of the KA of testing as a means for quality must include other as important objectives for testing, e. g. , reliability measurement, usability evaluation, contractor’s acceptance, for which different approaches would be taken. Note that the test objective varies with the test object, i. e. , in general different purposes are addressed at the different levels of testing. Introduction to Software testing techniques 36

Testing criteria • The test objective determines how the test set is identified, both with regard to its consistency -how much testing is enough for achieving the stated objective? - and its composition -which test cases should be selected for achieving the stated objective? -, although usually the "for achieving the stated objective" part is left implicit and only the first part of the two italicized questions above is posed. Criteria for addressing the first question are referred to as test adequacy criteria, while for the second as test selection criteria Introduction to Software testing techniques 37

C. Test Techniques • Several Test Techniques have been developed in the last two decades according to various criteria, and new ones are still proposed. "Generally accepted" techniques will be described from chapter three. Introduction to Software testing techniques 38

D. Testing measure • Sometimes, it can happen that confusion is made between test objectives and techniques. For instance, branch coverage is a popular test technique. Achieving a specified branch coverage measure should not be considered per se as the objective of testing: it is a means to improve the chances of finding failures (by systematically exercising every program branch out of a decision point). To avoid such misunderstandings, a clear distinction should be made between test measures which evaluate the test set, like measures of coverage, and those which instead provide an evaluation of the program under test, based on the observed test outputs, like reliability Introduction to Software testing techniques 39

E. Managing the Test Process • Testing concepts, strategies, techniques and metrics need to be integrated into a defined and controlled process, which is run by people. The test process should support testing activities and testing teams, from test planning to test outputs evaluation, in such a way as to provide justified assurance that the test objectives are met costeffectively. Issues relative to Managing the Test Process. Introduction to Software testing techniques 40

F. Test Tools • Software testing is a very expensive and laborintensive part of development. For this reason, tools are instrumental to support test activities. Moreover, in order to enhance cost-effectiveness ratio, a key issue has always been pushing test automation as much as possible. Existing tools and concepts related to automating the constituent parts of test process are addressed in different chapters Introduction to Software testing techniques 41

RECOMMENDED REFERENCES FOR SOFTWARE TESTING • K. Beck, Test-Driven Development by Example, Addison-Wesley, 2002. • P. C. Jorgensen, Software Testing: A Craftsman's Approach, second edition, CRC Press, 2004 • C. Kaner, J. Bach, and B. Pettichord, Lessons Learned in Software Testing, Wiley Computer Publishing, 2001. • S. L. Pfleeger, Software Engineering: Theory and Practice, second ed. , Prentice Hall, 2001