Lecture 7 Product Quality Verification Validation and Testing

Lecture 7 Product Quality: Verification, Validation and Testing (Based on Stevens and Pooley, 2006, Chapter 19) David Meredith d. meredith@gold. ac. uk www. titanmusic. com/teaching/cis 224 -2006 -7. html CIS 224 Software Projects: Software Engineering and Research Methods 1

Introduction • To build high-quality systems, we need techniques that focus on – The quality of the product – The quality of the process • Product-focused techniques for ensuring software is of high quality: – Verification • Making sure we’ve “built the product right” – Product satisfies design and specified requirements • “The purpose of Verification (VER) is to ensure that selected work products meet their specified requirements. ” (Capability Maturity Model Integration for Development, version 1. 2) – Validation • Making sure we’ve “built the right product” – Design satisfies real-world requirements and fits intended usage • “The purpose of Validation (VAL) is to demonstrate that a product or product component fulfills its intended use when placed in its intended environment. ” (Capability Maturity Model Integration for Development, version 1. 2) – Testing is used for both verification and validation 2

Quality review • Software is high quality if it meets the users’ requirements – Useful and usable • Makes people’s lives easier or better – Reliable • Few bugs • Must be easy to thoroughly test software – Flexible • Easy to change software in response to requirements change • Easy to maintain and debug software – Affordable • Not too expensive to buy and maintain • Implies easy and quick to develop and maintain – Available • Available for different platforms • Development must complete 3

How can high quality software be achieved? • Need to combine two orthogonal approaches – Focus on product • Testing for verification and validation – Focus on process • Faults in product may be due to faults in process – Need to evaluate process frequently and improve it when necessary 4

Verification • Involves testing product against specification • If using UML, then, in testing phase of each iteration, we might – Verify that use cases satisfy requirements – Verify that classes can provide use cases – Verify that code correctly implements classes • Includes “sanity checks” that ensure project satisfies basic requirements, e. g. , – code compiles cleanly – program runs without errors (e. g. , segmentation faults, unhandled exceptions) – UML diagrams syntactically correct 5

Verification • Verification can be – Informal: developer manually compares code with design and ensures they correspond • Best if tester not the same person as the programmer • Devise a checklist of most common problems to look for • Reviews – Formal: formal proof of equivalence of code and design • Rarely done in practice • Only possible if two things being compared have formal, welldefined semantics – UML does not have formal semantics, therefore formal proof of equivalence of UML with code is impossible in principle – UML can be guaranteed syntactically correct if produced using a UML modelling tool that enforces correct syntax – Equivalence of UML with code presumably ensured if using 6 MDA or executable UML

Validation • Validation harder than verification because don’t know exactly what you’re looking for – Have to identify ways in which system is less useful to the customer than it should be • Implies requires customer involvement – Worst errors arise from communication failure between customer and developer on what is required • Need to have frequent customer evaluations as in iterative process – May need to build rapidly a throw-away prototype to ensure that customer and developer have same idea of what is required 7

Usability • System must not just provide required functionality – Must also allow users to exploit functionality effectively – they must be able to • Carry out their tasks quickly and easily with minimum stress • Recover from errors • Developers bad at evaluating usability of their software – Real users make mistakes that developers could never imagine – If understand how something can be done, hard to imagine not understanding it – Developers usually not typical users • Typically understand system better than users and domain less well than users • Employ expert in usability and consult users frequently • Thomas Landauer proposes “user-centred design” – One day’s focus on usability can increase work efficiency of system by 25% – Landauer, Thomas K. (1996). The Trouble with Computers: Usefulness, Usability and Productivity. MIT Press, Cambridge, MA. 8

Testing • Testing contributes more to verification than validation • Can test designs and system components as well as complete running systems • Testing has three aims: – To find bugs (most important aim) – To convince customer that there are no bugs – To provide information for system evolution, e. g. • Information on future requirements • Information on present performance 9

Testing • A successful test is one that finds a bug • Different types of testing – Usability testing: checks if system is easy to use effectively – Module (unit) testing: tests individual modules of a system • in OO, modules will usually be classes – Integration testing: tests that parts of system work together properly – System testing: checks that system meets functional and nonfunctional requirements – Acceptance testing: validates that system is fit for purpose – Performance testing: checks satisfactory performance of module, collaborating modules or system – Stress testing: puts extreme loads on system to ensure that it degrades gracefully and does not fail catastrophically – Regression testing: tests that a change to a system does not introduce new bugs. Includes • Module tests for changed modules • Integration tests for subsystems including changed modules • Some whole-system tests (usability, system, acceptance, performance, stress) 10

Choosing and carrying out tests • Tests can be – Black box • Chosen by looking at specification of thing to be tested – White box • Chosen by looking at structure of thing to be tested – Petschenik, N. H. (1985). Practical priorities in system testing. IEEE Software, 2(5), pp. 18 -23. • It is more important to – Test whole system than its components – Check that the system can still do what it could do before than to check that new features work – To test typical cases than boundary value cases 11

Choosing and carrying out tests • When functionality changes, must carry out regression tests which must be – Repeatable – Documented (both tests and results) – Precise • In iterative development, same test carried out many times – Need automated testing – There exist specialist testing tools (e. g. , junit) – But can also automate tests using scripts (e. g. , Perl, shell scripts, DOS batch files, etc. ) • Write test specifications as soon as requirements are understood • Some bugs only show up after complex and unusual sequence of events – Have someone test software with the aim of breaking it, however sneakily • e. g. , IBM’s “Black team” (De. Marco, T. and Lister, T. (1987). Peopleware: Productive Projects and Teams. Dorset House, New York. ) – Especially hard to test GUIs systematically • Many problems with GUIs are to do with usability – Usability testing cannot be automated 12

Unit testing in an OO system • Special problems arise when testing an OO system: – What counts as a unit? – Checking hidden data in an encapsulated unit – Problems arising from inheritance and polymorphism 13

What counts as a unit? • Unit of test must be at least the class – Can be a collection of collaborating classes (e. g. , a package) • Class harder to test than a function – Cannot test each method in isolation because a method might change an object’s state in such a way that reveals a bug in another method – Can use state machine diagrams to identify important states and then test object on every state transition – Avoid classes with complex state diagrams since they are hard to understand test! 14

Testing and encapsulation • Encapsulation can reduce likelihood of bugs by not allowing access to details that client does not need to know about • But checking which state an object is in during a test may require access to private attributes • One solution might be to provide a method that is only used during testing – but then rely on correctness of this testing method! 15

Inheritance and testing • Class D overrides foo() operation in class G but not bar() operation? • Do we only need to test the foo() operation in class D? • NO! Why? • Because bar() may call foo() – because of dynamic binding, class D object’s foo() method might be called when bar() message sent to it • To be safe, all subclass’s methods must be tested – even inherited 16 ones!

Testing and polymorphism • Two interacting objects: a of class C and b of a subclass of D • Could be that class of b was invented after C • Must make sure that way in which b behaves when sent messages in the public interface of class D is consistent with how an object of class D would behave in response to same messages • This must be tested when class of b is defined • Class tightly coupled with its superclasses which creates difficulties when testing – USE INHERITANCE ONLY WHEN ITS ADVANTAGES OUTWEIGH ITS DISADVANTAGES! 17

Why is testing often done badly? • Most people find it boring! – Need to automate process as much as possible – Need to minimise bureaucracy • It’s expensive – Could take 30 -50% of project time • If left until end of project, then squeezed by deadline pressure – Hence should use iterative process in which testing done in each iteration and spread over whole project • Customers often don’t realise the importance of testing and pressure developers to deliver product even if it has not been thoroughly tested 18

Reviews and inspections • Formal technical reviews (FTR) help mostly with validation but also with verification – Meeting to find problems in a deliverable (could be code or design) • FTR participants include author, moderator, scribe, possibly one or two users • Artifact studied by participants before meeting • Meeting focuses on defects found, not solutions • After meeting, author addresses defects and then meets with moderator to discuss solutions • Problems with FTRs – Can feel like an attack on the developers – Can be very time-consuming – Can produce long lists of trivial defects and miss more fundamental problems 19

Summary • • • • • • To achieve high-quality product, need product-focused and process-focused techniques Concentrated on product-focused techniques in this lecture Verification - making sure the product satisfies the stated requirements Validatation - making sure the product if fit for its intended purpose in the environment in which it will be used Testing required to verify and validate a product Reviewed criteria for a high quality software system Useful, usable, reliable, flexible, affordable, available Verification involves testing product against specification can be informal (e. g. , manual comparison) or formal (e. g. , proof of equivalence) Validation involves identifying ways in which system is less useful to customer than it should be necessitates customer involvement need to have frequent customer evaluation Usability - consult users frequently and employ expert in usability Testing more relevant to verification than validation Different types of testing: usability, unit, integration, system, acceptance, performance, stress, regression Black box testing: specification White box testing: structure Special problems with unit testing in an OO system General problems with testing 20 Formal technical reviews