What Is the Value of Testing and How

What Is the Value of Testing and How Can We Increase It? Paul Gerrard Technical Director Systeme Evolutif Limited 9 Cavendish Place London W 1 G 0 QD, UK Tel: +44 (0)20 7636 6060 email: paulg@evolutif. co. uk http: //www. evolutif. co. uk/ © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 1

Themes “[A cynic is] A man who knows the price of everything but the value of nothing” Oscar Wilde (1854 -1900), Lady Windermere’s Fan. “To win without risk is to triumph without glory” Pierre Corneille (1606 -84), The Cid. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 3

Agenda l l l l Let’s play Marketing Quantity, Quality, Process or Product – Where is the Value? The Most Valuable Product of Testing is Project Intelligence Influences on the Value of Testing Increasing the Value of Testing Process “Improvements” and the Value of Testing Summary. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 4

Let’s Play Marketing l We need to consider: – – Who are the customers of testing? What exactly does the customer value? How can we increase the value of our testing? How can we increase their appreciation of our testing? – How do we select an “improvement” to make? – How do we judge the value of an “improvement”? © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 5

Quantity, Quality, Process or Product – Where is the Value? © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 6

Mine clearing l According to captured records, there are 100 mines in that minefield We have identified and removed seven mines I crossed it once but don’t remember how l Is it safe? l l © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 7

Test activity and information l I ran some tests – But I can’t remember what happened or how I could repeat those tests – Was this testing valuable? l I ran a hundred tests and the system failed 23 times – I can’t remember how I did it or where the system failed – Was this testing valuable? l The ACTIVITY of testing only has VALUE if it generates INFORMATION (this is the deliverable). © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 8

How much detail? l Which is best? – – – l “We ran lots of tests” “We found lots of faults” “ 10% of our tests caused the system to fail” “This test case caused a failure” “This set of test steps caused the system to fail at precisely this point, in precisely this way…” Test records should be detailed enough to plan, design, implement, execute tests and interpret test results and reproduce failure – AND NO MORE. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 9

Quantity or quality? l Which would you pay more for? – – l 100 page test plan or 200 page test plan? 100 incident reports or 200 incident reports? 100 requirements faults or 100 coding faults? 100 random tests or 100 ‘designed’ tests? Quantity is one thing, but the nature and quality of the information and the way it was prepared makes a huge difference to its value. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 10

Testing generates “project intelligence” l l l If you don’t know where you are, a map won’t help If you don’t know the status of the deliverables, the best plan won’t help you manage your project A project is like driving off-road in the dark – Testing provides the headlights for the journey Testing provides data on the status of deliverables and generates “PROJECT INTELLIGENCE” l Stakeholders and management need intelligence to makeway you think about testingcommand 11 © 2003 Systeme Evolutif Ltd Slide we’ll change the decisions, l

Does testing ‘change’ anything? Testing Product Testing doesn’t change the product in any way. We do it to get the ‘valuable stuff’. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Product (unchanged) Valuable Stuff Slide 12

What is the valuable stuff? l l l Test deliverables (infrastructure, test plans, specifications, scripts, data, results etc…) TEST RECORDS – test documentation (but do only testers read it? ) PROJECT INTELLIGENCE – rarely documented as such though. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 13

Military intelligence? © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 14

The Most Valuable Product of Testing is Project Intelligence © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 15

Documentation l PRODUCTS – – – – l Test Strategy Test plans, designs, specifications Procedures/scripts, expected results Actual results Incident reports Log End of Phase Report Etc. TANGIBLE, BUT LESS VALUABLE. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 16

Test infrastructure l PRODUCTS – – – – l Test process Configured test hardware Test databases Tools Manual and automated tests Skilled resources Etc… TANGIBLE, USUALLY AN OVERHEAD – (unless these are required deliverables). © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 17

TEST RECORDS l DATA – – – How the product SHOULD work (test specs) How the product ACTUALLY works (test results) Where the product is STABLE (incident trends) Where the product is UNSTABLE (incident trends) What they have COVERED in the product (test log) What they have NOT COVERED (test plan – test log) – How (and where) the product FAILS (incidents) l TANGIBLE AND VALUABLE, BUT… – (valuable to testers, developers, NOT management). © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 18

PROJECT INTELLIGENCE l KNOWLEDGE OF PROJECT RISKS/BENEFITS – RISKS » Requirements are poor and/or unstable » Less of the product is ready, product is not as good as planned » They should have been less ambitious, done it differently » The project will slip – BENEFITS » Some features/benefits are available » Other features/benefits are blocked by risk uncertainty l INTANGIBLE AND VALUABLE – Crucial to management and stakeholders © 2003 Systeme Evolutif Ltd Slide 19 we’ll change the way you think about testing – Intelligence often arrives TOO LATE TO BE OF

Where the value of testing lies l LOW VALUE – Documentation – Test infrastructure ** UNLESS these are specific objectives or deliverables of project l HIGH VALUE – Test Records – Project Intelligence We can increase the value of testing MOST by: • Improved INTELLIGENCE • Providing INTELLIGENCE EARLIER. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 20

Influences on the Value of Testing © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 21

Intangible value, tangible cost l l l Testers think testing is good and has undeniable value, but we are not independent! Not everyone thinks that (software suppliers, in -house developers? ) If you are not a tester, the value seems intangible – But the cost IS tangible – Exercise, flossing, house maintenance, life insurance, testing can all be put off till another day – If you don’t do it now you store up problems for later A bit like a lifestyle choice – it’s human nature to put things we’ll change the way you think about testing off. © 2003 Systeme Evolutif Ltd Slide 22 l

Key factors in the value of testing l Adding value depends on how well we achieve and communicate: 1. 2. 3. 4. l l Coverage Incidents Risk-based testing Benefits-based testing Time to deliver that value depends on the test process (more later) Willingness of the organisation to accept and act upon project intelligence is key (more later). © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 23

Coverage l Breadth of coverage – Are all aspects of the product tested? l Depth of coverage – Are critical aspects of the product tested more thoroughly? l l l Can the measure be applied systematically? Is the measure relevant to the faults to be detected or the risks of concern? Can we communicate coverage in a meaningful, understandable way? © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 24

Incidents l l Are the failures of most concern being found and logged as incidents? Can these failures be reproduced easily? Are the incidents being addressed in the right order? Are the incidents of most concern being resolved quickly? © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 25

Risk-based testing l l l Are product risks identified early in the project? Are stakeholders involved in the decision to rank these risks? Is the decision to include risks in scope for testing done by consensus? Is the decision to de-scope risks done by consensus? Is test design driven by the need to address risk? Is test progress reporting based on risks addressed? © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 26

Benefit-based testing l l Does the project have cardinal objectives (benefits) defined? Are all risks related to the benefits? Are tests related to the risks that block the benefits? Is test progress reporting based on the benefits that are threatened or available? © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 27

Increasing the Value of Testing © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 28

Contribution of the test process l l l Does testing generating valuable intelligence? Does testing generate intelligence early enough? Is testing flexible enough to react to new risks? Are the test communication channels clear? Improvement types: – Effectiveness – Efficiency – Perceived value of testing. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 29

Effectiveness improvements l l l Improvements to “intellectual” test activities enhance the quality of intelligence Implementing these activities earlier make intelligence available earlier Examples: – Formal test design improves coverage, and improves the quality of test records – Improvements that promote earlier test activities tend to make the value available earlier (e. g. more reviews). © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 30

Efficiency improvements l l l These reduce the cost of doing testing Improvements to the mechanical or clerical aspects of our test process make us more efficient (but don't affect the value of testing) For example – Using a tool to automate tests might make it cheaper to do a task, but does it add value? – It might save time, allowing us to do other useful activities l Ultimately, effectiveness improvements are more valuable. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 31

Organisational culture l l Willingness to accept and act on the intelligence that testers generate is key Primarily a cultural issue If the intelligence isn’t valued by the culture - it doesn't matter what value it has We need to EDUCATE our customers. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 32

Increasing the perceived value of testing No such thing as “absolute value” of testing l We must work on the PERCEIVED VALUE to the customers of our testing: 1. Improve the value in line with their expectations l – Provide richer, deeper, focused, relevant information – “Create more complex, subtle, interesting wine” 2. Increase their appreciation of what we do – Educate them to understand testing products – “Train their palate to detect the complex flavours” Improvements must you think about testing we’ll change the way address both issues. Slide 33 l © 2003 Systeme Evolutif Ltd

Process “Improvements” and the Value of Testing © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 34

Test process or test product assessment? l l Assessments of the “Test Process” may miss the point “Process healthchecks” are useless because they don’t focus on the VALUE of test deliverables Assessment of the perceived value of Test Products must be a key driver We must focus on process changes that: – add value to the test products – enhance the perceived value of test products – make test products available earlier. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 35

The challenges of testing improvement l Barriers are organisational, personal, not technical: – – – l l changing management perceptions so they support testing and improvement overcoming management and practitioner resistance to change design and implementation of workable processes and management controls People, not tools, implement test strategies Change is easier if people see VALUE being added. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 36

Problems or symptoms? l Need to separate problems from symptoms: – – – l l “Management doesn't understand the value of testing” “The cost of testing is high but difficult to pin down” “Developers, testers, users never trained” “The quality of the product delivered into testing is poor, so takes longer to system test” “The perceived value of testing is low” Need to understand symptoms to identify the source of problem and fix the problem Improvements often tackle the symptoms only. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 37

Right improvement mix is key l A mix of improvements is most likely to be required: – – – management awareness tester training developer training improved definition of the test stages and their objectives measurement of the quality of the product at each stage etc. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 38

Constraints and expectations l Not all improvements a good idea straight away – – some improvements are expensive some save time, but bring dramatic change some improve the quality of the testing, but take longer to implement some tidy up ‘loose ends’ but have negligible value Very few improvements save time, improve quality, cause minimal change and pay back after two weeks l Need to set/agree clear management expectations. © 2003 Systeme Evolutif Ltd Slide 39 l we’ll change the way you think about testing

Do tools increase the value of testing? l l With tool vendors present, I’d better be careful! Need to distinguish between – Quantity, quality and the information provided l l l A tool might allow you do to testing faster, but it probably won’t increase the value of tests It might make a task interesting, so a valuable task gets done It might give you more time to do other valuable tasks. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 40

To Summarise… © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 41

How to increase the value of testing l l The value of testing lies in the project intelligence it generates Produce documentation with purpose: – To allow testers to do their job (THE MEANS) – To communicate intelligence (THE END) l l Process ‘improvements’ selected to deliver better intelligence, earlier, faster and cheaper You need to train your customers to appreciate the project intelligence that testers produce. © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 42

Themes revisited “A poor tester knows the challenge and cost of testing but not its value” – Don’t be cynical – testing isn’t just a costly activity to be gotten through, testers should aim to deliver value “To win without risk is to triumph without glory” – Risks are inevitable - the testers’ role isn’t to avoid them, but inform management and reduce their uncertainty – Satisfaction (and recognition) comes from overcoming risks, not doing the mundane tasks of development – Make your management aware of software product risks and report progress towards their goal, not just © 2003 Systeme Evolutif Ltd Slide 43 we’ll change the way you think about testing incidents.

What Is the Value of Testing and How Can We Increase It? Paul Gerrard www. evolutif. co. uk www. riskbasedtesting. com © 2003 Systeme Evolutif Ltd we’ll change the way you think about testing Slide 44