Just-In-Time Testing Workshop Robert Sabourin Ami. Bug. Com, Inc. Montreal, Canada rsabourin@amibug. com 19 March 2018 © Robert Sabourin, 2012 Slide 1 Ami. Bug. Com, Inc.
Overview Welcome Some Philosophy Be Pre[pared Testing Ideas Test Triage 19 March 2018 © Robert Sabourin, 2012 Slide 2 Ami. Bug. Com, Inc.
Just In Time Testing • Robert Sabourin , Software Evangelist • President • Ami. Bug. Com Inc. • Montreal, Quebec, Canada • rsabourin@amibug. com • www. amibugshare. com 19 March 2018 © Robert Sabourin, 2012 Slide 3 Ami. Bug. Com, Inc.
Just-In-Time Testing Welcome 19 March 2018 © Robert Sabourin, 2012 Slide 4 Ami. Bug. Com, Inc.
Just-In-Time Testing • Pain points? – What hurts? – How Much? 19 March 2018 © Robert Sabourin, 2012 Slide 5 Ami. Bug. Com, Inc.
Just-In-Time Testing Just In Time For ____? 19 March 2018 © Robert Sabourin, 2012 Slide 6 Ami. Bug. Com, Inc.
Just-In-Time Testing Some Philosophy 19 March 2018 © Robert Sabourin, 2012 Slide 7 Ami. Bug. Com, Inc.
Fundamental Question • How do you know when you are finished? 19 March 2018 © Robert Sabourin, 2012 Slide 8 Ami. Bug. Com, Inc.
Crosby on Quality • “Quality is defined as conformance to requirements” • “Quality is not a measure of GOODNESS” – Phil B. Crosby, Quality is Free 19 March 2018 © Robert Sabourin, 2012 Slide 9 Ami. Bug. Com, Inc.
Joseph Juran “Quality is fitness for use” Quality Control Handbook 19 March 2018 © Robert Sabourin, 2012 Slide 10 Ami. Bug. Com, Inc.
Gerald M. Weinberg “Quality is value to some person” Exploring Requirements Quality Before Design Dorset House 19 March 2018 © Robert Sabourin, 2012 Slide 11 Ami. Bug. Com, Inc.
Edsger W. Dijkstra • “Program testing can be used to show the presence of bugs, but never to show their absence” 19 March 2018 © Robert Sabourin, 2012 Slide 12 Ami. Bug. Com, Inc.
Quiz • Application screens are selected with three controls: (a) has 5 options (b) has 6 options (c) has 2 options • How many screens can a user choose? 19 March 2018 © Robert Sabourin, 2012 Slide 13 Ami. Bug. Com, Inc.
Quiz • Total Combinations = 6 x 5 x 2 = 60 • To exercise each combination once a total of 60 tests would be required. 19 March 2018 © Robert Sabourin, 2012 Slide 14 Ami. Bug. Com, Inc.
Quiz • How many tests would be required to exercise all possible screens in every possible order? . Slide 15 19 March 2018 © Robert Sabourin, 2012 Ami. Bug. Com, Inc.
Pop Quiz • To exercise all screens in every possible order would require 60! Test cases 60! = 60 x 59 x 58 x. . . 3 x 2 x 1 60! ≈ 8. 32 x 10**81 19 March 2018 © Robert Sabourin, 2012 Slide 16 Ami. Bug. Com, Inc.
Pop Quiz How many atoms are in the observable universe? From 7. 0 × 10**79 To 1. 5 × 10**82 19 March 2018 © Robert Sabourin, 2012 Slide 17 Ami. Bug. Com, Inc.
Control Flow Testing • Model flow – Create control flow diagram – Find basis paths • Minimal set of transactions • Exercise at least once – Each step – Each decision 19 March 2018 © Robert Sabourin, 2012 Slide 18 Ami. Bug. Com, Inc.
Control Flow Testing A B – Minimal basis paths • N – number of nodes • E – number of edges • P – number of basis paths • P=E–N+2 • Mc. Cabe Cyclomatic Complexity C D N=9 E=10 P=10 -9+2 P=3 E F G H I 19 March 2018 © Robert Sabourin, 2012 Slide 19 Ami. Bug. Com, Inc.
Control Flow Testing A B – Finding basis paths: 1. Start with a typical baseline 2. Flip first decision keep rest as similar as possible 3. Continue flipping decisions on baseline 4. After all decisions on baseline have been flipped continue on next path 5. Stop when all paths have been exhausted C D N=9 E=10 P=10 -9+2 P=3 E F G H I 19 March 2018 © Robert Sabourin, 2012 Slide 20 Ami. Bug. Com, Inc.
Control Flow Testing 19 March 2018 © Robert Sabourin, 2012 Slide 21 Ami. Bug. Com, Inc.
19 March 2018 © Robert Sabourin, 2012 Slide 22 Ami. Bug. Com, Inc.
ST Di AD AE AG AC X Y Z AA AB AM AF AH AJ AL AI AN AK AO AP AQ L M A J O B C N H P Q R D E S T U V 19 March 2018 I K © Robert Sabourin, 2012 G F W Slide 23 Ami. Bug. Com, Inc.
19 March 2018 © Robert Sabourin, 2012 Slide 24 Ami. Bug. Com, Inc.
Purpose of Testing • Common definition: – To find bugs before our customers do! • Broader definition: – The role of testing is to provide objective input to facilitate business decisions! – Keeps stakeholders aware of all issues or concerns that relate to shipping a product! 19 March 2018 © Robert Sabourin, 2012 Slide 25 Ami. Bug. Com, Inc.
Bug Defined • To make our job more fun, whenever we have a concern with software, we call it a “bug”. 19 March 2018 © Robert Sabourin, 2012 Slide 26 Ami. Bug. Com, Inc.
Just-In-Time Testing • It’s all about people! (and the occasional bug too) 19 March 2018 © Robert Sabourin, 2012 Slide 27 Ami. Bug. Com, Inc.
Just-In-Time Testing Context Drivers 19 March 2018 © Robert Sabourin, 2012 Slide 28 Ami. Bug. Com, Inc.
Context Drivers - BTO • Business – Value – To whom? – Why? • Technology – Solutions • Organization – Corporate Structure – Team Structure – Roles and Responsibilities 19 March 2018 © Robert Sabourin, 2012 Slide 29 Ami. Bug. Com, Inc.
19 March 2018 © Robert Sabourin, 2012 Slide 30 Ami. Bug. Com, Inc.
Context Listeners • • Find Sources Monitor Drivers Anticipate Change React 19 March 2018 © Robert Sabourin, 2012 Slide 31 Ami. Bug. Com, Inc.
Just In Time Testing Get Ready, Get Set, Cause here it comes 19 March 2018 © Robert Sabourin, 2012 Slide 32 Ami. Bug. Com, Inc.
Just-In-Time Testing Turbulence 19 March 2018 © Robert Sabourin, 2012 Slide 33 Ami. Bug. Com, Inc.
Just-In-Time Testing Unprepared 19 March 2018 © Robert Sabourin, 2012 Slide 34 Ami. Bug. Com, Inc.
Just-In-Time Testing Sharpen Testing Skills • Thinker • Detective • Reporter • Diplomat • Negotiator • Cheer Leader • Pragmatist 19 March 2018 © Robert Sabourin, 2012 Slide 35 Ami. Bug. Com, Inc.
Philosophy We have precious little time to run tests! We must always be prepared! 19 March 2018 © Robert Sabourin, 2012 Slide 36 Ami. Bug. Com, Inc.
Time 19 March 2018 © Robert Sabourin, 2012 Slide 37 Ami. Bug. Com, Inc.
Just In Time Testing Test Triage 19 March 2018 © Robert Sabourin, 2012 Slide 38 Ami. Bug. Com, Inc.
Plan to support change Yoda "No! Try not, Do. Or do not. There is no try. " 19 March 2018 © Robert Sabourin, 2012 Slide 39 Ami. Bug. Com, Inc.
Testing Ideas Plan to support change • Collect all testing ideas you can find! – – List Sort Organize Shuffle 19 March 2018 © Robert Sabourin, 2012 Slide 40 Ami. Bug. Com, Inc.
Testing Ideas Plan to support change • How to find them? – – – – Does system do what it is suppose to do? Does the system do things it is not supposed to? How can the system break? How does the system react to it’s environment? What characteristics must the system have? Why have similar systems failed? How have previous projects failed? 19 March 2018 © Robert Sabourin, 2012 Slide 41 Ami. Bug. Com, Inc.
Testing Ideas Capture testing ideas • Collect testing ideas • From testing ideas build a series of testing objectives – Each can be assigned as work to testers – Each can include all, part of, or multiple testing ideas 19 March 2018 © Robert Sabourin, 2012 Slide 42 Ami. Bug. Com, Inc.
Testing Ideas Capture testing ideas • I often use Index Cards – Unique id – One testing idea per card – Colour indicates source – Shuffled and reviewed – Organized and reorganized – Sorted, grouped, prioritized and collected 19 March 2018 © Robert Sabourin, 2012 Slide 43 Ami. Bug. Com, Inc.
Capture testing ideas Test Idea Sources • • • Capabilities Failure Modes Quality Factors Usage Scenarios Creative Ideas States Data Environments White Box Taxonomies 19 March 2018 © Robert Sabourin, 2012 Slide 44 Ami. Bug. Com, Inc.
Testing Ideas Capture testing ideas • Investigative approaches – We become truffle snorting pigs and try to find useful information in all evidence we discover – We can even get good ideas from out of date sources 19 March 2018 © Robert Sabourin, 2012 Slide 45 Ami. Bug. Com, Inc.
Testing Ideas Capture testing ideas • Capabilities – Use cases – Functional requirements – Documented requirements – Implicit requirements 19 March 2018 © Robert Sabourin, 2012 Slide 46 Ami. Bug. Com, Inc.
Testing Ideas Capture testing ideas • Failure Modes – What can break? – Reaction to invalid input? – How does software behave in constrained environment? • • • Memory Disk Space Network Bandwidth CPU capacity Shared resources – Stress, Load, Volume 19 March 2018 © Robert Sabourin, 2012 Slide 47 Ami. Bug. Com, Inc.
Capture testing ideas Quality Factors Importance Different Application Types 19 March 2018 © Robert Sabourin, 2012 Slide 48 Ami. Bug. Com, Inc.
Testing Ideas Capture testing ideas • Usage Scenarios – Identify classes of users – Identify how users will use system – Describe scenarios – Use Story board or similar approaches – Identify variations 19 March 2018 © Robert Sabourin, 2012 Slide 49 Ami. Bug. Com, Inc.
Testing Ideas Capture testing ideas • Creative approaches – – Action verbs Mind Maps Soap Operas Lateral Thinking 19 March 2018 © Robert Sabourin, 2012 Slide 50 Ami. Bug. Com, Inc.
Testing Ideas power up Capture testing ideas service needed no cups OR no coffee OR sensor jam reset button coin inserted idle coin return cup removed coin return make coffee button pushed inserting coins right amount entered user choose State Models 19 March 2018 © Robert Sabourin, 2012 Slide 51 Ami. Bug. Com, Inc.
Testing Ideas Capture testing ideas • Data – – – Flow Structure Create Update Change 19 March 2018 © Robert Sabourin, 2012 Slide 52 Ami. Bug. Com, Inc.
Testing Ideas Capture testing ideas • Environment – – – – Hardware Software Operating systems Locales Browsers Plug-ins Co-dependent software 19 March 2018 © Robert Sabourin, 2012 Slide 53 Ami. Bug. Com, Inc.
Testing Ideas Capture testing ideas • White Box – Design – Internal structure – Code 19 March 2018 © Robert Sabourin, 2012 Slide 54 Ami. Bug. Com, Inc.
Testing Ideas Capture testing ideas • Bug taxonomies – Collections of possible bugs – Appendix A of Testing Computer Software, Kaner, Falk, Nguyen – Boris Biezer Taxonomy Otto Vinter manages – Shopping cart taxonomy Giri Vijayaraghavan 19 March 2018 © Robert Sabourin, 2012 Slide 55 Ami. Bug. Com, Inc.
19 March 2018 © Robert Sabourin, 2012 Slide 56 Ami. Bug. Com, Inc.
Which test? • Impact estimation Triage testing ideas –For each test idea guesstimate: • • benefit of implementation consequence of implementation benefit for not implementing consequence of not implementing –How credible is the information? 19 March 2018 © Robert Sabourin, 2012 Slide 57 Ami. Bug. Com, Inc.
Which test? Test Idea Rejection – What If? Triage testing ideas –If the cost/benefit does not make business sense then consider implementing: • part of the test, could that lead to part of the benefit at a more reasonable cost? • more than the stated test, would that generate more benefit? • a different test than the stated idea, could that generate more benefit for less cost? 19 March 2018 © Robert Sabourin, 2012 Slide 58 Ami. Bug. Com, Inc.
Test Triage • Test Triage Meeting – Review Context Triage testing ideas • Business • Technical • Organizational – New Information • Test results • Bug results • New testing ideas 19 March 2018 © Robert Sabourin, 2012 Slide 59 Ami. Bug. Com, Inc.
Test Triage testing ideas • Allocate Testing Assignments to Testers – – – Make sure testers know context Best thing to test Best person to test it Best people to explore it Best lead Are subject matter experts required 19 March 2018 © Robert Sabourin, 2012 Slide 60 Ami. Bug. Com, Inc.
Test Triage Life of a test idea Triage testing ideas a. b. c. Comes into existence Clarified Prioritized a. b. c. d. e. f. d. Test Now (before further testing) Test before shipping Nice to have May be of interest in some future release Not of interest in current form Will never be of interest Integrate into a testing objective 19 March 2018 © Robert Sabourin, 2012 Slide 61 Ami. Bug. Com, Inc.
Which test is next? Triage testing ideas • Questions – Given state of project, state of business, state of technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next? – How much effort are we willing to spend continuing to test this project? – Can we ship yet? 19 March 2018 © Robert Sabourin, 2012 Slide 62 Ami. Bug. Com, Inc.
Deciding what not to test? Triage testing ideas • Time pressure – Should we skip a test? – If test failed could system still be of value to some stakeholder? – If test was skipped could important bugs have been otherwise found? 19 March 2018 © Robert Sabourin, 2012 Slide 63 Ami. Bug. Com, Inc.
Guidelines and Decisions Get Started Right • To each stakeholder – – – risk of failure consequence of failure value of success how much certainty do we have is it a wild guess or an absolute truth? 19 March 2018 © Robert Sabourin, 2012 Slide 64 Ami. Bug. Com, Inc.
Bottom Line Get Started Right • My experience is that it is better to omit a test on purpose than to skip it because you ran out of time or forgot about it! • Systematically collecting, evaluating and triaging testing ideas helps me decide what not to test - at least for now? 19 March 2018 © Robert Sabourin, 2012 Slide 65 Ami. Bug. Com, Inc.
Thank You • Questions? 19 March 2018 © Robert Sabourin, 2012 Slide 66 Ami. Bug. Com, Inc.
Скачать презентацию Just-In-Time Testing Workshop Robert Sabourin Ami Bug Com
82f27d984282715feb8d11099be06923.ppt