IBM Software Group The Testing Profession Geoffrey

IBM Software Group Obligatory Dilbert 2

IBM Software Group Companies want YOU! 3

IBM Software Group Agenda § The Value of Testers to the Organization 4

IBM Software Group The bad old days…. Part One Fundamental Project Determinants Feature Set Time to Delivery locked early spent early Cost Quality Feature Set CHOOSE ONE Quality Time to Market 5

IBM Software Group The bad old days…. Part Two Software engineering is the only engineering discipline for which defects are accepted 6

IBM Software Group The bad old days…. Part Three Should be ready Ok, try now Here, we’re serious this time Uh, you’ve got two days “Iterative Development” “Unit Testing” “Process driven” 7

IBM Software Group The bad old days…. Part Four § No formal Tester training 4 Not treated as a profession 4 Self-fulfilling prophesy – novice testers, underappreciated value 4 Developers weren’t being trained to test either…. 8

IBM Software Group So what changed? Gartner estimates that, on average, a failed application costs an enterprise approximately $100, 000 per hour. And, with the National Institute of Standards and Technology ("The Economic Impact of Inadequate Infrastructure for Software Testing, " May 2002) reporting an estimated $22. 2 billion to $59. 5 billion lost per year as a result of poorquality software, catastrophic failures are here to stay. Approximately 25 percent of all applications deployed on distributed systems are effectively tested using modern tools and techniques. Through the second half of 2004, the number of tested deployed applications will remain close to 25 percent. 9

IBM Software Group Agenda § The Value of Testers to the Organization § Testing in Various Development Processes 10

IBM Software Group Legacy Development Processes § Code & fix 4 Purely an engineering effort 4 No testers at all 4 Fully dependent on skills of the programmers 11

IBM Software Group Modern Development Processes…. § Waterfall 4 First proposed in 1970 by W. W. Royce 4 Development proceeds linearly through phases of requirements analysis, design, implementation, testing (validation), integration and maintenance 4 Risk is considered mitigated via problem analysis 4 Testing is typically direct derivative of requirements 4 Considered discredited for use as a Real World process § However, in practice, the process rarely proceeds in a purely linear fashion 12

IBM Software Group Modern Development Processes § V model 4 Portrays several distinct testing levels / illustrates how each level addresses a different stage of the lifecycle 4 Testing process also involves identifying what to test (test conditions) and how they'll be tested (designing test cases), building the tests, executing them and evaluating the results, checking completion criteria and reporting progress. 4 Says nothing about how or how much requirements and designs should be documented 4 Doesn’t define when tests of any level should be designed or built. 13

IBM Software Group Modern Development Processes § Spiral/Iterative 4 Defined by Barry Boehm 4 Each cycle considers critical stakeholder objectives and constraints, product and process alternatives, risk identification and resolution, stakeholder review, and commitment to proceed. 4 Level of effort, degree of detail within each cycle driven by risk considerations 4 Each phase starts with a design goal and ends with the client reviewing the progress thus far. 4 Using analysis and measurement as drivers of the enhancement process is one major difference between iterative enhancement and the current Agile Methods. 14

IBM Software Group Modern Development Processes § Agile 4 Distillation of Scrum/Feature Driven/XP/Crystal family/Adaptive Software Development/DSDM 4 Considered “lightweight” process family 4 Have short iterative cycles, feature-based planning, constant feedback, change tolerance, team proximity, customer intimacy, and a focus on the overall team ecology 4 Requires § Uncertain or volatile requirements § Responsible and motivated developers § Customer who understands and will get involved. 4 Minimizes risk by ensuring that software engineers focus on smaller units of work. 4 Advocates developer testing, scenario-based testing, user testing http: //agilemanifesto. org/ 15

IBM Software Group Standards, Not Processes § Capability Maturity Model (CMM) - http: //www. sei. cmu. edu/cmm/ 4 model for judging the maturity of the software processes of an organization § No rules for creating such an organization though 4 Five levels of maturity (Initial, Repeatable, Defined, Managed, Optimizing) 4 Recently replaced by CMMI (CMM Integrated) § ISO 9000 - http: //www. iso. ch/iso/en/iso 9000 -14000/index. html 4 Family of standards primarily concerned with "quality management“ 4 Focused on what the organization does to fulfil: § the customer's quality requirements, and § applicable regulatory requirements, while aiming to § enhance customer satisfaction, and § achieve continual improvement of its performance in pursuit of these objectives § Six Sigma - http: //www. isixsigma. com/ 4 methodology that uses data and statistical analysis to measure and improve a company's operational performance by identifying and eliminating "defects" in manufacturing and service-related processes 4 commonly defined as 3. 4 defects per million opportunities § ITIL - http: //www. itil. co. uk/ 4 best practice for IT Service Management - series of books giving guidance on the provision of quality IT services, and on the accommodation and environmental facilities needed to support IT 16

IBM Software Group Clarification…. 17

IBM Software Group Agenda § The Value of Testers to the Organization § Testing in Various Development Processes § Testing Approaches 18

IBM Software Group Various Black Box Testing Approaches All tests involve choices and tradeoffs in at least these five areas: Coverage: What you’re testing Risks: Problems you are trying to find Evaluation: How you know a test passed or failed Activities: How you test Testers: Who is testing How you think about these issues dictates what combinations of test techniques you will use. 4 Function testing 4 Domain testing 4 Specification-based testing 4 Risk-based testing 4 Stress testing 4 Regression testing 4 User testing 4 Scenario testing 4 State-model based testing 4 High volume automated testing 4 Exploratory testing Caner, Bach, Pettichord “Lessons Learned in Software Testing” 19

IBM Software Group Agenda § The Value of Testers to the Organization § Testing in Various Development Processes § Testing Approaches § Being a Good Tester 20

IBM Software Group A Good Test Engineer § Has a 'test to break' attitude, an ability to take the point of view of the customer, a strong desire for quality, and an attention to detail § Tact and diplomacy are useful in maintaining a cooperative relationship with developers, and an ability to communicate with both technical (developers) and non-technical (customers, management) people is useful § Previous software development experience can be helpful as it provides a deeper understanding of the software development process, gives the tester an appreciation for the developers' point of view, and reduce the learning curve in automated test tool programming “Quality is never an accident; it is always the result of high intention, sincere effort, intelligent direction and skillful execution; it represents the wise choice of many alternatives. " — Willa A. Foster § Judgment skills are needed to assess high-risk areas of an application on which to focus testing efforts when time is limited. http: //www. softwareqatest. com/qatfaq 2. html 21

IBM Software Group Succeeding as a Tester and as a Test Organization § Facilitate communication across the team 4 Raise team consciousness about quality, best practices 4 Possible because they are not insulated from the customer context § Clarify requirements through your understanding of the customer 4 From customer point of view 4 From user point of view 4 From programmer’s point of view 4 Mismatches between these views 4 Identify assumptions “There is nothing as mysterious as something clearly seen. ” Robert Frost § Understand team characteristics to determine suitability of approaches to testing 4 Skills 4 Team size 4 Management 4 Schedule 4 Severity of failure 4 Testability needs 22

IBM Software Group Succeeding as a Tester, as a Test Organization § Plan, plan 4 Refactoring tests as you go along without a goal is just bad refactoring 4 Where is the finish line? 4 Test design finds errors (Boris Beizer, Software Testing Techniques (Thomson Computer Press, 1990) § Collaborate to help design for testability 4 We can learn from hardware…. 4 Simplifies automation efforts for regression testing § Know the dimensions of test case quality 4 Information § Reduction of uncertainty. How much do you expect to learn from this test? 4 Power § If two tests have the potential to expose the same type of error, one test is more powerful if it is more likely to expose the error. 4 Credibility § Failure of the test should motivate someone to fix it, not to dismiss it. 4 Feasibility § How hard is it to set up the test, run the test, and evaluate the result? “In preparing for battle I have always found that plans are useless, but planning is indispensable. ” Dwight D. Eisenhower 23

IBM Software Group Agenda § The Value of Testers to the Organization § Testing in Various Development Processes § Testing Approaches § Being a Good Tester § How IBM Rational Can Help 24

IBM Software Group Quality Drivers in the IBM SWG Product Line Analy st Archite ct Define Design Developer Tester Operations Code Test Debug Functional Test Load Test Deplo Monitor y RUP – Clear. Case (CM) – Clear. Quest (DCT) – Requisite. Pro (RM) – Test. Manager (TM) – Project. Console Web. Sphere Business Integrator Rational Rose XDE Rational Purify. Plus Rational Test Real. Time Rational Rose XDE Visual Trace Rational Robot Rational XDE Tester Tivoli TMTP Rational Performance Tester WSAD tracing 25

IBM Software Group Continuously Ensure Quality Uniting innovative solutions and best practices to prevent, detect, diagnose and remove defects all across the iterative software application development and deployment lifecycle Quality-enabling activities must start here and involve the entire team Analy st Archite ct Developer IT Operations must be tied into the lifecycle Tester Operations Integrated, Optimized Quality Solutions Define Design Code Test Debug Functional Test Load Test Deplo Monitor y Data and metrics must be open, accessible, traceable 26

IBM Software Group IBM Rational - Defining and Implementing Standards § Extends UML 2. 0 to accommodate static and dynamic test specification § Enables sharing of profiling and testing information between tools, vendors § Establishes same test abstraction benefits as UML provides to developers Consortium lead 1 st implementation! § Open source, integrated test, trace and monitoring infrastructure § Promotes testing across lifecycle by opening door for innovation § Avoids lock-in forced by vendors § Already shipping in WSAD Founding member and primary contributor Laying groundwork for process and data integration 27

IBM Software Group The IBM Software Development Platform Develop Iteratively Analyst Pervasive Model, and simulate, Embedded assemble, and monitor business processes IBM and 3 rd Party Servers Project Manager Focus on Architecture Architect Model application s and data Continuously Ensure Quality Developer Tester Visually construct, transform, integrate and generate code Design, create, and execute tests § Follow a common process § Track project status § Manage requirements Microsoft Manage Change and Assets Industry Standards Deployer DB 2 Provision, Lotus configure, Rational tune and Tivoli troublesh Web. Sphere oot applicatio ns § Manage change and assets § Manage quality Training & Mentoring Partners 28

IBM Software Group Agenda § The Value of Testers to the Organization § Testing in Various Development Processes § Testing Approaches § Being a Good Tester § How IBM Rational Can Help § Resources 29

IBM Software Group Reading Suggestions § Better Software Magazine http: //www. stickyminds. com/Better. Software/magazine. asp § Software Test and Performance - http: //www. stpmag. com/ § ASQ Software Quality Professional http: //www. softwarecertifications. com/ § Journal of Software Testing Professionals – http: //www. testinginstitute. com/journal. php § Quality Techniques Newsletter - http: //www. soft. com/News/QTNOnline/ § Methods and Tools - http: //www. martinig. ch/mt/index. html § Software Development Magazine - http: //www. sdmagazine. com/ 30

IBM Software Group Organizations § SEI - http: //www. sei. cmu. edu/ § IEEE - http: //shop. ieee. org/store/default. asp? tabtype=stand § American Society for Quality - http: //www. asq-software. org/ 4 Certification - http: //www. asq. org/cert/types/csqe/index. html § Quality Assurance Institute - http: //www. qaiusa. com/ 4 Certification - http: //www. softwarecertifications. com/ § International Institute for Software Testing http: //www. testinginstitute. com/ 31

IBM Software Group 32

IBM Software Group 33