Applications Services Brazil IBM Fundamentals of Executing

Applications Services – Brazil IBM Fundamentals of Executing Tests Elton Paz Test Automation Analyst June, 2005 IBM Confidential © Copyright IBM Corporation 2005

AGENDA 4 Execute Tests - Setting up systems and software - Distributing test cases - Executing test cases 4 Analyze test results - Identify verification tasks to complete prior to logging a defect - Use techniques for analyzing failures 4 Document results of test execution - Use test logs to record test execution - Use the Can Pig Ride techniques in creating effective incident reports - Create a test-incident report to document problems Application Services Brazil June, 2005 Page 2

Introduction 4 The previous classes have focused on designing and implementing tests that verify that the implementation of the project meets the requirements. In this class, you will learn how to execute the tests. Application Services Brazil June, 2005 Page 3

Definition 4 Test execution is the process of setting up the test environment, distributing test cases, and manually or programmatically running the test cases Application Services Brazil June, 2005 Page 4

Execute Tests 4 The key to setting up the testing environment is to establish a set of standards that everyone in the organization can adhere to. A strict structure needs to be put in place for where and how to store the components of the testing environment. Components of the testing environment include: - Supporting software - An inventory of hardware - The code under test - A workbook or execution logs Application Services Brazil June, 2005 Page 5

Distributing Test Cases 4 You must distribute each test script or series of scripts used for a test case to the appropriate environment. This may involve walking from client to client and installing the test scripts in the appropriate directory on each machine, or it can involve using automated tools, such as the Software Testing Automation Framework, to distribute the scripts. Application Services Brazil June, 2005 Page 6

Executing Test Cases 4 After you set up the environment and distribute your test cases, you are ready to execute your tests. You must execute each test exactly as defined in the manual or scripted test-case implementation. Application Services Brazil June, 2005 Page 7

Applying your knowledge Application Services Brazil June, 2005 Page 8

What’s next? 4 Now that you understand the basics of executing tests, you can move on to learn about verifying and analyzing your test results. Application Services Brazil June, 2005 Page 9

Verifying and analyzing your results 4 Identify verification tasks you need to complete prior to logging a defect 4 Use techniques to analyze failures Application Services Brazil June, 2005 Page 10

Verifying and analyzing your results 4 Before you begin analyzing the problem as a defect, follow these steps to validate each discrepancy between actual and expected results: Application Services Brazil June, 2005 Page 11

Analyzing the defect 4 After you have confirmed the discrepancy between actual and expected results, investigate further and gather as much information as possible about the symptoms. Try to determine the nature and location of the underlying problem indicated by the symptoms. In analyzing your results, you will gather more data, find simple situations that reveal the defect, note situations that do not reveal the defect, and collect additional data for test cases already run. 4 In analyzing the results of test execution, it is important to be able to gauge the severity of the defect. The severity level of the problem helps to identify the priority that the team gives to fixing the defect. To understand the severity, you will sometimes need to do some exploratory testing to investigate other effects of the defect. Application Services Brazil June, 2005 Page 12

Determining defect severity 4 With experience, you gain a better sense for identifying follow-up testing opportunities that may help to determine the defect severity. Follow-up testing opportunities involve 4 Changing the conditions under which an error occurred by changing what you do. As an example, if the problem seems related to your entering data very quickly, try slowing your rate of input. 4 Changing the options and settings of the program under test. As an example, if the problem seems related to desktop screen-resolution settings, try varying the settings. 4 Changing the software and hardware environment. For example, if the defect seems related to a low-end hardware configuration, try using the lowest configuration specified in the requirements. 4 Changing the data used by the program. As an example, if the defect seems related to strings exceeding a certain number of characters in length, create a second set of test data to check the hypothesis. 4 It is important to document the variations and the results of executing the tests so that you can trace the results back to the variation that caused it. Characterizing the defect with additional testing often leads to a hypothesis as to the cause of the defect. A better understanding of the conditions under which the defect occurs, and a hypothesis as to the cause, will help determine the defect severity by allowing the project team to better understand the true impact of the defect on the business. Application Services Brazil June, 2005 Page 13

Analyzing the defect - Severity definition’s example 4 Severity 4 - Minor problem, not significant to the customer’s operations - The Customer may be able to circumvent the problem - Testing of the selected component or system can continue 4 Severity 3 - Minor problem - The Customer can use the Product with some functional restrictions, but it does not have a severe or critical impact on the customer’s operations - Testing of the selected component or system can continue with most functions Application Services Brazil June, 2005 Page 14

Analyzing the defect - Severity definition’s example 4 Severity 2 - Major problem - The Customer can use the Product, but an important function is not available or the customer’s operations are severely impacted - Testing of the selected component or system cannot continue with the impacted function but can continue with other functions 4 Severity 1 - Critical problem - The Customer cannot use the Product or there is a critical impact on the customer’s operations which requires an immediate solution - Testing of the selected component or system cannot continue Application Services Brazil June, 2005 Page 15

Analyzing failures by varying conditions, options and settings 4 Varying the conditions involves trying actions that relate to the task that failed. For example, if you were testing the Self Assessment System application with the displayed section of questions and found that all responses in the section cleared when you clicked the Save Draft button, you might try entering certain fields and leaving others blank to see if everything clears after every click of the Save Draft button. To analyze a defect, you can also try varying options and settings that look as though they might be relevant to the defect. You can try reproducing the defect with different values for variables, different selections for how the program uses memory, different screen resolutions or other changes relevant to the defect you are analyzing. Application Services Brazil June, 2005 Page 16

Analyze nonreproducible errors 4 So what should you do when you are unable to reproduce an error? When you realize that you are unable to reproduce an error, document everything you can remember. Be as detailed as possible. Also let the responsible developer know that you encountered a problem that you cannot reproduce and ask if she has any ideas for follow-up tests that you can run 4 The fact that a defect is not reproducible is data. To reproduce a problem, you must be able to put the code through the same conditions that existed when the problem manifested. If you cannot reproduce the failure, you are missing some conditions. 4 There are many reasons why a defect may not be easily reproducible, including causes related to program interactions, speed or specific order of actions, and environmental factors. Application Services Brazil June, 2005 Page 17

Program interactions 4 Programs have various degrees of coupling. When two modules share the same variable, defects can arise in one of the two modules if the other changes variable values in unexpected ways. Interrupts may share data with main routines in ways that cause the defects to manifest only after a specific interrupt. 4 Defects caused by interrupts received at unexpected times or cases in which the program receives data before or after it expects to receive the data can also be hard to reproduce. 4 Another hard-to-reproduce error occurs when your program depends on a specific version of a DLL, but a different program loads a different version of the same DLL into memory. Depending on which program you run first, the defect may or may not appear. 4 Defects that depend on the crash or exit of an associated process can also be difficult to reproduce. Application Services Brazil June, 2005 Page 18

Speed or specific order of actions 4 Another type of defect that is hard to reproduce is one that is related to speed or specific order of actions. For example, a defect may occur only the first time you run the program or the first time you perform a task after booting a program. To re-create the defect, you might need to reinstall the program. Or, if the program does not uninstall cleanly, you might have to install it on a fresh machine before you can see the problem. 4 Defects that are dependent upon a certain speed of input can also be hard to re-create. For example, you may be typing too slowly to recreate a defect as a result of too carefully trying to reproduce the problem. Application Services Brazil June, 2005 Page 19

Environmental factors 4 Another example of a defect type that can be difficult to reproduce is one that results from specific environmental factors. That is, a defect might appear only at a specific date or time of day, such as on a weekend or at month end, quarter end or year end. 4 Another example of difficult-to-reproduce defects is one that is a side effect of a hardware failure, such as an unstable power supply that can create nonreproducible failures. 4 In general, communicate informally with unit or component developers and other testers prior to issuing a formal problem report, especially if the nature and location of the fault is in doubt. Application Services Brazil June, 2005 Page 20

Applying your knowledge Application Services Brazil June, 2005 Page 21

Applying your knowledge Application Services Brazil June, 2005 Page 22

Summary 4 Verification techniques to perform prior to logging a defect. These techniques include verifying that the element under test, the execution environment, and the test-case implementation are consistent with the test-case design and that the test-case design is consistent with the requirements. 4 Techniques to identify failures. You can identify failures by varying conditions, options and settings. Also, you can identify nonreproducible errors through program interactions, speed or specific order of actions, and environmental factors. Application Services Brazil June, 2005 Page 23

What's next 4 Now that you understand how to verify and analyze your test results, you can move on to learn about documenting your execution results. Application Services Brazil June, 2005 Page 24

Documenting your execution results 4 Clearly communicating the results of your tests is the key to improving software quality. Let’s discuss three opportunities to communicate the results of executing your tests through documentation: test logs, descriptions of the incidents included in defect reports and the overall test incident report that you file to document a defect. Application Services Brazil June, 2005 Page 25

Logging execution of tests 4 A test log is a chronological record of relevant details about the execution of tests. The purpose of the test log is to share the test-case execution information among testers, users, developers and others. The test log facilitates replication of situations encountered during testing to help you perform an analysis of problems. 4 With automated tests, test logs are created, modified and deleted according to the design of test logging, which is implemented through the test script or test automation. With manual tests, logging execution of your tests means manually writing your results to your test logs according to the standards and schemes established during test implementation. Application Services Brazil June, 2005 Page 26

Using Can Pig Ride criteria for effective incident reports 4 Defect reports are among the most important deliverables that the test team produces. These reports are as important as the test plan and have more impact on the quality of the product than many other deliverables from test. Therefore, it is worth the effort to learn how to write effective defect reports. 4 Effective defect reports: - Reduce the number of defects returned from development. - Improve the speed of getting defect fixes. - Improve the credibility of the test team. - Enhance teamwork between test and development groups. Application Services Brazil June, 2005 Page 27

Using Can Pig Ride criteria for effective incident reports 4 The objective is not to write the perfect defect report, but to write an effective defect report that conveys the proper message, gets the job done and simplifies the process for everyone. Application Services Brazil June, 2005 Page 28

Condense the incident report 4 Condense 4 Say it clearly but briefly. First, eliminate unnecessary wordiness. Second, don't add in extraneous information. It is important that you include all relevant information, but make sure that the information is relevant. In situations where it is unclear how to reproduce the problem or the understanding of the problem is vague for whatever reason, you will probably need to capture more information. Keep in mind that irrelevant information can be just as problematic as too little relevant information. Application Services Brazil June, 2005 Page 29

Example of condensing the incident report 4 The following is an example of an incident report that can be condensed: - I was setting up a test where the real intent was to detect memory errors. In the process I noticed a new GUI field that I was not familiar with. I decided to exercise the new field. I tried many boundary and error conditions that worked just fine. Finally, I cleared the field of any data and attempted to advance to the next window, and then the program abended. Several retries revealed that anytime there is not any data for the "product description" field, you cannot advance to the next window or even exit or cancel without abending. 4 This report can be condensed as follows: - The exit, next and cancel functions for the Product Information window abend when the Product Description field is empty or blank. Application Services Brazil June, 2005 Page 30

Exercise 1: Condensing the incident report 4 Martin is a tester responsible for Function Verification Testing of the Self Assessment System. He is exercising the code for the login page with a series of verification triggers as shown. Martin's incident report offers the following detail: I started the login testing by entering a user ID that is not yet registered and then entering a valid password for that user ID. This produced the expected result of displaying a message that first-time users must register. However, when I entered an invalid user ID and entered a password, the system also responded with the message "First-time users must register. " Subsequently, I left the user ID blank and entered a password, which produced the expected results. I reentered an invalid user ID and a password to double-check the results and found that the result was indeed "First-time users must register. " I also confirmed with the database administrator that the user ID I was using, mbtest, was not a valid user ID. Application Services Brazil June, 2005 Page 31

Application Services Brazil June, 2005 Page 32

Application Services Brazil June, 2005 Page 33

Accurately describe the incident Accurate 4 Make sure that what you are reporting is really a bug. You can lose credibility very quickly if you get a reputation of reporting problems that turn out to be set-up problems, user errors or misunderstandings of the product. Before you document the problem, make sure that you do your homework. Before documenting the problem, ask yourself: - Is there something in the setup that could have caused this? For example, are the correct versions installed and all dependencies met? Did you use the correct login, security, command-task sequence and so forth? - Could an incomplete cleanup, incomplete results or manual interventions from a previous test cause this? - Could this be the result of a network problem or some other environmental problem? - Do you really understand how the product is designed to work? Application Services Brazil June, 2005 Page 34

Accurately describe the incident 4 Numerous influences can affect the outcome of a test. Make sure that you understand what these influences are and consider the effect they have on the perceived bug you are reporting. This is one area that quickly separates the experienced tester from the novice. If you are unsure about the validity of the problem, it may be wise to consult with an experienced tester or developer prior to documenting the problem. 4 As a rule of thumb, it helps to remember the adage that "it is a sin to overreport, but it is a crime to underreport. " Don't be afraid to document problems. Do your best to ensure that they are valid problems. When you discover that a problem you have opened turns out to be an incorrectly reported problem, make sure that you learn from the experience. Application Services Brazil June, 2005 Page 35

Example of inaccurately describing an incident Application Services Brazil June, 2005 Page 36

Exercise 2: Accurately describe the incident Application Services Brazil June, 2005 Page 37

Exercise 2: Accurately describe the incident Application Services Brazil June, 2005 Page 38

Exercise 2: Accurately describe the incident Application Services Brazil June, 2005 Page 39

Use neutral language in the incident report Neutral 4 State the problem objectively. Don't try to use humor and don't use emotionally charged language. What you think is funny when you write the defect may not be interpreted as funny by a developer who reads your report while working overtime and stressed by deadlines. Emotional statements create barriers to communication and teamwork. Even if the developer doubted you and returned your previous defect and now you have proof that you are correct and they are wrong, just state the problem and the additional information that will be helpful to the developer. In the long run, this added bit of professionalism will gain respect and credibility for your work. Application Services Brazil June, 2005 Page 40

Example of using neutral language Application Services Brazil June, 2005 Page 41

Exercise 3: Using neutral language 4 The following is a snapshot of a self-assessment that is in progress. In this example, the tester has determined that the Save Draft & Next button and the Save Draft & Previous button do not take the user forward or backward through the application. Essentially, the user cannot progress through the application. The tester reporting this defect was hoping to provide some humor with the following incident report: Though this is a very nice window and people should enjoy the interaction and may even be saddened when they have to leave this window, you may want to consider providing some sort of exit from this window. Application Services Brazil June, 2005 Page 42

Be precise in the incident report Precise 4 Try to make the problem description as easy as possible to read quickly. 4 First, in the description, describe exactly what you perceive the problem to be. Some descriptions detail a series of actions and results, for example, "I hit the enter key and action A happened. Then I hit the back arrow and action B happened. Then I entered the xyz command action C happened. "In this case, the reader may not know if you think all three resulting actions were incorrect, or which one, if any, is incorrect. In all cases, but especially if the description is long, you need to summarize the problems at the beginning of the description. Don't depend on an abstract in a different field of the defect report to be available or used by everyone who reads the problem description. Don't assume that others will draw the same conclusions that you do. Your goal is not to write a description that is possible to understand, but to write a description that cannot be misunderstood. The only way to make that happen is to describe explicitly and precisely the problem rather than just giving a description of what happened. Application Services Brazil June, 2005 Page 43

Example of being precise when writing incident reports Application Services Brazil June, 2005 Page 44

Isolate the problem Isolate 4 Each organization has its own philosophy and expectations about how much the tester is required to do to isolate the problem. Regardless of what is required, always invest a reasonable amount of effort into isolating the problem. Application Services Brazil June, 2005 Page 45

Example of isolating the problem 4 In this example, a tester creates an incident report that states, "Printer hangs if a job is canceled while printing. " 4 Take into account the following considerations before submitting the incident report: - Does the printer hang for all document types? PCL? PS? - Does it hang for all document sizes? Small? Big? - Does it matter when I cancel printing? Start? Middle? End? - Are there traces or logs that add useful insight? Application Services Brazil June, 2005 Page 46

Exercise 4: Isolating the problem Application Services Brazil June, 2005 Page 47

Application Services Brazil June, 2005 Page 48

Application Services Brazil June, 2005 Page 49

Generalize the problem Generalize 4 Sometimes the developers will fix exactly what you report, without even realizing the problem is a more general problem that needs a more general fix. If you have already investigated and know that a problem can be generalized, your report can save the developer time and enhance the possibility of a better fix to handle the general case. When you detect a problem, take reasonable steps to determine if it is more general than is immediately obvious. Application Services Brazil June, 2005 Page 50

Example of generalizing the problem 4 For example, a tester may report that a new word-processor save file function failed and the word processor abended when trying to save the file myfile. A little more investigation may have revealed that this same failure occurs anytime it saves a zero-length file. Or, perhaps with this release it abends on every save to a remote disk, a read-only disk and so forth. Your incident report can be generalized to enable the developer to solve the general problem. Application Services Brazil June, 2005 Page 51

Exercise 5: Generalizing the problem Application Services Brazil June, 2005 Page 52

Explain how to re-create the problem 4 Re-create 4 Some bugs are easy to re-create and some are not. If you can re-create the problem, you need to explain exactly what is required to replicate it. List all the steps, including the exact syntax, file names and sequences that you used to encounter or re-create the problem. If you believe that the problem will happen with any file, any sequence, etc. , then mention that, but still provide an explicit example that can be used to replicate the problem. If, in your effort to verify that you can re-create the problem, you find a shorter and more reliable means of re-creating it, document the shortest, easiest means of recreation. Application Services Brazil June, 2005 Page 53

Explain how to re-create the problem Application Services Brazil June, 2005 Page 54

Explain the impact of the problem Impact 4 What is the impact if the bug were to surface in the customer environment? The impact of some problems is obvious, for example, "entire system crashes when I hit Enter. " However, some defects are not so easy to identify. For example, you may discover a misspelling in a GUI. While it may seem minor, even trivial, every time someone uses your product, this GUI is the first thing they see and the misspelling results in an offensive word. In this case, even though it is just a typo, it may be something that absolutely must be fixed prior to shipping the product. 4 In short, use your best judgment. If you think it is possible that this defect will not get sufficient priority, then state the potential impact to try to convince the developer to fix the defect. Don't be overzealous, but make sure the readers of the defect have an accurate understanding of the probable impact on the customer. Application Services Brazil June, 2005 Page 55

Example of explaining the impact Application Services Brazil June, 2005 Page 56

Provide steps for debugging the problem Debug 4 When creating your incident report, provide all of the steps and information that the developer will need to debug the problem. If there are traces, dumps or logs, capture them and make them available through the defect report. Document what information has been captured and how this information can be accessed. Application Services Brazil June, 2005 Page 57

Explain evidence for the error Error 4 When creating your report, include all details that prove the existence of the error. Provide both the expected results and the actual results as evidence of why you believe there is an error. Because you are writing a problem report, it is obvious that you believe a problem exists. Provide any information you can to convince others that this is a valid problem. Evidence may take the form of documentation from user guides, specifications, requirements and designs. It may be past comments from customers, de facto standards from competing products or results from previous versions of the product. 4 Don't assume everyone views information in the same way you do. Don't expect people to read between the lines and draw the same conclusions as you. Don't assume that three weeks from now you will remember why you thought this was a bug. Think about what it is that convinced you that this is a problem and include that in the report. You will have to provide even more evidence if you suspect this situation may not be readily accepted by all as a valid bug. Application Services Brazil June, 2005 Page 58