15 1 Introduction n n Test execution is

15. 1 Introduction n n Test execution is situated on the critical path to product introduction. Test automation is used, for instance, to minimize the time needed for test execution. Automation can be profitable in terms of time, money, and/or quality. In principle, the execution of almost every test can be automated. In practice, only a small part of the tests are automated. Test automation is often used in the following situations: n n tests that have to be repeated many times; a basic test with a huge range of input variations – the same steps have to be executed every time but with different data (for instance, evolutionary algorithms); very complicated or error-prone tests; testing requires specialized equipment that generates the appropriate input signals, activates the system, and/or captures and analyzes the output.

15. 1 Introduction n Changes in the system, design, and requirements are common and are also a threat to the usability of the automated tests. The consequences of these changes can be: n n n n additional test cases; changed test cases; different result checks; changed system interface; changed functionality; different signals; different target platform; different internal technical implementation.

15. 2 The technique of test automation 15. 2. 1 Design for maintainability n n To avoid a test suite from becoming shelfware it is designed to be as independent as possible of changes – changes in the test object, the technical connection between the test suite and the system under test, and the tests and data used. This is established by reducing the dependencies: n n n test data is stored outside the test suite; the description of test actions is separated from the technical implementation of these actions; the process of how to handle an automated test is implemented independent of its environment and system under test – this explains why any test suite needs several essential components; the communication between the system under test and the test suite is implemented independent of the central process of the test suite. These issues will be discussed in more detail in the following sections. The following terms will be used: n n Automated testing. Executing tests using an automated test suite. Test suite. Everything necessary to execute a test by pushing just one button. Data driven. An approach where physical test actions are separated from test data, and the test data is the impulse for the test. Framework. A library of reusable modules and design.

15. 2. 1. 1 Test cases n The test cases are stored outside the test suite (see Figure 15. 1). They can be stored in a database or spreadsheet. An additional test case means adding only this test case to the storage of the test cases. The test suite doesn’t have to be changed.

15. 2. 1. 2 Test actions n n This implementation should be hidden from the normal user of the test suite – the only thing to know is which test actions are available (see Figure 15. 2). By choosing the right test actions the right test script can be composed. If the functionality of the system under test stays the same but the technical implementation is changed (for instance, another interface is used) only the technical implementation must be changed. The user can still use the same script as if nothing has happened. Table 15. 1 shows an example where test data and actions are combined in one test script. This combination is quite common.

15. 2. 1. 2 Test actions

15. 2. 1. 3 Essential components n n n Those basic components are part of the generic design of a test suite (see Figure 15. 3) which is called a blueprint – this has three major sections. The first is the input side with test scenarios and scripts. The last is the output side with status reports and progress reports. The processing unit, or test suite kernel, is situated in between. This kernel makes it possible to test with one push of a button. The test suite continues working without human intervention until the complete test scenario (see section 6. 4. 3) is executed. Appendix C provides a more detailed description of the blueprint.

15. 2. 1. 4 Communication n The system under test can have the following forms: executable code on the same machine as that on which the test suite is installed; n executable code on a test target (separated from the test suite environment); n system under test in a simulation environment; n system under test as pre-production system. n

15. 2. 1. 4 Communication n The test case will stay the same while the communication, or technical implementation of the communication, is changed. There has to be a communication layer (see Figure 15. 3) for all forms.

15. 2. 2 Maintenance n Due to certain changes, the automated tests may not run correctly any more and maintenance becomes necessary. The blueprint is developed to keep maintenance to a minimum. The parts possibly affected by changes are: n n the test data; synchronization, error recovery, checks and initialization; application specific modules; the communication layer.

15. 2. 2 Maintenance n In the introduction to this chapter, a list of possible changes was presented. The impact of those changes on the test suite are listed below. n Adding test cases. Only the data store has to be changed. n Changed test cases. The data store has to be changed. Sometimes it is necessary to add new test actions to the framework. n Different result checks. Only the data store has to be changed if only the output prediction is changed. Otherwise checks and or test actions have to be changed or added. n Changed system interface. The communication layer has to be changed. If the signal definition is changed, the test data should also be changed if these test data contain information about the signals. n Changed functionality. This change can have an effect on the test data because new test cases have to be added. The synchronization, error recovery, checks, and initialization can be changed because the system dialog and sequence of functionality can have changed. New functionality means new test actions. The new functionality can also affect the communication layer. n Different signals. There have to be changes in the communication layer. Sometimes it is also necessary to change the test data. Different target pla affect the communication layer. n Different internal technical implementation. If the external interface and functionality are not changed then this can only affect the software implementation which bypasses the external interface.

15. 2. 3 Testability To make test automation easier it is essential that testability is a quality attribute recognized during the design and implementation of the system. n It is rather time consuming and expensive to built a dedicated communication layer because the behavior of the system is impossible to monitor. n

15. 3 Implementing test automation n Test tools are used to enhance the quality, lower the costs of test execution, and/or save time. The automated test suite must be “repaired” and “upgraded” to make it run once again and cover the changed system behavior that must be tested. In bad cases, the repair costs more than executing the test manually. The objectives of quality, cost, and time are no longer realized and the overall perception is one of failure.

15. 3 Implementing test automation n n Investment in the implementation of these kinds of tools is high and so the complete test set should be executed three to six times at least to achieve pay back. Most problems can be overcome by using a lifecycle model with proper analyzing activities and decision points (see Figure 15. 4). This lifecycle has three consecutive phases: n n n initiation; realization; exploitation.

15. 3 Implementing test automation

15. 3 Implementing test automation n Test automation objectives are quantified, the test objectives are described in detail, and the technical implications of the system under test are investigated. The output of this process biases the design process. According to the design, the automated test environment is developed. The end product of this phase will be an implemented test automation environment capable of realizing the test automation objectives.

15. 3. 1 Initiation n The test automation project starts with a feasibility study. The main objective of this is to get enough information to decide if test automation is the proper thing to do. In order to get the right information, three items are analyzed: n n n n test automation objectives; system under test; test organization. The test automation objectives are the expectations of the organization about test automation. The objectives are described in terms of quality, cost, and time – there should be a balance between these. High quality generally leads to lower cost and time saving. The objectives are also the criteria for the determination of the success of the test automation project. Other objectives include in which development stage should automated testing be implemented, and what should be tested with this automated test environment.

15. 3. 1 Initiation The system under test is investigated technically to find out how it should be tested and whether additional equipment, such as simulators and signal generators, is necessary. n It is also necessary to know whether the interfaces, internal technical structure, and the functionality of the system are regularly changed during development or consecutive releases. n

15. 3. 1 Initiation n n It would be rather nice if this format could be used in the test automation environment. The proper use of test tools demands a repeatable test process. This means that there are well-described test cases derived by using test design techniques and a test strategy which decides on the relative importance of the system parts and quality attributes. To use and develop the automated test environment, special knowledge is necessary. If this knowledge is not available then employees have to be trained. Another consequence can be that the usage of the automated test environment is made as simple as possible by developing an intuitive user interface. This will hide the complexity of the automated test environment.

15. 3. 1 Initiation n n In addition to all these primary conditions one should also know if there is a tool available to use for the system itself. This will depend on the system itself, the target, the instantiation of the system, and also the program language. Figure 15. 5 shows the different stages in the test tool selection process.

15. 3. 1 Initiation n The ultimate consequence is the development of a dedicated tool or no automated test execution. The following activities have to be carried out. n n 1. Set goals and define the scope. Goals can include improving the test process, shortening test execution time, diminishing test execution capacity, or enhancing quality test execution. Be aware that goals such as solving organizational problems or attention-free testing can not be realized by tools. The scope shows whether the automated test environment should be used for only one particular system, a range of systems, or across the company. 2. Define knockout criteria. These criteria are mostly based on the technical implications of the system under test. A tool that cannot meet these criteria is useless and is immediately deleted from the list. 3. Long list and RFI. With the knockout criteria in mind, a long list of tools is prepared. All tool vendors get a Request For Information (RFI). This is a list of questions a vendor should answer. The accuracy of this answer is also an indicator of the vendor’s service level – tool vendors that do not answer are discounted. 4. Detailed criteria. A detailed list of selection criteria is prepared. There are three kinds of criteria: n n n technical (compatibility, stability, script language, etc. ); functional (object recognition, synchronization, report functionality, operating system, target platforms, etc. ); demands to tool vendors (continuity, education, support, installed base, etc. ). The criteria are grouped by knockout criteria, demands, and wishes.

15. 3. 1 Initiation n 5. Short list and RFP. Based on the detailed criteria, tools are selected from the long list. The selected tools are placed on a short list which is sent to the vendors with detailed criteria and a Request for Proposal (RFP). This gives an indication of the costs, and the possibility of support and training. Based on this information, two or three tool vendors are invited to show the capabilities of their tools on a demonstration test environment. A list of capabilities each vendor should show with their tool should be prepared. 6. Proof of concept. Before using the tool in practice, and investing much effort in its implementation, a pilot can be started. If this pilot is successful the tool can be implemented. Many tool vendors offer the chance to use the tool on trial for a certain period. 7. Decision. The results of the proof of concept give enough information to decide if an implementation of the tool will meet expectations. The outcome could be that the objectives can only be met with a dedicated developed test tool.

15. 3. 1 Initiation n The benefits can also be quantified in terms of saving money by spending less time and labor on testing. The enhancement of the quality of the test process is the subjective part of the benefits. Be aware that the direct cost of tools is high but the cost of labor during implementation, usage, and maintenance are usually much higher. This whole process should aid the enhancement of awareness and positive support for implemention as a major side effect.

15. 3. 2 Implementation n The first is the preparation of a plan of action – the purpose being to show the activities, people involved, and time estimation. Then the design and development will start. In this process, six activities are identified (see Figure 15. 6).

15. 3. 2 Implementation n Based on this design, the engineer develops the automated test environment and is responsible for implementing the necessary functionality to execute the delivered test cases. The functional testers develop the test cases based on test design techniques. In the last step, a final test of the automated test environment is performed. Now, the design and all other documentation about the automated test environment is brought up to date. Together with the automated test environment, this is deployed and used in the maintenance phase. In additional to these major activities, parallel activities can be executed as necessary.

15. 3. 3 Exploitation n The automated test environment is now ready for use. The impact on the test process of the availability of an automated test environment is shown in Figure 15. 7.

15. 3. 3 Exploitation n n During specification, the test cases should be described in a readable format for the automated test environment, and the environment should be adjusted due to changes in the test object. The test execution is partly covered by the automated test environment and in the completion phase all changes are frozen and stored. It is essential that a complete copy of the automated test environment is subject to configuration management.

15. 3. 3 Exploitation n The major objective of the exploitation phase is to maintain and use the automated test environment. During this phase, the organization can profit from the benefits of the automated test environment. To meet the test automation objectives and to be able to benefit from this environment, in the next release the focus must be on maintenance of the environment. By neglecting this part of the job the environment will be useless after just one or two releases – and that is probably before the investment is paid back.