Software testing techniques Automated software testing Presentation on

Software testing techniques Automated software testing Presentation on the seminar Kaunas University of Technology

Topics • Automated software testing. What? Why? • Automated vs. Manual • Common types of automated testing methods • Automated software testing framework • Summary 2

What is automated software testing? A process which covers: • the use of software to control the execution of tests • the comparison of actual outcomes to predicted outcomes • the setting up of test preconditions • other test control and test reporting functions. 3

Why testing is critical? 4

Why automated testing? • A time savings translates directly into cost savings • Improves testing productivity • Improves accuracy • Increases test coverage • Does what manual testing cannot do 5

Effort and Cost – – – – Let’s assume 6 test cases Effort required to run all 6 manually => 10 min Effort required to write unit tests for all 6 cases => 10 min Effort required to run unit tests for all 6 cases => < 1 min Number of testing iterations => 5 Total manual testing time => 50 min Total unit testing time => 10 min Release Manual Test Auto Test Manual Test Cumulative 1 10 10 10 2 10 0 20 3 10 0 30 4 10 0 40 5 10 0 50 6

Automated vs. Manual testing • Not all tests can be automated and most times is difficult to decide what to automate and what to manually test. 7

Advantages Automated Testing • Need to run a set of tests repeatedly • Helps performing "compatibility testing" (on different configurations and platforms) • Long term costs are reduced Manual Testing • Test cases have to be run a small number of times • Allows the tester to perform more specific tests • Short term testing costs are reduced • Possible to run regressions on • The more time tester spends a code that is continuously testing a module the grater changing and in shorter time chance to find real bugs 8

Disadvantages & other factors Automated Testing Manual Testing • It's more expensive to automate (bigger initial investments) • You cannot automate everything, some tests still have to be done manually • Manual tests can be very time consuming • For every release you must rerun the same set of tests which can be tiresome Other Factors • The performance of your test tools • The knowledge level of your testing team • The continuous growth of software to be tested 9

Common types of automated testing methods • Monkey testing • Capture / playback • Code-based (Unit) testing • Intelligent test automation 10

Monkey testing • Randomly selecting inputs from a large range of values and monitoring if exceptions are thrown. • For example, a monkey test can enter random strings into text boxes to ensure handling of all possible user input. • It applies not only for GUI or WEB testing, but also for Unit testing. 11

Tools example Monkey – Android testing tools. The simplest way to use the monkey is with the following command, which will launch the application and send 500 pseudo-random events to it. $ adb shell monkey -v -p your. package. name 500 12

Capture / playback • It’s a set of software programs that capture user inputs and stores it into a script for later replay. + Repeated testing can be performed quickly. + Does not require programming skills. - When the GUI changes, input sequences previously recorded may no longer be valid. - Difficult to determine location of bugs. 13

Tools example Autohotkey - Free keyboard macro program. Supports hotkeys for keyboard, mouse. 14

Code-based (Unit) testing • Individual units of source code are tested to determine if they fit for use. • Ideally, each test case is independent from the others + Bugs can be found in early development stage + Easy to test boundary cases - Not effective for the integrated system testing 15

Unit test-case generation method • Random • Genetic • Other… 16

Random • Input values generated randomly • Run with generated inputs • User checks output values (pass/fail) and saves them for later automatic checking (Oracle). 17

Genetic This methods approach is to combine JML (JAVA model language) and genetic algorithms. JUnit is used as a test execution platform. 18

Tools examples Parasoft Jtest is an automated Java testing and static code analysis product Test. Gen is a collection of open-source tools that automatically generates unit test cases. 19

Intelligent test automation • A model-based method for generating tests from a description of an application’s behavior (UML) + Models are simple to modify, so modelbased tests are economical to maintain over the life of a project. - Most of these methods are researched only theoretically. 20

OCL + UML Inputs generator Constraints UML + OCL Tested software Oracle Outputs Constraints pass / fail Report 21

Automated testing framework • A framework is an integrated system that sets the rules of automation of a specific product. • This system integrates the function libraries, test data sources, object details and various reusable modules. • The framework provides the basis of test automation and simplifies the automation effort. 22

Common types of frameworks • Data-driven testing • Keyword-driven testing • Hybrid testing 23

Data-driven testing • Data-focused automation • User defines just data sets to run tests with • Data is defined in external data source and decoupled from script • Good fit for a single application testing, featuring multi-environment, big datasets, and rarely changing Test Cases + - Good usability and reusability Quality and coverage strongly depends on the automation skills of implementing person Good test flow coverage Continuous maintenance issues due to a big amount of the code Multi-environment support Manual validation is required 24

Data set Example Test script 25

Keyword-driven testing • Good fit for a multi-application testing featuring multi-environment, big datasets, and a big number of short and straight Test Cases • Users de-compose test cases into reusable action keywords • Non-coders are able to author automated test cases with action keywords + - Test development doesn’t require programming skills Due to keywords limit doesn’t allow covering very complicated Test Cases Good test flow coverage Meta-language specific training is required for staff Compact code Manual validation is required 26

Example 27

Hybrid testing • Combines the best of data-driven and keyword -driven testing: – User defines data sets to run tests with – User also defines flow control through action keywords – Data is separated from script and stored in external data source with action keywords + Test Plan / Test Scenario creation doesn’t require programming or training on the Testing Tool Compact code and scalable architecture Built-in strict validation Requires initial investment for design and implementation of the framework 28

Framework components • Test generation (data and script generators). Tools might create specialized data such as randomized email messages, or populate databases etc. • System configuration. Tools might preserve or reproduce system parameters. • Test execution. Tools might operate the software itself, either simulating a user working through the GUI and using an alternative testable interface. • Oracles. An oracle is any mechanism by which we detect failure or success. Tools might automatically detect certain kinds of error conditions in a product. • Activity recording & coverage analysis. Tools might watch testing as it happens and retrospectively report what was and was not tested. • Test management. Tools might record test results, organize test ideas or metrics. 29

General framework structure 30

Common functional automated test tools Tool IBM / Rational Functional Tester (RFT) + • Built as Eclipse plug-in with full IDE, Java support and source management • Supports Web-based applications, . Net, Java, Siebel, SAP®, Power. Builder, AJAX, Adobe Flex, Adobe PDF documents • Full GUI Object Map repository HP / Mercury • Supports Web 2. 0, GWT, Dojo, YUI, Quick Test Silverlight, Flex, Ajax, JAVA, . NET Pro (QTP) applications • Full GUI Object Map repository • Seamless integration with Quality. Center Selenium RC & IDE • Good browser support • Good language support (JAVA, Ruby, C#) • Can be easily extended as JUnit suite • Open-source (no license cost) • License cost • No IDE (changing in release 10) • License cost • Visual. Basic scripting is limited • No GUI Object repository • Only web-based application support • Tool support is 31 limited

Tools examples 32

Summary • Automated testing improves productivity, accuracy, increases test coverage, saves time and cost • Need to decide what to automate and what to manually test • The framework provides the basis of test automation and simplifies the automation effort. 33

More information • http: //www. slideshare. net/lfingerman/test-automation-bestprcatices-with-soa-test-approach • http: //opuntia. cs. utep. edu/utjml/utest. html • http: //www. slideshare. net/yogindernath/understanding-ofautomation-framework-presentation • http: //www. ibm. com/developerworks/rational/library/591. ht ml • http: //www. satisfice. com/presentations/agileauto. pdf • http: //automation-beyond. com/2009/05/30/test-automation -practices-recordplayback/ • http: //en. wikipedia. org/wiki/Test_automation • http: //www. slideshare. net/directi/automated-testing-vs 34 manual-testing-391121

Thank you! 35

Question ? ? ? • What is the benefit of automated testing? • What are the common types of automated testing methods? • What are the common types of frameworks? • What components come to framework? • What are the examples of automated testing tools? 36