Business Case for Test Automation S Janardhanan Chief

Business Case for Test Automation S. Janardhanan Chief Technology Officer ITC Infotech India Limited © Company Confidential www. itcinfotech. com

Agenda • Test Automation – Myths and Reality • Return on Investments • Key take outs and summary © Company Confidential www. itcinfotech. com

Test Automation – Myths and Reality © Company Confidential www. itcinfotech. com

Test Automation Myths • Test Automation is simple, every tester can do it – This myth is promoted by the tool sales people. They are trying to promote the following test automation process: • • Record the script Enhance the script by adding functions and data driving Run the scripts Report results – Under the influence of this myth a QA manager can proudly report: All our testers are developing test automation. © Company Confidential www. itcinfotech. com

Reality - Test automation is a software development task • Automation should be designed, developed and tested • You need to have some kind of a programming background to implement test automation. Test Automation is not as complex as C++/C#/Java development. • Test automation standards should be developed • Automated test components are assets that should be treated like application source code, unit tested, integration tested and performance aspects considered. © Company Confidential www. itcinfotech. com

Myth #2 – Commercial test tools are expensive • Under the influence of this myth some companies, especially the small ones: – – Try to develop their own test automation tools Use scripting languages like Perl and Ruby Use shareware test tools Do not consider test automation at all © Company Confidential www. itcinfotech. com

Reality – Commercial tools are cheap • Per seat license for most expensive automation tool is $8 K • This tool will be used for 5 years. • Maintenance/Support fees are 20% of tool cost or $1, 800 per year • The cost of this tool is $8 K/5+$1, 800 = $3, 100 per year • The automation developer cost with overhead is $100 K per year • The cost of this tool is just 3% of the person who uses it, but productivity gain can be very significant © Company Confidential www. itcinfotech. com

Commercial Tool Benefits: • Customer support. Many of the open source tools come and go with little to no support • Most commercial tools are constantly being updated as technologies change • Most commercial tools usually have more functionality (QTP can test various GUI applications: Web, . Net, Java, VB, C/C++, Power. Builder, etc. vs. WATIR – Web only) • Commercial tools usually have a large community of users, which translates into better availability of qualified resources • Commercial tools require less advanced programming • More test automation frameworks are available for commercial tools • Commercial tools are integrated with Test Management tools which make reporting and execution much simpler. © Company Confidential www. itcinfotech. com

Return On Investment (ROI) © Company Confidential www. itcinfotech. com

Classic ROI Calculation ROI = BENEFIT/COST Automation Cost = Price Of HW + Price of SW + Development Cost + Maintenance Cost + Execution Cost Manual Testing Cost = Development Cost + Maintenance Cost + Execution Cost ROI = (Manual Testing Cost - Automation Cost)/Automation Cost Looks right, doesn’t it? © Company Confidential www. itcinfotech. com

Problems with Classic ROI Calculation • You can’t compare Automated Testing and Manual Testing. They are not the same and they provide different information about the AUT. • You can’t compare cost of multiple execution of automated tests vs. manual tests. You would never dream of executing that many test cases manually. © Company Confidential www. itcinfotech. com

Automation Real ROI • ROI value IS NOT the value of Automation vs. Cost of executing these tests manually • Automation ROI value IS the benefit of this type of testing, and it can be: • • • Reducing Time to Market / Execution Time Increased Test Efficiency (Productivity) Increased Test Effectiveness & test coverage Decrease in test defects escaping to production Improved test repeatability © Company Confidential www. itcinfotech. com

Reduced Time to Market • Can get a greater market share • Makes people available to work on other projects • Higher margins, if no competitive products are currently available © Company Confidential www. itcinfotech. com

Productivity and Effectiveness • More testing gets done faster, increasing the odds of finding defects • Defects found early have better chances to be fixed • Manual Testers can concentrate on clever ways to finding defects, instead of typing test inputs and verify output. • About 7% of bug fixes create new bugs, sometimes in already tested parts of the system. With automation you can rerun tests for those modules. This almost never happened when testing done manually. © Company Confidential www. itcinfotech. com

Manual VS Automated Testing Example • There is a suite of manual test cases that have been deemed as candidates for automation. When run manually, the execution can start immediately (ITC = 0 days). It takes the manual testing team seven business days to execute all of the test cases (TEC = 7 days). The Total Testing Time will always be a linear factor of seven days. • If the test cases are to be automated it will take 30 business days to complete (ITC = 30 days). Once the test scripts have been written it takes half of a business day to complete the tests with the necessary hardware resources (TEC = 0. 5 days). Test Cycle Manual Execution Time (days) Automation Execution Time (days) 1 7 30. 5 3 21 31. 5 5 35 32. 5 7 49 33. 5 9 63 34. 5 © Company Confidential www. itcinfotech. com

ROI summary • Each project requires different types of automation - there is no easy formula available to calculate ROI • Performing ROI calculation can help to determine upfront what type of automation, what level of skills, what tools will be required. © Company Confidential www. itcinfotech. com