Скачать презентацию Workshop on Advances Innovations in Software Testing Скачать презентацию Workshop on Advances Innovations in Software Testing

1d0acf66d093619be74231ecd2dbcaff.ppt

  • Количество слайдов: 43

Workshop on “Advances & Innovations in Software Testing” Importance of Test Automation in System Workshop on “Advances & Innovations in Software Testing” Importance of Test Automation in System Testing Date: Tue, May 8 th ‘ 07 Rel. Q Software © 2007 Venue: Fed. Ex Institute of Technology, Memphis

Test Automation, Tools and Enablers Speaker: Sudheer Mamadapur, PMP Rel. Q Software © 2007 Test Automation, Tools and Enablers Speaker: Sudheer Mamadapur, PMP Rel. Q Software © 2007

Agenda § Introduction § Test Automation § Tools & Enablers § Challenges § About Agenda § Introduction § Test Automation § Tools & Enablers § Challenges § About Rel. Q § Rel. Q’s Experience § Conclusion § Q&A Rel. Q Software © 2007 3

Introduction Rel. Q Software © 2007 Introduction Rel. Q Software © 2007

About the Speaker Sudheer Mamadapur Project Management Professional Test Manager – Rel. Q Software About the Speaker Sudheer Mamadapur Project Management Professional Test Manager – Rel. Q Software Inc. § 8 & half years of experience in Rel. Q focused on Software V&V and Test Automation. § Architected framework for Test Automation across various domains, technologies, automation methodologies and tools. § Roles in Rel. Q have varied from taking Test Automation challenges head-on to performing duties as Test Lead, Onsite/Offshore Coordinator, Test Automation Practice Manager and Test Manager. Rel. Q Software © 2007 5

System Testing • System testing is testing conducted on a complete, integrated system • System Testing • System testing is testing conducted on a complete, integrated system • Types of System Testing: • Functional • Usability • Performance (Load, Stress, Volume) • Compatibility • Security Testing • Smoke Testing • Exploratory/Adhoc Testing • Regression Testing • Reliability Testing • Recovery Testing • Installation • Accessibility Rel. Q Software © 2007 6

Why Test Automation? § Software Testing is an integral, costly and time-consuming activity in Why Test Automation? § Software Testing is an integral, costly and time-consuming activity in the SDLC. § The challenge of the day for IT companies is to improve productivity and reduce the cost of Testing. § Test Automation plays a pivotal role in effective testing and reduction in test effort. § Changes in environment like new patches, service packs etc. also call for testing/test automation. § Keep pace with development activities § A strategic approach for Test Automation using tools, framework and methodologies will improve test coverage in regression cycles and reduce the test effort in subsequent release cycles. “Rapidly changing software is a moving target, but Test Automation can still hit it“ Rel. Q Software © 2007 7

Test Automation Rel. Q Software © 2007 Test Automation Rel. Q Software © 2007

Benefits of Test Automation § Cost Effective and Efficient Process Consistent and Accurate Test Benefits of Test Automation § Cost Effective and Efficient Process Consistent and Accurate Test Cycles Reduces Overall Test Cycle Time Mitigates Risks of Manual Testing Increases Overall Product Quality Early Time to Market Rel. Q Software © 2007 9

What to Automate? Governs the Quality of tests Analyze Design Intellectual Performed once Good What to Automate? Governs the Quality of tests Analyze Design Intellectual Performed once Good to Automate Construct Test Deploy / Maintain Rel. Q Software © 2007 Repeated many times Clerical 10

What to Automate? (contd…) Which & What Test Should be Automated First ? Most What to Automate? (contd…) Which & What Test Should be Automated First ? Most important tests A set of breadth tests Test for the most important functions Tests that are easy to automate Tests that will give the quickest payback Tests that are run more often Alternatively, test candidates that are identified as part of risk-based testing strategy may as well be considered for test automation. Rel. Q Software © 2007 11

Test Automation Lifecycle Tools Processes 6. Maintain 5. Deploy 4. Test Skilled Resources Rel. Test Automation Lifecycle Tools Processes 6. Maintain 5. Deploy 4. Test Skilled Resources Rel. Q Software © 2007 1. Analyze 2. Design 3. Construct Methodologies 12

Rel. Q’s Approach © Rel. Q Software © 2007 13 Rel. Q’s Approach © Rel. Q Software © 2007 13

Tools & Enablers Rel. Q Software © 2007 Tools & Enablers Rel. Q Software © 2007

Tools Mercury Quick Test Professional Win. Runner IBM Rational Robot Functional Tester Borland Silk Tools Mercury Quick Test Professional Win. Runner IBM Rational Robot Functional Tester Borland Silk Test Compuware QA Run Automated. QA Test. Complete Rel. Q Software © 2007 15

Tool Characteristics & Criteria Capability Capacity Compatibility Learnability Operability Rel. Q Software © 2007 Tool Characteristics & Criteria Capability Capacity Compatibility Learnability Operability Rel. Q Software © 2007 16

Enablers Methodologies Data Driven Functional Decomposition Keyword Driven Framework Test Framework Process Framework Rel. Enablers Methodologies Data Driven Functional Decomposition Keyword Driven Framework Test Framework Process Framework Rel. Q Software © 2007 17

Level 1: Basic Automation Rel. Q Software © 2007 Level 1: Basic Automation Rel. Q Software © 2007

Record & Playback Approach: Test scripts are recorded on the initial version of the Record & Playback Approach: Test scripts are recorded on the initial version of the application. The same scripts are executed on the next version. The scripts need some modification for the changes happening on the application during every version of the application. The test scripts repository keeps growing as the application goes through changes. 19

Record & Playback (contd…) An automation tool generates scripts by recording user actions. The Record & Playback (contd…) An automation tool generates scripts by recording user actions. The generated scripts can be played back to reproduce the exact user actions. Advantages: • Less effort for automation • Quick returns • Does not require expertise on tools Limitations: • High dependency on the GUI of AUT • Difficult to maintain the scripts Rel. Q Software © 2007 20

Data-Driven Methodology Approach: Scripts are freed from hardcoded data. Test input and output values Data-Driven Methodology Approach: Scripts are freed from hardcoded data. Test input and output values are read from data files (data pools, ODBC sources, cvs files, Excel files, DAO objects, ADO objects, and such) and are loaded into variables in captured or manually coded scripts. Variables are used for both input values and output verification values. Navigation through the program, reading of the data files, and logging of test status and information are all coded in the test script. Rel. Q Software © 2007 21

Data-Driven Methodology (contd…) Advantages: • Reduces the overall number of scripts needed to cover Data-Driven Methodology (contd…) Advantages: • Reduces the overall number of scripts needed to cover the test cases • Data is separated from script and stored in a data file; easy to maintain • The volume of test data and its combination can be increased for exhaustive testing Limitations: • Application must be available to start scripting • Highly depends on the GUI of AUT • When the GUI design changes, the Automation scripts require change • Maintenance of the test script and data files are very high Rel. Q Software © 2007 22

Level 2: Advanced Methodologies Rel. Q Software © 2007 Level 2: Advanced Methodologies Rel. Q Software © 2007

Functional Decomposition Create small, independent scripts that represent modules, sections, and functions of the Functional Decomposition Create small, independent scripts that represent modules, sections, and functions of the application-under-test. Scripts are taken and combined in a hierarchical fashion to construct larger tests. Use of this methodology yields a higher degree of modularization and add to the overall maintainability of the test scripts. Advantages: • Provides separation between data and scripts • Delivers script reusability • Provides one maintenance point for each functionality or screen Limitations: • Longer implementation time • Implementation team must have programming background Rel. Q Software © 2007 24

Keyword-Driven Methodology Approach: The discrete functional business events that make up any application can Keyword-Driven Methodology Approach: The discrete functional business events that make up any application can be described using a short text description (Keyword) and associated parameter value pairs (arguments). By designing Keywords to describe discrete functional business events, the testers begin to build up a common library of Keywords that can be used to create Keyword test cases. This is really a process of creating a language (Keywords) to describe a sequence of events within the application (Test Case). Rel. Q Software © 2007 25

Keyword-Driven Methodology (contd…) Advantages: Development of Keywords, Action Files can begin before the application Keyword-Driven Methodology (contd…) Advantages: Development of Keywords, Action Files can begin before the application is ready The User can easily create the test scenarios by grouping a set of Keywords in the Action file. The User has multiple options of test case selection for execution. Minimum coding required for addition / modification of test cases. Maintenance is easy, changes in functionality require changes in Action Files only Keywords can be reused for other applications’ testing Limitations: Requires extensive programming background Requires longer implementation time Rel. Q Software © 2007 26

Benefits of Automation Methodology Effort Initial Development Ongoing Maintenance Record/ Replay Data. Driven Modular Benefits of Automation Methodology Effort Initial Development Ongoing Maintenance Record/ Replay Data. Driven Modular Keyword. Driven Methodology Rel. Q Software © 2007 27

Level 3: Framework Rel. Q Software © 2007 Level 3: Framework Rel. Q Software © 2007

Framework-driven automation is an innovative approach to software testing that leverages proprietary automation frameworks Framework-driven automation is an innovative approach to software testing that leverages proprietary automation frameworks and tool sets from leading vendors, supported by additional programming to maximize the benefits of test automation. Types: Test Framework Focus on creating reusable test automation components. Process Framework Focuses on automation of process in addition to the type of test. Hybrid Framework Most effective framework that combines the features of process and test frameworks. Rel. Q Software © 2007 29

Test Framework DESIGN CONSIDERATIONS • Modularity The framework to be modularized in line with Test Framework DESIGN CONSIDERATIONS • Modularity The framework to be modularized in line with the application architecture. • Maintainability The Scripts to be normalized and freed from any kind of data within. Coding Standards to be defined, followed and audited. Functional decomposition method to be used to identify commonly used functions and the same to be kept outside the main script. The configuration files and data files to be maintained in user-friendly format. Rel. Q Software © 2007 30

Test Framework (contd…) • Flexibility The framework to accommodate automation on different interfaces; GUI Test Framework (contd…) • Flexibility The framework to accommodate automation on different interfaces; GUI (Web/CS), Character Based Interface, Mainframe, . . • Scalability Accommodate addition of new test cases to the framework with minimum effort • Extensibility The framework to be extended to accommodate different types of Validations; GUI, Database, File, … • Reusability Functions / scripts to be designed to facilitate reusability across tools & technologies Rel. Q Software © 2007 31

Process Framework § End-to-End Automation • • • § Identification of Test Environment Installation Process Framework § End-to-End Automation • • • § Identification of Test Environment Installation of Build(s) Test Set(s) Selection Test Execution Result Reporting Advantages: • Useful when different tests have the similar test setup steps. • Provide common Interface to Plug-in Multiple types of tests. • Consistent set of commands to execute different tests. Rel. Q Software © 2007 32

Test Automation Challenges Rel. Q Software © 2007 Test Automation Challenges Rel. Q Software © 2007

Challenges § Organization § Poor testing practice Business: Unrealistic expectations Achieving ROI False sense Challenges § Organization § Poor testing practice Business: Unrealistic expectations Achieving ROI False sense of security Application: Frequent changes to the User Interface / Business Rules § Tool: Object Recognition Limitations Support for new technologies (. NET, . . . ) Technical support from tool vendor / user forum Rel. Q Software © 2007 34

About Rel. Q Software © 2007 About Rel. Q Software © 2007

About Rel. Q § Rel. Q Software is an Independent Software Verification & Validation About Rel. Q § Rel. Q Software is an Independent Software Verification & Validation (IV&V) Services Company. Rel. Q provides software testing and related Quality Assurance services to IT Organizations globally, to help them deliver reliable and quality software on time to their customers. § Established in 1998, Rel. Q has been a pioneer in this segment in the Indian IT Industry. § Rel. Q has around 1, 800 person-years of experience in Software Testing across 350+ projects. § Currently, 800+ Rel. Q engineers are working on testing assignments for clients across the globe. Rel. Q Software © 2007 36

Rel. Q and Test Automation § Rel. Q is focusing on Test Automation since Rel. Q and Test Automation § Rel. Q is focusing on Test Automation since 1998. § Rel. Q has experience of more than 1, 200 person-months on Test Automation across domains, technologies and tools. § The knowledge acquired in the projects has helped Rel. Q in defining the methodologies for test automation and improve upon the same over a period of time. § The metrics collected in the projects has helped Rel. Q build reliable models for effort estimation in Test Automation. § Rel. Q’s experience – 75% of the application testing projects required Test Automation! Rel. Q Software © 2007 37

Rel. Q’s Experience in Test Automation Rel. Q Software © 2007 Rel. Q’s Experience in Test Automation Rel. Q Software © 2007

Levels of Test Automation Level 3 ADVANCED METHODOLOGIES: Data-Driven, Keyword-Driven, Hybrid Level 2 Level Levels of Test Automation Level 3 ADVANCED METHODOLOGIES: Data-Driven, Keyword-Driven, Hybrid Level 2 Level 1 Rel. Q Software © 2007 FRAMEWORK: Test / Process / Hybrid: Methodologies, Libraries, Re-usable components BASIC: Record & Playback, Functional Decomposition 39

Statistics Level Type of Applications Effort spent on Automation Tools Used Methodology Level 1 Statistics Level Type of Applications Effort spent on Automation Tools Used Methodology Level 1 Client Server, Web 200 personmonths Rational Robot, Record & Playback, Win. Runner, Functional Silk. Test, Decomposition QTP Level 2 Client Server, Web, Mainframe, Devices, ERP Packages 700 personmonths Rational Robot, Data Driven, Win. Runner, Keyword Driven, Silk. Test, Hybrid QTP, Test Complete Level 3 Multilayered, Complex Applications 300 personmonths QC, BPT, QTP, Silk. Test, Perl, Shell Scripting Rel. Q Software © 2007 Framework, Libraries, Reusable Components 40

Conclusion System Testing is a costly & time-consuming activity Test Automation helps overcome the Conclusion System Testing is a costly & time-consuming activity Test Automation helps overcome the difficulties of manual testing thus reducing the effort and time required for testing. A framework based approach for test automation supports maintainability, extensibility, scalability, … A strategically developed framework based automation suite can even be used for other similar applications/platforms thus maximizing the ROI. Integrated with a test management solution, provides complete solution for automated testing, reporting, test coverage, defect logging, … Rel. Q Software © 2007 41

Q&A Rel. Q Software © 2007 Q&A Rel. Q Software © 2007

End of Presentation Rel. Q Software © 2007 End of Presentation Rel. Q Software © 2007