Test Automation Framework Designs Designing standardized frameworks instead

Test Automation Framework Designs Designing standardized frameworks instead of simple scripting By Martin Lienhard

The Great Divide • There has been a divide separating QA and Dev due in large part by 3 rd party tool vendors and their proprietary test tools • Open source tools have helped significantly to bridge that divide, so that we can work together to create better software • Most QAs have the bigger picture of what they want to test, and how to do it, but have trouble building robust automation frameworks, because they lack the technical skills. • Most Developers focus on testing code in isolation, but lack the testing experience from a system-wide or end-to-end perspective, so they have trouble building a test automation framework that suits the needs of QA.

1 st Phase of Test Progression • Most QA start out creating test scripts by using record & playback tools • Lack conditional logic and looping • Contain static data, UI locators, URLs, etc. • Cannot be executed iteratively • Cannot be reusable modules • Should not be chained together as dependencies • Lack configuration and global properties • Cannot abstract away from the test tool or test runner

Record & Playback Test Scripts

2 nd Phase of Test Progression • QA usually improves their tests by adding data-driven techniques • Create variables and arrays to store data • Create parameterized functions or methods to store data • Add looping to recorded scripts

3 rd Phase of Test Progression • QA usually improves their tests by creating a business function library • Modular, reusable test libraries of functions • Create file readers that pull in parameters from text files, CSV files, etc. • Abstract out the data from the tests by externalizing data • Create parameterized or iterative tests that are data-driven • Abstract out the UI locators from the tests by externalizing UI maps • Create global properties and configuration files • A vast array of disparate data pools and UI maps are created, which usually become unmanageable after a couple of years

4 th Phase of Test Progression • QA usually improves their tests by creating a keyword-driven library: • Create ODBC functions that read data from spreadsheets and databases • Create spreadsheets of keywords that represent business domain language – Low-level keywords describe test operations – High-level keywords describe business language – Intermediate-level keywords map business functions to test operations – Tests become readable by business users and nontechnical QA

…continued • Create spreadsheets of data that drive the tests • Query data from the application database as input and verification points for tests • A hierarchy of abstracted keyword spreadsheets are created • Now non-technical BAs and QAs are creating automated tests • A vast array of disparate keyword spreadsheets are created, which usually become unmanageable after a couple of years • Nearly all 3 rd party vendor test tools live in this area today – Most of these tools use legacy scripting languages, which are not compatible with today’s technology

5 th Phase of Test Progression • QAs become more technical and improve their tests by using object oriented testing techniques: • Objects are designed and created to further modularize and reuse code with very low maintenance • Test code is stored in objects – The page object design is used – Component object designs are used • Test data are stored in objects • Configuration and properties are stored in objects • Calls are made to the application and developer written APIs to make tests “smarter”

…continued • Continuous integration environments are utilized to run tests • QA moves into white box testing to drive quality deeper into the application – UI tests are ran “headless” to make them execute faster • QA expands the test framework to include multiple test tools and test runners – Databases, web services, and APIs are created to reuse code and data across multiple test tools – Test tools and 3 rd party utilities can be upgraded to newer versions with enhanced functionality with minor impact to the test framework

…continued • Developers can now run the tests, and contribute to the overall test suites • More 3 rd party vendors are rewriting their test tools to include object oriented languages to accommodate these technical QAs – Due to the cost of the tools and their proprietary nature, it is difficult to place these tools into a continuous integration environment to share with developers

What is a test automation framework? • A test automation framework is an application that allows you to write a series of tests without worrying about the constraints or limitations of the underlying test tools

Benefits of building a Test Framework • Tool agnostic – Abstracts away low level commands – Utilize many test tools • Test tools such as Selenium, Web. Driver, etc. • Test runner such as unit test frameworks • Other common utilities – Perform multiple levels and types of testing • Functional, regression, load, performance, unit, integration, etc.

…continued • Generic test API – Common codebase and database – Exception and error handling – Modularized, reusable, and maintainable code and data – Standardized test idioms • Test configuration – Configurable test suites – Global test properties

…continued • Automatic test versioning – No need to get a specific version from the source control repository • Always get the latest version – Eliminates deployment to multiple machines for testing different versions of an application • Deploy to a single server machine

…continued • Multi-threading – Run tests in parallel – Runs on test machines in the grid or cloud – Test across multiple environments and application versions simultaneously • Continuous integration – Run in a continuous integration environment – Share tests and collaborate with other teams

Test Framework Architecture

Test Suite • A collection of tests to run • Have test parameters – i. e. : application, environment, version, etc. • Have test groups – i. e. : regression, build, etc.

Test Suite example

Test Configuration • Any configurable test item that is separate from the test suite – i. e. : web site URLs, database URLs, usernames, passwords, lab machine configurations, etc. • Separated from the rest of the framework for information security and regulatory compliance

Test Configuration example

Test Properties • Are thread specific • Have global scope and can be accessed by any class method

Test Properties example

Test Data Repository • Abstracts away changes made only to test data • Data can be stored in databases and files such as XML, JSON, XLS, TXT, CSV, etc. • Data can be retrieved from data sources such as databases, web services, APIs, EDI, etc. • Metadata can be used to describe additional information about the test data such as versions, application names, required fields, invalid formats, etc. • Share data across multiple screens, applications, and tools

Test Data example

Parameterized Test example JUnit 4 Test. NG

Test Parameters • Uses a hash map of key-value pairs – Very useful for both functional and load testing • Hash maps eliminate the need for – Creating fixed method parameters that change frequently – Changing data access objects that change frequently • Hash maps can be easily populated from any data source • Retrieve the test data and process the format for input to various test tools • Handles version differences between test data

Test Parameters example

User Interface (UI) Mapping • Abstracts away changes made only to UI object locators • UI object identifiers can be stored in databases and files such as XML, JSON, XLS, TXT, CSV, etc. • Metadata can be used to describe additional information about the UI objects such as versions, application names, form fields, etc. • Share UI identifiers across multiple screens, applications, and tools

UI Map example

Test Controls • Retrieve the UI locators and process the format for input to various test tools • Handles version differences between UI locators • Develop custom controls to mimic UI objects • Self test verification

Test Control Interface example

Test Control Class example

Test Components • A class object that represents a piece of a form, page, or screen • Groups of common objects that can be found across more than one page • Standardized component verification tests

Test Components example

Test Forms • A class object that represents a piece a form • Groups of components and controls that mimic the UI form • Standardized form verification tests

Test Form Interface example

Test Form Class example

…continued

Test Page or Screen • A class object that represents a web page or screen – commonly referred to as “page objects” • Groups of forms, components, and controls that mimic the UI page or screen • Self test verification

Test Page Class example

Instances when NOT to return a new Page Object for each Page • Submitting a page can return one of many different pages, or versions of pages, which would be represented by many different classes – Page A could return B, C, or D – Page A could return B v 1. 0, B v 2. 0, or Cv 1. 0, C v 2. 0, or D v 1. 0, D v 2. 0, etc. – This means that you would need to do the following: • Create multiple methods with similar names that return different page objects – Page A. submit. B() returns B – Page A. submit. C() returns C

…continued • Return an page object of an inherited type, and then cast the type of the page in the script – Page A. submit() returns page X – Pages B and C inherit from page X – Script checks the type of X and casts to B or C – This can get very ugly in a script… • Submitting a page may initiate an AJAX call, which can rewrite the same page – Page A could display an AJAX (modal) dialog – Page A could display some other AJAX widget – Page A could be rewritten to look like page B

…continued • If the page flow is not what is being tested, then an unexpected page or AJAX call could fail and kill the test – Test case: • We are trying to test the functionality of page Z • The primary page flow is as follows: – A to B, B to C, C to D, and D to Z • The alternative page flows could also correctly occur, but are unpredictable base on many unknown variables: – A to H, H to L, L to M, and M to Z – A to R, R to S, and S to Z – A to X, X to Y, and Y to Z

…continued – What if page A unexpectedly returned page H, R, or X instead of page B? – If page H, R, or X verified itself and asserted a failure, then how would the test ever reach page Z?

Java. Script & HTML DOM • There are many instances when we need the test framework to mimic a human reading the content of a page and then making a decision on what action to perform next – Verifying the list of items on a search results page – Selecting items to purchase from a list – Selecting elements to edit, update, or delete • Scenarios like these usually occur as a result of some previous action, and display content generated dynamically to the user • The test framework may have to query an application database or web service to provide the input to trigger the dynamic page content

…continued • The test framework will then need to interrogate the page to determine if the expected elements, attributes, or content are present, and then make a decision on whether to perform an action on some element • Java. Script can be used to inspect the HTML DOM to find these elements and their attributes and content • Java. Script expressions can be evaluated by the test tool, and then return some result that the test framework can use to make a decision • Checking if an element is present or getting a value will not work on a dynamically generated element

JS DOM example HTML Java. Script

Test Application Container • A class object that represents the application, and contains the major page and component objects by version • It provides the test services available to the test scripts • Is thread specific

Container example Interface Class

Parallel Environment Tracks with Different Build Versions • Parallel development tracks competing to deploy to production • Run automated build verification tests • Run automated functional and regression tests • Run automated user acceptance tests • Run automated tests for production patches and fixes • Run automated sanity tests against production postdeployment • Run automated tests across versions for production rollback strategy

Parallel Environment Tracks with Different Build Versions

Test Scripts • Test scripts can be written using standard unit test frameworks • Classes and methods are written using domain language • Script details and complexity are abstracted to the framework layers • Scripts are shorter and easier to read

Test Script Class

Test Reports and Logs • Reports can be generated from – The unit test framework – External reporting tools – Custom report utilities can be written • Test execution activity can be written to standard logging utilities • Test failures, errors, and exceptions can be written to the reports and logs

Q&A Thank you for your time! Martin Lienhard lienhardtestautomation@gmail. com