web Methods Testing at Corporate Express Sonam Chauhan

web. Methods Testing at Corporate Express Sonam Chauhan Corporate Express sonam. chauhan@ce. com. au

Overview • Introduction • Change Process • General Requirements for Testing • Old Testing Process • New Testing Process • Implementation Tips • Future Steps and Trends • References

Introduction • Constant pressure for change • Changes need to be tested for bugs • Testing has business impact – – Good testing costs money Bad testing costs more money! • Need to find the “Golden Mean”

Introduction • web. Methods issues that relate to testing • Dev/Test/UAT/Live Builds Much Variation » • Source Code Management Rudimentary » • Support planned for interfacing to SCMs Complex Change Deployments Risky » • Due to IS’s multiple packages feature Partially addressed by w. MDeployer Testing Support Absent » w. M aren’t adding it

Change Process Requirements Specification Requirements Definition Development Changes Validated Changes Change Deployment (Dev Test UAT Live) Review and Testing

General Requirements for Testing • Like development, testing is an engineering skill – Allocate limited resources for max. benefit. • Black-box Integration testing for Regressions Best return for your testing buck – – High level tests simulate live use Tests can be external to code • – Easy to retrofit to existing codebase Regression are the costliest bugs • – All changes must “first do no harm” This is not Unit Testing • • Unit tests can’t assure system functionality This is integration/system-level testing

General Requirements for Testing • Developers can be testers! – Commitment to quality • • – • Automate testing Automated Regression Testing Harness – Without automation, testing is tedious work • – – – Automation increases productivity and efficiency Permanently add hard-won insight to corporate memory • • Make test development part of development process Make reviews of the test code part of code review process And replay it on demand Essential for managing change in complex systems Active testing catches problems early Load testing often very important – Some bugs can only be "teased out" under load.

Old Testing Process • Implemented Circa 2001 • Mostly manual • Repository of test cases • Test case input submitted to system being tested • System output compared to stored test case outputs and “Diff” generated (image at right) • “Diff” manually inspected and signed off

Old Testing Process • Advantages – Thorough Inspection of Changes • A human eyeballs all differences • Disadvantages – Expensive • Due to tedium, testing is only done when required – Not Scalable • Testing time increases significantly as platform grows more complex – Limited Coverage • No load testing • Fixed issues are not re-tested – Errors • Quality of QA suffers as most testing is manual and tedious

New Testing Process The ‘metatest’ Testing Architecture • • Implemented Circa 2004 Drivers: – – • Increasing Complexity Need for Automation ‘metatest’ Testing Architecture – Perl command line engine – Open source components – Controls test engines that execute parameterized test scripts – Supports multiple test engines, test parallelization (in addition to threading within tests), test retargeting and test load variation – Test scripts execute sequence of steps, with assertions made at end of each step. – Entire test runtime and test script database in SCM system (CVS) • Effort: 1 man-months (excluding test scripts) with one developer (me) • Enhanced over a year

New Testing Process Sample Test Script – JMeter Test Engine Load testing part of same test script Documents retrieved from TN database to test document transform assertions • JMeter [4] test scripts are executed by JMeter instances controlled by metatest

New Testing Process • Advantages – Cheap • Open-source software, no new hardware cost • Completely test automation - no tedium – Scalable • No time increase with platform complexity - tests run in parallel – Coverage • Includes load testing • Can run selective tests, or run regression test an entire system in 5 minutes – No Human Errors • Quality of QA improves – New Benefits • Detailed test results, Test Driven Development, Continual testing • Disadvantages – Typically, only specific assertions are automatically proven • All differences not examined – this may be mitigated by using ‘full-document’ assertions regular expressions adjusting for variances

New Testing Process New Benefit – Detailed Test Results • Automatic archival of detailed results • Data useful for: – Load and stress performance validation (eg: image on the right) [5] – Capacity planning – Trend Detection – Analysing software component failure rates • New analytical methods can be applied to stored data in future

New Testing Process New Benefit –Test Driven Development • While not Unit Testing, our approach has several of the benefits of ‘classical TDD’: – – – Tests and code concurrently developed Bugs replicated as tests and fixed Bugfix tests added to test database Simple tests Brief run time for tests Immediate testing feedback

New Testing Process New Benefit – Continual Testing • Continual Testing – Dev/Test/UAT tested nightly – Test & UAT also load tested (Load imposed = 4 x production load) – Results emailed to developer list so they see it first thing in the morning (see image on right) • Benefits – All platforms fully regression and load tested each night – Differential testing – identical tests run on different systems each night. When the results differ, bugs are identified by comparing the two systems. In example at right, failures are only on ‘test’ in ‘load’ testing… – The load imposed in load testing is updated to reflect growth. Thus, bottlenecks are identified before they hit production.

Implementation Tips • Documentation very important – Documentation crucial to getting developer buy-in – ‘metatest’ architecture has 35 pages of user documentation (in addition to documentation for the tests themselves) • When using open source components, expect bugs – – – • Version control everything to do with testing – – – • Interact with developers of the component Contribute if possible Document problems, link to Bugzilla DBs etc Test scripts Even test engines binaries Advantages: Branching/ Merging, new developer can be setup in minutes Leverage existing infrastructure – Most software can simulate endpoints. For eg: WM C 1 On. Ramp can create a Commerce. One envelope from XML • Get a domain expert to write your tests

Future Steps and Trends • Plans to improve the ‘metatest’ architecture – – – Implementing other types of testing Integration with Wm. Deployer Integration with CECM Verification Tool [6] • Other Testing Resources in the WM space – – – Customware’s Wm. Unit [1] Wmusers. com article on testing using JUnit [2] Solstice Software [3]

Future Steps and Trends • web. Methods Testing Support – WM response to group request for testing support: “Currently no plans exist add unit testing or test service generation services to the developer. ” – However, WM’s IDE is moving to the Eclipse platform. This should open up interesting possibilities with Eclipse plug-ins. – Server support: Ideally, on throwing an particular exception, a WM server would save the pipeline in a testcase format for bug replication on development server. (WM tool does this already? )

References [1] Customware website: http: //www. customware. net/ [2] wm. Users. com testing article: http: //www. wmusers. com/ezine/2003 feb_parvanitis_1. shtml [3] Solstice software website: http: //www. solsticesoftware. com/ [4] JMeter website: http: //jakarta. apache. org/jmeter [5] Article on analysis of JMeter logfiles in stress testing: http: //wiki. apache. org/jakarta-jmeter/Log. Analysis [6] CECM change verification tool described in: http: //www. wmusers. com/software/uploads/2820 B 2 B_Change_ Management_Presentation. ppt