The Pain and Gain of Test Automation

The Pain and Gain of Test Automation – the early days Andy Redwood Portman Building Society redwooda@msn. com 1

Portman Group • • 3 rd Largest Building Society in UK 2000+ staff, 142 branches, HO in Bournemouth 2005 Total assets increased by 15% to £ 17. 8 billion 2005 Residential mortgage balances up by 16% to £ 13. 5 billion • 2005 New residential lending up by 18% to £ 4. 3 billion • 2005 £ 237 m of net retail inflows (Society) • 2005 Retail savings balances increased to £ 11. 4 bn 2

Agenda • • 3 Starting position Evaluation Buy-in Solution – first stage People What’s going well? Core issues Where are we going next?

The first 10 days • 35 observations • 43 suggestions • Cross-referenced to risk table • 19 actions agreed with Internal Audit • 18 of 19 completed to date 4

Problem Statement June 2005 • Need to be more ‘merger friendly’; • Need to accommodate a diverse and changing architecture; • Business applications must be resilient to change; • Life-cycle processes are reactive; • Testing takes too long; • Accountability over the testing is weak; • High Defect Rate in later test phases; • High re-work rate; • Testing Skill sets are being questioned; 5

Buy-in • Management • • • 6 • Team dynamic • • • Bigger Picture CIO and executive Mandate for change Budget for tools Resource Strategy Training programme Test management Flexible process Multi-project focus Pragmatics Prioritising Communications

Gaining Control – Defining the End Game Days Define the problem – Decision point - Define the End Game Action Now - Test Leadership, Proactive Risk-focus Quick Wins 3 - 6 months Change Programme Life-cycle changes Revised Testing Strategy (by workshop) Decision point – Automation (Winrunner/Quick Test Pro) Pilot Programme – Champion required Decision point – good to go? Phased Improvement 7 Staged Iterative Improvements By Programme

2006 Test Strategy • 57 page Test Policy • ‘The way we work’ • ‘The way we want others to work with us’ • Test Approach by exception • Supported by Standards and User Guides 8

Testing – Red Flags Insufficient Reporting Lack of early involvement INITIATION DEFINITIONANALYSIS DESIGN BUILD TEST EXECUTION IMPLEMENTATION Manual Data Environment Tests held In Access Test can take months to complete Environment Rebuild time Process is Long & error prone 9 Test does not Target risk Test Execution Restarts Every time Defects are missed Dependencies unknown Limited E 2 E Capability 1 person 1 test Tests are duplicated

June 2005 Team Structure Test Manager Test Coordinator 11 Manual Testers 10 Test coordinator Environment Coordinator

IS Testing Flexible Working Group Business Manager IS Testing Test Manager Mortgage Channel Test Manager BAU Projects Test Manager Environment and Tools Test Manager Investment Projects Test Manager Regulatory Projects Test Manager Broker Channel Flexible Resource Pool – IS, IST, Business Secondments, External Resource 11

Stage 1 Solution • • • Write, publish and sign off the Strategy; Standardised but flexible process; Testing Services function; Accountability and ownership of the test; Flexible test resourcing from professional testing organisations – annual tenders; • Integrated Test management Toolset – Test Director; • Optimised Automated Functional and regression; 12

Flexible Resourcing High Test Management focus Lower Cost Defined Business Role Retained Expertise Hire Testers on demand Risk-based Flexibility High Test Execution focus 1 Test Manager 3 Coordinators 12 Testers Higher Cost 13 Lower Test Execution focus No go zone Lower Test Management focus

Informed Deployments Risk Decision Point deploy now? Law of Diminshing Returns Cumulative Defects Low Detection Rate Test on or implement now Undetermined time saving High Detection Rate Time Requirements 14 Analysis Design Code Test Implementation date

Extended Testing Day Higher Test Volume Higher error detection rate Fewer Testers Production Stability Lower Costs No. of Tests Run Automated Testing In 14 -18 hr day Requirements 15 Manual Testing In 8 hr day Analysis Design Code Test Implementation date

Where are we now? Test Advise in Early stages INITIATION Enterprise Test Strategy DEFINITION Continuity of Standards & Process ANALYSIS DESIGN BUILD Automated Informed Decisions Measures & Metrics TEST IMPLEMENTATION EXECUTION On Demand Just enough Regression Testing Data Environment Test Input Test data Can be Scripted Tests Held in TD 8 16 Tests defined to a multi-project Repository Real Time Prep by many Testers Data Load is Scripted Automatic Environment Sanity Check Environment Performance Monitoring Execution is Unattended n-desktops Executions can be networked

Next Steps • Integrate more across departments • Roll out TD 8 into development – move dynamic defect focus into Unit Testing • Comparable execution toolsets across platforms • Test Process Toolbox – more detail on the process • Sort our environments 17

Thanks for inviting us to your Si. Gi. ST Andy Redwood redwooda@msn. com 18