1 Test Learn as a Tool

1

Test & Learn as a Tool to Optimize Call Center Performance Stephanie Denny SVP & Inbound Channel Sales Manager Bank of America Card Services

Test and Learn: The Practical Use of a Six Sigma Tool • Opportunity: Optimize the sales process in our Credit Card Customer Service call centers • Challenge: Use the same “test and learn” discipline in the inbound call channel as we do in direct mail marketing 3

Why Model After Direct Mail? • Direct mail measures everything – Offer, message, creative and response rates – Uses a Champion-Challenger approach to continually improve response rates • It uses a disciplined process of measurement from beginning to end 4

Our Challenge • Existing call center desktops are designed for customer service • Sales engine -- limited measurement capabilities • We needed a different way to test, separate from a production environment 5

Enter Design of Experiment • What is DOE? – An engineering tool, originally used in manufacturing environments – It’s a way to efficiently test many variables within a process, and still get statistically valid results – DOE tests many variables, with the objective to reduce variability • So how could we use it to build an effective sales process? 6

We Used DOE to Design Sales Tests • DOE is a way to test many variables, or attributes, at the same time • Within each attribute, there may be different variations, or levels, to be tested • For this experiment we have 23 = 8 combinations Levels 7 Variables

Start with a Hypothesis • Examples: – The words we use to offer a balance transfer affect the customer’s response – If Contractual Rate balance transfer offers are more profitable, we should offer them first – If we give a rebuttal, we will get a better response if we use the words “valued customer” • These are all variable factors within a single customer interaction. How can we efficiently test all of them with confidence in the results? 8

Setting Up a Test Design • Below are all the possible combinations of the variables in the experiment: • Each variable is tested two or more times to see whether or not it makes a difference in the results 9

Six Sigma Talk These are all the Xs =Y (Response Rate) • Design of Experiment looks at the cause-and-effect relationship between different variables, or Xs (inputs) • The output is Y 10

Measurement of the Xs • DOE enables predictability and optimization • Our Information Analysis and Modeling team can build an equation to predict a response, based on using a combination of the winning variables 11

And the Winner Is. . . Results: Balance Transfer Dialogue Test For demonstration purposes only – these were not our actual response rates. • With a statistically valid sample size, we can be confident we have a winning message 12

So How Do We Conduct Our Tests in the Call Centers? • We engaged a vendor with Web-based software designed for this purpose – Capable of managing complex experimental designs – Manages many concurrent tests – Measures each step of the process independently – Dynamic scripting capability – Next-day reporting enables us to quickly identify trends 13

We Call Our Experiment “Test & Learn” • Rapid-cycle testing: many small “test pilots” running concurrently • Randomization of components to ensure valid tests – We measure all parts of the sales process – We work with our information analysis group to determine valid sample sizes for each offer – We continually challenge previous results with new tests for further improvement • Objective: Predict and control the optimal settings for the sales process 14

What is Test & Learn? • Our Test Lab is a team of Customer Service associates, the “Test Pilots” – They have opted to be part of this pilot – Some of their scorecard metrics are different • No emphasis on call handle time • Qualified Good Contacts (QGC) rates are measured (offers made) • Their primary metric is for adherence to the tests 15

What Does It Look Like? All inbound credit card calls enter VRU Accounts flagged as marketable route to Test Lab first Small percent of transactions in Test Lab All other accounts have normal routing Normal service routing Test Lab call overflow routes back to general servicing 16

Test & Learn as a Performance Measurement Tool • Our Test Lab has enabled us to measure associate sales performance • We could then monitor and coach efficiently Conversion Rate • We combined two metrics to identify the best and worst performers Conversion Rates are measured as the ratio of sales to offers made (QGCs) VP Above Average RC RCH AR QP YV JH KB Avg. = 2. 5% DR SB NP NM • Imagine the power of applying this discipline to other Customer Service processes VV DM CB Below Average QGC Rate 17 GM VR ET SW CC

Test & Learn Uses Six Sigma Discipline Define Measure Analyze Improve Control Design Experiment Measure Each Test Analyze Results Deploy Learnings Monitor Results • Test & Learn is a continuum in the DMAIC process • First you learn what to do –Identify best offers –Develop optimal environment • Then you deploy and control • Then the cycle begins again to enhance the process 18

Key Learnings • Know Your Metrics: they drive everything – Metrics are based on the objective of your tests – Your metrics may be something other than profitability (such as customer retention rates) • Your guesses are often wrong – use statistically valid data to make decisions • It’s much less costly to fail in a test environment than in a production environment 19

Consider the Possibilities • Test & Learn can be a powerful tool to improve many Customer Service processes • Control of customer dialogue reduces variability • Ideas for tests will happen as more associates are exposed to the process – ask for their ideas • Test & Learn will become an important part of the business-as-usual process 20

Contact Information Stephanie Denny SVP: Consumer Card Inbound Channel Sales Phone: 602. 597. 5656 Fax: 602. 597. 4725 Email: stephanie. a. denny@bankofamerica. com Bank of America Card Services AZ 9 -502 -01 -06 1825 E. Buckeye Road Phoenix, AZ 85034 21