Clarity in Voice Performance Measurements with Apdex Session

Clarity in Voice Performance Measurements with Apdex Session 45 A CMG International Conference San Diego, California December 5, 2007 Ken Harker Consultant Service Level Management Keynote Systems, Inc. ken. harker@keynote. com (512) 335 -7172

Keynote Overview Largest global on demand test & measurement § Founded 1995, public in 1999 (NASDAQ: KEYN) § 2, 600 corporate customers § 170, 000 member panel representative of end users § “Software On Demand” Vo. IP and Streaming - Accelerate services rollout and ensure service quality § Mobile - Improve mobile data services, applications and content delivery § Web - Improve online experience and site performance § 40+ distributed testing and monitoring services § Unique On Demand T&M Solutions Unique Capabilities The “Keynote Network” § 240 locations § 160 metropolitan areas worldwide § 2, 400 measurement computers deployed worldwide § More than 160 million online performance measurements taken daily Slide 2

Some of What We Measure § Audio Responsiveness § Avoid conversational disruption § One-way delay in milliseconds § Audio Clarity § How good does it sound? § ITU-T P. 862 PESQ standard § PESQ has a high correlation to human perception § MOS values are well understood in the industry Slide 3

Why the Average is not Enough § Today there is no ‘standard’ way of aggregating measurements for reporting § § Arithmetic means and medians Geometric means and medians Arithmetic and geometric standard deviation Percentages over a threshold § Averaging loses outliers, which may be important § Example, 500 calls, 450 at MOS 4. 2, 50 at MOS 3. 0 § Average MOS for 500 calls = 4. 08 § SLA Threshold of 4. 0 is satisfied, but 10% of users unhappy § Standard deviation is difficult for many to understand use Slide 4

The End User is Everything § We need to translate service level performance data into a quantifiable measurement of end-user perception! § If the Average of MOS over the past month is X, and the standard deviation is Y, will the end user be happy? Unhappy? How do I know? Slide 5

Apdex Solves Two Problems at Once § Apdex is better at capturing the distribution of the end user experience § Two data sets with identical average values can have different Apdex values § Apdex is better at examining performance data against industry standards of end user satisfaction § We can set the Satisfied and Tolerating thresholds to be industry-standard values § A unit-normalized index is accessible to those whose lives do not revolve around the technical infrastructure Slide 6

Industry Standards § MOS: Telecommunications Industry Association Technical Services Bulletin 116 “Voice Quality Recommendations for IP Telephony” § Audio Delay: International Telecommunications Union standard ITU-T G. 114 “One-way transmission time” Slide 7

Business Problem § Vo. IP service provider § Nationwide USA customer base § Internal monitoring tools lack visibility into the open Internet and other networks § Needs to understand the service level performance of their Vo. IP product as experienced by the end user § Does not know where their product performance places relative to the competition Slide 8

Keynote Monitoring Solution § Four agents in two cities § DSL and Cable network connections § Long distance and local calls § Measure five competitors from the same locations § Monthly benchmark reporting § Portal access to current measurement data and visualization tools Slide 9

Comparisons Using Averages and Thresholds § Six voice services providers compared from the same measurement locations § Providers appear to fall into two performance groups in Average Audio Delay and two other performance groups in Average MOS Slide 10

Apdex applied to Audio Delay § Providers C and E still have the best performance, as the previous table illustrated § Providers A and B are now clearly differentiated from providers D and F Slide 11

Apdex applied to MOS § In our previous approach to reporting, Providers C and F were in the top performance tier § Provider F is meeting the Apdex target for end user satisfaction, but has many Frustrated measurements Slide 12

Market Breakdowns § Both local calling routes appear to have similar Audio Delay performance, as do both long distance routes § The Average MOS on all routes are very similar to one another – just shy of a 4. 0 average Slide 13

Apdex applied to Audio Delay § Now, call performance from routes that originate from New York look very much alike § SFO-NY is a much worse end user experience than NY -SFO despite the fact that both routes have averages in the 150 -400 ms range Slide 14

Apdex applied to MOS § One of these call routes is not like the others § The SFO-SF call route had an average MOS just onetenth of a point lower than the other routes, but a much lower Apdex Slide 15

Summary Apdex uncovers performance problems Delay Ave Best Middle C E A B F D MOS Apdex C E B A D F Ave Best Middle Worst B E A D F Best C B Best F C Apdex Middle E A D Worst Slide 16

Customer Success § Addition of Apdex enriched the value of monthly reports § Responsiveness Apdex results helped customer identify problems with ATAs using incorrect gateway SBCs § MOS Apdex results helped customer identify problem with SBC settings in San Francisco § Competitive information was valuable in making strategic investment decisions Slide 17

Thank You Ken Harker Consultant, Service Level Management ken. harker@keynote. com