Application Performance Index Apdex Standard CMG International Conference

Application Performance Index (Apdex) Standard CMG International Conference Orlando, Florida December 7, 2005 A program of the IEEE-ISTO Peter Sevcik Net. Forecast, Inc. 955 Emerson Drive Charlottesville, VA 22901 peter@netforecast. com www. netforecast. com 434 249 1310

Outline § § The Apdex Alliance How Apdex Works Getting Started With Apdex Using Apdex Reports © 1990 -2005 Net. Forecast, Inc. , All rights reserved. © 2005, Apdex Alliance, all rights reserved. Slide 2

Alliance Overview § The Apdex Alliance is a program of the IEEE-ISTO § IEEE Industry Standards and Technology Organization § The MISSION of the Apdex Alliance is to develop open standards that define standardized methods to report, benchmark, and track application performance. § The Alliance was formed in October, 2004 § Currently participation from more than 20 companies § Information at www. apdex. org © 2005, Apdex Alliance, all rights reserved. Slide 3

Alliance Operations § Organization § § Executive Director Management Board provide business and policy leadership Technical Working Groups develop specifications and documents IEEE-ISTO administers the finances and membership system § Activities § Research effects of application performance on users & businesses § Specify methodologies for application performance measurement & reporting § Advance IT management processes through the use of Apdex § Accomplishments – Ratified Documents § Application Performance Index (Apdex) Final Technical Specification § Apdex Alliance Product Certification Process © 2005, Apdex Alliance, all rights reserved. Slide 4

Current Members § Contributor Member § § § § Akamai Compuware Coradiant Expand F 5 (Swan Labs) Fine. Ground Juniper Keynote Net. Forecast Netli Net. Qo. S Network Physics Packeteer Symphoniq Wild. Packets © 2005, Apdex Alliance, all rights reserved. § Executive Director § Peter Sevcik, Net. Forecast § Management Board § § § Tom Welch, Compuware Chris Loosley, Keynote Systems Dave Trowbridge, Network Physics Chris Yeh, Symphoniq Scott Haugdahl, Wild. Packets § Advisory Board § § § Peter Christy, Internet Research Group Kerry Field, 3 M IT Mike Pennacchi, Net Protocol Specialists Jeff Sult, INS Michael Kennedy, JP Morgan Chase Doug Mc. Clure, Earth. Link Slide 5

Outline § § The Apdex Alliance How Apdex Works Getting Started With Apdex Using Apdex Reports © 1990 -2005 Net. Forecast, Inc. , All rights reserved. © 2005, Apdex Alliance, all rights reserved. Slide 6

Application Performance Index – Apdex § Problem § § Enterprises are swimming in numbers, but have no insight There is little linkage between IT and the business objectives Enterprises do not know user satisfaction levels Response time numbers are meaningless without context § Solution § Apdex is a numerical measure of user satisfaction with the performance of enterprise applications § Converts many measurements into one number § Uniform 0 -1 scale, 0 = no users satisfied, 1 = all users satisfied § It is a comparable metric across all applications © 2005, Apdex Alliance, all rights reserved. Slide 7

Deconstructing Application Transactions Session = Period of time that a user is “connected” to an application End or suspend the application Start the application Process = A group of user interactions that accomplish a goal Idle Get new email, add an employee, check on inventory status, etc. Task = Each interaction with the application during the session Type or choose User waits Enter or click User reads or thinks System responds © 2005, Apdex Alliance, all rights reserved. Type Enter or click Wait Read System responds Slide 8

Deconstructing Application Transactions (con’t) Wait Turn = Each application client-and-server software interaction needed to generate a system response Protocol = Each TCP Open, ACK, retransmission, etc, required to operate a Turn and move Payload Packet = Each packet as seen on the wire in support of the above © 2005, Apdex Alliance, all rights reserved. Slide 9

The Task Defined § Task response time is the elapsed time required for an application system to respond to a human user input such that the user can effectively proceed with the process they are trying to accomplish § Time when the user is waiting in order to proceed § User feels the responsiveness of the application § Long Task time makes the user less productive § The Task is what a user can time with a stopwatch © 2005, Apdex Alliance, all rights reserved. Slide 10

How Users View Application Task Performance § Satisfied § User maintains concentration § Performance is not a factor in the user experience § Time limit threshold is unknowingly set by users and is consistent § Tolerating § Concentration is impaired § Performance is now a factor in the user experience § User will notice how long it is taking § Frustrated § Performance is typically called unacceptable § Casual user may abandon the process § Production user is very likely to stop working © 2005, Apdex Alliance, all rights reserved. Slide 11

How Apdex Works § Start with a sufficient number of Task measurement samples § Target response time “T” defines the satisfied zone (0 -T sec) § T is shown as a subscript of all Apdex values (for example 0. 80 T ) § Count the number of samples within three performance zones § Satisfied, Tolerating, Frustrated Given Target response time T and Sufficient response time measurement samples Then Tolerating count Satisfied count + 2 Apdex. T = Total samples Note Frustrated samples are not in numerator but are counted in total samples Index 0 = Failure; 1 = Perfection (all users satisfied) © 2005, Apdex Alliance, all rights reserved. Slide 12

Apdex Ratings Excellent 1. 00 T 0. 94 T Good 0. 85 T Fair 0. 70 T Poor 0. 50 T Unacceptable (Unax) 0. 00 T © 2005, Apdex Alliance, all rights reserved. Slide 13

Putting it All Together 1. Define T for the application T = the application target time (threshold between satisfied and tolerating users). F = threshold between tolerating and frustrated users is calculated (F = 4 T). Existing Task Response Time 2. Define a Report Group (details available are tool dependent). Measurement 3. Extract data set from existing measurements for Report Group. Samples 4. Count the number of samples in three performance zones. 1. 00 T 5. Calculate the Apdex formula. Report Group: 0. 94 T 6. Display Apdex result (T is always shown as part of the result). Excellent Application 0. 85 T User Group Good Time Period 2 3 Fair Apdex. T= Tolerating 1 T Satisfied © 2005, Apdex Alliance, all rights reserved. Tolerating 2 Total samples Satisfied + 0. 50 T 6 Unacceptable 4 F Poor 5 Frustrated 0. 70 T 0. 00 T Slide 14

Adpex Examples On-line stock trading company Within the United States 0. 793 Outside the United States Fair 3 Unax 3 Poor 3 Fair 3 0. 923 Good 3 0. 714 Fair 4 Good 4 Excel 4 0. 433 Retirement funds manager by their customers’ ISP SBC (many dial-up users) 0. 623 Sprint (mix of access types) 0. 723 MCI (only corporate broadband access) Supply chain management system Original system across the United States Adding content compression 0. 904 Adding transparent turns reduction © 2005, Apdex Alliance, all rights reserved. 0. 944 Slide 15

Outline § § The Apdex Alliance How Apdex Works Getting Started With Apdex Using Apdex Reports © 1990 -2005 Net. Forecast, Inc. , All rights reserved. © 2005, Apdex Alliance, all rights reserved. Slide 16

Get Started Now! § You just learned the basic process and formula § Start with a spreadsheet using your existing data § The standard is open § You can learn all the details at www. apdex. org § The power of Apdex § Apdex transcends applications and organizations § The first uniform way to know how well you are doing Bank and Store deliver equal service Bank better at customer service, Store better at human resources Store Bank Customer Service 0. 918 0. 915 Customer Service 0. 918 0. 965 Human Resources 0. 916 0. 9110 Human Resources 0. 916 0. 7810 © 2005, Apdex Alliance, all rights reserved. Slide 17

Where to Get the Samples Data Center User’s Net Internet Servers LANs Server Software Symphoniq Control/ Accel. Device Expand F 5 Fine. Ground Juniper Packeteer Passive Monitor Network Active Agent Accel. / Delivery Service Compuware Keynote Akamai Coradiant Netli Net. Qo. S Network Physics Wild. Packets Apdex Members © 2005, Apdex Alliance, all rights reserved. Control/ Accel. Device Remote Probe Agent Software Compuware Expand Wild. Packets F 5 Juniper Compuware Packeteer Slide 18

How to Set T § T is set by the Apdex report operator § Only needs to be understood for a handful of tasks users perform very frequently § Method 1: Empirical setting § Measure Apdex at well known locations (e. g. , headquarters) – Measure during the business day and study users at these locations § Determine how well performance is meeting business needs § Find T that returns the proper Apdex for this well studied group § Turn on Apdex reporting for all users without changing T § Method 2: Net. Forecast user behavior model § Count elements of interest to the user – Number of objects, fields, sentences the user will notice/read – One, Few, Several, Many § Rate repetitiveness of the session or process – Very High (many short tasks), High, Low, Very Low (user is browsing, no specific process being performed) § Find T on table in next slide © 2005, Apdex Alliance, all rights reserved. Slide 19

Satisfied-Tolerating Threshold User is satisfied if task completes by T seconds Number of Elements Viewed Task Repetitiveness 1 2 3 4 Very Low 4 8 12 16 Low 3 6 9 12 High 2 4 6 8 Very High 1 2 3 4 © 2005, Apdex Alliance, all rights reserved. Slide 20

What Affects Apdex § Server performance § Servers can slow down under load § Typically occurs in busy periods of the day § Typically adds time to all samples Time Of Day § Network performance § Distance between server and user (speed of light) § Network effects (congestion, hop count, re-routing) § Adds time to some samples © 2005, Apdex Alliance, all rights reserved. User Location Slide 21

Outline § § The Apdex Alliance How Apdex Works Getting Started With Apdex Using Apdex Reports © 1990 -2005 Net. Forecast, Inc. , All rights reserved. © 2005, Apdex Alliance, all rights reserved. Slide 22

The Effect of the Long Tail on User Performance Probability of Experiencing the Time 16% North American broadband users accessing a San Francisco business site 14% 12% Major e. Commerce site ($4 B annual on-line sales) Apdex = 0. 7310 = Fair 52% Satisfied 10% 8% 6% 4% 42% Tolerating 2% 0% 0 6% Frustrated 20 40 60 80 100 Load Time of a Typical Business Page (sec) © 2005, Apdex Alliance, all rights reserved. 120 Slide 23

Apdex Daily Summary Report Time of Day The e. Commerce Application Midnight-6 AM 6 AM-5 PM 5 PM-9 PM 9 PM-Midnight User Location Headquarters Western Eastern Campus Region 1. 00 10 0. 95 10 0. 88 10 0. 96 10 0. 74 10 0. 61 10 0. 98 10 0. 88 10 0. 71 10 0. 99 10 0. 93 10 0. 81 10 Overall Apdex for this day = 0. 7310 Note: Overall Apdex is weighted by the number of user sessions in each cell © 2005, Apdex Alliance, all rights reserved. Slide 24

Aligning IT to the Business: 1 – Rank The Applications Poor Fair Good Exc. High Unacceptable Customer Orders Business Rank Call Center CRM Product Design Factory System Inventory Database Payroll System Low HR Portal Corporate e. Mail 0. 10 © 2005, Apdex Alliance, all rights reserved. 0. 20 0. 30 0. 40 0. 50 0. 60 Apdex. T 0. 70 0. 80 0. 90 Slide 25 1. 00

Aligning IT to the Business: 2 – Generate Apdex Report Poor Fair Good Exc. High Unacceptable Customer Orders Business Rank Call Center CRM Not Aligned Product Design Factory System Inventory Database Payroll System Low HR Portal Corporate e. Mail 0. 10 © 2005, Apdex Alliance, all rights reserved. 0. 20 0. 30 0. 40 0. 50 0. 60 Apdex. T 0. 70 0. 80 0. 90 Slide 26 1. 00

Aligning IT to the Business: 3 – Apdex Report After Changes Poor Fair Good Exc. High Unacceptable Customer Orders Business Rank Call Center CRM Product Design Factory System Inventory Database Payroll System Low HR Portal Aligned Corporate e. Mail 0. 10 © 2005, Apdex Alliance, all rights reserved. 0. 20 0. 30 0. 40 0. 50 0. 60 Apdex. T 0. 70 0. 80 0. 90 Slide 27 1. 00

Thank You More information is available at: Articles and reports on performance measurement, analysis, and management are available at www. netforecast. com Current status and news about Apdex and joining the Apdex Alliance is at www. apdex. org A program of the IEEE-ISTO