Metrics for MMP Development and Operations Lessons Learned

Metrics for MMP Development and Operations Lessons Learned: The Sims Online Larry Mellon GDC, Spring 2004

Metrics: Catch-22 Useful Careful… Hard Data GI / GO Optimization Tool Expensive

Importance of Metrics is Relative Lord Kelvin Measure Everything Mark Twain Measure “Just Enough”

Invest No More Than You Need Kelvin $$$ Twain MMP Complexity Scale Precision $

MMP: Measurement Focal Points Operational Costs Infrastructure Player Actions Economy

Similar Use Case: Casinos (Harrah’s “Total Reward”) Unified Player Action DB Casino Table ///Machine Table Machine Track every Player Action Player

Highly Profitable, Highly Popular Unified Player Action DB Analyze: Profit (per Casino, per Player) Patterns of Play Modify: Casino Operations Player Awards Program “This is one of the best investments that we have ever made as a corporation and will prove to forge key new business strategies and opportunities in the future. “ John Boushy (Harrah's CIO, 2000)

TSO: Live Monitors, Summary Views Designers Community Managers Engineers Player Actions & Persistent Data Operations Server Reactions Metrics System Embedded Profiler (Server Side) Automated Report Generators

Outline Background: Metrics & MMP Implementation Overview Metrics in TSO Applications & Sample Charts Wrapup Lessons Learned Conclusions Questions

Implementation: Driving Requirements Low overhead Common Infrastructure Ease of use

Esper Architecture User Live Server CPUs Process esper. Probe esper. Log

Event-level Sampling, Aggregated Reporting esper. Probes t 1 t 2 esper. View t 3 Min, Max, Av, Count esper. Log t 4 t 5 esper. Store DBImporter esper. Fetch

Esper Probes • Self-organizing: “class” hierarchy • Data driven: new probes and/or new game content immediately visible on web • Example: ESPER_PROBE – (“Object. interaction. %s”, chair->picked) – (“Object. interaction. puppet. %s”, self->picked) • Human-readable intermediate files

Esper. View: Web-Driven Presentation Daily Reports Report Generator Graph Caching & Archiving Filtering & Meta Data

Esper. View: Hierarchical Presentation Process-Level Collection Server Cluster Process Class Process Instance

Outline Background: Metrics & MMP Implementation Overview Metrics in TSO Applications & Sample Charts Wrapup Lessons Learned Conclusions Questions

Applications of Metrics Load Testing (Realistic Inputs) Beta Testing & Live Operations (Game Tuning, Community Management) Load Testing & Live Operations (Server Performance)

Load Testing: “Monkey See / Monkey Do” Sim Actions (Player Controlled) Sim Actions (Script Controlled) Live Beta Testers Alpha. Ville Servers Test Servers

Applications of Metrics Load Testing: Realistic Inputs Beta Testing & Live Operations: Tuning/Mngmnt Load Testing & Live Operations: Server Performance

History Make Friends, Shake Hands beats out Give Money / Get Money Least Used: Disco Dancing Meta Data

Top: TD Dance, Woohoo Bottom: Dance

Players per Lot 0 to 70 players: < 2 / lot 70 to 400 players: > 3 / lot

Top: Metrics Bug (sorta) Next: Garden Gnome, Toilet Bottom: Buffet Table

Beta: num. Players by num. RMates 1. Most have one Roommate 2. Hard-core players: 8

Highest 1. Using Skill Objects 2. Using Pizza Maker 3. Selling Objects Lowest 1. Visitor Bonus 2. New Avatar 3. Object “Rentals”

Economy: Detailed View 1. Using Objects = $$$ 2. Visitor Bonus == $

Visitor Bonus: Who Makes Money? 1. Most getting no V. Bonus 2. Hard-core players: $$

4 of top 5: windows? ?

House Categories (Beta Test) 1. Not a well-used feature 2. Shopping least of all

Community Management Community Actions & Trends Influencing Player Activity Free Content Tracking Problem Players

Marketing In-Game Brand Exposure Special Events Press Release Teasers

NYEve: Kiss Count Esper Cities All Cities (extrapolated) ====================== New Year's Kiss 32, 560 271, 333 Be Kissed Hotly 7, 674 63, 950 Be Kissed 5, 658 47, 150 Be Kissed Sweetly 2, 967 24, 725 Blow a Kiss 1, 639 13, 658 Be Kissed Hello 1, 161 9, 675 Have Hand Kissed 415 3, 458 ====================== Total 52, 074 433, 949

Applications of Metrics Load Testing: Realistic Inputs Beta Testing & Live Operations: Tuning/Mngmnt Load Testing & Live Operations: Server Performance

DB byte count oscillates out of control

A single DB Request is clearly at fault

Most-Used DB Queries (unfiltered) 11, 000 level Queries need attention, and drown out others

DB Queries (Filtered) Filters on 11, 000 level Queries show patterns of 7, 000 level Queries

Incoming & Outgoing Packets

Outline Background: Metrics & MMP Implementation Overview Metrics in TSO Applications & Sample Charts Wrapup Lessons Learned Conclusions Questions

Lessons Learned • • • Implement early Ownership, senior engineers Aggregated probes vs event-level tracking Automation: collect / summarize / alarms “There can be only one”…

Conclusion: Very Useful! Game Design Realistic Load Testing Engine Fixes, Optimization Data Mining On Players: Untapped Gold… Server Cost, Launch Timing Critical Feature: Accessibility

Questions Slides available @ www. maggotranch. com/MMP