Automated Testing: Better, Cheaper, Faster, For Everything Larry Mellon, Steve Keller Austin Game Conference Sept, 2004
What Is A MMP Automated Testing System? n n Push-button ability to run large-scale, repeatable tests Cost n n Hardware / Software Human resources Process changes Benefit n n n Accurate, repeatable measurable tests during development and operations Stable software, faster, measurable progress Base key decisions on fact, not opinion
MMP Requires A Strong Commitment To Testing System complexity, non-determinism, scale n Tests provide hard data in a confusing sea of possibilities n Increase comfort and confidence of entire team n n Tools augment your team’s ability to do their jobs n Find problems faster n Measure / change / measure: repeat as necessary n Production / exec teams: come to depend on this data to a high degree
How To Get There n Plan for testing early Non-trivial system n Architectural implications n Make sure the entire team is on board n n Be willing to devote time and money
Automation: Architecture Startup & Control Repeatable, Sync’ed Test Inputs System Under Test Manager Test Selection/Setup Control N Clients RT probes Collection & Analysis System Under Test Scripted Test Clients Report Managers Emulated User Play Sessions Multi-client synchronization Raw Data Collection Aggregation / Summarization Alarm Triggers
Outline Overview: Automated Testing Definition, Value, High-Level Approach Applying Automated Testing Mechanics, Applications Process Shifts: Stability, Scale & Metrics Implementation & Key Risks Summary & Questions
Scripted Test Clients n Scripts are emulated play sessions: just like somebody plays the game n Command steps: what the player does to the game n Validation steps: what the game should do in response
Scripts Tailored To Each Test Application Unit testing: 1 feature = 1 script n Load testing: Representative play session n n The average Joe, times thousands Shipping quality: corner cases, feature completeness n Integration: test code changes for catastrophic failures n
“Bread Crumbs”: Aggregated Instrumentation Flags Trouble Spots Server Crash
Quickly Find Trouble Spots DB byte count oscillates out of control
Drill Down For Details A single DB Request is clearly at fault
Process Shift: Applying Automation to Development Earlier Tools Investment Equals More Gain Amount of work done Not Good Enough Project Start MMP Developer Efficiency Strong test support Weak test support Time Target Launch
Process Shifts: Automated Testing Can Change The Shape Of The Development Progress Curve Stability Keep Developers moving forward, not bailing water Scale Focus Developers on key, measurable roadblocks
Process Shift: Measurable Targets, Projected Trend Lines Target Complete Core Functionality Tests, Any Feature (e. g. # clients) Time First Passing Test Now Any Time (e. g. Alpha) Actionable progress metrics, early enough to react
Stability Analysis: What Brings Down The Team? Test Case: Can an Avatar Sit in a Chair? use_object () buy_object () enter_house () buy_house () create_avatar () login () Failures on the Critical Path block access to much of the game. Worse, unreliable failures…
Impact On Others
Pre-Checkin Regression: don’t let broken code into the Critical Path.
Monkey Test: Enter. Lot
Non-Deterministic Failures
Stability Via Monkey Tests Continual Repetition of Critical Path Unit Tests Code Repository Compilers Reference Servers
Process Shift: Comb Filter Testing Sniff Test, Monkey Tests - Fast to run - Catch major errors - Keeps coders working Smoke Test, Server Sniff Full Feature Regression, Full Load Test - Is the game playable? - Do all test suites pass? - Are the servers stable under a light load? under peak load conditions? - Do all key features work? $ New code ready For checkin Full system build $$$ $$ Promotable to full testing Promotable to paying customers • Cheap tests to catch gross errors early in the pipeline • More expensive tests only run on known functional builds
Process Shift: Who Tests What? n Automation: simple tasks (repetitive or large-scale) n n Manual: judgment / innovative tasks n n Load @ scale Workflow (information management) Full weapon damage assessment, broad, shallow feature coverage Visuals, playability, creative bug hunting Combined n n n Tier 1 / Tier 2: automation flags potential errors, manual investigates Within a single test: automation snapshots key game states, manual evaluates results Augmented / accelerated: complex build steps, …
Process Shift: Load Testing (Before Paying Customers Show Up) Expose issues that only occur at scale Establish hardware requirements Establish play is acceptable @ scale
Resource Metrics Load Testing Team Debugging Data Client Metrics Load Control Rig Test Test Client Client Test Client Test Driver CPU Game Traffic System Monitors Server Cluster Internal Probes
Client-Server Comparison
Highly Accurate Load Testing: “Monkey See / Monkey Do” Sim Actions (Player Controlled) Sim Actions (Script Controlled) Live Beta Testers Alpha. Ville Servers Test Servers
Outline Overview: Automated Testing Definition, Value, High-Level Approach Applying Automated Testing Mechanics, Applications Process Shifts: Stability, Scale & Metrics Implementation & Key Risks Summary & Questions
Data Driven Test Client Load Regression Reusable Scripts & Data Single API Test Client Single API Key Game States Script-Specific Logs & Metrics Pass/Fail Responsiveness
Scripted Players: Implementation Test Client Game Client Script Engine Game GUI State Commands Presentation Layer Client-Side Game Logic
What Level To Test At? Mouse Clicks Game Client View Presentation Layer Logic Regression: Too Brittle (UI&pixel shift) Load: Too Bulky
What Level To Test At? Game Client View Internal Events Presentation Layer Logic Regression & Load: Too Brittle (Churn Rate vs Logic & Data)
Automation Scripts == QA Tester Scripts Basic gameplay changes less frequently than UI or protocol implementations. View Logic Null. View Client Presentation Layer Chat Route Avatar … Enter Lot Use Object
Common Gotchas n Setting the Test bar too high, too early n n n Non-determinism n n n Race conditions, dirty buffers/process. State, … Developers test with a single client against a single server: no chance to expose race conditions Not designing for testability n n n Feature drift == expensive test maintenance Code is built incrementally: reporting failures nobody is prepared to deal with wastes everybody’s time Testability is an end requirement Retrofitting is expensive No senior engineering committed to the testing problem
Outline Overview: Automated Testing Definition, Value, High-Level Approach Applying Automated Testing Mechanics, Applications Process Shifts: Stability & Scale Implementation & Key Risks Summary & Questions
Summary: Mechanics & Implications n n n Scripted test clients and instrumented code rock! Collection, aggregation and display of test data is vital in making decisions on a day to day basis Lessen the panic n n Scale&Break is a very clarifying experience Stable code&servers in development greatly ease the pain of building a MMP game Hard data (not opinion) is both illuminating and calming Long-term operations: testing is a recurring cost
Summary: Process n Integrate automated testing at all levels n n Don’t just throw testing over the wall to QA monsters Use automation to speed & focus development Stability: Sniff Test, Monkey Tests n Scale: Load Test n
Tabula Rasa Pre. Checkin Sniff. Test Keep Mainline Working Hourly Monkey Tests Baseline for Developers Dedicated Tools Group Executive Support Load Test: Early & Often Easy to Use == Used Radical Shifts in Process Break It Before Live Distribute Test Development & Ownership Across Full Team
Cautionary Tales Flexible Game Development Requires Flexible Tests Signal To Noise Ratio Defects & Variance In The Testing System
Questions (15 Minutes) Overview: Automated Testing Definition, Value, High-Level Approach Applying Automated Testing Mechanics, Applications Process Shifts: Stability, Scale & Metrics Implementation & Key Risks Slides online @ www. maggotranch. com/MMP
Скачать презентацию Automated Testing Better Cheaper Faster For Everything Larry
2cda0b541ed5b6cebaaf46e4dfd935b2.ppt