Скачать презентацию Automated Testing Better Cheaper Faster For Everything Larry Скачать презентацию Automated Testing Better Cheaper Faster For Everything Larry

2cda0b541ed5b6cebaaf46e4dfd935b2.ppt

  • Количество слайдов: 39

Automated Testing: Better, Cheaper, Faster, For Everything Larry Mellon, Steve Keller Austin Game Conference Automated Testing: Better, Cheaper, Faster, For Everything Larry Mellon, Steve Keller Austin Game Conference Sept, 2004

What Is A MMP Automated Testing System? n n Push-button ability to run large-scale, What Is A MMP Automated Testing System? n n Push-button ability to run large-scale, repeatable tests Cost n n Hardware / Software Human resources Process changes Benefit n n n Accurate, repeatable measurable tests during development and operations Stable software, faster, measurable progress Base key decisions on fact, not opinion

MMP Requires A Strong Commitment To Testing System complexity, non-determinism, scale n Tests provide MMP Requires A Strong Commitment To Testing System complexity, non-determinism, scale n Tests provide hard data in a confusing sea of possibilities n Increase comfort and confidence of entire team n n Tools augment your team’s ability to do their jobs n Find problems faster n Measure / change / measure: repeat as necessary n Production / exec teams: come to depend on this data to a high degree

How To Get There n Plan for testing early Non-trivial system n Architectural implications How To Get There n Plan for testing early Non-trivial system n Architectural implications n Make sure the entire team is on board n n Be willing to devote time and money

Automation: Architecture Startup & Control Repeatable, Sync’ed Test Inputs System Under Test Manager Test Automation: Architecture Startup & Control Repeatable, Sync’ed Test Inputs System Under Test Manager Test Selection/Setup Control N Clients RT probes Collection & Analysis System Under Test Scripted Test Clients Report Managers Emulated User Play Sessions Multi-client synchronization Raw Data Collection Aggregation / Summarization Alarm Triggers

Outline Overview: Automated Testing Definition, Value, High-Level Approach Applying Automated Testing Mechanics, Applications Process Outline Overview: Automated Testing Definition, Value, High-Level Approach Applying Automated Testing Mechanics, Applications Process Shifts: Stability, Scale & Metrics Implementation & Key Risks Summary & Questions

Scripted Test Clients n Scripts are emulated play sessions: just like somebody plays the Scripted Test Clients n Scripts are emulated play sessions: just like somebody plays the game n Command steps: what the player does to the game n Validation steps: what the game should do in response

Scripts Tailored To Each Test Application Unit testing: 1 feature = 1 script n Scripts Tailored To Each Test Application Unit testing: 1 feature = 1 script n Load testing: Representative play session n n The average Joe, times thousands Shipping quality: corner cases, feature completeness n Integration: test code changes for catastrophic failures n

“Bread Crumbs”: Aggregated Instrumentation Flags Trouble Spots Server Crash “Bread Crumbs”: Aggregated Instrumentation Flags Trouble Spots Server Crash

Quickly Find Trouble Spots DB byte count oscillates out of control Quickly Find Trouble Spots DB byte count oscillates out of control

Drill Down For Details A single DB Request is clearly at fault Drill Down For Details A single DB Request is clearly at fault

Process Shift: Applying Automation to Development Earlier Tools Investment Equals More Gain Amount of Process Shift: Applying Automation to Development Earlier Tools Investment Equals More Gain Amount of work done Not Good Enough Project Start MMP Developer Efficiency Strong test support Weak test support Time Target Launch

Process Shifts: Automated Testing Can Change The Shape Of The Development Progress Curve Stability Process Shifts: Automated Testing Can Change The Shape Of The Development Progress Curve Stability Keep Developers moving forward, not bailing water Scale Focus Developers on key, measurable roadblocks

Process Shift: Measurable Targets, Projected Trend Lines Target Complete Core Functionality Tests, Any Feature Process Shift: Measurable Targets, Projected Trend Lines Target Complete Core Functionality Tests, Any Feature (e. g. # clients) Time First Passing Test Now Any Time (e. g. Alpha) Actionable progress metrics, early enough to react

Stability Analysis: What Brings Down The Team? Test Case: Can an Avatar Sit in Stability Analysis: What Brings Down The Team? Test Case: Can an Avatar Sit in a Chair? use_object () buy_object () enter_house () buy_house () create_avatar () login () Failures on the Critical Path block access to much of the game. Worse, unreliable failures…

Impact On Others Impact On Others

Pre-Checkin Regression: don’t let broken code into the Critical Path. Pre-Checkin Regression: don’t let broken code into the Critical Path.

Monkey Test: Enter. Lot Monkey Test: Enter. Lot

Non-Deterministic Failures Non-Deterministic Failures

Stability Via Monkey Tests Continual Repetition of Critical Path Unit Tests Code Repository Compilers Stability Via Monkey Tests Continual Repetition of Critical Path Unit Tests Code Repository Compilers Reference Servers

Process Shift: Comb Filter Testing Sniff Test, Monkey Tests - Fast to run - Process Shift: Comb Filter Testing Sniff Test, Monkey Tests - Fast to run - Catch major errors - Keeps coders working Smoke Test, Server Sniff Full Feature Regression, Full Load Test - Is the game playable? - Do all test suites pass? - Are the servers stable under a light load? under peak load conditions? - Do all key features work? $ New code ready For checkin Full system build $$$ $$ Promotable to full testing Promotable to paying customers • Cheap tests to catch gross errors early in the pipeline • More expensive tests only run on known functional builds

Process Shift: Who Tests What? n Automation: simple tasks (repetitive or large-scale) n n Process Shift: Who Tests What? n Automation: simple tasks (repetitive or large-scale) n n Manual: judgment / innovative tasks n n Load @ scale Workflow (information management) Full weapon damage assessment, broad, shallow feature coverage Visuals, playability, creative bug hunting Combined n n n Tier 1 / Tier 2: automation flags potential errors, manual investigates Within a single test: automation snapshots key game states, manual evaluates results Augmented / accelerated: complex build steps, …

Process Shift: Load Testing (Before Paying Customers Show Up) Expose issues that only occur Process Shift: Load Testing (Before Paying Customers Show Up) Expose issues that only occur at scale Establish hardware requirements Establish play is acceptable @ scale

Resource Metrics Load Testing Team Debugging Data Client Metrics Load Control Rig Test Test Resource Metrics Load Testing Team Debugging Data Client Metrics Load Control Rig Test Test Client Client Test Client Test Driver CPU Game Traffic System Monitors Server Cluster Internal Probes

Client-Server Comparison Client-Server Comparison

Highly Accurate Load Testing: “Monkey See / Monkey Do” Sim Actions (Player Controlled) Sim Highly Accurate Load Testing: “Monkey See / Monkey Do” Sim Actions (Player Controlled) Sim Actions (Script Controlled) Live Beta Testers Alpha. Ville Servers Test Servers

Outline Overview: Automated Testing Definition, Value, High-Level Approach Applying Automated Testing Mechanics, Applications Process Outline Overview: Automated Testing Definition, Value, High-Level Approach Applying Automated Testing Mechanics, Applications Process Shifts: Stability, Scale & Metrics Implementation & Key Risks Summary & Questions

Data Driven Test Client Load Regression Reusable Scripts & Data Single API Test Client Data Driven Test Client Load Regression Reusable Scripts & Data Single API Test Client Single API Key Game States Script-Specific Logs & Metrics Pass/Fail Responsiveness

Scripted Players: Implementation Test Client Game Client Script Engine Game GUI State Commands Presentation Scripted Players: Implementation Test Client Game Client Script Engine Game GUI State Commands Presentation Layer Client-Side Game Logic

What Level To Test At? Mouse Clicks Game Client View Presentation Layer Logic Regression: What Level To Test At? Mouse Clicks Game Client View Presentation Layer Logic Regression: Too Brittle (UI&pixel shift) Load: Too Bulky

What Level To Test At? Game Client View Internal Events Presentation Layer Logic Regression What Level To Test At? Game Client View Internal Events Presentation Layer Logic Regression & Load: Too Brittle (Churn Rate vs Logic & Data)

Automation Scripts == QA Tester Scripts Basic gameplay changes less frequently than UI or Automation Scripts == QA Tester Scripts Basic gameplay changes less frequently than UI or protocol implementations. View Logic Null. View Client Presentation Layer Chat Route Avatar … Enter Lot Use Object

Common Gotchas n Setting the Test bar too high, too early n n n Common Gotchas n Setting the Test bar too high, too early n n n Non-determinism n n n Race conditions, dirty buffers/process. State, … Developers test with a single client against a single server: no chance to expose race conditions Not designing for testability n n n Feature drift == expensive test maintenance Code is built incrementally: reporting failures nobody is prepared to deal with wastes everybody’s time Testability is an end requirement Retrofitting is expensive No senior engineering committed to the testing problem

Outline Overview: Automated Testing Definition, Value, High-Level Approach Applying Automated Testing Mechanics, Applications Process Outline Overview: Automated Testing Definition, Value, High-Level Approach Applying Automated Testing Mechanics, Applications Process Shifts: Stability & Scale Implementation & Key Risks Summary & Questions

Summary: Mechanics & Implications n n n Scripted test clients and instrumented code rock! Summary: Mechanics & Implications n n n Scripted test clients and instrumented code rock! Collection, aggregation and display of test data is vital in making decisions on a day to day basis Lessen the panic n n Scale&Break is a very clarifying experience Stable code&servers in development greatly ease the pain of building a MMP game Hard data (not opinion) is both illuminating and calming Long-term operations: testing is a recurring cost

Summary: Process n Integrate automated testing at all levels n n Don’t just throw Summary: Process n Integrate automated testing at all levels n n Don’t just throw testing over the wall to QA monsters Use automation to speed & focus development Stability: Sniff Test, Monkey Tests n Scale: Load Test n

Tabula Rasa Pre. Checkin Sniff. Test Keep Mainline Working Hourly Monkey Tests Baseline for Tabula Rasa Pre. Checkin Sniff. Test Keep Mainline Working Hourly Monkey Tests Baseline for Developers Dedicated Tools Group Executive Support Load Test: Early & Often Easy to Use == Used Radical Shifts in Process Break It Before Live Distribute Test Development & Ownership Across Full Team

Cautionary Tales Flexible Game Development Requires Flexible Tests Signal To Noise Ratio Defects & Cautionary Tales Flexible Game Development Requires Flexible Tests Signal To Noise Ratio Defects & Variance In The Testing System

Questions (15 Minutes) Overview: Automated Testing Definition, Value, High-Level Approach Applying Automated Testing Mechanics, Questions (15 Minutes) Overview: Automated Testing Definition, Value, High-Level Approach Applying Automated Testing Mechanics, Applications Process Shifts: Stability, Scale & Metrics Implementation & Key Risks Slides online @ www. maggotranch. com/MMP