Helping Customers Write Acceptance Tests — Making It

Helping Customers Write Acceptance Tests - Making It Easy Jeremy Stell-Smith jeremy@thoughtworks. com

Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J 2 EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out… 2

Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J 2 EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out… 3

Why Test? • • What’s the point? To make change SAFE To lead to a flat cost of change curve As such, testing is the basis of XP 4

The Cost of Change Cost Traditional Development e. Xtreme Programming Time 5

“Test-Time” checking • Why do we like type safety? • As code changes, the compiler catches errors, and these errors are easy to fix • But, compile-time checking can’t check everything • There are runtime errors, and worse, bugs • Tests CAN guard against these • If it compiles, there are no compile-time errors • If the tests pass, there are no errors period 6

Types of Tests • Fine-grained vs. coarse-grained tests • Developer vs. customer tests • Fine-grained, developer tests (unit tests) are the ideal • Fast, come the closest to 100% coverage • Low duplication -> Low cost of change, low coupling • Use coarse-grained tests to protect yourself against holes in your bricks • Use customer tests to protect yourself from building the wrong thing right 7

An Acceptance Test is a Customer Test • Does not have to be coarse • Get your customers to write tests and take ownership • Must be in the language of the customer • Your “customer” may be an analyst or QA • Acceptance tests are not specific to XP 8

Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J 2 EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out… 9

Web Service Project • XML in / XML out • Stories are in terms of XML API • …so tests should be in terms of XML 10

Example Story • Gate In Container Story • I would like record a container as Gated In to a facility by submitting this message. • The syntax should be: 11

Our Framework • Simplest thing that could possibly work… • Built on top of Junit • Got UI for free • In the “language” of our analysts & customers • Setup / Teardown of Data 12

Example Test #3: Container Number was not provided Required element container was not found 13

Things We Added To It • Includes, to remove some duplication • db. Validation section to check against the db to be a bit more thorough • Let’s take a look 14

Things We Did Right • Analysts wrote the stories & the tests • At the beginning of every iteration, we got both • We kept it simple • Error handling • Analyst owned / developer maintained 15

Things We Did Wrong • Too much duplication – lot of copy / paste • Occasionally EVERY test broke – not good • Partly because XML isn’t type-safe 16

J 2 EE Project with Swing Client • • Stories were at the client level Client was untested Client had lots of functionality Big Project – QA Department 17

First Pass • • • One weekend Simple, simple Used XML to script tests against value objects. Not general, very specific to our app. Gave us coarse-grained developer tests • Wrong language • Didn’t solicit input 18

Example Test … 19

Second Pass • Now testing against swing components • Single threaded • Better language, but still difficult • Generic, but didn’t handle everything 20

Third Pass • • Significant effort JFCUnit Added a recorder Added an editor • modify, run, modify, run • Had been too hard to debug • Added data population • Scaling became impossible because it wasn’t tested… • Language – Good • Ease of use 21

Example Test 22

Forth Pass • Marathon • Uses Jython instead of XML • Access to all java code • Functions, classes, imports, external libraries, etc. • Interpreted • Test-First • Open source • JEdit 23

Example Test from default. Fixture import * def test(): window('Calculator Example') click('5') click('1') click('+') click('3') click('=') click('JText. Field') assert. Text('JText. Field', '54') click('+') click('5') click('=') assert. Text('JText. Field', '59') close() 24

Testing Strategies • Tests that ran with the build • Unit tests • Smoke tests : coarse-grained tests that provided safety net for our unit tests - 1/3 of our acceptance tests • Cruise Control • Nightly build of other tests • Developer maintained 25

Things We Did Right • Once and Only Once • Encouraged feedback • Bought where we could 26

Things We Did Wrong • • Analysts didn’t write the tests We didn’t integrate close enough w/ QA Used them as a substitute for unit tests Disconnect with stories 27

Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J 2 EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out… 28

Lessons Learned • In terms of stories • Acceptance tests are NOT a substitute for unit tests • Buy don’t build • Junit, Jython, etc • Iterative • Make it modular – once and only once • Language of Analysts • Focus on readability == maintainability • Don’t EVER want to re-record tests 29

Lessons Learned (cont’d) • • Yes you have to test your testing framework… Use a scripting language to write scripts User friendly Pair with your customers to write tests Developers maintained Error handling Needs to be Customizable -> Open Source? 30

Things To Watch Out For • • How to deploy When to run What to do when they break Are tests independent? Data? Acceptance tests are not all passing? Long running tests 31

Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J 2 EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out… 32

FIT • • • http: //fit. c 2. com/ Open source general purpose functional testing tool Totally generic, and customizable Allows fine-grained testing Remember, customer tests are not necessarily end to end tests • If you want to GUI level tests, I wouldn’t use FIT • Ideal for test first stuff 33

Marathon – GUI Test Runner • http: //marathonman. sourceforge. net/ • More suited for testing GUIs • Recorder & editor • Can’t use a recorder for Test First tests • But you can use it help maintain them • Modular without leaving script 34

Agenda • Why Test? • Types of Tests • Evolving a Framework • On a Web Service Project • On a J 2 EE Project with a Swing Client • Lessons Learned • Other Frameworks • FIT • Marathon • Let’s Try It Out… 35