Test and Verification Solutions Improved time to market

Test and Verification Solutions Improved time to market through automated software testing Mike Bartley, TVS Test and Verification Solutions 28 October 2009 1

Agenda • • The potential benefits & drawbacks of automation Compare with HW verif Can automation reduce TTM? Automating static & dynamic testing Deciding what to automate Choosing your automation tools Case study Test and Verification Solutions 28 October 2009 2

The potential benefits of automation • • Reduced costs Improved quality Improved employee motivation Improved TTM Test and Verification Solutions 28 October 2009 3

The potential drawbacks of automation • • Shelf-ware! Maintenance Skills base Missing failures Test and Verification Solutions 28 October 2009 4

HW verification The mechanics of complex test bench Functional Coverage Test Coverage Scoreboard Monitor Assertions constraint addr Stimulus generator data Driver Design Under Test assert Test and Verification Solutions 28 October 2009 Coverage 5

The mechanics of finding a bug Stimulate Propagate …. . 0101 01100101…. . 01001101 11110101…. . 00010101…. . 10011010 …. . 01001101 Design Under Test Expected Results Actual Results Observe Compare Test and Verification Solutions 28 October 2009 6

Can automation reduce TTM? • • Getting test results sooner Catching bugs earlier Stabilising the software sooner Assessing the stability of the software Test and Verification Solutions 28 October 2009 7

Automating static testing • • • Types of bug found A quick win Beware false failures Introduce filters slowly Developers not testers (check in checks, IDE) • No testing of intent Test and Verification Solutions 28 October 2009 8

Automating dynamic testing • • Test plan Test design specification Test case specification Test procedure specification Test execution Test log Test incident report Test summary report Test and Verification Solutions 28 October 2009 9

Deciding what to automate • • • Complexity of the test environment The test level Frequency of execution Ease of automating pass/fail Test stability and repeatability Test and Verification Solutions 28 October 2009 10

Choosing your automation tools • • Who writes test? Matching skill set Separate orthogonal concerns Portability Ease of adding/banning tests Integration into configuration/build system Cost of the tool? Test and Verification Solutions 28 October 2009 11

Case Study – Background • Includes chips, accelerator cards, and software • All work with industry-standard x 86 -based systems Test and Verification Solutions 28 October 2009 12

Case Study – Background • Low level host driver code – Code on host computer communicates with the chip on the board in the PCI slot • Libraries of code – Runtime library – Mathematics acceleration libraries (ACML, MKL, …) • Application code – Linpack, Montecarlo, …. • Code running on the chip – This had parallel C extensions not supported by our chosen tool • Software models of the hardware • In-house compiler for the code running on the chip • Mainly C and C++, some assembler Test and Verification Solutions 28 October 2009 13

Case study – automation details • • Integrated into configuration/build system Automated mostly lower level tests Developed internally by SW Engineers Offshored manual tests Offshored some automated test development Integration of static analysis Test maintenance Test and Verification Solutions 28 October 2009 14

Case study - results of automation • • • Improved quality Shortened release schedules Release schedules hit Reduced TTM ROI? Test and Verification Solutions 28 October 2009 15

Questions? • Mike Bartley • mike@tandvsolns. co. uk • Linked. In • 07796 307958 Test and Verification Solutions 28 October 2009 16