Automated Testing Environment Concepts required for testing embedded

Automated Testing Environment Concepts required for testing embedded systems adopted in this course (quizzes, assignments and laboratories) 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 1

To be tackled today n Why test, and what kinds of tests are there? n Difference between n TLD – Test Last Development n TDD – Test Driven Development (Test First Development) n How do you add tests? n More details on the testing process E-TDD and the testing toll E-UNIT Other kinds of available tests – time and access n Design of custom TESTs and ASSERTS n 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 2 / 28

Why test? n Unit test n I AM CURRENTLY BUILDING A NEW FUNCTION – does it work the way I expect? n Testing my and my partner’s understanding of the Design n WE ARE DESIGNING A NEW PRODUCT -- If we can’t explain (to ourselves via a test) the product ideas – how do we know when product is working? n Regression testing n MY PARTNER ADDED A NEW FEATURE to a product – how do I know it did not “break something” I wrote? n System testing n PROOF TO THE CUSTOMER that the product (Lab. or assignment) works the way that was requested. 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 3 / 28

What are the problems with this sort of test? 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 4 / 28

Visual. DSP development environment (IDDE) Processor Debugging Visual. DSP++ Interface program HPPCIce Lab. station (ICT 318 / 320) Debugging Interface “JTAG” boundary scan operation Peripherals BF 533 Evaluation Board Each time we want to send a message (via printf( ) or cout) we must “stop” the processor from doing what we want it to do, and then send messages over the interface – Slow, plus the testing can impact on peripheral performance. E. g. no sound while printing in Assignment 2 and Lab. 1 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 5 / 28

More issues with this sort of test! n Tests are mixed up “with production code” n Must remove tests before release to customer n n n 3/18/2018 Unreliable – may result in test messages at wrong time in released code Difficult to “add” and “remove tests” Can’t be automated – therefore unlikely to have tests run often during development – leads to unreliable product Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 6 / 28

E-TDD Tool n E-TDD – embedded test driven development n n Build-the-tests-first (during design) ideas for product development has been made popular by the “Agile” approach Requires an automated testing approach n In this class, we are going to use the E-Unit tool for its automated behaviour n n n 3/18/2018 Will use in both “test last” development (what most people currently do) and in “test first” development. I (the customer) will provide tests so that you can check that you have developed the “right” stuff at the “high” level You will build additional tests to check your own code at the “low level” Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 7 / 28

Test Environment n Can download “Lab. Test. Environment. zip” file from the web (Check the September webpages for exact name of file) n Lab. X directory is where you keep your “final” code n Lab. XTests is where you keep the tests for Lab. X E-UNITIncludes n E-UNITIncludes directory contains “everything” needed to make test environment work 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 8 / 28

Build a new VDSP project “Assignment 1 Test” and place in “Assignment. Tests” directory n Now add certain standard files to the project – same set of files for all assignments and laboratory n All found in E-UNITIncludes directory E-UNIT name replaces E-TDD name 07. dlb 07. h Expect minor changes (names and dates) as we “improve” E-UNIT as part of our own going research work on embedded systems 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 9 / 28

Now add your Assignment 1 subroutines files to the Assignment 1 Test project n Don’t move or copy the files into the test directory – just “add” them to the project n Note the use of “Assignment 1. h” which has the prototypes for all “C++” and “assembly code” functions used (from Assignment dir. ) E-UNIT name 3/18/2018 replaces E-TDD name 07. dlb This files are “added” to the “Test” project and NOT copied from the Assignment 1 directory Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 10 / 28

Now we need to develop and add “Testmain. cpp” E-UNIT name replaces E-TDD name 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 11 / 28

Now we need to develop and add Assignment 1 Tests. cpp” 1 “. . /E-UNITIncludes/E-UNIT_Tests 1 Sept 07. h” 2 3 4 5 6 7 8 9 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 12 / 28

Tests have a standard format 1 “. . /E-UNITIncludes/E-UNIT_Tests 1 Sept 07. h” 3 TESTGROUP NAME 4 TEST NAME TEST TYPE Add further tests at this location 8 TESTGROUP NAME, 9 3/18/2018 TEST TYPE Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 13 / 28

Tests then need to be customized 1 “. . /E-UNITIncludes/E-UNIT_Tests 1 Sept 07. h” 07 h 2 3 4 5 6 7 8 9 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 14 / 28

Test Components n The ACTUAL Test (This is a TEST MACRO) CHECK(result 1 == result 2) n Would provided better test messages if we “self-documented the code” CHECK(result_CPP == result_ASM); n Test CONTROL statements (ALL MACROS) n TEST_CONNECT( test_name) n LINK_TEST( test_name, test_type) n TEST_LEVEL( which_test_level) 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 15 / 28

Stages to “automating” the tests 1) Activate the E-TDD Gui (E-TDDIncludes directory) This causes the GUI to appear, and adds new “menus” to Visual. DSP 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 16 / 28

Stages to “automating” the tests 2) Select E-UNIT Connection | Refresh Connection List This automatically builds the E-Unit Test Info file 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 17 / 28

Now “build and run” the tests Quick test failure visualization Detailed test failure information 3/18/2018 Clicking on the “detailed test info” automatically takes you to the “test” that failed Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 18 / 28

Test information modes are controlled via “Activate. Tests. cpp” n Show Failures only | BTC_CONNECTION n Show Failures and Successes | BTC_CONNECTION 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 19 / 28

Add a separate test to show that result 2 (from the ASM code) is equal to 7 “. . /E-UNITIncludes/E-UNIT_Tests 1 Sept 07. h” Add further tests at this location 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 20 / 28

Write the tests for “self-documenting” messages (What does “result 1 == result 2” mean if 200 tests present? ) HAVE WANT 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 21 / 28

Write the “CPP” code to get to work but “Write the ASM for speed” “. . /E-UNITIncludes/E-UNIT_Tests 1 Sept 07. h” FAIL SUCCESS 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 22 / 28

Did the ASM code “destroy” or even “use” the CPParray[ ] defined in CPPassignment 1. cpp? 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 23 / 28

Available ASSERTS LONG INT tests – 32 bit – typical for “C++” n CHECK(condition) – Success if passes n CHECK_EQUAL(expected, actual) n EXPECTED_FALSE(condition) – Success if fails n ARRAYS_EQUAL(expected, actual, length) n CHECK_VALUES(expected. Temp, condition, actual. Temp) DOUBLES_EQUAL(expected, actual, threshold) 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 24 / 28

ASSERTS for embedded systems n Short int tests – 16 bit – typical for embedded systems n n SHORTS_EQUAL(expected, actual) SHORTS_CHECK(expected, condition, actual) USHORTS_EQUAL(expected, actual) USHORTS_CHECK(expected, condition, actual) n Timing of routine – Blackfin Specific n MAXTIME_ASSERT(max_time, function) n Useful utility n MEASURE_EXECUTION_TIME(timetaken, function) n Examples available in the tests associated with Lab. 1 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 25 / 28

New ASSERTS can be added to customize the tests or speed the tests ASSERT FORMAT #define CHECK_VALUES(expected. Temp, condition, actual. Temp) if (!((expected. Temp) condition (actual. Temp))) { FAILURE 1(#expected. Temp##" !"#condition##" "#actual. Temp); } else { SUCCESS 1(#expected. Temp##”""#condition##" "#actual. Temp); } 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 26 / 28

Practice – Assert Design n Design an assert that will determine whether the lower 8 bits of a 32 -bit variable is identical to the lower 8 bits of a second 32 bitvariable. n The two variables should remain unchanged by the assert (since they may be reused later) n The upper 24 bits of each variable MUST be ignored in the assert n Does it matter if the variables were defined in the. data section of an assembly code routine or in an C++ code routine? 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 27 / 28

Handled today – some easy “nonprogramming” quiz and midterm questions n Why test, and what kinds of tests are there? n Difference between n TLD – Test Last Development n TDD – Test Driven Development (Test First Development) n How do you add tests for Assignment 1? n More details on the testing process E-TDD and the testing tool E-UNIT Other kinds of available tests – time and access n Design of custom TESTs and ASSERTS n 3/18/2018 Concept of Test Driven Development applied to Embedded Systems M. Smith University of Calgary, Canada 28 / 28