Unit Testing Test Coverage Why What Who

Unit Testing & Test Coverage Why, What, (Who) Andrea Guarise INFN Torino Milano, June 7 th, 2004 Andrea. Guarise@to. infn. it

Introduction • Unit Testing: It is the procedure that furnishes the capability to test a software module in a reprudicible, efficient and automated way. • Test Suite: A collection of unit tests. • Test Coverage: The measurement of how many parts of the code are really tested, it is an expression of the goodeness of the Test Units. Milano, June 7 th, 2004 Andrea. Guarise@to. infn. it

Why The Four reasons for Unit Testing: 1) Before writing code, it forces you to detail your requirements in a useful fashion. 2) While writing code, it keeps you from over-coding. When all test cases pass, the function is complete. 3) When refactoring code, it assures you that the new version behaves the same way as the old version. 4) When maintaining code, it helps you cover your ass when someone comes screaming that your latest change broke their old code. (“But Sir, all the unit tests passed when I checked It in. . . ” Mark Pilgrim, from “Dive Into Python” Milano, June 7 th, 2004 Andrea. Guarise@to. infn. it

Why (2) Why Test Coverage: Simply because it is useless to have a beautiful test suite that doesn’t test all your code. A good test suite should always have instruments to tell you if you forgot to test something. So that at least you know where most likely the users of your code will find bugs. This is a direct consequence of the Murphy’s law. Milano, June 7 th, 2004 Andrea. Guarise@to. infn. it

What Unit Test Framework: It can be really useful to find a single test framework for all our software modules. But there is a little problem ahead of this: We use many different programming languages Therefore it’s necessary to change our perspective: We need to find a set of testing framework (one for each language) that behaves all in the same manner. Milano, June 7 th, 2004 Andrea. Guarise@to. infn. it

What (2) The x. Unit Framework: x. Unit is not a real test framework. It is the name used to address all the test frameworks that adhere to a common set of “capabilities” and test patterns. It isn’t yet a formalised standard but there are chanches that it will be. One inportant thing about x. Unit framework is that test code must always be separate from tested code. There are x. Unit compliant test frameworks for more or less every programming language. Our goal is to find one of these framework for each of the languages we use: C++, C, Java, Perl, Python (did I forget any? ) Milano, June 7 th, 2004 Andrea. Guarise@to. infn. it

What (3) Some General rules: The general rules we have to follow are: ● ● Each module must use only one unit test framework. Unit test code goes under the module/test/ directory. The unit test output is in XML format and goes under the module/reports diroctory. The build fails if the unit test fails. Milano, June 7 th, 2004 Andrea. Guarise@to. infn. it

What (4) C++: There are many implementations available, we analyzed some of them, and the choice has been: ● cpp. Unit: http: //cppunit. sourceforge. net Milano, June 7 th, 2004 Andrea. Guarise@to. infn. it

What (4) Java: The choice for java is really easy the de facto standard is j. Unit: The Idea of x. Unit itself is borrowed from the j. Unit framework. Frameworks for the other languages, a part from boost. test, are portings of j. Unit. It only adds dependencies at build level. This is one of the requirements of the x. Unit framework, the test units must not interfere with the code itself. There is no need to instrument the code. Milano, June 7 th, 2004 Andrea. Guarise@to. infn. it

What (5) Perl: We tried the: Test: : Unit unit testing framework available on http: //www. cpan. org It is intended to be a porting of j. Unit for OO based perl programming, altough not all the classic x. Unit functionalities are present yet. The documentation is in “POD” format. There is the documentation of the APIs some simple example and the usage instruction for the tools shipped with the framework. It only adds dependencies on the “build” node, no dependencies are needed on the machine where the tested code is installed. Milano, June 7 th, 2004 Andrea. Guarise@to. infn. it

What (6) Python: We did not try the framework, but the standard implementation is py. Unit: http: //pyunit. sourceforge. net It is fully documanted, with API and examples. It doesn’t add dependencies to the executables, but only on the build node. We did not know about any currently maintained test coverage analysis tool for Python. Milano, June 7 th, 2004 Andrea. Guarise@to. infn. it

Test Coverage As for unit testing, we need tools to measure test coverage for all of the above mentioned languages. Unfortunately tools for different languages report their results in different way, and all of them requires some effort in order to elaborate the results to get some usable statistics. So it may not be straightforward to really exploit these tools: Milano, June 7 th, 2004 Andrea. Guarise@to. infn. it

Test Coverage (2) C/C++: GNU GCC + GCOV It works, reports the percentage of code covered, and how many times each useful line of code has been run. GNU GCC + COVTOOL: http: //covtool. sourceforge. net On the paper seems most powerful, but it doesn’t build on gcc 3. x. x Java: Hansel: http: //hansel. sourceforge. net No. Unit: http: //nounit. sourceforge. net Not tested (does any one want to spare some time? ) Python: There is a module named coverage. py available at: http: //www. garethrees. org/2001/12/04/python-coverage/ But it is rather old and seems to be not official (if someone wants to check. . ). Perl: Code coverage is available under CPAN, some tests have been performed and it seems to work fine. Milano, June 7 th, 2004 Andrea. Guarise@to. infn. it

(Who? ) The short answer: We The long answer: (This is what we proposed in last meeting, but it needs to be reviewed, see conclusions) For each SW module we have two team of developers: Team A (The developers): responsible of developing the module and writing the basic unit tests (at least 50% code coverage), excuting them each time they feel it is necesssary, write basic Documentation. Team B (The Guardian Angels): study Team A code and tests, perform test coverage analysis on team A tests and add missing cases (90% code coverage), propose changes to both code and tests, help Team A in maintaining the unit tests and the test suites. Review and finalize Documentation. And then a Test coordinator: Integrate the test suites with the help of each “team B”. Check the global test coverage analysis and that the unit tests adhere to the project standards. Milano, June 7 th, 2004 Andrea. Guarise@to. infn. it

Conclusions In the meeting we learned how to use basic cpp. Unit functionalitites in our cpp tests, with a simple generic example, and a real life example with socketpp classes. We also learned how to use code coverage analysis tools to discover missing test cases. We realized that: It is possible to test currently available client/server sw pieces only relying on real instances of the servers (or of the clients) instantiated from whithin the test suites. This is not really compliant with true Unit Testing practices, but has been agreed with Alberto Di Meglio. This approach however has some disadvantages, like difficulties in tracking down the source of problems and most important it creates dependencies between client and server modules. A reimplementation of the server interfaces allowing the creation of mock implementation of the servers, for testing purposes has been proposed, and we agree that this would be the right solution, but it is definitely a medium/long term solution. Milano, June 7 th, 2004 Andrea. Guarise@to. infn. it

Conclusions(2) So for the short term we plan to follow the joint client/server test approach But we need to agree with Alberto the details of this approach and moreover the fact that the coverage analysis be performed to the client and server modules jointly. We defined the initial test cases for the LB service. We also started analyizing test cases for the NS service. Questions: Deadlines? What about the fact that we will go to WS (it is a waste of time to write unit. Tests for code that has to be thrown away? ) Test cases: who starts: the new people (the GA), that aren’t trained with the code and will need more time? Or the developers that can quickly implement the skeleton for the tests and then let the remaining work to the GA? How much effort are we expected to put in this activity? Who (The names) C framework, we have to choose it. (I will discard the not compliant one) Report schema: Alberto has to give us the XML schema for the reports, otherwise we aren’t able to go on. (I’ll send the request to Alberto. ) Milano, June 7 th, 2004 Andrea. Guarise@to. infn. it