Test-Driven Learning in Early Programming Courses David Janzen

Test-Driven Learning in Early Programming Courses David Janzen California Polytechnic State University San Luis Obispo, CA Hossein Saiedian University of Kansas Lawrence, KS SIGCSE’ 08

I Have a Dream • That all students will some day… – learn to design and test as they learn to program • “. . . we have as many testers as we have developers. And testers spend all their time testing, and developers spend half their time testing. ” Bill Gates in Information Week, May 2002 SIGCSE’ 08

How to Accomplish the Dream? • Test-Driven Development • TDD is a design (and testing) approach involving short, rapid iterations of Unit Test Code Refactor Forces programmer to consider use of a method before implementation of the method SIGCSE’ 08

TDD in Action class Rectangle. Test { @Test public void test. Area() { Rectangle r = new Rectangle(5, 8); assert. Equals(40, r. get. Area()); } } @Test public void test. Perim(){ Rectangle r = new Rectangle(5, 8); assert. Equals(26, r. get. Perim()); } } class Rectangle { private length, width; public Rectangle(int l, int w) { length = l; width = w; } public int get. Area() { return length * width; } } public int get. Perim() { return (length * 2) + (width * 2); } } SIGCSE’ 08

Predicting Evolution Adoption the Future Objects TDD Research Labs Dahl & Nygaard NASA Project Mercury Industry Smalltalk, C++ Industry XP SW Process Any “beyond the visionary Grad & and into the phase Advanced UG early mainstream” IEEE Software 2007 Grad & Advanced UG First Year CS CS SIGCSE’ 08

Questions We Want To Answer 1. Can we teach automated unit testing in first year programming courses without sacrificing other first year topics? 2. Is it better to teach test-first (TDD) or testlast programming to first year students? 3. What do students think about TDD? SIGCSE’ 08

How Do We Teach TDD? • Test-Driven Learning involves teaching testing by example XXXXXXX System. out. println(“Result: ” + my. Object. calculate. Result()); assert. Equals(25, my. Object. calculate. Result()); SIGCSE’ 08

Experiment Design • Proposed TDD/TDL integration into CS 1 and CS 2 courses at University of Kansas in 2006 as part of Ph. D research • We don’t always get what we ask for – Given permission to teach two labs and assign two projects • Weeks 6 and 7 in a 16 -week CS 1 course • Weeks 1 and 3 in a 16 -week CS 2 course SIGCSE’ 08

Experiment Design Project 1 solution with tests provided CS 1 CS 2 TF/TL Instruction & Survey Project 1 CS 1 Students Assigned by ID Survey Project 2 Test-First 40 Test-Last 66 Test-First 6 Test-Last 30 Project 1 solution with tests provided CS 2 Students Self-selected SIGCSE’ 08

Experiment Context • CS 1 and CS 2 both used C++, vi/emacs, and g++ Project Topics 1 st Project 2 nd Project CS 1 Arrays, functions, Classes, text I/O CS 2 Classes, array. Classes, pointerbased list, text I/O SIGCSE’ 08

Goal: Keep Unit Testing Simple #include class Exams {. . . }; int main() { run_tests(); } void run_tests() { { //test 1 Minimum of empty list is 0 Exams exam 1; assert(exam 1. get. Min() == 0); } //test 1 { //test 2 Exams exam 1; exam 1. add. Exam(90); assert(exam 1. get. Min() == 90); } //test 2 } SIGCSE’ 08

CS 1 Results: Grades & Effort • Project grades were virtually the same • Time spent on project was ~10% higher for test-first programmers Time (min) 1 st Project 2 nd Project TF mean 285. 29 343. 48 TL mean 260. 20 308. 40 % diff p-value 10%. 60 11%. 47 SIGCSE’ 08

CS 1 Results: Unit Tests • Test-first programmers wrote more tests on the first project, but fewer on second • Remember that students switched approaches # asserts TF mean TL mean % diff p-value 1 st Project 2 nd Project 5. 85 1. 89 3. 85 3. 10 52%. 11 -39%. 11 SIGCSE’ 08

CS 1 Results: Confidence • No difference in confidence on first project • On second project, test-first programmers were more confident of their solutions (P 1 tests) 2 nd Project TF mean TL mean % diff p-value Quality 3. 98 3. 25 Changes 3. 90 3. 06 Reuse 3. 69 2. 88 22%. 05 27%. 02 28%. 02 SIGCSE’ 08

CS 1 Results: Student Opinions SIGCSE’ 08

CS 2 Results: Grades • Project grades were higher for test-first • No significant difference in previous experience or gpa Score TF mean TL mean % diff p-value 1 st Project 2 nd Project 88. 83 90. 17 79. 47 72. 83 12%. 09 24%. 04 SIGCSE’ 08

CS 2 Results: Effort • Time spent on project was lower for testfirst programmers Score TF mean TL mean % diff p-value 1 st Project 2 nd Project 12. 42 16. 00 14. 70 18. 06 -16%. 30 -11%. 42 SIGCSE’ 08

CS 2 Results: Unit Tests • Test-first programmers wrote more tests • Not significant, high standard deviation • Students not graded on tests # asserts TF mean TL mean % diff p-value 1 st Project 2 nd Project 53. 67 47. 67 13. 43 25. 03 300%. 08 90%. 39 SIGCSE’ 08

CS 2 Results: Confidence • No significant difference in confidence on either project Both Projects Quality TF mean 3. 83 TL mean 3. 21 % diff p-value 16%. 14 Changes 4. 00 3. 89 Reuse 3. 67 3. 71 3%. 87 -1%. 74 SIGCSE’ 08

CS 2 Results: Student Opinions SIGCSE’ 08

Reluctance to adopt test-first despite perceived benefits 11% vs 63% would choose test-first SIGCSE’ 08

Validity Concerns • Study too small to draw any conclusions • Lack of randomization in CS 2 • Confounding factors: – – New concepts introduced with each project Solutions with tests provided after 1 st project Simple assert-based unit tests, no framework C++ • However, many of the results line up with results from industry and more advanced courses SIGCSE’ 08

Conclusions • Unit testing can be integrated into CS 1/2 courses without taking time away from other topics • Test-first programmers tend to write more tests • Test-first programmers may do as well (CS 1) or better (CS 2) on projects • Test-first programmers may spend more time (CS 1) or perhaps less time (CS 2) on projects • Early programmers are reluctant to use test-first, even after positive experiences with it SIGCSE’ 08

Future Work • Currently conducting a study in CS 1. 5 – Java and JUnit – TDD from beginning – TDD throughout 10 -week quarter • Early Results – TDD takes no extra instruction time – Existing course materials can be rewritten w/TDD – TDD could influence course refactoring • e. g. delay coverage of I/O SIGCSE’ 08

Questions • Resources at http: //www. simexusa. com/tdl/ • If you are an early adopter of TDD in CS 1/2, I’d like to know, email me at djanzen (at) calpoly. edu • Acknowledgements – SIGCSE Special Projects Grant – seed funding – Lockheed Martin – current (future work) study SIGCSE’ 08