Open Distributed and Collaborative Software Development Bill Lorensen

Open, Distributed and Collaborative Software Development Bill Lorensen GE Research lorensen@research. ge. com NCRR NIBIB PI Meeting June 22, 2005

"If you come to a fork in the road, take it. " NCRR NIBIB PI Meeting June 22, 2005

The Software Fork Closed Open

Google Hits “Open Source”: 117, 000

A Personal Look Back 1976 - NASA’s Computer Software Management and Information System (COSMIC) l 1978 - MOVIE. BYU l 1984 - GE Research Workstation l 1994 - Visualization Toolkit l 2000 - Insight Toolkit l NCRR NIBIB PI Meeting June 22, 2005

And a look forward http: //www. bisti. nih. gov/ncbc/ NCRR NIBIB PI Meeting June 22, 2005

NIH National Centers For Biomedical Computing l …NIH does have goals for software dissemination… n …software should be freely available … n …permit the commercialization of enhanced n or customized versions … …include the ability of researchers outside the center and its collaborating projects to modify the source code and to share modifications … NCRR NIBIB PI Meeting June 22, 2005

Outline Open Source Software l Inside Insight l Observations l

Flattener #4 Open Sourcing Self-organizing collaborative communities NCRR NIBIB PI Meeting June 22, 2005

Open Source Anecdote * IBM Manager: “So, walk me through the development process for e -commerce. What’s the underlying web server? ” Developer: “It’s built on top of Apache. ” Manager: “Apache? ” Developer: “A shareware program for web server technology, produced for free by a bunch of geeks working online in some open-source chat room. ” Manager: “How do you buy it? ” Developer: “You download it off a Web site for free. ” Manager: “Who supports it if something goes wrong? ” Developer: “I don’t know, it just works!” *from The World is Flat NCRR NIBIB PI Meeting June 22, 2005

Successful Open Source Projects NCRR NIBIB PI Meeting June 22, 2005

Visualization Toolkit - vtk Open source toolkit for scientific visualization, computer graphics, and image processing www. vtk. org

Computer Vision Library - vxl l l Based on Target jr and Image Understanding Environment Numerics Imaging Geometry Camera models vxl. sourceforge. net NCRR NIBIB PI Meeting June 22, 2005

Slicer l l Developed by Brigham and Womens Surgical Planning Lab User interface plus plug-ins for applications n n n Segmentation Registration Image Guidance NCRR NIBIB PI Meeting www. slicer. org June 22, 2005

SCIRun l l Developed by U Utah SCI Institute Computational Workbench Visual Programming Modeling, Simulation and Visualization www. sci. utah. edu NCRR NIBIB PI Meeting June 22, 2005

Open Source Menu for Success l l l l A Community with a common vision A pool of talented and motivated developers/scientists A mix of academic and commercial An organized, light weight approach to software development A leadership structure Communication A business model NCRR NIBIB PI Meeting June 22, 2005

Inside Insight (itk) NCRR NIBIB PI Meeting June 22, 2005

National Library of Medicine Segmentation and Registration Toolkit $14 million over 6 years Leading edge algorithms Open Source Software www. itk. org NCRR NIBIB PI Meeting June 22, 2005

What is itk? l l A common Application Programmers Interface (API). l A framework for software development l A toolkit for registration and segmentation l An Open Source resource for future research A validation model for segmentation and registration. l A framework for validation development l Assistance for algorithm designers l A seed repository for validation case studies NCRR NIBIB PI Meeting June 22, 2005

Insight - Open Source Products NCRR NIBIB PI Meeting June 22, 2005

itk by the Numbers l l l March 2000 l First code checkin 1600 l # of nightly builds 910 l # tests run nightly 23 l # of platforms 730 l # of classes 2000 l # of files with code NCRR NIBIB PI Meeting l l l 400 K l # of lines of code 100 K l # of lines of test code 45 K l # of lines of examples 160 K l # of lines of Applications 220 l # weekly t-cons 50 l # unique developers June 22, 2005

itk by the Numbers l l l 186 l # of subscribers to the developers mailing list 900 l # of subscribers to the users mailing list 350 l # of monthly posts to users-list NCRR NIBIB PI Meeting June 22, 2005

A Common Vision Create a dynamic, self-sustaining, public domain and extensible toolkit that will empower researchers throughout the world to develop new segmentation and registration algorithms and create new applications that leverage the NLM’s investment in the Visible Human Male and Female data sets NCRR NIBIB PI Meeting June 22, 2005

The Original Team l l l Six prime contractors Industrial l Kitware (Schroeder) l Insightful (Ng) l UPenn (Gee) l GE Research (Lorensen) l Brigham and Womens (Kikinis) Academic l UNC (Aylward) l Pitt (Stetton) l Utah (Whitaker) l Rutgers (Metaxas) l Columbia (Imielenski) l UPenn (Udupa) NCRR NIBIB PI Meeting June 22, 2005

Insight Consortium l l A competitive process selected the development team Each group was evaluated individually, without regard to how they might integrate with each other Each team had strengths in software development, validation or algorithm development, but no one group had the necessary skills to do the entire project Distributed software developers with varying software engineering experience NCRR NIBIB PI Meeting June 22, 2005

Insight Software Architecture l l l Object-oriented design Generic Programming Design Patterns Frameworks Separation of Algorithms from Interfaces NCRR NIBIB PI Meeting June 22, 2005

Object-Oriented Design l l l Dominated software systems throughout the 1990’s Continues to be the accepted software design technique Particularly useful for dealing with complexity Provides programmatic abstractions to deal with generalization and encapsulation C++ and Java have mechanisms to support OOD NCRR NIBIB PI Meeting June 22, 2005

Generic Programming l l l Organize libraries consisting of generic— or reusable—software components. The essential ideas of generic programming are containers to hold data, iterators to access the data, and generic algorithms that use containers and iterators to create efficient, fundamental algorithms. ITK uses generic programming to process n-dimensional “images”. NCRR NIBIB PI Meeting June 22, 2005

Design Patterns l l Good object-oriented software systems have recurring designs (patterns) that occur frequently ITK employs a number of powerful design patterns l object factories l command/observer l smart pointer memory management NCRR NIBIB PI Meeting June 22, 2005

Frameworks l l l Define how a group of participants can be put together to solve a particular task. Particularly suitable for describing complex flows or algorithms that have a number of steps that can be varied ITK Frameworks l A demand-driven data processing pipeline that connects algorithms to process n-dimensional image data l Registration framework l Level-set framework NCRR NIBIB PI Meeting June 22, 2005

Registration Framework NCRR NIBIB PI Meeting June 22, 2005

Level-Set Framework NCRR NIBIB PI Meeting June 22, 2005

Separation of Algorithms from Interfaces l l l Implement the algorithms with a clear separation from the applications and especially the user interfaces. Uses the Command/Observer design pattern that permits applications to watch for significant events during the execution of an algorithm ITK has no built-in visualization, but has been interfaced to several systems including 3 D Slicer, Analyze, Sci. Run and Volview. NCRR NIBIB PI Meeting June 22, 2005

In search of a new software development process NCRR NIBIB PI Meeting June 22, 2005

Google Hits “Extreme Programming”: 698, 000

Extreme Programming NCRR NIBIB PI Meeting June 22, 2005

A Light Weight Software Engineering Process l Based on the new Extreme Programming process l High intensity design, test, implement cycle l Supported with web-enabled tools l Automated testing integrated with the software development NCRR NIBIB PI Meeting June 22, 2005

Extreme Programming Compresses the standard analyze, design, implement, test cycle into a continuous process NCRR NIBIB PI Meeting June 22, 2005

A process supported by a suite of portable, open source tools l l l Apache, perl, php l Web services cvs, subversion l Revision control Wiki l Collaborative content capture Doxygen l Automated documentation CMake l Cross-platform program build Dart l Continuous and distributed test reporting NCRR NIBIB PI Meeting June 22, 2005

Extreme Programming The community owns the code Although the identity of the original author is kept, other developers are free to correct defects and enhance each other's code In the end, all of the software should appear as though one author wrote it NCRR NIBIB PI Meeting June 22, 2005

Extreme Programming Release early, release often Although developers are tempted to keep their code under wraps until it is perfect, the process encourages them to release their code as soon as it passes some minimum tests The longer the code is visible to the community, the better integrated it will be NCRR NIBIB PI Meeting June 22, 2005

Extreme Programming Continuous integration There is no scheduled porting to computer platforms All new software builds supported platforms every evening NCRR NIBIB PI Meeting June 22, 2005

Extreme Programming All developers agree to keep the software defect free. Although everyone is encouraged to submit their code early, the code must compile and pass tests nightly A continuous build process sends e-mails to developers who check in code that does not compile More effectively, the community enforces the commitment though peer pressure NCRR NIBIB PI Meeting June 22, 2005

Insight - Development Cycles l Daily – dashboard l l Weekly – telephone conferences Periodic – architecture reviews Quarterly – developer meetings Yearly – work assignments NCRR NIBIB PI Meeting June 22, 2005

Extreme Programming Daily Testing Is The Key l l l Testing anchors and drives the development process (Dart) Opens up the development process to everyone Developers monitor the testing dashboard constantly Problems are identified and fixed immediately Developers receive e-mail if they “Break the Build” NCRR NIBIB PI Meeting June 22, 2005

How DART Enables Collaboration Results posted on web (the dashboard) CVS maintains source code revisions DART compiles source code, runs tests CVS Developers check-in code NCRR NIBIB PI Meeting Developers review results June 22, 2005

Dart NCRR NIBIB PI Meeting June 22, 2005

NCRR NIBIB PI Meeting June 22, 2005

Multi-Platform Builds NCRR NIBIB PI Meeting June 22, 2005

Regression Testing NCRR NIBIB PI Meeting June 22, 2005

Continuous System Monitoring NCRR NIBIB PI Meeting June 22, 2005

Someone broke the build! NCRR NIBIB PI Meeting June 22, 2005

Insight Observations: The Good l l l Good mix of commercial and academic Communication is critical The daily rhythm of Extreme Testing The Whole >>> Sum of the parts Process automation reduces the burden on developers This process can (and will) be repeated NCRR NIBIB PI Meeting June 22, 2005

Insight Observations: The Bad l l l Communication l Early in the project, communication was limited to the mailing list and the Quarterly meetings. Almost a year passed before we added the weekly telephone conferences. Developers felt isolated and the impersonal electronic communication methods failed to build the personal relationships required for any software development process. Lack of Design Reviews l The lightweight process has its drawbacks. The current process emphasizes coding and testing. This process is well defined and benefits from automation. But, the design process is not well supported by the current mechanisms. Unstable Application Programming Interfaces (API’s) l The nightly test/build was so effective in empowering programmers to make changes, that API changes occurred too frequently without the necessary buy-in from the development community. NCRR NIBIB PI Meeting June 22, 2005

Insight Observations: The Bad l l Insufficient Automation l Automation is critical to maintain consistent style and robust software. Although the project provided many automatic techniques to catch developer errors, the project would benefit from more automation. Complexity and Feature Creep l Like most software development projects, this software suffers from complexity and feature creep. Early designs and implementations used many features of the C++ language to an “extreme”. The team realized and began removing the complexity. NCRR NIBIB PI Meeting June 22, 2005

Insight Observations: The Ugly l l Still need a stick l Someone needs to monitor the daily process. XP is not everyone’s cup of tea l Almost all bought into the process. A few tolerated it. NCRR NIBIB PI Meeting June 22, 2005

Parting Thoughts (from Robert Fulghum) l l l l l Share everything Play fair Don’t hit people Put things back where you found them Clean up your own mess Don’t take things that aren’t yours Say you’re sorry when you hurt someone Wash you hands before you eat and… Flush NCRR NIBIB PI Meeting June 22, 2005

Open Source Resources l l l l www. opensource. org sourceforge. net www. itk. org www. vtk. org vxl. sourceforge. net www. slicer. org www. sci. utah. edu NCRR NIBIB PI Meeting June 22, 2005

"If you don't know where you are going, you will wind up somewhere else. " NCRR NIBIB PI Meeting June 22, 2005

Open, Distributed and Collaborative Software Development Bill Lorensen GE Research lorensen@research. ge. com NCRR NIBIB PI Meeting June 22, 2005