No Respect Danny, Be the Ball National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 1
NA-MIC National Alliance for Medical Image Computing http: //na-mic. org Dart 2 Quality Framework Software Quality: Past, Present, Future Dan Blezek Jim Miller Bill Lorensen
A brief history of quality Very First VTK Dashboard • Update information • Builds – Irix, Solaris, Win. NT • Regression tests • Win. NT “Catastropic failure” National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 3
Last VTK Dashboard • 8 Platforms • 650 Nightly Tests • 70% coverage (Still) • Nightly Purify National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 4
A Bad Day National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 5
The Continuous Build National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 6
My first VTK checkin Continuous Update National Alliance for Medical Image Computing http: //na-mic. org Continuous Build Salt Lake City AHM February, 2005 7
The Big Bat of Quality National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 8
Bill “Yogi” Lorensen National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 9
Lessons learned • • • Test on different platforms Test nightly Make it easy to add a test Track changes daily Keep historical information National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 10
NA-MIC National Alliance for Medical Image Computing http: //na-mic. org Two roads diverged in a wood, and I-I took the one less traveled by, And that has made all the difference. Robert Frost Closed Source
Dart (v 1) • • • Tests Reports Dashboards Captures state of the system Distills data into information – Convert build log errors/warnings – Summarize test execution – Rollup Coverage statistics National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 12
National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 13
National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 14
Someone broke the build! National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 15
Someone broke the build! National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 16
National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 17
Dart’s Power • • • Distributed testing If I don’t have a platform, you do Distill data from many tools Distributed Extreme Programming Know the state of the system Instant feedback on changes National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 18
Quality Statistics Original VTK dashboard • 8 platforms / 650 tests • 13. 6 G over 4 years Current VTK dashboard • 29 Nightly platforms / 500 tests • 1 -2 G / week Insight dashboard • 60 Nightly builds / 970 tests • 1 -2 G / week National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 19
Dart 2 Design Goals • One Server, multiple Projects • Simple, flexible setup and management • Configurable presentation • Persist data on dashboard over time • Aggregate Dashboards • Authenticated submission, if desired • Resource management tools National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 20
Implementation • Java – Many, many available packages – Cross-platform • Everything in one package – No extra OS packages required – Distribute as Jar and/or platform exe • Should be easily extensible – Even to non-Java programmers National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 21
Packages National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 22
Components/Concepts • • • Client, Submission Test Hierarchy Results RDBMS XML-RPC Server Task Manager Scheduler Tracks HTTP / Template Engine National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 23
Client, Submission • Client: a unique platform – Need to define criteria – Currently Site / Build. Name • Submission – One Time. Stamped set of Test data – Particular to a sub-Project • slicer. itk • slicer. vtk National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 24
Test Hierarchy • • Test is a logic group of Results Has a Pass/Fail/Not. Run status May contain other Tests Has Hierarchial naming convention – itk. common. Print. Self • Sub. Test information rolled up National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 25
Results • Data produced by a Test • Examples: – Log of standard out – Image – Execution. Time • Typed – text/string, text/url, text/xml, text/text – numeric/integer, numeric/double – image/png, image/jpeg National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 26
RDBMS • Core of Dart 2 – Bundled with Derby embedded RDBMS – Any JDBC compliant DB works • Stores “small” data – Images, large blocks of text in files • Jaxor – Object – Relational Bridge package – No fancy SQL required – Creates objects from rows in DB National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 27
XML-RPC Server • Accepts Submissions • Administrative functions • HTTP transport – Easy submission through firewalls • Digester used to process XML – Executes code when tags found National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 28
Task Manager • Tasks are units of work for the server – Project and Server Tasks – Scheduled, Event driven • When a Submission arrives, a Task is queued • Queue. Manager executes Tasks • Plug-ins allow Project specific Tasks – Simply implement the Task Interface National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 29
Scheduler • Quartz Enterprise Scheduler • Executes Tasks – Uses enhanced “cron” syntax • Uses – Regular DB maintained – Purge unnecessary data – Archive aging data National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 30
Tracks • Groups of Submissions – Dashboard consists of intersecting Tracks • Temporal Tracks – Time based, i. e. 12 am start, 24 hr duration • “Most Recent” Track – Last 5 Continuous builds • Project specific National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 31
HTTP / Template Engine • Jetty is HTTP/Servlet server • Free. Marker – Data prepared in Servelet – Template processed – Returned to client via HTTP • Flexible • Easy to add new pages • No XSLT! National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 32
Dart 2 Current Status • Alpha version ready • Test server – http: //www. na-mic. org: 8081/Insight/Dashboard/ – Populated with Build & Test from public. kitware. com • Subversion Code Repository – svn co http: //svn. na-mic. org: 8000/svn/Dart – Web SVN: http: //www. na-mic. org: 8000/websvn/ National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 33
Acknowledgements • • • Andy Cedilnik Bill Hoffman Will Schroeder Ken Martin Amitha Perera Fred Wheeler National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 34
NA-MIC National Alliance for Medical Image Computing http: //na-mic. org Dart 2 Quality Framework Software Quality: Past, Present, Future Dan Blezek Jim Miller Bill Lorensen
Desirable Qualities • Frequent testing – Identify defects as soon as they are introduced – Hard to find cause if not done frequently • Minimally invasive to daily activities • Automated testing • Automated report generation/summaries – Must be concise yet informative • Track results over time National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 36
NA-MIC National Alliance for Medical Image Computing http: //na-mic. org NAMIC Software Process Dan Blezek Jim Miller Bill Lorensen
Motivation • Many algorithms, many platforms – VTK, ITK, Slicer, LONI – Linux, Windows, Mac OSX, Solaris, SGI • Many users • Many datasets • Many sources of problems! National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 38
Motivation Negative example • MIT codes ITK algorithm for LONI pipeline • UCLA developer changes LONI • GE changes ITK • Time for release, everything’s broken National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 39
Motivation Ensuring high quality software • System’s state must be known – If UCLA knew about MIT code, they would have been more careful w/changes • All the code works, all the time – As often as is feasible, compile and test the code National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 40
Extreme Programming National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 41
NAMIC Process • Light weight • Based on Extreme Programming – High intensity cycle • Design • Test • Implement – Supported with web-enabled tools – Automated testing integrated with the software development National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 42
Software Process • • • Design Process Coding Standards Testing Bug Tracker Communication – Mailing lists, Discussion forum, Wiki – Tcons • Documentation • Releases National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 43
Design Process • • Take the time to design a good API Plan for future use Plan for future extension Two routes – Code something, check it in • Others will tear it down & make it better – Put together a strawman • Solicit ideas, implement National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 44
Coding Standards • Follow the package’s rules • ITK has certain coding standards – Style guidelines – Naming conventions – Macros National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 45
Testing • If it isn’t tested, it’s broken. • Tests – Ensure your code works – Documents expected results – Others free to change National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 46
Bug Tracker • • Bugs assigned / taken by developers Tracks progress to releases Captures feature requests Communication mechanism National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 47
Documentation • Doxygen – Automatic API documentation – Algorithm references – Implementation details • Books / Manuals – Insight Book National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 48
Communication • • Email lists Discussion forum Wiki Tcon National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 49
Extreme Programming • Compression of standard analyze, design, implement, test cycle into a continuous process National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 50
Daily Testing Is The Key • Testing anchors the development process (Dart) • Developers monitor the testing dashboard constantly • Problems are identified and fixed immediately • Developers receive e-mail if they “Break the Build” National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 51
Daily rhythm Design, implement algorithm • write regression test • check it in Repeat Dart takes over • Code is updated, built, tested • Guilty parties are blamed Core 2 • Build breakers are notified • Work on cleaning up dashboard National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 52
Dart • • Testing Reports Dashboards Central site for state of the system – Updates – Builds – Test – Coverage National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 53
National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 54
National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 55
Someone broke the build! National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 56
National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 57
Conclusion • Have fun • Process extends your impact – Many can use your code – Many can improve your code • Communicate, Communicate National Alliance for Medical Image Computing http: //na-mic. org Salt Lake City AHM February, 2005 58
NA-MIC National Alliance for Medical Image Computing http: //na-mic. org NAMIC Software Process Dan Blezek Jim Miller Bill Lorensen
Скачать презентацию No Respect Danny Be the Ball National Alliance
a0a3473067a459c92837e44dc4a77ade.ppt