How to progress fast towards a new OO

How to progress fast towards a new OO HEP library? Some ideas on prerequisites and impediments February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 1

Outline Disclaimer Some thoughts Summary and Outlook February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 2

Disclaimer This is not a talk about prerequisites: l l software project management software engineering software architecture data models I have no new ‘silver bullet’ to offer! l go and read (one of many good books): “How to run successful projects II” by Fergus O’Connel February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 3

Disclaimer(2) This is only my biased, imperfect view. The whole talk is meant to trigger discussion. February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 4

A few observations on software projects Fraunhofer Institute for Software and Systems Engineering (Berlin, Germany): 33% of software projects fail Of those which don’t fail: l 80% last substantially longer than planned and cost more than estimated Track record in HENP is of the same order of magnitude!! February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 5

Software projects: A comparison If you want to build a summer house it is sufficient to employ a bricklayer who will do it using his experience. If you want to build a skyscraper or a bridge you need foremen, engineers, architects and an agreed process. Nobody would do it with a 1000 bricklayers only!!! (except physicists) A bridge lasts 100 years, very good software lasts 10 -20 years at most, much software in HEP experiments survives < 5 years. February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 6

How to progress fast towards a new OO HEP library? Using OO in HENP for now is a fact, and a requirement. We should careful evaluate which parts are HENP specific. CERNLIB style libraries: l Quite some code and algorithms are ‘reinvented’ over and over => incompatible February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 7

Fergus O’Connel: The silver bullet (1) Planning the [software, mka] project 1: Visualize the goal; set your eyes on the prize 2: Make a list of jobs to be done 3: There must be one leader 4: Assign people to jobs 5: Manage expectations, allow a margin error, have a fallback position February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 8

Fergus O’Connel: The silver bullet (2) Implementing the plan/Achieving the goal 6: Use an appropriate leadership style 7: Know what is going on 8: Tell people what is going on 9: Repeat 1 -8 until 10 10: The prize (“oh what fun, my job is done”) February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 9

Planning - 1: Visualize the goal A new OO HEP library? !! l Contains all experiment-non-specific but HEPspecific algorithms l l l Elements for common solutions for HEP problems Robust, debugged, available for different OS’s compatible to all or most commonly used non-HEP packages l l Leverage knowledge from public domain or IT industry allows HEP to concentrate on HEP-specific problems May need adaptation for HEP February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 10

Example: CMS Software Tasks This talk Why not have a HEP framework? February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 11

Issues for Core Software Types Non - HEP software l l Version evolution on multi-platforms Distribution and possible licensing Cost Long term availability of commercial product HEP Software l l Manpower costs for development and maintenance Technical requirements for “common items” l l l Layered Architecture Standard Application Program Interfaces (APIs) Phasing: early linkage with key experiment-specific architecture decisions February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 12

Example: LHC ++ LHC++ is a HEP-wide effort to provide a suite of OO software covering roughly the functionality of CERNLIB l Use public domain or commercial SW when appropriate l l Develop own HEP software if necessary Collect and distribute SW; take care of possible licensing Develop a well-engineered toolkit of software components l Build interface layers for HEP usage l l Develop/acquire, maintain and replace components separately Build different applications from components Use standards and standard solutions l Use of standards may help in maintaining and developing software over the long lifetime of the experiment LHC++ has ‘one big’ problem: already the name indicates that is it not meant to be a HEP-wide solution ; -) CHEP 2000, Padova February 8, 2000 M. Kasemann, FNAL 13

LHC++ components Standard components l l C++ libraries, STL: Object Space Basic mathematics: NAG C library ODBMG database: Objectivity/DB Standard graphics: Open. GL, Open. Inventor, Qt, . . . HEP components l l l Simulation tools: Pythia 7, Geant 4 Data analysis, graphics, visualisation: HEPVis, HEPTags, … Mathematical libraries: Gemini, HEPfitting Histogramming: HTL Foundation level class libraries: CLHEP, Hep. ODBMS February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 14

Planning - 3: There must be one leader A leader is required l l l who can lead who is leading who is identified with the project and wants to be successful One and only one leader is required, not a committee. Bridges are not build by 1000 bricklayers who do what they want. February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 15

Implementation - 7: Know what is going on Use the plan to guide the project. February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 16

Implementation 8: Tell people what is going on Progress reviews, updated ‘work-plans’ or WBS’s. Good and bad surprises should be published instantly in order to adapt. February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 17

Implementation - 9: Repeat 1 -8 until 10 Be prepared to take long term commitment to finish project there is no shortcut to finish the project without cutting back in requirements. l need to compromise on short-term goals l February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 18

Implementation - 10: The prize Now the maintenance starts: needs resources and commitment, again: l conflict of interest for scientists who want to operate experiment and do HEP science. We have to get serious: l l l Learn to involve professionals from IT disciplines (like engineers in detector building). HEP software designers needs a career path like detector designers and builders. We should assign a higher status to those who release their scientific code with open source licenses. February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 19

How to progress fast towards a new OO HEP library? My preferred model is: l l l an idea/goal is connected to a person (leader, PI) an approval process happens resources must be allocated l l at universities, IT-divisions, in experiments progress is tracked feedback and early buy-in by customers operations and maintenance phase must be covered. Use ‘silver bullet’ steps 1 -10. This is how experiments are build, also applied to successful software projects. February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 20

How to convince experiments to use a new OO HEP library? Experience shows: It is hard to impossible to coordinate or enforce usage of common products. Different time-scales and decisions in experiments make common development difficult. Chance for Success: Persuade with good quality software and functionality. As systems become more complex, calculations that are verifiable in principle are not verifiable in practice without public access to the code. Authors of open scientific code should be recognized in the same way as they are recognized for scientific papers. February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 21

Summary and Outlook We have to develop and grow the understanding of the benefit of a common library to achieve a common long-term goal. Good ideas and leadership must be acknowledged and supported with resources. We want the best algorithms and modules available to HEP, this needs the best people! The academic career path for HEP software builders must be better developed. Publishing code should be equivalent to publishing a paper! February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 22

Planning - 2: Make a list of jobs to be done February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 23

Planning - 4: Assign people to jobs Each job has a name February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 24

Planning - 5: Manage expectations, allow a margin error, have a fallback position February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 25

Implementation 6: Use an appropriate leadership style February 8, 2000 CHEP 2000, Padova M. Kasemann, FNAL 26