Jet Propulsion Laboratory The Conundrums of the Costing

Jet Propulsion Laboratory The Conundrums of the Costing World Jairus Hihn October 28, 2008 Phoenix SMAP 23 rd International Forum on COCOMO and Systems/Software Cost Modeling The research described in this paper was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration

Background The Jet Propulsion Laboratory is a Federally Funded Research & Development Center operated by the California Institute of Technology for the National Aeronautics and Space Administration. As part of the NASA team, JPL enables the nation to explore space for the benefit of humankind by developing robotic space missions to: Explore our own and neighboring planetary systems. Search for life beyond the Earth's confines. Further our understanding of the origins and evolution of the universe and the laws that govern it. Enable a virtual presence throughout the solar system using the Deep Space Network and evolving it to the Interplanetary Network of the future. 2

Measurement Estimation & Analysis “Creating a Quantitative SW Management Culture” Helping Projects Estimation & Planning Measuring Development Performance Size Growth & Stability Resources & Cost • SW Cost Estimation Handbook • Software Cost Estimation Class • Estimation Support • Estimation Tools – Flight SW Cost Model (Team X) – SCAT: Probabilistic COCOMO – SLIC: Code Counter Schedule & Progress Product Quality • Project SW Measures Guide • Managing using Software Metrics Class • Measurement Support • Software Repositories – Cost – Defects – Foundation Measures • SW Engineering Models to support task planning Organizational and Process Measures • JPL SW baselines and trends • Measure Impact of SQI – Cost & Schedule Growth • Process Metrics – Productivity • Benchmarking – Reuse – Defect Density – Cycle Time Hihn MEs. A 1 -3

Background (cont. ) JPL has around 5, 000 employees 800 work with software in some capacity SQI has 15 FTE MEs. A 4 FTE Assessed at Maturity Level 3 in September 2007 Scope was Class B and C Mission Software 4

Cost Estimation Issues: Point 1 All the people with power know just enough to be dangerous Cost Estimators and Modelers must become change agents 5

Models Work But Getting Buy-in is Still a Struggle Jet Propulsion Laboratory The research described in this paper was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration

OCM Key Points Organizational Change does not come quickly or easily The improvement process needs to be approached with many of the same deliberate methods and practices that are used in actual system development. There are similarities between change at the individual and organizational level The onus is on us to proactively reach out to customer/others and promulgate change It is essential to proactively reach out to customers instead of merely waiting for them to come to you. 7

The Onus Is On Us Think about it, what are the alternatives? Blaming others Why won’t they tool/methodology? (colleagues) use our Why don’t those #$@@ managers force them to adopt our new approach? Repeatedly OCM studies have documented that this approach never works The Change Agent must accept responsibility for proactively reaching out to ‘customers’ and promulgating change 8

Percentage in Each Adopter Category Excerpted from http: //www. valuebasedmanagement. net 9

OCM Stages We Are Here Internalization COMMITMENT TO CHANGE Institutionalization Adoption Initial Use Understanding Awareness Contact 2 4 5 6 7 Mechanisms to support sustaining the change Mechanisms to support wider rollout of change Mechanisms to support measured success in piloting Mechanisms to assure understanding 3 Mechanisms to promote awareness 1 TIME Hihn Adapted from Out from Dependency: Thriving as an Insurgent in a Sometimes Hostile Environment, Su. Z Garcia and Chuck Myers, SEPG Conference, 2001 1 -10

Parametric Cost Estimation Evolution Year SW Cost Activities Organizational Reaction OCM 19861989 Softcost: developed by Tauseworth and Reifer for DSN but not used much Collecting COCOMO metrics for ISS, SCT: probabilistic COCOMO JPL had very limited interest and did not want to be bothered. Contact 19891991 JPL’s first SPI (SORCE) provided funding to collect COCOMO data with effort borken out and detailed write ups Focus on non-NASA projects, later 2 GDS were willing to participate. Awareness 19901995 DSN Implementation Office paid us verify cost using SCT and identify risks, collect metrics, 10 page reports. Begrudging recognition was a good thing Initial Use 19962000 No or little institutional funding, isolated special requests for some larger GDS projects. Cost Growth Study Two of the Mgrs we worked with used models for main estimates Limited Use Awareness in Program Dir. 20012007 SDR requires multiples estimates All day cost and metrics class SCAT is advertised as one of SQI’s best products Team X: Flight SW Cost Model and Chair PSSE position is required More and more managers and reviewers expect to see model based probabilistic estimates. Projects often requested to explain differences. Adoption to Institutionalization Today Growing number of very vocal supporters and independent users of SCAT, especially in Flight SW community SDSPs being revised and include direct references as a requirement for model based estimates Costing is ahead of rest of SQI and we have mostly engaged early majority. Now facing late majority and laggards. Hints of Internalization in early adopters, but they are changing! Early Majority at Initial Use 11

Cost Estimation Issues: Point 2 Budget ‘bogies’ get set very early in lifecycle. Sometimes based on casual conversations. You will typically get held to this number!! Current proposal and planning process encourages/ demands under-estimating in early stages of lifecycle 12

What Should One Cost? Development Cost What Flies Proposed Performance Index In our environment we need to know what it will cost after all the *X!*! happens 13

Cost Estimation Issues: Point 3 We improve our models but rarely do we improve our techniques Our models have too many inputs If samples are small not only is the data not normal it is not log normal either 14

Key Research Results Part 1 • Following is based on research with Tim Menzies using data mining techniques to analyze COCOMO • 2004 -2008 • The results are … • Our models have too many inputs – Measures of RE go up with over specified models – Problem is what is best varies • Manual stratification does not lead to the ‘best’ model – E. g. a combination of flight SW and Class B ground produces a ‘better’ model then just selecting all your flight records and doing LC. – Nearest Neighbor searches for analogous records based on your current project model inputs 15

Key Research Results Part 2 The results are … • The same approach is never best but some combination of the following always wins – Local Calibration – Column Pruning – Row Pruning based on Nearest Neighbor • Which is best is determined case-by-case • Recalibrating every time you estimate 16

Some Codicil Thoughts • There is no such thing as a point estimate that means anything, must account for uncertainty • We built SCAT a probabilistic COCOMO in Excel • Often the earliest simplest estimate is the best • Too often we talk ourselves into unrealistic low costs or get beaten into it. • Give Me Multiple high level estimates any day – Detailed estimates get lost in the trees and in an environment of requirements volatility it is a house of cards. • Whatever you do be consistent 17