Introduction Wilson Rosa AFCAA wilson rosa pentagon af mil

Introduction Wilson Rosa, AFCAA wilson. rosa@pentagon. af. mil CSSE Annual Research Review March 8, 2010

Agenda • Overview 2

Project Background • Goal is to improve the quality and consistency of estimating methods across cost agencies and program offices through guidance, standardization, and knowledge sharing. • Project led by the Air Force Cost Analysis Agency (AFCAA) working with service cost agencies, and assisted by University of Southern California and Naval Postgraduate School • We will publish the AFCAA Software Cost Estimation Metrics Manual to help analysts and decision makers develop accurate, easy and quick software cost estimates for avionics, space, ground, and shipboard platforms. 3

Stakeholder Communities • Research is collaborative across heterogeneous stakeholder communities who have helped us in refining our data definition framework, domain taxonomy and providing us project data. – – Government agencies Tool Vendors Industry Academia SLIM-Estimate™ True. Planning® by PRICE Systems 4

Research Objectives • Establish a robust and cost effective software metrics collection process and knowledge base that supports the data needs of the United States Department of Defense (Do. D) • Enhance the utility of the collected data to program oversight and management • Support academic and commercial research into improved cost estimation of future Do. D software-intensive systems 5

Software Cost Model Calibration § Most program offices and support contractors rely heavily on software cost models § May have not been calibrated with most recent Do. D data § Calibration with recent data (2002 -Present) will help increase program office estimating accuracy 6

AFCAA Software Cost Estimation Metrics Manual Table of Contents Chapter 1: Software Estimation Principles Chapter 2: Product Sizing Chapter 3: Product Growth Chapter 4: Effective SLOC Chapter 5: Historical Productivity Chapter 6: Model Calibration Chapter 7: Calibrated SLIM-ESTIMATE Chapter 8: Cost Risk and Uncertainty Metrics Chapter 9: Data Normalization Chapter 10: Software Resource Data Report Chapter 11: Software Maintenance Chapter 12: Lessons Learned 7

Manual Special Features • Augment NCCA/AFCAA Software Cost Handbook: – Default Equivalent Size Inputs (DM, CM, IM, SU, AA, UNFM) – Productivity Benchmarks by Operating Environment, Application Domain, and Software Size – Empirical Code, Effort, and Schedule Growth Measures derived from SRDRs – Empirically Based Cost Risk and Uncertainty Analysis Metrics – Calibrated SLIM-Estimate™ using most recent SRDR data – Mapping between COCOMO, SEER, True S cost drivers – Empirical Dataset for COCOMO, True S, and SEER Calibration – Software Maintenance Parameters 8

Manual Special Features (Cont. ) • Guidelines for reconciling inconsistent data • Standard Definitions (Application Domain, SLOC, etc. ) • Address issues related to incremental development (overlaps, early-increment breakage, integration complexity growth, deleted software, relations to maintenance) and version management (a form of product line development and evolution). • Impact of Next Generation Paradigms – Model Driven Architecture, Net-Centricity, Systems of Systems, etc. 9

Agenda • • • Overview (Barry Boehm) Data Analysis Software Sizing Recent Workshop Results Conclusions 10

Do. D Empirical Data • Data quality and standardization issues – – – – No reporting of Equivalent Size Inputs – CM, DM, IM, SU, AA, UNFM, Type No common SLOC reporting – logical, physical, etc. No standard definitions – Application Domain, Build, Increment, Spiral, … No common effort reporting – analysis, design, code, test, CM, QA, … No common code counting tool Product size only reported in lines of code No reporting of quality measures – defect density, defect containment, etc. • Limited empirical research within Do. D on other contributors to productivity besides effort and size: – – – Operating Environment, Application Domain, and Product Complexity Personnel Capability Required Reliability Quality – Defect Density, Defect Containment Integrating code from previous deliveries – Builds, Spirals, Increments, etc. Converting to Equivalent SLOC • Categories like Modified, Reused, Adopted, Managed, and Used add no value unless they translate into single or unique narrow ranges of DM, CM, and IM parameter values. We have seen no empirical evidence that they do… 11

SRDR Data Source 12

Data Collection and Analysis • Approach – Be sensitive to the application domain – Embrace the full life cycle and Incremental Commitment Model • Be able to collect data by phase, project and/or build or increment • Items to collect – SLOC reporting – logical, physical, NCSS, etc. – Requirements Volatility and Reuse • Modified or Adopted using DM, CM, IM; SU, UNFM as appropriate – Definitions for Application Types, Development Phase, Lifecycle Model, … – Effort reporting – phase and activity – Quality measures – defects, MTBF, etc. 13

Data Normalization Strategy • Interview program offices and developers to obtain additional information not captured in SRDRs… – Modification Type – auto generated, re-hosted, translated, modified – Source – in-house, third party, Prior Build, Prior Spiral, etc. – Degree-of-Modification – %DM, %CM, %IM; SU, UNFM as appropriate – Requirements Volatility -- % of ESLOC reworked or deleted due to requirements volatility – Method – Model Driven Architecture, Object-Oriented, Traditional – Cost Model Parameters – True S, SEER, COCOMO, SLIM 14