Using Process Improvement and Knowledge Management for Better

Using Process Improvement and Knowledge Management for Better Predictive Analysis Capability Rick Hefner Marilee J. Wheaton 310. 812. 7290 310. 813. 6510 rick. hefner@trw. com marilee. wheaton@trw. com TRW One Space Park - R 2/2130 Redondo Beach, CA 90278 Presented to Sixteenth International Forum On COCOMO and Software Cost Modeling

Agenda Motivation for Improvement CMMI Level 4 Requirements and Expectations Organizational Metrics Process Database Design and Implementation Lessons Learned Systems Architecture Vision 16 th COCOMO Forum 2

Motivation for Improvement to Level 4 Level 3 ensures well-defined, repeatable processes Competition demands better quantitative understanding of the contributors to cost and schedule • Process productivity • Product quality (which drives rework) Needed better measures, data, and analytic techniques for critical process and product characteristics: • Determine whether processes are behaving consistently or have stable trends (i. e. , are predictable) • Identify improvement in TRW’s standard processes • Identify project practices which may be best practices • Understand the cost-quality-schedule tradeoffs 16 th COCOMO Forum 3

How Does Six Sigma Fit with ISO 9001 and CMMI? Process Improvement Best-Practices Six Sigma Quality Mgmt. ISO 9000 Business Measures Voice of the Customer Change Management SW CMM ISO 9001 Methods & Tools DMAIC DFSS Process Management CMMI • Capability Maturity Model Initiative (CMMI) and ISO 9001 establish the change management and process management framework needed for Six Sigma • Six Sigma methods and tools assist in the quantitative analysis needed at CMMI Levels 4/5 16 th COCOMO Forum 4

CMMI-SE/SW Staged Representation Level 5 Optimizing Focus Continuous process improvement 4 Quantitatively Quantitative Managed management 3 Defined 2 Managed Process standardization Basic project management Process Areas Causal Analysis and Resolution Organizational Innovation and Deployment Quantitative Project Management Organizational Process Performance Organizational Process Focus Organizational Process Definition Organizational Training Integrated Project Management Risk Management Decision Analysis and Resolution Requirements Development Technical Solution Product Integration Verification Validation Requirements Management Project Planning Project Monitoring and Control Supplier Agreement Management Measurement and Analysis Process and Product Quality Assurance Configuration Management 1 Performed 16 th COCOMO Forum 5

Level 4 Changes in the CMMI SW-CMM process areas are split by process and product quality • Quantitative Process Mgmt Identify and correct special causes of process variation • Software Quality Mgmt Develop a quantitative understanding of the quality of the project's software products CMMI process areas are split by organization and project • Organizational Process Performance Maintain a quantitative understanding of the performance of the organization’s set of standard processes, and provide the process performance data, baselines, and models to quantitatively manage the organization’s projects. • Quantitative Project Management Quantitatively manage the project’s defined process to achieve established quality and process performance objectives. 16 th COCOMO Forum 6

Organizational Process Performance (CMMI) Establish and maintain a quantitative understanding of the performance of the organization’s set of standard processes, and to provide the process performance data, baselines, and models to quantitatively manage the organization’s projects. Required Goals • SG 1 Establish Performance Baselines and Models Baselines and models that characterize the expected process performance of the organization's set of standard processes are established and maintained. • GG 3 Institutionalize a Defined Process The process is institutionalized as a defined process. 16 th COCOMO Forum Expected Implementation Practices Expected Institutionalization Practices 7

Expected Implementation Practices • SP 1. 1 Select Processes Select the processes or process elements in the organization's set of standard processes that are to be included in the organization's process performance analyses. • SP 1. 2 Establish Process Performance Measures Establish and maintain definitions of the measures that are to be included in the organization's process performance analyses. • SP 1. 3 Establish Quality and Process Performance Objectives Establish and maintain quantitative objectives for quality and process performance for the organization. • SP 1. 4 Establish Process Performance Baselines Establish and maintain the organization's process performance baselines. • SP 1. 5 Establish Process Performance Models Establish and maintain the process performance models for the organization's set of standard processes. 16 th COCOMO Forum 8

Approach Our Level 3 process database supported cost estimation and process improvement • Surveyed management team to establish business drivers • Defined measures needed to characterize process performance and quality at the organizational level • Defined measures needed to characterize project satisfaction of organizational goals • Identified sub-processes amenable to quantitative management • Defined project measures needed for quantitative management of those sub-processes • Examined improvements in the organization standard process needed to stabilize the process or make measures meaningful • Examined improvements in the project’s defined processes 16 th COCOMO Forum 9

Example Organizational Metrics Collected and Derived Collect base measures • Size • Effort by activity • Cost by activity • Number of defects (By phase) Derive other measures • Defect density (defects/size) • Productivity (size/effort) • Defect containment (defects saves/escapes from defects by phase) • Rework effort • SPI/CPI (planned vs. actual effort) 16 th COCOMO Forum 10

How Projects Use the Organizational Process Assets Process Asset Organizational Standard Process & Organizational Procedures Library (PAL) Organizational Policies Organizational Database Organizational Training Office CMM Senior Management Review Organization Project Plans Project Defined Process & Procedures Project Schedules & Budgets 16 th COCOMO Forum Project Results 11

Project Comparative Data Base (PCDB) PCDB contains: • Cost data from a certified accounting system • Project validated technical characteristics data PCDB relates: • Cost data to standardized (WBS) work elements • Cost data to the technical characteristics of work performed The Office of Cost Estimation uses the PCDB to: • Calibrate the parametric models • Derive cost estimating relationships • Provide historical data for proposals, project planning/replanning • Define risk affordability/analysis • Characterize process performance and quality 16 th COCOMO Forum 12

What is the PCDB ? Accounting Data Project Inputs • PCDB Standard WBS JN Mapping • PCDB Standard WBS Alternate Hours/Cost Mapping Accounting & Descriptive Data • Project Management • Systems Engineering • Hardware Development • Software Development • Systems Integration & Test Accounting Data • Site Activation • Labor hours, dollar costs • Integrated Logistics • Non-labor costs • Configuration Management • Breakdown by PCDB WBS • Data/Documentation element, by labor category Management • Quality Assurance S/W Development Descriptive Data Examples • Development Support Facility • SLOC, other size measures • Operations and Maintenance • ESLOC (derived) • Specialty Customer • Labor Required to Develop Services SLOC/ESLOC – Software Development • Other Activities – Other Project Disciplines • Cost Model Parameters COCOMO Forum 16 th 13

PCDB Support to Cost Modeling and Proposals RFP Functional Requirements Project and Proposal Team Support First Order Estimates Project Comparative Data Base (PCDB) Parametric Cost Models Model Calibration Sanity Check Project Certified Technical Accounting Description Data & Metrics Final Estimates Advanced Pricing System (APS) Cost Volume Final Estimates BOEs CERs Sanity Check Project Data Actuals Management Estimation Information System Notebook (AMIS) 16 th COCOMO Forum Guidelines BOE Generation Support 14

PCDB Data Submittal Process No Project Prepares and Submits Descriptive Data Inputs Place Data in PCDB in the Input State Start Proj Yes OK? Place Data in PCDB in the Review State Retrieve Accounting System Data Project Prepares and Submits Accounting Data Inputs OCE Generate Derived Data and Summary Reports Project Staff Data Review and Validation Update Input Data No Proj & OCE OK? Yes End Perform Parametric Validation if Applicable 16 th COCOMO Forum Place Data in the Reportable State 15

Parametric Validation/Calibration Additional selective validation for software development History Data Points Creation of parametric cost model baseline Input from Project • Descriptive Data Forms: Software Development OCE provides guidance and assistance Provided to Project • Software Parametric Model (i. e. , Costar (COCOMO II), SEER-SEM, Price S, or SLIM) cost estimation validation – Iteration with project to achieve validation within 10% 20% of actual effort Results in calibrated baseline for future estimates 16 th COCOMO Forum 16

Lessons Learned - 1 Determining a common set of metrics is tough because there are different needs (and frequency, granularity, accuracy, etc. ) • Project management decisions • Customer insight • Senior management oversight • Organizational process performance characterization • Process improvement Proper support for metrics collection requires changing the culture to “management-by-data” • Project managers must use the data to manage their projects • Senior managers must use the data to meet organizational business objective • Data collectors must be confident that data will not be used against them 16 th COCOMO Forum 17

Lessons Learned - 2 Understanding variation in process performance allows more insight into estimation • What’s the likely cost of this work? • What’s the probability we can perform the work for $____? Project resistance to data collection is primarily due to the time and effort required to collect and report the data • Must be integrated and consistent with the process Data collection mechanisms require clear instructions, to ensure the desired information is captured and validated • Collection should be a combination of automated and manual methods for cost and accuracy 16 th COCOMO Forum 18

Systems Architecture Vision Current Situation Organizational Metrics Data Project Risk Assessment What if Modeling Financial Data Predictive Analysis Project Risk Assessment Project Mgmt Data What if Modeling Risk Radar Decision Support Layer Organizational Metrics Data Financial Data Proposal Data The End Objective Integrate Metrics with Financial, Project, proposal and other information to support trend and risk analysis 16 th COCOMO Forum Group Common Data Repository Project Mgmt Data Proposal Data 19

Conclusions The process database developed at Level 3 is a key asset in achieving Level 4 The many uses of metrics places additional emphasis on innovative database design and usage Characterizing the organization’s process performance requires: • Definitizing your business goals • Selecting the right metrics • Stabilizing the organization and projects’ processes • Collecting and analyzing the metrics Management and decisions that are data driven result in better predictive analysis capability 16 th COCOMO Forum 20