Approved for Public Release; Distribution Unlimited. #06 -0697 Risk Analysis & Estimating Uncertainty …and what this has to do with the price of milk in Mc. Lean. The Society of Cost Estimating & Analysis (SCEA) Greater Washington DC Chapter Phil Beenhouwer May 17, 2006 MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only.
Approved for Public Release; Distribution Unlimited. #06 -0697 Agenda l The Problem l Variability & Standard Deviation l Briefing Goals l An Uncertainty Primer l Definitions l Distributions and Simulation… l Excel Prowess l Benefits, Headlines, Other Disciplines MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 1
Approved for Public Release; Distribution Unlimited. #06 -0697 My Premise… There are three kinds of people in the World……. , ……………. those who can count, and those who can’t. The lack of risk/uncertainty analysis and poor cost/schedule estimating (across all levels of costing) are the primary reasons that Programs are over-budget. MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 2
Approved for Public Release; Distribution Unlimited. #06 -0697 "It's tough to make predictions, especially about the future. " -- Yogi Berra. The Problem Combined Cost Modeling and Cost Driver Uncertainty Cost = a + b. Xc Cost Modeling Uncertainty Cost Estimate $ Historical data point Cost Driver Uncertainty Input variable MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. Cost estimating relationship Standard percent error bounds Cost Driver Source: Timothy P. Anderson, The Aerospace Corporation, 2005 Do. DCAS Symposium 3
Approved for Public Release; Distribution Unlimited. #06 -0697 “Figures lie "There are lies, and liars damned lies figure…” and statistics. " “Variability” and “Standard Deviation” l When someone says it’ll cost you $100… l The average American carries $9, 000 in credit card debt. 1 - In reality, most Americans owe nothing to credit card - 1 companies. Most households that carry balances owe $2, 000 or less. Only about 1 in 20 American households owes $8, 000 or more on credit cards. Card. Web. com, a service that tracks credit card trends. The Economic and Decision Analysis Center MITRE © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 4
Briefing Goals l The “threshold” is for you to think in terms of risk, uncertainty, and variability… Hey Phil. . . , there’s a good chance that we’ll have to replace the hardware every two to three years… Timmy’s saying that the soccer trip to Europe will only cost three grand, but… You know, Phil…. . , we may have to place a sensor suite anywhere from 1. 2 to 3. 1 miles… Phil…, I think we should plan for a worst-case of 2. 5 million transactions per year… l The “objective” is for you to speak in terms of risk, uncertainty, and variability…, and then to apply it… MITRE The Economic and Decision Analysis Center Approved for Public Release; Hmmm…, Unlimited. #06 -0697 Distribution this Contractor’s saying that the kitchen remodel will cost twenty grand, but… © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 5
Approved for Public Release; Distribution Unlimited. #06 -0697 An Uncertainty Primer… (Slide 1 of 4) SDLC Phase l Concept and Business Case l Resolve the lack of milk at home l Initiation and Authorization l Spousal approval/funding l Project Definition l Buy a gallon of milk l System Design l Find merchant along route home l Construction l Drive to store and purchase milk l Acceptance l Did you have enough money? l Operational Readiness MITRE Action l Get a clean glass! The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 6
Approved for Public Release; Distribution Unlimited. #06 -0697 An Uncertainty Primer… (Slide 2 of 4) l Gallon of Milk Data Collection - Convenience Store: $3. 49 - Warehouse Club #1: $2. 59 - Warehouse Club #2: $2. 69 - Grocery Store #1: $2. 69 - Grocery Store #2: $3. 19 - Grocery Store #3: $2. 89 - Grocery Store #4: $2. 79 - Specialty Store #1: $3. 09 - Specialty Store #2: $2. 99 - Specialty Store #3: $3. 29 MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. ED AT C UN R T 7
Approved for Public Release; Distribution Unlimited. #06 -0697 An Uncertainty Primer… (Slide 3 of 4) l So, if you budgeted $3. 00 for milk (the mean), you can go to… - Warehouse Club #1: $2. 59 - Warehouse Club #2: $2. 69 - Grocery Store #3: $2. 89 - Grocery Store #1: $2. 69 - Specialty Store #2: $2. 99 I could go - Grocery Store #4: $2. 79 here with l But you can’t go to… - Convenience Store: $3. 49 - Specialty Store #3: $3. 29 - Grocery Store #2: $3. 19 - Specialty Store #1: $3. 09 MITRE The Economic and Decision Analysis Center 21 cents more… 80% of the stores sell milk for less than $3. 21 per gallon. © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 8
Approved for Public Release; Distribution Unlimited. #06 -0697 An Uncertainty Primer… (Slide 4 of 4) l What can you afford to buy at these stores? l. But you can’t go to… - Conv. Store: $3. 49 - Is a half-gallon acceptable? - Spec. Store #3: $3. 29 - Grocery Store #2: $3. 19 - Will you even leave with milk? - Spec. Store #1: $3. 09 - How will ‘Operational Readiness’ go when you get home? - Can you use ‘legacy’ orange juice in tomorrow’s cereal instead? - Will you need to reduce the number of cereal users? - Will you need to cut all cereal training from the budget? - Will there be a GAO report on your pillow in the morning? l “Sure”, you say, “if I end-up at the Convenience Store with only $3. 21, I can find the 28 cents I need from under the driver’s seat…”. But what if these weren’t dollars, but billions of dollars…? Could you find $28 M under the Program Manager’s seat? - MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 9
Approved for Public Release; Distribution Unlimited. #06 -0697 “Program Risk” versus “Estimating Uncertainty” l Total Risk = Risk. Program + (+/- Uncertainty. Estimation)1 l For the purposes of quantitative risk analysis, I have defined: - “Program Risk” – the probability and severity of loss linked to hazards 2 (e. g. , software development cannot begin if the environment is not ready, system testing might fail, etc. ) “Estimating Uncertainty” -- the estimated amount or percentage by which an observed or calculated value may differ from the true value 3 (e. g. , the number of users could be 25% less, the COTS license cost could be $1, 000 more, training could take one week longer, etc. ) l Should also consider benefits and schedule, in addition to cost - MITRE 1 Source: Keith Horenstein, The Economic and Decision Analysis Center The MITRE Corporation © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 2 Source: Do. D Dictionary, www. dtic. mil/doctrine/jel/doddict/ 3 Source: www. answers. com 10
Approved for Public Release; Distribution Unlimited. #06 -0697 Definitions l Simulation: any analytical method meant to imitate a real-life system. 1 l Monte Carlo simulation: a simulation that randomly generates values for uncertain variables over and over to simulate a model. 1 l xth percentile: the percentage at which x% of all outputs are at, or below, the associated cost value i. e. , the 80 th percentile in the gallon of milk example means that 80% of the values are at, or below, $3. 21. Conversely, 20% of the values exceed $3. 21. - MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 1 Source: Decisioneering’s website; www. decisioneering. com 11
Approved for Public Release; Distribution Unlimited. #06 -0697 Description of Monte Carlo Tools l “Crystal Ball applications transform your spreadsheets into dynamic models that solve almost any problem involving uncertainty, variability and risk. ” 1 l “Simply by running a simulation, @RISK takes your spreadsheet model from representing just one possible outcome to representing thousands of possible outcomes. ” 2 MITRE 2 Source: Palisade website The Economic and Decision Analysis Center www. palisade. com/ © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 1 Source: Decisioneering’s website; www. decisioneering. com 12
Approved for Public Release; Distribution Unlimited. #06 -0697 Triangular Distributions… Perfectly symmetrical; used far too often Probably the most realistic…. , ‘low’ and ‘most likely’ are the same!! Getting better…. , skewed to the right MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 13
Approved for Public Release; Distribution Unlimited. #06 -0697 …Triangular Distributions… 80 th Percentil e MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 14
Approved for Public Release; Distribution Unlimited. #06 -0697 Monte Carlo Simulation MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 15
Approved for Public Release; Distribution Unlimited. #06 -0697 Monte Carlo Output MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 16
Approved for Public Release; Distribution Unlimited. #06 -0697 …Other Common Distributions… Uniform distribution: Normal distribution: bellrepresents a range with curved shape represents no ‘most likely’ value exam. scores, height of the people in this room, etc. Discrete distribution: distinct points represents the roll of a die Another discrete distribution: represents the unique costs for four pieces of hardware MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 17
Approved for Public Release; Distribution Unlimited. #06 -0697 With a little Excel prowess… With every iteration, the ‘number of years’ changes, and this formula computes the annual cost based on the ‘number of years’. MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 18
Approved for Public Release; Distribution Unlimited. #06 -0697 With a little Excel prowess… MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 19
Benefits to Quantifying Risk/Uncertainty & Using Monte Carlo Approved for Public Release; Distribution Unlimited. #06 -0697 l Identify and apply risk/uncertainty within a model where it really exists (I. e. , risk/uncertainty does not really exist “+/- 10%” around software integration!) l Sensitivity analysis l Risk-adjusted estimates are included in the individual items of the model instead of as a bottom-line “tax” Just present all your numbers at the 80 th percentile! Makes it harder for decision-makers to remove the “risk” risk line-item - MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 20
Approved for Public Release; Distribution Unlimited. #06 -0697 Headlines l GAO Testimony: “CAPITOL VISITOR CENTER -- Results of Risk -based Analysis of Schedule and Cost”, GAO-06 -440 T, February Cost 15, 2006 l GAO Report: “INFORMATION TECHNOLOGY -- Agencies Need to Improve the Accuracy and Reliability of Investment Information”, GAO-06 -250, January 2006 Information l GAO Report: “DEFENSE MANAGEMENT -- Additional Actions Needed to Enhance DOD’s Risk-Based Approach for Making Resource Decisions”, GAO-06 -13, November 2005 Decisions MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 21
Approved for Public Release; Distribution Unlimited. #06 -0697 Relationship to Other Disciplines l Budgeting / Investment Planning Provides risk-adjusted requirements of the E 300 and other planning activities - l Acquisition Cost, Schedule, and Performance are part of defining an acquisition strategy - l Contracting Provides a cost basis for negotiation - l Program Management Provides insight into Program risks; helps prioritize mitigations It’s just good Program Management !! - l Earned Value/Baseline Management Provides an input to the management reserve level More objective inputs to the EVMS than the typical Integrator provides Quantification of risks and uncertainties will result in less re-baselines - l Engineering Offers an approach to incorporate uncertain aspects of the engineering design - MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 22
Approved for Public Release; Distribution Unlimited. #06 -0697 MS Project Monte Carlo Analysis MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 23
Approved for Public Release; Distribution Unlimited. #06 -0697 Conclusion l “Think” and “speak” in terms of risk and uncertainty (and then apply it…) Collect uncertainty inputs when you gather data Use Monte Carlo applications (e. g. , Crystal Ball, @Risk, etc. ) $ These are relatively inexpensive compared to other applications in a coster’s toolkit $ There is even a $15 application that we are currently investigating (Excel Business Solutions’ XL Modeling: www. excel-modeling. com/index_007. htm) l Include “Program Risk” and “Estimating Uncertainty” in cost, benefit, and schedule analyses - MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 24
And if all else fails…. . , re-define the word “outlier”… MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. Approved for Public Release; Distribution Unlimited. #06 -0697 25
Approved for Public Release; Distribution Unlimited. #06 -0697 Questions / Comments MITRE The Economic and Decision Analysis Center © 2006 The MITRE Corporation. All rights reserved. For MITRE Internal Use Only. 26
Скачать презентацию Approved for Public Release Distribution Unlimited 06 -0697
8f85b7c82c14e51677324ddef7639010.ppt