1 Cumulative IOT E Results Through FY 2008

1

Cumulative IOT&E Results Through FY 2008 Ideal Effective Number Effective And Suitable FY 08 Suitable Total: 28% of Systems Not Suitable 2007: 4 of 8 (50%) Not Suitable 2008: 2 of 6 (33%) Not Suitable Cumulative Number of BLRIP Reports 2

DSB DT&E Taskforce Main Conclusion May 2008 • “. . . the single most important step necessary to correct high suitability failure rates is to ensure programs are formulated to execute a viable systems engineering strategy from the beginning, including a robust reliability, availability, and maintainability (RAM) program, as an integral part of design and development. No amount of testing will compensate for deficiencies in RAM program formulation. ” 3

Section 231 Report to Congress Core T&E Principles 1. T&E should concentrate on measuring improvements to mission capability and operational support based on user needs; 2. T&E programs should experiment. . learn and understand the strengths and weaknesses of a system and its components, and the effect on operational capabilities and limitations; 3. DT and OT activities should be integrated; 4. T&E should begin early, be more operationally realistic, and continue through the entire system life-cycle; 5. Evaluation should be conducted in the mission context expected at time of fielding to the user. . . in terms of operational significance; 6. Evaluations should include a comparison against current mission capabilities; 7. Evaluations should take into account all available data and information; 8. T&E should exploit the benefits of appropriate M&S. 4

New Acquisition/T&E Policies Mc. Queary/Finley Memo on Assessment of Op Test Readiness (21 May 2007) • The DUSD(A&T) shall conduct an independent Assessment of Operational Test Readiness (AOTR) for all ACAT ID programs and special interest programs designated by the USD(AT&L) • The CAE shall consider the results of the AOTR prior to making a determination of materiel system readiness for IOT&E. 5

New Acquisition/T&E Policies Young Memo on Competitive Prototyping (19 Sep 2007) • All acquisition strategies requiring USD (AT&L) approval must be formulated to include competitive, technically mature prototyping through MS B. 6

New Acquisition/T&E Policies Young-Mc. Queary T&E Policy Letter - (22 Dec 2007) • DT and OT test activities shall be integrated and seamless • Evaluations shall include a comparison with current mission capabilities • T&E should assess improvements to mission capability and operational support based on user needs • To more effectively integrate DT and OT, evaluations shall take into account all available and relevant data and information, including contractor data • Operational evaluators will continue to fulfill their statutory roles in providing assessments of operational effectiveness, operational suitability, and survivability to the Milestone Decision Authority • To realize the benefits of modeling and simulation, T&E will be conducted in a continuum of live, virtual, and constructive environments. 7

New Acquisition/T&E Policies Mc. Queary-Finley Memo on Reliability Improvement WG (15 Feb 2008) • Ensure programs are formulated to execute a viable systems engineering strategy, including a RAM growth program. • Ensure government organizations reconstitute a cadre of experienced T&E and RAM personnel. • Implement mandated integrated DT and OT, including the sharing and access to all appropriate contractor and government data and the use of operationally representative environments in early testing. 8

New Acquisition/T&E Policies Mc. Queary-Finley Memo defining Integrated Testing (May 2008) • “Integrated testing is the collaborative planning and collaborative execution of test phases and events to provide shared data in support of independent analysis, and evaluation. ” 9

New Acquisition/T&E Policies Young Memo on RAM Policy (July 2008) • The Service Secretaries are directed to establish Service policy to do the following: – Effective collaboration between the requirements and acquisition communities – Development contracts and acquisition plans must evaluate RAM during system design. – Evaluate the maturation of RAM through each phase of the acquisition life cycle. 10

Senior Leadership Buy-In of New Reliability/T&E Policies “Having performance is important, but not as important in most cases, as having reliability. ” - Hon. Donald Winters, Secretary of the Navy (Sept 3, 2008) 11

Reliability, Availability, Maintainability and Cost Manual (RAM-C Manual) DOT&E on JCIDS Functional Control Boards GEIA Standard 009, RFP and Contract Language, Investment Model Reliability Growth in design phase RAM growth monitoring for incentives, Young/Bolton memos RAM program Evaluation and Standards, testing KPP RAM field data collection, feedback Ri Ri Ri gh gh an an t t. D t. R t. V t. D d d Co ev eq al Re es In n id ce tr el ui de ig at op nt ac re si n io iv t m m gn n es en en ts t gh Ri gh t. N ex t. I nc re m en t Ownership Cost Number of Failures in the Field Initiatives to Improve Reliability, Maintainability, and Availability

Basic Model Improvement Ratio 10 R 2 =. 81 8 6 5 4 3 2 1 0. 8 2. 119 0. 6 0. 5 Investment = Reliability Improvement Ratio 0. 4 X APUC 0. 3659 0. 3 0. 2 = Major System 0. 6 0. 8 1 2 3 4 5 6 7 8 10 20 30 40 50 60 80 100 200 300 400 500 700 1000 2000 Investment/ APUC Investment/APUC 13

Notional Example Effect of Reliability Investment on System Cost (UAV) 3000 20 -year Cost, $M $ M 20 -Year Cost, 2500 2000 1500 1000 500 0 0 20 40 60 80 100 120 Investment in. Reliability Improvement, $M Reliability, $ M 14