COMOPTEVFOR Operational Test Director s Course Navy Systems Acquisition

COMOPTEVFOR Operational Test Director’s Course Navy Systems Acquisition Process Mr. Dave Erickson ericksod@cotf. navy. mil (757) 282 -5546 x 3295 DSN 564 Dec 2009 1

Who I Am! – Dave Erickson Background • Two Fleet A-6 Squadron Tours • A-6 FRS Instructor • DT Tour • CVN Department Head (Weps) • COTF Experience (2 tours) • Attack Weapons & Platforms OTC • Plans & Policy 2

Objectives - Introduce Key Documents, Concepts - Explain the Process for System Acquisition - Definitions - Introduce the Principal Players and Roles in T&E 3

Warning-Acronyms Ahead! ASN/RDA ACAT TEMP AOA MNS CDD USD/AT&L 4

Key Guiding Documents • DOD Directive 5000. 1 12 May 2003 The Defense Acquisition System • DOD Instruction 5000. 02 02 DEC 2008 Operation of the Defense Acquisition System • Interim Defense Acquisition Discretionary Best Practices Guidebook 16 Jun 2009 • SECNAVINST 5000. 2 D 16 Oct 2008 Implementation and Operation of the Defense Acquisition System & JCIDS • COTFINST 3980. 1 OTD Manual 5

DODD 5000. 1 and DODI 5000. 02 • Focus on ACAT I (big ticket) programs • DOD Will Set Few Rules For ACAT II and III • Weapons Systems and AIS Systems all Covered by 5000 Series • Acquisition Knowledge Sharing System (AKSS) • Web site https: //akss. dau. mil/default. aspx 6

DODD 5000. 1 and DODI 5000. 02 • Capability Documents replace MNS and ORD • Initial Capabilities Document (ICD) at MS A • Capability Development Document (CDD) at MS B • Capability Production Document (CPD) at MS C • Test & Evaluation Strategy at MS A • TEMP required at MS B 7

SECNAVINST 5000. 2 D (Oct 08) • Identifies service unique processes; adds further guidelines to DOD 5000 series instructions • Defines MDA for all ACAT programs • Defines ACAT IV program requirements • Chapter 5 Integrated T&E – Defines roles & responsibilities – Defines OTRR process – Requires OT reports in 60 days (90 days for ACAT I) 8

OTD Manual • Provides guidance for all OT&E testing policy and execution • Updated via Policy Information Notices (PINs) and Standard Operating Procedures (SOPs) • Provides sample documents • On Line - www. cotf. navy. mil 9

OTD Manual • Eight Chapters: 1) Introduction 2) T&E Policy 3) Foundation Documents 4) Resources 5) TEMP 6) Test Planning 7) Test Operations 8) Evaluation Reports 10

Evolutionary Acquisition (EA) Strategy • Initial system developed and fielded with less than full capability (60 -80% solution). The “core” capability. • Increment 1 capability must be militarily useful • Subsequent developments or increments to improve the capability • ICD -- provides broad, time-phased , operational goals and describes requisite capabilities • CDD -- Prior to MS B. Users shall identify and requirements authority shall validate a minimum set of key performance parameters to guide phase efforts. • CPD -- Prior to MS C. Documents final requirements to be used in production and used for OPEVAL 16

Evolutionary Approach Do. D Strategic Guidance Gap Analysis Joint Operating Concepts Joint Functional Concepts DAB DAB Materiel Solution A Technology C B EMD ICD MDD CPD 1 CDD 1 Analysis Increment 1 Development Ao. A JROC DAB A Technology CDD 2 Development JROC DAB B DAB EMD CPD 2 Increment 2 C JROC DAB DAB A Technology B C EMD CDD 3 CPD 3 Increment 3 Development JROC Continuous Technology Development and Maturation 17

Some Terms To Know Milestone Decision Authority (MDA) - Approves Program Structure/Baseline - Approves Progression to Next Phase of Acquisition - Approves Production (Either Limited or Full) The MDA for a program can be at the SECDEF, Component Secretary or lower level, depending on the cost of the program and the degree of Congressional interest. Programs are categorized by Acquisition Category (ACAT). 18

Some Terms To Know Milestone Acquisition Decision Point Milestone III/ Full Rate Production Decision(FRPD) Decision point to authorize entrance into production Developing Agency (DA) Program Manager’s Office Acquisition Category (ACAT) DOD categorization of programs 19

Some Terms To Know Initial Operational Test and Eval (IOT&E) OT phase conducted on production or production rep system to support FRPD Follow-On Test and Evaluation (FOT&E) All OT that occurs after FRPD Low Rate Initial Production (LRIP) Production of initial articles for purpose of production ramp-up and OT Beyond LRIP (BLRIP) Production over and above that approved for LRIP. Usually associated with FRPD decision. 20

Program Decision Forums Defense Acquisition Board (DAB) Do. D level forum for addressing program decisions Navy Program Decision Meeting (NPDM) Navy level forum for addressing program decisions 2 Pass / 6 Gate CNO/CMC &ASN RDA level forum for assessing program health Review Process 21

2 Pass/6 Gate Review Process • SECNAVNOTE 5000 26 FEB 2008 • Based on a 2 year study to improve program success • Applies to all ACAT I and selected ACAT II programs – Pass 1 led by CNO/CMC with 3 “requirements” gates – Gate 1 reviews/approves DON ICD for submission to Joint Staff – Gate 2 reviews Ao. A results and approves DON preferred alternative – Gate 3 reviews/approves DON CDD for submission to Joint staff – Pass 2 led by ASN RDA with 3 “acquisition” gates – Gate 4 review/approve SDS and proceed to MS B – Gate 5 review/approve EMD RFP release to industry – Gate 6 review overall program health & approve CPD 22

Program Acquisition Categories Acquisition Category (ACAT) ID (Defense) IC (Component) II Milestone Criteria IVT IVM Decision Authority Documents (MDA) Above $365 M RDT&E or $2. 19 B procurement OUSD(AT&L) REQs Doc, AOA, TEMP, TP (FY 00 Dollars) SECNAV REQs Doc, AOA, Same, but delegated to Component (Usually Delegated to ASN(RD&A) TEMP, TP $140 M RDT&E or ASN(RD&A) $660 M Procurement (Or Individual Delegated by ASN(RD&A) REQs Doc, AOA, (FY 00 Dollars) III Required PEO, Syscom, or DRPM decides whether ACAT III or ACAT IV based on military characteristics. PEO, Syscom, or If ACAT IV, COTF gets input on whether IVT or IVM PEO, Syscom, Designated Flag/SES, or PM TEMP, TP REQs Doc, TEMP, TP Designated Flag/SES REQs Doc, TEMP, TP 23

ACAT-I Programs Weapons and C 4 I Programs AIS Systems Ø 365 M RDT&E Ø 378 M Total Life Cycle Cost Or 2. 190 B Procurement Or 126 M Program Costs Or if USD(ATL) designates Or 31. 5 M in any program year MDAP = Major Defense Acquisition Program MAIS = Major Automated Information System 24

ACAT-I Programs ACAT-IA ACAT-ID (Defense) ACAT-1 AM (Major) ACAT-IC (Component) ACAT-1 AC (Component) MDA = USD(AT&L) MDA=ASD(NII) MDA=ASN(RDA) MDA=DON CIO 25

ACAT-II Programs “Major” Programs Weapons and C 4 I Systems: Trips no criteria for ACAT-I and. . . >140 M RDT&E Or 660 M Procurement Or if CAE designates CAE = Component Acquisition Executive = MDA 26

ACAT III and IV Programs ACAT-III = any program that falls short of ACAT-II but… Affects mission characteristics of ships or aircraft or combat capability ACAT IV IVT = OT IVM = Monitor DT 27

Developmental Testing - The design test of any weapon system, equipment, or munitions (or key component) under both laboratory and normal installation conditions (with or without contractor assistance) to ensure the system or equipment meets design technical specifications. Operational Testing - The field test, under realistic operating conditions, of any weapon system, equipment, or munitions (or key component) for the purpose of determining its effectiveness and suitability for use in combat by typical military users; and the evaluation of the results of such tests to resolve stated critical operational issues (COIs). 28

Integrated Test (IT) • Pre-IOT&E test phase in which DT and OT testers share assets and data • OT Framework can be the Test Plan • Independent evaluation of data and reporting 29

Early Operational Assessment (EOA) • Conducted during Concept and Technology Development (OT-I/A) • Uses advanced development models, prototypes, surrogate systems • Provide early operational insight of a system’s significant areas of risk to Effectiveness and Suitability • Continue Program Development recommendation 30

The Defense Acquisition Management Framework EOA 31

Operational Assessments (OA) • OT when maturity of system will not support a full OPEVAL • Identify Potential Areas of Risk • Continue Program Development recommendation • Requires Test Plan and Report 32

The Defense Acquisition Management Framework OA 33

IOT&E (OPEVAL) • Navy’s Independent Evaluation of new systems • Last phase of OT prior to FRPD • Must use production representative system; including personnel and documentation • Operational scenarios • Effective and Suitable determination • Recommendation concerning fleet release 34

The Defense Acquisition Management Framework IOT&E (OPEVAL) 35

DT Assist • • DT Assist is not a phase of formal OT! No OT # assigned COIs not resolved Not used for Fleet release Effectiveness and Suitability not addressed Usually a “test and fix” effort OK to use the data collected - maybe 36

Capabilities Document • Purpose: To identify system-specific performance capabilities, • Replaced Mission Needs Statements (MNS) • Three documents throughout the acquisition life cycle • Initial Capabilities Document (ICD) • Provides broad, time-phased , operational goals and describe requisite capabilities • Capabilities Development Document (CDD) • Prior to MS B. Users shall identify and requirements authority shall validate a minimum set of key performance parameters (KPP) to guide phase efforts. • Capabilities Production Document (CPD) • Prior to MS C. Documents final requirements to be used in production and used for OPEVAL 37

OT Framework • • The primary output of MBTD Basis for IT planning and TEMP Defines adequate OT – what we need For IT events, may serve as the formal test plan –no other test plan needed! 38

Test and Evaluation Master Plan (TEMP) Purpose: To ensure a plan exists for a logical sequence of developmental and operational testing • Some refer to the TEMP as a contract, but the TEMP can be changed. • Typically updated prior to each new Milestone • A joint effort by PM and OTA 39

Test & Evaluation Master Plan (TEMP) Content described in the Interim DAG (9. 6. 2) No required format, however…. (9. 10 Recommended TEMP format) • Introduction • Test Program Mgmt & Integrated Schedule • Test and Evaluation Strategy • Resource Summary 40

Test Plan • Required for each phase of testing • Includes: – Purpose of test – Capabilities/functions of COI’s to be examined – How test will be conducted – Evaluation criteria against which results will be measured – Resources required – Data collection/data analysis methods – E-tests & S-tests to resolve specific COIs 41

Test Plan (Cont. ) • For Non Do. D Oversight Programs - test plan to be signed by COMOPTEVFOR 30 days prior to test • For Do. D Oversight programs: – Concept of test and evaluation brief 120 days prior to commencement of test – Submit COMOPTEVFOR signed Test Plan to DOT&E 60 days prior to commencement of test 42

Final Report • COMOPTEVFOR’s report to CNO regarding each phase of operational test • Contains conclusions regarding: – Operational effectiveness and operational suitability – Recommendation for fleet release • Reports test results and evaluation criteria to support conclusions • Resolution of COIs • 60 Days to complete after completion of OT (90 for ACAT I) 43

DOCUMENT SEQUENCE Initial Capabilities Document Mission Need Statement (MNS) (ICD) Test Phase Test Plan Test Operational Requirements Capabilities Development & Document (ORD) Production Documents (CDD/CPD) Test Final Report OT Framework Test and Evaluation Master Plan (TEMP) 44

Key Players 45

User/User rep. = “Sponsor” • Sets Requirements/ Provides Funding • Signs ORD/ICD/CDD/CPD • Makes Cost/Performance Tradeoff Decisions • Owns the “Checkbook” • Note: Congress really owns the checkbook! 46

Program Manager • Responsible for All Facets of Program Management • Negotiates & Manages Contracts • Directs Developmental Testing (DT) • Recommends Production 47

COMOPTEVFOR • Early Involvement • Designate Test Parameters, based on Requirements • Conduct Operational Testing (OT) • Evaluate Results and Report • Recommend Fleet Introduction - Or Not 48

Service Operational Test Agencies • Navy - Operational Test & Evaluation Force (OPTEVFOR) Norfolk, VA • Army- Army Test & Evaluation Command (ATEC) Alexandria, VA • Air Force - Air Force Operational Test and Evaluation Center (AFOTEC) Albuquerque, NM • USMC - Marine Corps Operational Test and Evaluation Activity (MCOTEA) Quantico, VA 49

Operational Test Director (OTD) • Primary source of information on all aspects of a project • Monitor developmental testing (DT) and if we elect write a report to CNO based on our observations • Conduct detailed test planning, supervision of test execution, and documenting of test results • Draft TEMP, Test Plans, and Final Reports • Brief project 50

Operational Test Coordinator (OTC) • Headquarters focal point on project matters • Primary interface with the Program Managers (PM) and System Commands (SYSCOM) and outside organizations • Liaison VX OTD/ Section Head 51

Service T&E Coordinator (CNO – N 091 for Navy) - Coordinates Processing of TEMPs - Arbitrates Occasional Disagreements (TECG) - Key Interface between OPNAV and acquisition management structure - Funding and Budget issues wrt T&E and S&T - Resolves T&E Issues for CNO 52

DT&E and OT&E Compared DT&E OT&E • Test of the Combat System • Operational Effectiveness • Performance Specs Verification • Operational Suitability • Certification for OT&E • Fleet Introduction • Technical • Operational/Tactical • Controlled Variables • Projected Threat • Contractor/ Lab Technicians Environment: • Test of the Weapon • Technical Risk Assessment Purpose: • Enemy Countermeasures • Fleet Operators/ Maintainers Agents: • Developing Agency (PM) • COMOPTEVFOR (OTD) 53

In Conclusion • Introduced Key Documents and Concepts • Explained the Process for System Acquisition • Definitions • Introduced the Principal Players and Roles in T&E 54

Questions? 55

Break 56