One Army One Team One Vision Operational Research

One Army, One Team, One Vision Operational Research An Introduction to the Development, Application, Uses and Limitations of STAMPER: Systematic Task Analysis for Measuring Performance and Evaluating Risk Eugenia Kalantzis Directorate Land Combat Development Operational Research Team Canadian Department of National Defence 21 st ISMOR Norton Manor, Sutton Scotney, Hampshire, UK September 2 nd, 2004 Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Agenda • • • Sponsor Mandate and Objectives The Seminar Wargame Series OR Roles and Objectives Methodology STAMPER Concluding Remarks Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research The Sponsor • Director General Land Combat Development (DGLCD) – Director Land Strategic Concept – Director of Army Doctrine Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research DGLCD Mandate Linking the Army of Today to the Interim Model, Army of Tomorrow and the Future Army. Today Tomorrow Interim model Une Armée, une équipe, une vision Recherche opérationnelle Future

One Army, One Team, One Vision Operational Research Force Employment Seminar Wargame Series • Series of Seminar Wargames to evaluate evolving FE concepts Subsequent Iterations Baseline Iterations Four Corners War Fighting Domestic Non-Combatant Evac Peace Support Current Structures NEO Four Corners NEO Coy Gp BG/ Bn Gp Complex PSO BG/ Bn Gp Coy Gp NEO BG/ Bn Gp Domestic Coy Gp NEO BG/ Bn Gp Bde Coy Gp Complex PSO Warfighting Bde Area/Bde Domestic Warfighting Complex PSO Four Corners Domestic Bde Complex PSO Four Corners Area/Bde Complex PSO Four Corners Coy Gp NEO BG/ Bn Gp Warfighting Bde Area/Bde Domestic Baseline Performance Updated Concepts Warfighting Updated Performance x MCF Brigade Group HQ Updated Structures x Command Support MCF Brigade Group HQ Comand Sense Act Shield Sustain Comand Sense Act Shield Command Support Sustain MP MP Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Force Employment Seminar Wargame Series • Seminar wargame format – 5 days and 45 participants • Detailed scenarios, vignettes and forces • Joint Conflict and Tactical Simulation (JCATS) for situational awareness • STAMPER • Structures judgments and insights sessions Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Traditional Combat in Open HEAVY FEATHERS RABBIT REDUX WILD HONEY LEATHER STOCKINGS Complex Terrain (but not urban) X HERA Urban Operations Baku MEDUSA HEAVY FEATHERS APHRODITE WILD HONEY Une Armée, une équipe, une vision Gormanskaya Recherche opérationnelle

One Army, One Team, One Vision Operational Research OR Roles and Objectives • Role: – Problem definition, guidance, data collection and analysis, presentation of results • Objectives: – Develop a comprehensive and robust methodology to measure performance and evaluate risk factors – Present results in a clear concise manner to be briefed to decision-makers Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Special Considerations • • Imperfect knowledge of models, equipment, structures, particularly for the Interim Army Transient nature of the wargaming teams Nature of the seminar wargame - hidden complexity in the verbal interplay Multiple levels participation Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Analysis plan – Two Tier Approach • Quantitative – • STAMPER Qualitative – – – Observation Sheets Strengths/ Weaknesses/ Issues Sheets Judgment & Insights Sessions Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research STAMPER – Systematic Task Analysis for Measuring Performance and Evaluating Risk • Development of the DGLCD Task Lists • Development of Survey Instruments • Ratings rolled-up for overall performance per operational function, and evaluation of risk • Interpretation of results • Comparison of results from one seminar to another Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research DGLCD Task Lists • Developed by Combat Developers and assessment teams • Comprehensive list of tasks per Operational Function: Command, Sense, Act, Shield and Sustain • Level 1, Level 2 and Level 3 Tasks Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Sample of Command Task List Breakdown Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research DGLCD Task Lists • Client-analyst collaboration was key • Process: Iterative “straw man” approach • Benefits: – Engagement of participants – sense of ownership – Constant review and refinement – Synergies achieved Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Survey Instruments • Based on the DGLCD Task Lists • One EXCEL based survey instrument per operational function • Assessors asked to rate tasks along two dimensions: – Performance – Impact/Importance Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Survey Instruments… Survey Questions Response Options List of tasks Assessor’s responses for all 6 vignettes Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Performance Measurement The simple model… Performance score of a task = weighted average of the performance scores of the tasks belonging to it. For example: The performance of a Level 2 task is the weighted sum of the performance scores of the Level 3 tasks that make up that Level 2 task. Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Roll-up of scores… Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Assessment of Risk For Level 3 tasks only… • Low score = higher risk • Color-coded for visual cuing • Numerical values for ranking Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Automated Tool Risk Performance Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Example 1. High Risk – Low Performance Command Tasks Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Example 2. Operational Function Overall Scores Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Strengths • Client/analyst collaboration • Structured evaluation of performance for combat developers • Confirmation of seminar discussion results • Visualization of results – Colour-coding and graphs – Quick identification of performance/risk deltas • Near real-time results – Available for discussion at end-of-day J&I – Clarification of remaining issues Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Caution… • Should not compare results across op functions – Within operational function – Across scenarios/vignettes (e. g. open vs. urban) – Across capability sets (i. e. across seminar wargames) • Deltas should be used as indicators, pointers to an underlying phenomenon requiring investigation • Results to be interpreted along with qualitative results Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Abusing the Simple Model Ex. 1 Assuming all tasks are performed at an acceptable level Ex. 2 Assuming equal improvement/ investment across operational functions Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Concluding Remarks • Seminar wargame format appropriate for scope of investigation • Complimentary data collection methodologies: quantitative and qualitative (STAMPER) provided complete picture • STAMPER – – – Met the objectives: consistent, repeatable, and simple Structured evaluation Good communications tool Results should be interpreted with caution Provided an indication of performance/risk to a level deemed appropriate, and not beyond Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Questions? Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Une Armée, une équipe, une vision Operational Research Recherche opérationnelle

One Army, One Team, One Vision Operational Research From Baseline to Next Iterations Baseline System: MCF Mission: Objectives, threat, environment, etc. New Structure New System: Modified Capability Set Same mission Mission: Objectives, threat, environment, etc. Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Specification Error Models, Complexity and Error Model Error Complexity Measurement Error Minimize Model Error Complexity Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Specification Error Models, Complexity and Error Model Error Complexity Measurement Error Minimize Model Error by decreasing complexity Complexity Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Survey Instrument Questions Question 1: Performance Given your experience and the discussions held during the seminar wargame, please indicate the level of overall performance you believe we would most likely achieve in the completion of this task during this mission, given the capability-set assumed for this seminar wargame. Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Survey Instrument Questions Question 2: Impact For this particular mission, how would you characterize the impact or importance of this task in the overall performance of the higher-level task it belongs to. Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Point Allocation Matrix Score = Sum of points Sum of weights Assessor Response to Question 1 Une Armée, une équipe, une vision Assessor Response to Question 2 Recherche opérationnelle

One Army, One Team, One Vision Operational Research Example 4. Distribution of Sense L 3 Scores For Level 3 Tasks: Q 1 results Q 2 results Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Example 2. Comparison Across Seminars Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Sponsor Objectives • To assess/measure the impact of changes in structures, equipment and capabilities that will come into effect during the Interim Army timeframe with a view to further refining the force employment concept in preparation for field trials at Canadian Manoeuvre Training Centre and eventual incorporation into doctrine • To identify and prioritize discrete issues requiring additional analysis and to direct the examination of these spin-off activities to the appropriate organization Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Survey Instruments Responses Provided by Assessors Derived from the Roll-Up Question 1: Performance Level 3 Tasks Question 2: Impact Level 3 Tasks Level 2 Tasks Level 1 Tasks Une Armée, une équipe, une vision Recherche opérationnelle Operational Function

One Army, One Team, One Vision Operational Research Future Development • Refinement of DGLCD Task Lists • Refinement of weights using techniques such as ranking, pair-wise comparison, etc. • Increase number of surveys collected – statistical analysis • Hand-held data entry devices for instantaneous collection of surveys Une Armée, une équipe, une vision Recherche opérationnelle

One Army, One Team, One Vision Operational Research Abusing the Simple Model Ex. 3 Assuming improvement in Command exceeds improvement in other operational function; basing investment decisions on this Une Armée, une équipe, une vision Recherche opérationnelle