New Processes and Estimation Methods for Acquiring 21

Скачать презентацию New Processes and Estimation Methods for Acquiring 21

8e31e7dd2a64f46bcd499b4770f670e5.ppt

Количество слайдов: 84

New Processes and Estimation Methods for Acquiring 21 st Century Software-Intensive Systems of Systems Barry Boehm boehm@cse. usc. edu Jo Ann Lane jolane@usc. edu Winsor Brown awbrown@cse. usc. edu Acquiring 21 st Century SISOS University of Southern California Center for Software Engineering COCOMO Forum 2005 COCOMO Forum – October 2005 © USC CSE 2005

Overview • Characteristics of 21 st century software-intensive systems of systems (SISOS) • Major SISOS acquisition risks – Addressal via risk-driven spiral model – Associated SISOS estimation challenges • New processes for 21 st century SISOS • New estimation methods for 21 st century SISOS • Case study Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 2

What is a “System-of-Systems”? • Very-large systems developed by creating a framework or architecture to integrate – Existing systems – Systems currently under development – New systems to be developed • So. S system components independently developed and managed – Some outsourced by LSI – Some externally evolving Acquiring 21 st Century SISOS © USC CSE 2005 • Business Domain: enterprise -wide and corss-enterprise integration to support core business enterprise operations across functional and geographical areas • Military Domain: dynamic communications infrastructure to support operations in a constantly changing, sometimes adversarial, environment • So. S activities often planned and coordinated by a Lead System Integrator (LSI) COCOMO Forum 2005 3

What is a “Lead System Integrator”? • Organization (or set of organizations) selected to accomplish the definition and acquisition of So. S components, and the continuing integration, test, and evolution of the components and So. S • Typical activities – – – – Lead concurrent engineering of requirements, architecture, and plans Identify and evaluate technologies to be integrated Conduct source selection Coordinate supplier activities and validate So. S architecture feasibility Integrate and test So. S-level capabilities Manage changes at the So. S level and across the So. S-related IPTs Manage evolving interfaces to external systems • Typically do not develop system components to be integrated (possible exception: So. S infrastructure) Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 4

What Does an SISOS Look Like: Future Combat Systems Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 5

What Does an SISOS Look Like: Network Centric Operations and Warfare Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 6

The Need for Software-Intensive Systems of Systems (SISOS) • Lack of integration among stove-piped systems causes – – Unacceptable delays in service Uncoordinated and conflicting plans Ineffective or dangerous decisions Inability to cope with fast-moving events • Increasing SISOS benefits – – See first; understand first; act first Network-centric operations coordination Transformation of business/mission potential Interoperability via Integrated Enterprise Architectures Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 7

Integrated Enterprise Architectures Federal Enterprise Architectural Framework (FEAF) DOD Architectural Framework (DODAF) Zachman Framework Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 8

System Acquisition Trends Traditional Acquisition Current/Future Trends • Standalone systems • Everything connected (maybe) • Stable requirements • Rapid requirements change • Requirements determine capabilities • COTS capabilities determine requirements • Control over evolution • No control over COTS evolution • Enough time to keep stable • Ever-decreasing cycle times • Failures locally critical • Failures globally critical • Reductionist systems • Complex, adaptive, emergent So. Ss • Repeatability-oriented process, maturity models • Adaptive process models Result: Sequential acquisition practices are increasingly inadequate • Fixed-requirements, -cost, -schedule contracting • Waterfall legacies (e. g. , MIL-STD-1521 B, Reviews and Audits) Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 9

Complexity of Solution Spaces • Size: 10 -100 MLOC • Number of external interfaces: 30 -300 • Number of “Coopetitive” suppliers: 20 -200 – Even more separate work locations • Depth of supplier hierarchy: 6 -12 levels • Number of coordination groups: 20 -200 – Reviews, changes, risks, requirements, architecture, standards, procedures, technologies, -ilities, integration, test, deployment, personnel, infrastructure, COTS, … – Key personnel spend 60 hours/week in meetings • Unprecedentedness • Emergence • Rapid change Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 10

Complexity of Solution Spaces - Breadth, Depth, and Length Width Platform N • • F • Platform 1 T DO Infra C 4 ISR 1. 0 2008 LP M 2. 0 3. 0 4. 0 5. 0 2012 2014 2016 … Command Control Situation Assessment Info Fusion Sensor Data Management Sensor Data Integration Sensors Sensor Components : Legend: DOTMLPF C 4 ISR Depth Acquiring 21 st Century SISOS © USC CSE 2005 Length COCOMO Forum 2005 Doctrine, Organization, Training, Materiel, Leadership, Personnel, Facilities Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance 11

Need Simultaneous Agility and Discipline • Discipline for planning and structure – Foundations (architecture, organizations) • Agility to handle the environment – Rapid, continuous change – Concurrency of development – Many suppliers, coordination groups, external interfaces • Use risk analysis to determine how much agility, discipline is enough Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 12

Top-10 Risks: Software-Intensive Systems of Systems - Cross. Talk, May 2004 1. Acquisition management and staffing 2. Requirements/architecture feasibility 3. Achievable software schedules 4. Supplier integration 5. Adaptation to rapid change 6. Quality factor achievability and tradeoffs 7. Product integration and electronic upgrade 8. Software COTS and reuse feasibility 9. External interoperability 10. Technology readiness Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 14

How Much Architecting Is Enough? -A COCOMO II Analysis 10000 KSLOC Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution Added Schedule Devoted to Rework (COCOMO II RESL factor) Total % Added Schedule Sweet Spot 100 KSLOC Sweet Spot Drivers: Rapid Change: leftward 10 KSLOC High Assurance: rightward Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 15

Effect of Unvalidated Requirements -15 Month Architecture Rework Delay $100 M Arch. A: Custom many cache processors $50 M Arch. B: Modified Client-Server Available budget Original Spec 1 After Prototyping 2 3 4 5 Response Time (sec) Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 16

Effect of Unvalidated Software Schedules • Original goal: 18, 000 KSLOC in 7 years – Initial COCOMO II, SEER runs showed infeasibility – Estimated development schedule in months for closely coupled SW with size measured in equivalent KSLOC (thousands of source lines of code): – Months =~ 5 * 3√KSLOC - KSLOC 300 1000 3000 10, 000 - Months 33 50 72 108 • Solution approach: architect for decoupled parallel development; Schedule As Independent Variable (SAIV) process Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 17

The SAIV* Process Model – Cross Talk, January 2002 (http: //www. stsc. hill. af. mil/crosstalk) 1. Shared vision and expectations management 2. Feature prioritization 3. Schedule range estimation and core-capability determination - Top-priority features achievable within fixed schedule with 90% confidence 4. Architecting for ease of adding or dropping borderline-priority features - And for accommodating past-IOC directions of growth 5. Incremental development - Core capability as increment 1 6. Change and progress monitoring and control - Add or drop borderline-priority features to meet schedule *Schedule As Independent Variable; Feature set as dependent variable. Also works for cost, schedule/cost/quality as independent variable. Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 18

COTS Upgrade Synchronization and Obsolescence • • Many subcontractors means an increasing number of evolving COTS interfaces Aggressively-bid subcontracts can deliver obsolete COTS – New COTS released every 8 -9 months (GSAW) – COTS unsupported after 3 releases (GSAW) – An actual delivery: 120 COTS; 46% unsupported • • Emphasize COTS interoperability in source selection process Develop contract/subcontract provisions/incentives to ensure – Consistency and interoperability across contractor and subcontractordelivered COTS software products – Such products are recent-release versions • • Develop a management tracking scheme for all COTS software products in all CSOS software elements Develop a strategy for synchronizing COTS upgrades Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 20

“Spiral Development” Definition • A risk-driven process model generator • Used to guide concurrent engineering • Two distinguishing features: – Cyclic approach for growing system definition – Anchor point stakeholdercommitment milestones Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 21

Risk Patterns Determine Spiral Specifics Example 1: Boeing 777 • System requirements, architecture well understood • Major risks – Assembly line, tooling rework; long-lead components; supplier interface mismatches • Thorough PDR, CDR – Many suppliers developing to inconsistent requirements, interfaces • Complete, validate these early – Some human interface, safety risks • Early prototypes, analysis • Risk-driven waterfall variant of spiral Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 22

Risk Patterns Determine Spiral Specifics Example 2: Business Decision Support • System architecture provided by Enterprise Resource Planning (ERP) package • Major risks – Requirements not well understood; emergent with system use • Human interface; decisionrelevant data and processing – Rapid requirements change • Evolutionary development on ERP package Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 23

Risk Patterns Determine Spiral Specifics Example 3: Complex C 4 ISR • Major risks – Requirements not well understood; emergent with system use – Architecture not well understood – Many suppliers developing to inconsistent interfaces – Ambitious schedule; rapid change – Many technical, program, resource risks • Full spiral development with anchor point milestones, increments Inception Elaboration Constr-1 Constr-2 … – Example: automated IFF Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 24

Different Risk Patterns Generate Different Spiral Realizations Waterfall (777) Product Spec Completeness CDR Full Spiral (M-S C 4 ISR) Iter. 3 Iter. 2 Source Selection PDR Iter. 1 LCA Iter. 0 Evolutionary (ERP) SRR LCO Product Completeness Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 25

Need for Emergent (Evolutionary, Spiral) Processes • Products and system requirements not prespecifiable – – • Selected COTS strengths and weaknesses Selected best-of-breed suppliers User IKIWISI (“I’ll know it when I see it…”) Emergent needs via usage Processes not pre-specifiable 1. Determine best COTS products 2. Compensate for COTS shortfalls Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 26

SISOS Challenges for Traditional Estimation Methods • Overlapping increments – Hard to classify effort, calibrate details • Test I 1, develop I 2, design I 3 • Cross-increment change management • Massive concurrency – Across breadth, depth, length, versions – Multiple process types with different drivers • Acquisition, systems engineering, COTS-based, agile, plan-driven – Hard to determine dependencies – Many potential sources of critical path slippage • Rapid change – Across breadth, depth, length, versions • Estimating additional LSI effort, schedule – 5%? 15%? 50%? – Effort and schedule drivers Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 27

Schedule Estimation Very Challenging • Synchronization of system schedules – Accounting for slippages, slack – No simple cube-root rules • Only tasks on critical path count – Overlaps, concurrency make critical path hard to determine • Deterministic estimation is optimistic estimation • Tradeoffs between hasty source selection and later integration delays Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 28

How much Architecting is Enough? - A COCOMO II Analysis 10000 KSLOC Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution Added Schedule Devoted to Rework (COCOMO II RESL factor) Total % Added Schedule Sweet Spot 100 KSLOC Sweet Spot Drivers: Rapid Change: leftward 10 KSLOC High Assurance: rightward Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 29

Future System Types and Processes • Mainstream enterprise operation – Domain-specific user programming on evolving ERP infrastructure • Evolving product lines – Software-intensive: business, public service, infrastructure – Hardware-intensive: cars, buildings, devices, robots – Scalable evolutionary spiral processes • Evolving, complex, net-centric systems of systems – Defense-, crisis-, transportation-management – Scalable evolutionary spiral processes • Unprecedented exploratory systems – Bio-computing, nanotechnology, virtual reality – Skill-intensive rapid prototyping Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 31

Emerging Scalable Spiral Process - For 21 st century systems engineering and acquisition • The C 4 ISR Metaphor for NCSOS Acquisition – Role of OODA loops – Example: Internet Spiral – Example: FCS Win Spiral Model • Risk-Driven Scalable Spiral Model – Increment view – Life-cycle view – Role of anchor point milestones • Acquisition management implications • People management implications Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 32

Acquisition C 4 ISR Via Spiral OODA Loops Observe new/updated objectives, Orient with respect to stakeholders • Usage monitoring • Risk/Opportunity analysis • Competition, technology, marketplace ISR • Business case/mission analysis constraints, alternatives Example: ARPANet/Internet Spiral priorities, feasibility, risks • Prototypes, models, simulations Operate as current system Accept new system Act on plans, specifications • Keep development stabilized • Change impact analysis, preparation for next cycle (mini. OODA loop) Decide on next-cycle capabilities, architecture upgrades, plans • Stable specifications, COTS upgrades • Development, integration, V&V, risk management plans • Feasibility rationale Life Cycle Architecture Milestone for Cycle Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 33

The Internet Spiral Process Ref: USAF-SAB Information Architectures Study, 1994 Approved Internet Standard Full Implementation Approved Draft Standard Approved Proposed Standard Widespread Implementation and Test Multiple Implementation and Test Working Group Evolution Equipment IESG Approval IETF Review Acquiring 21 st Century SISOS © USC CSE 2005 Unapproved Proposed Standard IESG Approval Working Group Review COCOMO Forum 2005 Unapproved Draft Standard Unapproved Internet Standard IESG = Internet Engineering Steering Group IETF = Internet Engineering Task Force 34

FCS Win Spiral Model Driven By: 2 1 b. Stakeholders Identify System Objectives, Constraints, & Priorities (OC&Ps); Alternative Solution Elements 2 a. Evaluate Alternatives with respect to OC&Ps Successcritical stakeholder s’ win 1 conditions 1 a. Identify 8 7 3 Success-Critical Stakeholders Build Bl 1 Stakeholders’ 3 2 L CA L CO 4 Spiral anchor point milestones 3. Elaborate Stakeholders’ Review 6 Acquiring 21 st Century SISOS © USC CSE 2005 Risk Managemen t 2 b. Assess, Address Risks Commitment 4. Verify and Validate Product and Process Definitions Progress Through Steps Product and Process Definition Feasibility Rationale 5 COCOMO Forum 2005 35

Risk-Driven Scalable Spiral Model: Increment View Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 36

Risk-Driven Scalable Spiral Model: Increment View Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 37

Risk-Driven Scalable Spiral Model: Life Cycle View System LCA System Inception System, DI 1 LCA System Elaboration DI 2 B/L LCA Changes Agile DI 2 (OO&D) Rebaselining Plan-Driven DI 1 Construction (A) DI 1 V&V Plan-Driven DI 2 Construction (A) LCA: Life Cycle Architecture IOC: Initial Operational Capability OO&D: Observe, Orient and Decide V&V: Verification and Validation DI: Development Increment B/L: Baselined Acquiring 21 st Century SISOS © USC CSE 2005 DI 2 LCA DI 2 V&V COCOMO Forum 2005 38

Risk-Driven Scalable Spiral Model: Life Cycle View System LCA System, DI 1 LCA System Inception DI 2 B/L LCA DI 3 B/L LCA Changes System Elaboration Agile DI 2 (OO&D) Rebaselining Plan-Driven DI 1 Construction (A) DI 1 V&V Changes Update DI 1 IOC DI 1 Trans’n DI 1 Usage DI 2 LCA Agile DI 3 (OO&D) Rebaselining Plan-Driven DI 2 Construction (A) DI 2 V&V Changes Update DI 2 IOC DI 2 Trans’n DI 2 Usage DI 3 LCA Agile DI 4 (OO&D) Rebaselining LCA: Life Cycle Architecture IOC: Initial Operational Capability OO&D: Observe, Orient and Decide V&V: Verification and Validation DI: Development Increment B/L: Baselined Acquiring 21 st Century SISOS © USC CSE 2005 DI 4 B/L LCA Plan-Driven DI 3 Construction (A) DI 3 V&V DI 3 IOC DI 3 Trans’n DI 3 Usage. . . DI 4 LCA. . . COCOMO Forum 2005 39

LCO (MS A) and LCA (MS B) Anchor Points Pass/Fail Criteria • A system built to the given architecture will – – – Support the operational concept Satisfy the requirements Be faithful to the prototype(s) Be buildable within the budgets and schedules in the plan Show a viable business case Establish key stakeholders’ commitment to proceed LCO: True for at least one architecture LCA: True for the specific life cycle architecture; All major risks resolved or covered by a risk management plan Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 40

Spiral Feasibility Rationale Deliverable • LCO, LCA reviews not just UML/Power. Point charts • Need to show evidence of product and process feasibility • Evidence provided by prototypes, production code, benchmarks, models, simulations, analysis – Sizing and cost/schedule model results for process feasibility • Evidence provided in advance to LCO/LCA review team – Key stakeholders, specialty experts • Lack of evidence risks destabilizing the process – Needs coverage by viable risk mitigation plan • Key new progress metric – Feasibility evidence progress vs. plans Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 41

Acquisition Management Implications - I • 20 th century build-to-spec contracting practices usable in part – Good fit for stabilized-increments team – But not for rebaselining, V&V teams • Time & materials or equivalent • Award fee based on cost/effectiveness • These apply all the way down the supplier chain • Need top-level award fee for cost-effective team balancing – No stable distribution of effort Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 42

Acquisition Management Implications - II • Don’t skimp on system definition phases – But avoid analysis-paralysis – Use Feasibility evidence generation as progress metric • Use more evidence-based source-selection processes – Competitive exercise as proof of capability – Preceded by multistage downselect • Use Schedule/Cost as Independent Variable processes – Prioritized features as dependent variable • Top priority: transformational empowerment of acquisition corps – Education, mentoring, tools, techniques Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 43

Staffing Implications • Critical success factors for system development organizations – Understanding, dealing with sources of system value • And associated stakeholders, applications domains – Software/systems architecting and engineering • Creating and/or integrating system components • Using risk to balance disciplined and agile processes – Ability to continuously learn and adapt • Getting the right people on the right teams—it takes different types of people to make a successful team – – Plan-driven teams: Thrive on order Agile teams: Thrive on chaos Verification and validation teams: Thrive on oversight All supported by a collaborative environment to encourage continuous learning, optimization, and adaptation Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 44

Overview • Characteristics of 21 st century software-intensive systems of systems (SISOS) • Major SISOS acquisition risks – Addressal via risk-driven spiral model – Associated SISOS estimation challenges • New processes for 21 st century SISOS • New estimation methods for 21 st century SISOS – Evolution of estimation methods – Cost estimation – Schedule estimation • Case study Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 45

Evolution of Estimation Methods to Support SISOS Acquisition Activities • Estimation approaches need to more closely reflect development approaches – Processes, both agile and traditional – Architecture – Factors for • • Security Reliability Building for reuse COTS incompatibilities – People factors • • • Politics Timeliness of key decisions Vendor compatibility Experience levels Resource availability • Need to integrate existing “stove -pipe” cost models to better capture relationships between various teams and activities – Integrate SE, COTS, software development, hardware development/manufacturing • Need to add in budgets for activities not covered in current cost models – LSI – Post development life cycle activities (e. g. , installation/deployment, operation, maintenance) Estimates evolve over time, starting with upfront “knowns”, then expanding to include tasks identified as part of architecture decisions Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 46

Observations on How Processes Are Adapting to the SISOS Environment • Traditional planning and scheduling – Plan activities as independent projects • Requires that up-front SISOS architecting be performed in sufficient detail to allow sub-projects to be somewhat independent of each other • Requires that risk-driven processes be used to identify and manage risks early at SISOS and sub-project levels – Blend traditional processes with more agile processes • Plan for stabilized evolutionary increments • Concurrently have agile change/risk/opportunity team – Performs acquisition intelligence/surveillance/reconnaissance functions – Rebaselines future increment solutions • Plan for early and continual verification and validation (V&V) – Competing priorities: use stakeholders to negotiate priorities with other on-going system component enhancements and maintenance Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 47

Observations on How Processes Are Adapting to the SISOS Environment (continued) • Project monitoring and control – Minimize impacts on key personnel – Prioritize oversight areas • Integrated project management – Identify key cross-cutting processes for standardization – Allow flexibility in other areas • Let organizations to use their own proven processes • Supplier organizations have been selected by the independent system component “owner” for their technical expertise and ability to produce • Decision making process – Need to reduce to the extent possible • Number of required SISOS-level decisions • Number of clearances or approvals required for each decision – Studies indicate that the probability of success decreases as the number of required decision approvals increases Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 48

Observations on How Processes Are Adapting to the SISOS Environment (continued) • Risk management – Cross-cutting risks need to be managed and balanced across system and organizational boundaries – Each risk needs a responsible “owner” and committed suppliers – Risk portfolios and “owners” to manage cross-cutting risks • Integrated product teams typically play a much larger role and have more responsibilities • The people processes are at least as important as the technical processes – Personal, organizational, and political motivations and priorities can impact the success of the project Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 49

SISOS Schedule Estimation LCO Schedule = Effort/Staff Try to model ideal staff size Source Selection RFP, SOW, Evaluations, Contracting Effort/Staff COSOSIMO -like risks, rework Customer, Users Similar, with added change traffic from users… COSOSIMO -like LCA 1 LCA 2 Develop to spec, V&V CORADMO risks, rework Risk-manage slow-performer, completeness LSI – Agile LSI IPTs – Agile Suppliers – Agile risks, rework Proposal Feasibility Acquiring 21 st Century SISOS © USC CSE 2005 Increments 2, … n Assess sources of change; Negotiate rebaselined LCA 2 package at all levels Effort/staff at all levels Proposals Degree of Completeness IOC 1 Increment 1 So. S Architecting Rework LCO LCA Packages at all levels Effort COSYSMO-like. Elaboration Assess compatibility, short-falls Inception LCA -like Integrate COSOSIMO -like COCOMO Forum 2005 Similar, with added rebaselineing risks and rework… risks, rework LCA 2 shortfalls Suppliers – PD – V&V LSI – Integrators 50

SISOS Cost Estimation • Many models exist to support estimation of various aspects of system development – – Systems engineering Software development COTS integration Hardware development and manufacturing • Others are currently under development – Lead system integrator effort – Security Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 51

System Engineering Cost Model: COSYSMO • Addresses first four phases of the system engineering lifecycle (per ISO/IEC 15288) Conceptualize Develop Oper Test Transition to & Eval Operation Operate, Maintain, or Enhance Replace or Dismantle • Considers standard Systems Engineering Work Breakdown Structure tasks (per EIA/ANSI 632) • Developed with USC-CSE Corporate Affiliate sponsorship and INCOSE participation Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 52

How is Systems Engineering Defined? • EIA/ANSI 632 • Processes for Engineering a System: – Acquisition and Supply • Product Realization – Implementation Process – Transition to Use Process • Supply Process • Acquisition Process • – Technical Management Technical Evaluation – – • Planning Process • Assessment Process • Control Process – System Design Systems Analysis Process Requirements Validation Process System Verification Process End Products Validation Process • Requirements Definition Process • Solution Definition Process Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 53

COSYSMO Operational Concept # Requirements # Interfaces # Scenarios # Algorithms + 3 adjustment factors Size Drivers Effort Multipliers - Application factors -8 factors - Team factors -6 factors Acquiring 21 st Century SISOS © USC CSE 2005 COSYSMO Effort Calibration COCOMO Forum 2005 54

My. COSYSMO* Capabilities • Jointly developed by USC/CSE and Raytheon • Provides costing using local rates as well as effort • Supports multiple levels of estimate formality/complexity – Budgetary estimate – Rough order of magnitude (ROM) – Proposal • Embeds local systems engineering program performance data – Systems Engineering size and productivity – Environmental data – Local site salary grade profiles • Provides for more consistent inputs and outputs, reduces variability * Developed by USC CSE Raytheon Affiliate, Gary Thomas Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 55

My. COSYSMO* Capabilities (continued) • Focus on risk, uncertainty • Provides user friendly Interface and documentation • Provides for local site expansion – Size drivers and effort multipliers – Site unique parameters • Historical data collection mode * Developed by USC CSE Raytheon Affiliate, Gary Thomas Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 56

How Much Effort to Architect and Integrate a System of Systems? System of Systems ? person-years (PY) Sensing 500 PY Command & Control 1000 PY Vehicles 500 PY Common 400 PY Infrastructure 600 PY • Systems developed by system contractors – Total effort 3000 person-years • System of systems integration functions – So. S abstraction, architecting, source selection, systems acquisition, integration, test, change management effort • How much to budget for integration? • What factors make budget higher or lower? • How to develop and validate an estimation model? Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 57

Constructive System-of-System Integration Cost Model (COSOSIMO) • Parametric model to estimate the effort associated with the definition and integration of softwareintensive “system of systems” components • Includes at least one size driver and 6 exponential scale factors related to effort • Targets input parameters that can be determined in early phases • Goal is to have zero overlap with COCOMO II and COSYSMO Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 58

Scope of Proposed So. S Cost Model Size Drivers Scale Factors COSOSIMO So. S Definition and Integration Effort Calibration • Characteristics of So. Ss supported by cost model – – Strategically-oriented stakeholders interested in tradeoffs and costs Long-range architectural vision for So. S Developed and integrated by an LSI System component independence • Size drivers and scale factors – Based on product characteristics, processes that impact LSI effort, and LSI personnel experience and capabilities Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 59

SISOS Schedule Estimation: A Composite Approach LCO Schedule = Effort/Staff Try to model ideal staff size Source Selection RFP, SOW, Evaluations, Contracting Effort/Staff COSOSIMO -like risks, rework Customer, Users Similar, with added change traffic from users… COSOSIMO -like LCA 1 LCA 2 Develop to spec, V&V CORADMO risks, rework Risk-manage slow-performer, completeness LSI – Agile LSI IPTs – Agile Suppliers – Agile risks, rework Proposal Feasibility Acquiring 21 st Century SISOS © USC CSE 2005 Increments 2, … n Assess sources of change; Negotiate rebaselined LCA 2 package at all levels Effort/staff at all levels Proposals Degree of Completeness IOC 1 Increment 1 So. S Architecting Rework LCO LCA Packages at all levels Effort COSYSMO-like. Elaboration Assess compatibility, short-falls Inception LCA -like Integrate COSOSIMO -like COCOMO Forum 2005 Similar, with added rebaselineing risks and rework… risks, rework LCA 2 shortfalls Suppliers – PD – V&V LSI – Integrators 60

Composite Approach: Inception Phase • COSYSMO estimate of system size, effort – – – – Medium difficulty for simplicity Requirements: 5, 000 * 1. 0 = 5, 000 Interfaces: 500 * 4. 3 = 2, 150 Algorithms: 50 * 6. 5 = 325 Scenarios: 200*22. 8 = 4, 560 Total Size: 11, 035 Plus 10% Rqts. Volatility 12, 138 • Nominal effort = 0. 254 * (12138)**1. 06 = 5420 PM Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 61

Composite Approach: Inception Phase -2 • Nominal effort = 5420 person months (PM) • Effort multipliers – Low requirements understanding = 1. 37 – Low architecture understanding = 1. 28 – High technology risk = 1. 32 • Product * Nominal PM = 2. 315 * 5420 = 12, 546 PM • COSYSMO Conceptual Phase Effort = 23% – 7% Inception; 16% Elaboration • Inception Phase Effort = 12, 546 *. 07 = 878 PM • Inception Phase Schedule – Months = 1. 5 * cube root (878) = 14. 4 months – Average staff size = 878/14. 4 = 61 people Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 62

Composite Approach: Elaboration Phase • Sum of schedules for systems engineering, source selection, and post -selection rebaselining • COSYSMO Elaboration effort = 12, 546 *. 16 = 2007 PM • Systems engr. schedule = 1. 5 * cube root (2007) = 19 months – Average staff size = 2007/19 = 106 people • Source selection schedule – Preparation in parallel: no added schedule – RFP finalization and publication: 1 month – Proposal responses: 3 months • Including prototypes, architecture, Feasibility Rationale – – Parallel evaluation, finalist selection: 2 months Finalist compatibility/feasibility, Q&A: 3 months Contracting: 1 month Total schedule: 10 months • Teambuilding, LCA rebaselining: • Total Elaboration Phase schedule: Acquiring 21 st Century SISOS © USC CSE 2005 6 months 35 months COCOMO Forum 2005 63

Composite Approach: Construction Phase • Multiple development increments – Usually 18 -24 months for broad, deep SISOS – Increments sequenced by user needs, cross-dependencies – Assume 18 months plus LSI integration time • COSOSIMO effort; cube root schedule estimator – Scope supplier increments to allow some slack for synchronization and stabilization • And interaction with V&V, agile teams • Agile team schedule fixed – Use COSYSMO Construction effort to baseline team size • OT&E, Transition schedules – Can do all but final increment in parallel with next increment – Use experience on earlier increments to schedule final increment Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 64

Case Study • Inception, Elaboration phase effort and schedule estimates for metropolitan area crisis management system • Using overall approach presented above See handout for case study overview and analysis Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 66

Backup Charts Acquiring 21 st Century SISOS COCOMO Forum 2005

Acquisition Management and Staffing: Effect of Software Under-Representation • Software risks discovered too late • Slow, buggy change management • Recent large project reorganization Software SW SW SW Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 SW 68

Need for CRACK Integrated Team Members - Cross. Talk, December 2003 • Not Collaborative: Discord, frustration, loss of morale • Not Representative: Delivery of unacceptable systems, late rework • Not Authorized: Authorization delays, unsupported systems • Not Committed: Missed action items, discontinuities, delays • Not Knowledgeable: Unacceptable systems, delays, late rework Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 69

Supplier Integration: Rapid Adaptability to Change • Depth of supplier chain increases communication and coordination effort • Inflexible subcontracting is a major source of delays and shortfalls • Develop subcontract provisions enabling flexibility in evolving deliverables. Develop an award fee structure based on objective criteria for: – – – – Schedule Preservation Cost Containment Technical Performance Architecture and COTS Compatibility Continuous Integration Support Program Management Risk Management Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 70

Rapid Adaptability to Change: Architecture • Architecture may be over-optimized for performance vs. adaptability to change • Modularize architecture around foreseeable sources of change – – – Identify foreseeable sources of change Technology, interfaces, pre-planned product improvements Encapsulate sources of change within software modules Change effects confined to single module Not a total silver bullet, but incrementally much better Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 71

Rapid Adaptability to Change: Evolving Software Architecture • Software architecture will need to change and adapt to rapidly changing priorities and architecture drivers – New COTS releases – Evolving enterprise standards and policies – Emerging technologies and competitor threats • Organize software effort to ensure the ability to rapidly analyze, develop, and implement software architecture changes – Empower a focal-point integrator of the software architecture and owner of the critical software infrastructure • Raise to a very high organizational level the – Owner of the software architecture and infrastructure – Owner of SOS software integration and test Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 72

Quality Factor Achievability and Tradeoffs • Quality factor (-ility) tradeoffs are success-critical, difficult to analyze, and incompletely formulated – These tradeoffs address competing requirements for performance, interoperability, security, safety, survivability, usability, modifiability, portability, reusability, accuracy and other attributes • Create critical-mass subcontracts for software tradeoff analysis organizations to conduct continuing modeling, simulation, and execution analyses of success-critical quality factor tradeoff issues Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 73

Rapid, Synchronous Software Upgrades • Out-of-synchronization software upgrades can be a major source of operational losses – Software crashes, communication node outages, out-of-synch data, mistaken decisions – Extremely difficult to synchronize multi-version, distributed, mobile-platform software upgrades – Especially if continuous-operation upgrades needed • Architect software to accommodate continuous-operation, synchronous upgrades – E. g. , parallel operation of old and new releases while validating synchronous upgrade • Develop operational procedures for synchronous upgrades in software support plans • Validate synchronous upgrade achievement in operational test and evaluation Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 74

COTS: The Future is Here • Escalate COTS priorities for research, staffing, education – It’s not “all about programming” anymore – New processes required Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 75

SISOS Compound Software Risks • Serious inter-system compound risks will be discovered late – Compound risks are frequently architecture-breakers, budgetbreakers, and schedule-breakers – Examples include closely-coupled immature technologies and closely-coupled, ambitious critical path tasks • Establish a hierarchical software risk tracking compound risk assessment scheme – The top level is Top-10 system-wide software risks – Tier down to system-level and subcontractor-level Top-10 risk lists – Valuable both for overall software risk management and for compound risk assessment. Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 76

SISOS Compound Software Risks (continued) • Develop high-priority plans to decouple high-risk elements and to reduce their risk exposure • Establish a SISOS Software Risk Experience Base – This is extremely valuable in avoiding future instances of previously experienced risks Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 77

Failure Mode I: Build-to-Spec Deliverables —Purchasing agent metaphor • Rapid change: heavy spec change traffic, slow contract changes • Plus deep supplier chain: slowdowns multiply, changes interact • Plus emergent requirements: many initial specs wrong; more changes • Plus build-to-spec award fee: supplier inertia • Bottom line: late rework, overruns, mission shortfalls Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 78

Why Software Projects Fail Overruns: 189% cost; 222% schedule; 61% of product Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 79

18 -24 Month Development Increments Need More Concurrency DIDR DIAR DIRR DIPR SOS Engineering DIDR System / HW Supplier Guidance DIPR SOS SW So. S SW DI Guidance Element Team (SW) SW Supplier Element Team / HW Supplier DIDR ET SW DI Guidance So. S HW DI Build Guidance DIPR LCO or SSR Arch Risk Design DIRR LCA or PDR DIAR DI Assessment Results Arch Risk Design TRR Arch Risk Design DIDR = Development Increment Definition Review DIPR = Development Increment Planning Review DIRR = Development Increment Readiness Review DIAR = Development Increment Assessment Review Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 80

Failure Mode II: Sequential Document-Driven Milestones—Waterfall, V-model, MIL-STD-1521 B • Requirements emergence, COTS: freeze requirements too early • Plus document-completion progress metrics: hasty point solutions, undiscovered risks • Plus rapid change: problems with Failure Mode I • Bottom line: more late rework, overruns, mission shortfalls Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 81

The Cost of Hasty Specifications 15 -Month Architecture Rework Delay $100 M Required Architecture: Custom; many cache processors $50 M Original Architecture: Modified Client-Server Original Cost Original Spec 1 After Prototyping 2 3 4 5 Response Time (sec) Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 82

Failure Mode III: Hasty Best-of-Breed Source Selection • Complexity, emergence: incomplete, unvalidated system architecture, solicitation SOWs • Deep, wide supplier chains: incompatible legacy solutions, COTS; critical-path modeling and simulation needs • Rapid change: rapid COTS evaluation, version obsolescence – GSAW: 8 -11 months/release; 3 supported releases • Bottom line: serious integration problems, overruns, mission shortfalls Over-ambitious startup schedule… Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 83

Failure Mode IV: Incremental Document-Driven Development • High assurance of –ilities: deferral to later increments • Complex NCSOS: early-increment architecture inadequate for later-increment –ilities. • Rapid change: destabilization of ambitious increment schedules; increment completion delays; next increment destabilized • Bottom line: serious security, safety, scalability problems; destabilized development; more rework, overruns, shortfalls Risk-insensitive “spirals”; ambitious increment schedules Acquiring 21 st Century SISOS © USC CSE 2005 COCOMO Forum 2005 84