Processes for COTS-Based Systems Tricia Oberndorf LA SPIN

Processes for COTS-Based Systems Tricia Oberndorf LA SPIN 3 September 2003 Software Engineering Institute Carnegie Mellon University Pittsburgh PA 15213 Sponsored by the U. S. Department of Defense © 2003 by Carnegie Mellon University page 1

Outline • Background • Process Overview • High-level Process • Example Low-level Activities Descriptions

Background Building systems today: • Few custom-built to order • Commercial products expected to play major role Diverse things influence the shape and function of a COTS-based system (CBS): • Stakeholder needs • Characteristics of products and marketplace • Degree of interaction with legacy systems Together, these compel many changes in system development and maintenance methods for CBSs. © 2003 by Carnegie Mellon University page 3

COTS Spheres of Influence Use cases Quality attributes Business model Stakeholder Needs / Business Processes Architecture/ Design Marketplace Vendors Market segments Available COTS Products Emerging technology © 2003 by Carnegie Mellon University Patterns Interfaces Infrastructure COSTS Risk management Programmatics/ Project management Risk Change management License negotiation page 4

Marketplace Affects CBS Approach Traditional Engineering Approach Requirement s Architecture & Design Implementati on Requirements-driven © 2003 by Carnegie Mellon University Required CBS Approach Stakeholder Needs/ Business Processes Simultaneous Architecture Marketplace Definition and Tradeoffs Design Programmatics/ Risk Negotiation-driven page 5

Conceptual Bases: Requirements 1 “Nail down the requirements first” will not work – the marketplace will not cooperate • Think of it as a new sphere of influence: System that is created must accommodate competing sources of gravity Constraints come from stakeholder needs AND marketplace imperatives © 2003 by Carnegie Mellon University page 6

Conceptual Bases: Requirements 2 Requirements (cont. ) • Must distinguish between negotiable and nonnegotiable • Keep the non-negotiable set as small as possible • Understand how to prioritize and trade-off the negotiables A risk-driven spiral approach is key: • Frequent use of prototypes • Considerable interaction with system’s end users • Gradual refinement of understanding of system © 2003 by Carnegie Mellon University page 7

Conceptual Bases: Knowledge CBS development and maintenance is dependent on an evolving Body of Knowledge. Non-negotiables Prioritized negotiables Legacy system context Marketplace offerings Bo. K End-user business processes Acquisition constraints © 2003 by Carnegie Mellon University Evolving System View Architecture and design constraints Risks page 8

Process Overview Three classes of activities: • Iterative: short-term, engineering-oriented - Discovery: gather and refine system knowledge - Assembly: construct a prototype - Assessment: determine success of iteration & plan • Pervasive: long-term, organizational scope - e. g. , CM, license management, vendor relationship management, contract tracking and oversight • Executive: event-driven, deal with decision making - e. g. , cost estimating, contract negotiation, project oversight Today we focus on the Iterative. © 2003 by Carnegie Mellon University page 9

Discovery Gather and Refine activities occur simultaneously. Gather: • Sources take many forms. - Stakeholders, business processes, legacy systems, COTS marketplace • Each is independent of the others. • There is no optimal order. Refine: • Analysis and harmonization of the gathered knowledge • Yields the technical definition of the emerging system • Finds gaps, negotiates conflicts © 2003 by Carnegie Mellon University page 10

Discovery Activities Knowledge is sufficient for constructing executable Go on? Continue Discovery Harmonized data Data from: stakeholders products … Gather data z Gather data y Gather data x Gather Mismatch Analyze Refine Negotiate conflicts Agreement Gap - need more data © 2003 by Carnegie Mellon University page 11

Assembly Produces a prototype that reflects what’s been discovered Reflects a traditional sequential process • Guided by local “ candidate requirements” - Iteration Objectives, Detailed Iteration Plan plus hypotheses and desired behaviors expect to derive from prototype - Vets “candidate requirements” • Produces an executable version of full system - Likely to have less than complete functionality - Executable, not paper or specification or small piece Does not preclude prototyping “in the small” throughout © 2003 by Carnegie Mellon University page 12

Evaluation and Assessment Occur at two levels: • Evaluation of the prototype in its own context - Measures actual outcome against expected outcome - Considers degree to which prototype satisfies its local “requirements” • Assessment of the iteration itself - Addresses issues at project (not iteration) level - Considers whether objectives were good ones - Determines whether iteration results indicate changes in project schedule, budget, etc. © 2003 by Carnegie Mellon University page 13

On-going Iterations of A/APCS Deployed System Bo. K Body of Knowledge Prototype © 2003

Top-Level View of A/APCS Manage Program: based on global program constraints, generates iteration objectives and fielding decisions and forwards the current System View to the next iteration Perform Iteration: based on iteration objectives, the marketplace, and current System View, generates a new version of the system, evaluates it, and updates the new System View Assess Iteration: based on iteration objectives and the evaluation of the system, assess the overall iteration © 2003 by Carnegie Mellon University page 15

Detail of Perform Iteration Manage Program Plan Iteration 1 Perform Iteration Construct System *

Detail of Construct System Detailed iteration plan Gather Data 2. 1 Description of needed knowledge Consolidated data to analyze System view (defined in previous iteration) Refine Data Discovery activities COTS products © 2003 by Carnegie Mellon University * System view (defined in this iteration) Replanning needed 2. 2 “Go” decision for Assembly Assemble Candidate System 2. 3 System (for evaluation and potential deployment) page 17

Low-level Activities of Refine Data Detailed iteration plan (to all) Data to analyze Agreements, Gaps & Data 2. 2. 1 Conflicts Resolutions & Negotiate Gaps Conflicts System View (defined in previous iterations) Emerging Gaps revisions to System View © 2003 by Carnegie Mellon University 2. 2. 2 Replanning needed Agreements Form Assembly “Go” System view View 2. 2. 3 Additional Gaps knowledge needed Resolutions Determine Description Data to be of needed Gathered knowledge 2. 2. 4 page 18

Low-level Activity Descriptions Construct System (Step 2) 2. 1 Gather Data: Gather Stakeholder Inputs Gather COTS Product Data Gather Business Process Data Gather External System Data 2. 2 Refine Data: Analyze Data Negotiate Conflicts Determine Data to be Gathered Form System View 2. 3 Assemble Candidate System: Create Detailed Design Perform Needed Modifications Write Custom Code Integrate Components Test © 2003 by Carnegie Mellon University page 19

Example: Gather COTS Product Data Purpose: Gather product data by evaluating COTS products Role: Evaluators, stakeholders Inputs/Entry Conditions: · Description of the kind of product data that is needed, including the level of detail needed as well as gaps that must be filled. This description forms the requirements for the product evaluation. Outputs/Exit Conditions: · Product data to the requisite level of detail, consolidated and formatted Description: This activity consists of methodically evaluating one or more COTS products. As with all of the “gather” activities, the input is the Need for Knowledge generated by the Refine Data activity. Appropriate evaluation criteria for this product data gathering effort are established; these then provide the basis for the quantification methods that guide the data collection effort. © 2003 by Carnegie Mellon University page 20

Example (cont. ) Notes on Planning: … Actions: 1. Define criteria 2. Prioritize criteria 3. Prepare for evaluation 4. Collect data from the sources 5. Record results 6. Consolidate and format data (described in section 7. 1. 5) Notes on the Actions: 1. A criterion consists of both a capability statement … 2. It is essential to rank criteria by importance. … 3. One necessary preparation is to assemble the needed … 4. Different data collection techniques may be needed. … 5. The raw scores earned by each of the examined products … 6. Product evaluation data is often diverse and … © 2003 by Carnegie Mellon University page 21

Converting This to a Process While developing this framework, we engaged with a customer who needed a process PDQ. We took these principles and, using RUP as a backbone, turned them into a process. Original name: ITSEP New name: Evolutionary Process for Integrating COTSbased systems (EPIC) • A negotiation-driven process that helps identify and manage the differences between what users want and what COTS products can deliver • Commercial and government EPIC pilots underway © 2003 by Carnegie Mellon University page 22

EPIC Conceptual Framework Stakeholder Needs/ Business Processes Simultaneous. Architecture/ Marketplace Definition Design and Tradeoffs Programmatics/ Risk Trade Space • • Decisions Converge Trade Space Time Trades are negotiation-driven with knowledge of marketplace Requirements formed based on knowledge of market/architecture Continuous awareness of end-user process changes Project business and contractual approaches support trades © 2003 by Carnegie Mellon University page 23

Knowledge Grows Incrementally Time Executable • Risk-based spiral development focuses and integrates diverse information - Prioritized stakeholder needs, end-user processes Organization and system architecture, design constraints Identified risks, programmatic constraints Marketplace offerings, product characteristics, other buyer usage • Frequent, evolving executable representations demonstrate current understanding © 2003 by Carnegie Mellon University page 24

Continuous Stakeholder Negotiation • Stakeholder needs mature with increased understanding of marketplace implications • Business processes change to leverage available products • End users committed to system solution • Quick resolution to discovered mismatches - Business process owners and end users System engineers and developers Vendors and suppliers Project managers Change agents Stakeholder Buy-in Increases Time © 2003 by Carnegie Mellon University page 25

EPIC: A Negotiation-Driven Approach Accumulate knowledge through risk-based spirals Drive strategic vision to sustainable solution Increase stakeholder buy-in and reconcile end-user processes with COTS-based system Coordinates operational change, system & software engineering, and project management to field initial capability in 6 -18 months © 2003 by Carnegie Mellon University page 26

Iterations Redefined for CBS Gather information Simultaneous Definition and Tradeoffs Plan iteration (1 to 8 weeks per outer cycle) © 2003 by Carnegie Mellon University by identifying and analyzing mismatches and negotiating among stakeholders Refine into harmonized set Assess iteration Assemble executable representation Executable page 27

Phases Bounded by Anchor Points Inception Elaboration Construction Transition Gather and define project scope Refine, experiment & select solution Implement selected solution Field and support solution Survey/try COTS Try/select COTS Apply/track COTS Track/update COTS Agree to business process changes Prototype business process changes Prepare to change business process Change business process … … 6 to 18 months Life. Cycle Objectives © 2003 by Carnegie Mellon University Life. Cycle Architecture Initial Operational Capability page 28

Inception Phase: Goal Achieve concurrence among stakeholders on project’s life-cycle objectives • Match segments of marketplace with critical use cases • Identify implications of potential business process changes • Establish acquisition strategy for appropriate vendor relationships Establish feasibility for the project through the business case that shows there is one or more candidate solutions Survey & Try COTS © 2003 by Carnegie Mellon University page 29

Inception Phase: Activities • Gather high-level information (critical use cases) - “Must have” stakeholder needs and required business processes - Market drivers, technologies, products - Architectural constraints and design alternatives - Programmatic constraints and risks - Incentives /inhibitors to business process changes • Refine information to form high-level candidate solutions • Assemble executable representation(s) © 2003 by Carnegie Mellon University page 30

Elaboration Phase: Goal Define a high-fidelity solution with predictable cost/schedule • Select and acquire COTS products • Business process implementation supported by all user roles Achieve sufficient stability of the solution across architecture, requirements, and marketplace Mitigate development and rollout risks Try & Select COTS © 2003 by Carnegie Mellon University page 31

Elaboration Phase: Activities Amplify each candidate solution . . . Amplify selected solution • Gather detailed knowledge Refine knowledge to form a stable baseline - Gaps/mismatches identified and negotiated - COTS integration mechanisms defined and validated - Business process changes implementation refined - Balanced across all 4 spheres • Assemble architectural prototypes - Business process changes prototyped © 2003 by Carnegie Mellon University page 32

Construction Phase: Goal Achieve a product quality release ready for the user community Prepare functional units for needed business process changes • Organization changes • Training Manage vendor relationships Balance development stability and potential obsolescence with marketplace volatility Apply & Track COTS © 2003 by Carnegie Mellon University page 33

Construction Phase: Activities • Gather marketplace information continually • Refine selected solution as necessary • Assemble selected solution for fielding - Develop any custom components - Tailor products (non-source code customizations) - Develop production quality COTS integration code/data sets - Implement business process changes for initial fielding © 2003 by Carnegie Mellon University page 34

Transition Phase: Goal Roll out and support selected solution set to the user community • Initial fielding (beta testing) • Full fielding • On-going support until solution release retired Implement business process changes across user community Balance operational stability with marketplace volatility Manage relationships with vendors Track & Update COTS © 2003 by Carnegie Mellon University page 35

Transition Phase: Activities • Gather marketplace information continually • Refine solution to accommodate the impact of new products, releases, or technology • Assemble maintenance releases - Re-tailor products (non-source code customizations) - Re-develop COTS integration code/data sets - Re-integrate and re-test © 2003 by Carnegie Mellon University page 36

Closing This presentation has discussed: • a framework for COTS-based systems processes • a specific process based in that framework The principles are broadly applicable, and you are invited to try them out. © 2003 by Carnegie Mellon University page 37

For More Information David J. Carney SEI Pittsburgh, PA 412 -268 -6525 djc@sei. cmu. edu Tricia Oberndorf SEI Pittsburgh, PA 412 -268 -6138 po@sei. cmu. edu Ceci Albert SEI Washington, DC 703 -908 -8215 cca@sei. cmu. edu © 2003 by Carnegie Mellon University Patrick Place SEI Pittsburgh, PA 412 -268 -7746 prp@sei. cmu. edu Lisa Brownsword SEI Washington, DC 703 -908 -8203 llb@sei. cmu. edu page 38

Acronyms A/APCS CBS CM COTS EPIC ITSEP SEI Assembly/Acquisition Process for COTS-based Systems COTS-based system configuration management commercial off-the-shelf Evolutionary Process for Integrating COTSbased systems Integrating Technology by a Structured Evolutionary Process Software Engineering Institute © 2003 by Carnegie Mellon University page 39

BACKUPS © 2003 by Carnegie Mellon University page 40

Guiding Principles 1 • The requirements process must celebrate flexibility: requirements definition must be delayed and negotiated, and the true requirements minimized. • Use of any COTS product necessitates documenting, and probably re-engineering of existing end-user process. • Any COTS–based system will be in a necessary state of continual system re-formation and evolution throughout its useful lifetime. © 2003 by Carnegie Mellon University page 41

Guiding Principles 2 • Hands-on product evaluations are mandatory; these must be budgeted, scheduled, and accepted as a fundamental project activity, as central as requirements and design. • Prototyping of the integrated collection of COTS products from the earliest possible moment throughout system lifetime is a basic necessity. • Satisfactory contractual commitments can only result from the active participation of end users, testers, and other stakeholders, continuously throughout the entire period from project inception through sustainment and maintenance. © 2003 by Carnegie Mellon University page 42