Optimizing IV V in Mature Organizations Dan Mc Caugherty

Optimizing IV&V in Mature Organizations Dan Mc. Caugherty, Frank Gmeindl, Pro. Logic, Inc. Christopher Fuhrman, ETS Technical Interchange and Initiative Review, NASA IV&V Facility 2003 -07 -29

Summary Slide • Objective • Related work • Research approach • Success Measures • Major Accomplishments to Date (1/2) • Open issues and concerns 2003 -07 -29 Technical Interchange and Initiative Review, NASA IV&V Facility 2

Objective (1) • What optimization strategies can be taken by IV&V planners, knowing the appraised level of maturity of the development organization? 2003 -07 -29 Technical Interchange and Initiative Review, NASA IV&V Facility 3

Objective (2) • What relationship exists (if any) between the level of effort spent on “managerial” IV&V activities and the assessed level of maturity of the development organization? 2003 -07 -29 Technical Interchange and Initiative Review, NASA IV&V Facility 4

Objective (3) • Provide a set of guidelines to help IV&V planners tailor activities to a development organization’s mature processes 2003 -07 -29 Technical Interchange and Initiative Review, NASA IV&V Facility 5

Related work (1) Ratio of IV&V effort non-technical (managerial) vs technical issues [1] 3 -4% 6 -8% 10 -15% IV&V Unable to <25% IV&V estimate non. IV&V technical IV&V Level 5 somewhat unpredictable Level 1 Level 2 Devel [1] Level 3 op f level o ed s asses er’s Level 4 m M CM in the aturity R. O. Lewis, Independent verification and validation: a life cycle engineering process for quality software. New York: Wiley, 1992. Technical Interchange and Initiative Review, NASA IV&V Facility 2003 -07 -29 6

Related work (2) • IV&V Cost Estimation, NAVAIR – Need for similar data (IV&V Task Details, Man-hours, Scalars) • Baseline Cost Model, Pro. Logic – Re-use of WBS, planning strategies 2003 -07 -29 Technical Interchange and Initiative Review, NASA IV&V Facility 7

Research approach (1) • Analyze the CMMI Level 2 and 3 “typical work products” from IV&V perspective (FY 03) 1. Identify evaluation criteria for determining their usefulness to IV&V 2. Analyze the work products according to the criteria 2003 -07 -29 Technical Interchange and Initiative Review, NASA IV&V Facility 8

1. Research approach (2) 1. Examine actual project planning data for one or more IV&V projects done with mature developers (FY 04) 1. See if or how the CMM/CMMI work products were used in the actual IV&V activities (prove useful relationship) 2. Calculate ratio of effort spent on “managerial” vs technical IV&V activities 2. Provide guidebook for IV&V planners (FY 04) 2003 -07 -29 Technical Interchange and Initiative Review, NASA IV&V Facility 9

Success Measures • Reduced level of effort spent on non-technical IV&V activities • Ability to tailor IV&V activities in relationship to a developer’s mature processes based on an appraised maturity level • Ability to avoid unnecessarily redundant V&V activities between internal V&V and independent V&V 2003 -07 -29 Technical Interchange and Initiative Review, NASA IV&V Facility 10

Schedule of Deliverables Deliverable Due Date (FY) Quarter Draft Report: Analysis of CMMI PAs from IV&V perspective 03 Q 2 Final Report: Analysis of CMMI PAs from IV&V perspective 04 Q 1 Define approach and selection of project for pilot study 04 Q 1 2003 -07 -29 Technical Interchange and Initiative Review, NASA IV&V Facility 11

Major Accomplishments to Date (1/2) • Adaptation of research initiative for the CMMI rather than the CMM • Draft analysis of CMMI PAs and work products in the context of IEEE 1012 – Criteria for analysis of artifacts: • Useful for IV&V planning? • Useful/necessary in IV&V technical tasks? • Possibility of compromising technical independence? 2003 -07 -29 Technical Interchange and Initiative Review, NASA IV&V Facility 12

Major Accomplishments to Date (2/2) • Two articles submitted to NASA Software Engineering Workshop (SEW) – Optimizing IV&V in Mature Organizations – Software V&V and the RUP 2003 -07 -29 Technical Interchange and Initiative Review, NASA IV&V Facility 13

Prioritized list of CMMI PAs Criteria for IV&V activities H=high M=medium L=low Process areas (CMMI-SW Level 2 and 3) Planning Execution Support Conflict H L L L/M M/H M L/M 5. Configuration Management (CM) M H L L 7. Measurement and Analysis (MA) M L L M 8. Organizational Process Definition (OPD) H L L N/A 11. Integrated Project Management (IPM) M L L L 12. Risk Management (RSKM) M M L H 13. Requirements Development (RD) H H H L/M 14. Technical Solution (TS) M H H L 15. Product Integration (PI) M M/H L 1. Project Planning (PP) 4. Requirements Management (REQM) 2003 -07 -29 Technical Interchange and Initiative Review, NASA IV&V Facility 14

Open issues and concerns • Availability of IV&V project planning data for an IV&V project with a mature developer (i. e. , one that has been assessed/ appraised at a minimum of level 2 in CMM/CMMI) – Typical IV&V metrics are related to defects, etc. • Existence of appraised developers (yet to be identified) – GSFC in-house projects, e. g. , Global Precipitation Measurement, might “count” as a mature developer if they have had a CMM/CMMI appraisal 2003 -07 -29 Technical Interchange and Initiative Review, NASA IV&V Facility 15

Questions? 2003 -07 -29 Technical Interchange and Initiative Review, NASA IV&V Facility 16