Standards Assessments CMMI ISO 9000 TL 9000

Standards & Assessments CMMI, ISO 9000, TL 9000 Sources: ASQ CSQE Primer Introduction to CMMI Distilled 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 1

August 4 Class CMMI Introduction & Configuration Management Appraisal ISO 9000/TL-9000 Due today (31 -July): Cycle 2 Design & Code, hand off to System Tester System Test Plan Inspected & Baselined Project notebook updates including inspection records, meeting minutes, etc. 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 2

Topics Audits & Assessments CMM / CMMI & SCAMPI ISO 9000: ISO 9001: 2000, ISO 9000 -3: 1997, Tick. IT Q 9000, TL 9000 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 3

Capability Maturity Model (CMM) Created in 1987 by Software Engineering Institute (SEI) 5 level model based on proficiency in Key Process Areas (KPAs) Migrating to Capability Maturity Model Integration (CMMI) Three source models: – CMM for Software – Systems Engineering Capability model – Integrated Product Development CMMI v 1. 1 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 4

What is it? 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 5

Why Would I want one? Required – Contractual – Senior Management Decree (e. g. ROI of 7 to 1) Sales Tool Want to improve 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 6

Schedule Example 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 7

Process Capability Ability of a process to produce planned results • Predictable • Measureable 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 8

Process Models CMMI is model based Model = structured collection of elements that describes characteristics of effective processes Process Area = cluster of related practices that when performed collectively, satisfy a set of goals considered important for making significant improvement in that area Processes selected are those proven by experience to be effective (i. e. best practices, practical knowledge from previous endeavors) Notes: A process area is not a process A model is not a process models show what to do, not how to do it! Philosophy “All models are wrong, some are useful” – George Box 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 9

CMMI Models Model Options: Software Engineering (SW) Systems Engineering + Software Engineering (SE/SW) Systems Engineering + Software Engineering + Integrated Process & Product Development (SE/SW/IPPD) … + Supplier Sourcing (SE/SW/IPPD/SS) Representation Options: Staged (Maturity Levels) Migration from CMM to CMMI Continuous (Capability Levels) Migration from EIA/IS-731 to CMMI Recommended order for process improvements, but not prescribed … 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 10

Levels Zero – Ad Hoc One – Doing it (in Continuous, Ad Hoc in Staged) Two – Process performed for individual projects Three – Process focus at organizational level Four – Projects and processes are quantitatively managed Five – Projects and processes being optimized based on performance data & results 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 11

Representations Revisited Continuous Model – 25 Process Areas each assessed at level 0 -5 Configuration Mgmt = capability level 3 Risk Mgmt = not done (capability level 0) Requirements Mgmt = capability level 2 – Result can be presented as a Kiviat chart Staged Model – 25 Process Areas assigned to each of 4 Maturity Levels (see next slide) – Result is a grade (1 -5) 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 12

4 Quantitatively Managed Process measured & controlled 3 Defined on zati ani Focus on process improvement ani Org 5 Optimizing Org zati on Staged Representation Maturity Levels (MLx) Process characterized for project & often reactive 1 Initial ct Pro 2 Managed je Pro ject Process characterized by organization is proactive Process unpredictable, poorly controlled & reactive 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 13

Staged Representation Process Area Mapping to Maturity Levels 5. Optimizing Continuous Process Improvement Organizational Innovation & Deployment Causal Analysis & Resolution 4. Quantitatively Managed Quantitative Management Organizational Process Performance Quantitative Project Management 3. Defined Process Standardization 2. Managed Basic Project Management 1. Initial 7/31/2007 Requirements Development Technical Solution Product Integration Verification Validation Organizational Process Focus Organizational Process Definition Organizational Training Risk Management Decision Analysis & Resolution Requirements Management Project Planning Project Monitoring & Control Supplier Agreement Management Measurement & Analysis Process & Product Quality Assurance Configuration Management None SE 652 2007_7_31_CMMI_Software_Quality. ppt 15

Continuous Representation Process Areas Engineering Process Management – – – Organizational Process Focus (OPF-3) Organizational Process Definition (OPD-3) Organizational Training (OT-3) Organizational Process Performance (OPP-4) Organizational Innovation & Deployment (OID 5) Project Planning (PP-2) Project Monitoring & Control (PMC-2) Supplier Agreement Management (SAM-2) Integrated Project Management (IPM-3) Risk Management (RSKM-3) Integrated Teaming (IT-3) Integrated Supplier Management (ISM-3) Quantitative Project Management (QPM-4) 7/31/2007 Requirements Management (REQM-2) Requirements Development (RD-3) Technical Solution (TS-3) Product Integration (PI-3) Verification (VER-3) Validation (VAL-3) Support Project Management – – – – Configuration Management (CM-2) – Process & Product Quality Assurance (PPQA – 2) – Measurement and Analysis (MA-2) – Decision Analysis and Resolution (DAR-3) – Organizational Environment for Integration (OEI-3) – Causal Analysis and Resolution (CAR-5) SE 652 2007_7_31_CMMI_Software_Quality. ppt 16

CMMI Assessment Cheat Sheet Institutionalization – Ingrained Way of Doing Business that an organization follows routinely as part of its corporate culture Specific Goals – Required model component that describes the unique characteristics that must be present to satisfy the process area Specific Practice – Expected model component that is considered important to achieving the associated specific goal. The specific practices describe the activities expected to result in achievement of the specific goals of a process area. (In continuous representation – every specific practice (SP) is associated with a CL, in staged – all SPs are treated equally) Generic Goal – Required model component that describes the characteristics that must be present to satisfy the institutionalization of the processes that implement a process area Generic Practice – Expected model component that is considered important in achieving the associated generic goal. The generic practices describe the activities that are expected to result in achievement of the generic goal and contribute to the institutionalization of the processes associated with a process area. 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 17

CMMI Assessment Cheat Sheet (continued) Managed Process: – – – – Performed process planned & executed in accordance with policy Employs skilled people Adequate resources Produces controlled outputs Involves relevant stake holders Monitored, controlled & reviewed Evaluated for adherence to process description Defined Process: – – – Managed process tailored from the organizational standard processes Maintained process description Contributes work products, measures & other process info to organizational process assets Performed Process – Accomplishes needed work to produce work products – Specific goals of the process area are satisfied Establish & Maintain – Includes documentation & usage: • Planned • Documented & • Used 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 18

Configuration Management (CM) Assessment 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 19

De. Marco & Lister on Process Organizations driving to be SEI Level 5 (at least level N+1) Standards are good, but … Most success centered around standard interfaces Mandating a “best practice” is a bad practice Process improvement is good, but process improvement programs aren’t Competent people improve processes all the time (pride, growth, etc. ) Formal process improvement moves responsibility from the individual to the organization Process improvement programs focus on process rather than product (making a poor product efficiently is often worse than making a good product poorly) Focus on process “level” tends to make organizations risk averse “The projects most worth doing are the ones that will move you down one full level on your process scale!” 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 20

Break 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 21

Quality Standard Rationale Customers want & need assessments of supplier quality Means: Individually audit (i. e. qualify) vendor: Specific products Processes (e. g. manufacturing, design & development, support) Alternative: Common Quality Assurance standards & audits 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 22

Major Audit Types First Party Audit Within own company (aka internal audit) Used to measure own performance, strengths & weaknesses against internally established procedures & systems Second Party Audit Performed by customer on their supplier (aka external audit) Third Party Audit Outside, independent auditor contracted to audit on behalf of company or a supplier (e. g. ISO 9000 registration audit) Assessments (e. g. SCAMPI) Similar to first party audit, but typically performed by external assessors 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 23

Other Audit Types System Audit – examination of bigger picture of organization &/or project Typical cross organizational, cross process & cross product Process Audit – verify inputs, actions & outputs in accordance with defined requirements (e. g. software inspections) Product Audit – final product or service for “fitness for use” Customer oriented Compliance Audit Regulatory – audit to government regulations Management – audit to organizational rules, effectiveness & conformance Quality – systematic & independent of quality activities vs. established procedures 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 24

ISO 9001: 2000 Objective Provide confidence that vendor can produce quality products Assumptions: good practices will produce good products Standard for assessing organization’s Quality Management System (QMS) – Processes – Activities – Behaviors – Training But, ISO focuses on Quality Assurance not Quality Control ISO-9001 certification does not guarantee quality products! 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 25

Tenants of ISO 9001 1) Say what you do 2) Do what you say 3) Prove it! 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 26

ISO 9000 Audits Customers write requirements for current ISO-9001 certification into purchasing contracts Organizations apply for 3 rd party audit, end result is ISO-9001 certification ISO International Accreditation Forum (IAF) board Audits national accreditation boards (i. e. one board each nation) Who register individual registrars (e. g. Lloyd’s, DNV) Who audit organization internal auditors (e. g. Lucent Optical Networking) & spot check Who audit design, development, manufacturing & support teams within the organization 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 27

ISO Alphabet Soup ISO 9000: 2000 Overall framework, fundamentals of quality management systems & terminology ISO 9001: 2000 Requirements for quality management systems (qms) & what is required to demonstrate compliance ISO 90003 2004 (previously 9000 -3) Guidelines for the application of ISO 9001: 2000 to computer software ISO 19011 Guidelines for auditing quality and environmental management systems 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 28

What is wrong with ISO 9001? Vendors ISO-9001 certified, but quality still elusive! No visibility into supplier quality levels Not getting quality levels they wanted Solution: TL 9000 (Quest forum, telecommunications) QS 9000 (automotive) 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 29

TL 9000 ISO on steroids Wholly subsumes ISO 9001 -2000 Requires vendors prove they are actually improving Metrics focused on cost drivers of service providers: Know vendor is measuring Visibility into quality improvement results 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 30

TL 9000 Top Management Requirements Monitor & improve customer satisfaction Set long & short term objectives for organization effectiveness Set targets for TL 9000 product performance metrics Use an explicit life-cycle model Establish a quality improvement program Periodic management review of quality system 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 31

TL 9000 Metrics Cross-discipline metrics – – # of problem reports Problem report fix response time Overdue problem report fix responsiveness On-time delivery Hardware & Software measurements – System Outages Hardware measurements – Return rates Software measurements – – 7/31/2007 Software installation & release application aborts Corrective patch quality Feature patch quality Software update quality SE 652 2007_7_31_CMMI_Software_Quality. ppt 32

TL 9000 Common Audit Questions • Do you know how to find your Quality Policy, QMS and the processes you should be using for your work? • Do you know your organization’s product delivery & improvement goals and what you must do to support them? • Do you know what skills you should have? • Do you know what you have to do to approve/baseline/finalize your documents, designs & code? • Do you know how to store & find records of reviews, inspections, key decisions, etc. ? • Do you know what to do if a problem is found with the product or process? • Do you know your organization’s performance with respect to customer satisfaction, quality of delivered products & process execution? 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 33

TL 9000 Sample Requirements Planning – Must have methods for estimating & tracking – Determine where you will do reviews & tests – Risk management plans, customer, user & supplier involvement in reviews & evaluation Software Outputs – Requires architecture, detailed designs, code & user documentation – Each design thread must be reviewed at some point prior to integration or system test Software Testing – All testing must have test plans; test process must be documented – Plans must include test cases with inputs, output & test success criteria – Plans must include types of testing, requirements traceability, coverage definition & measurement, test environment, defect handling, et. al. – Integration testing specifically required 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 34

Team Project Postmortem Tracking process improvements during project Process Improvement Proposals (PIP) Port-Mortem Areas to consider Better personal practices Improved tools Process changes 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 35

Postmortem process Team discussion of project data Review & critique of roles 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 36

Postmortem process Review Process Data Review of cycle data including SUMP & SUMQ forms Examine data on team & team member activities & accomplishments Identify where process worked & where it didn’t Quality Review Analysis of team’s defect data Actual performance vs. plan Lessons learned Opportunities for improvement Problems to be corrected in future PIP forms for all improvement suggestions Role Evaluations What worked? Problems? Improvement areas? Improvement goals for next cycle / project? 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 37

Cycle Report Table of contents Summary Role Reports Leadership – leadership perspective Motivational & commitment issues, meeting facilitation, req’d instructor support Development Effectiveness of development strategy, design & implementation issues Planning Team’s performance vs. plan, improvements to planning process Quality / Process discipline, adherence, documentation, PIPs & analysis, inspections Cross-team system testing planning & execution Support Facilities, CM & Change Control, change activity data & change handling, ITL Engineer Reports – individual assessments 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 38

Role Evaluations & Peer Forms Consider & fill out PEER forms Ratings (1 -5) on work, team & project performance, roles & team members Additional role evaluations suggestions Constructive feedback Discuss behaviors or product, not person 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 39

Project Notebook Updated Requirements & Design documents Conceptual Design, SRS, SDS, System Test Plan, User Documentation* Updated Process descriptions Baseline processes, continuous process improvement, CM Tracking forms ITL, LOGD, Inspection forms, LOGTEST Planning & actual performance Team Task, Schedule, SUMP, SUMQ, SUMS, SUMTASK, CCR* 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 40

August 4 Class CMMI Introduction & Configuration Management Appraisal ISO 9000/TL-9000 Due July 31: Cycle 2 Design & Code, hand off to System Tester System Test Plan Inspected & Baselined Project notebook updates including inspection records, meeting minutes, etc. Deliverables for August 7 Project Postmortem (cycle report) Cycle 2 presentations Peer Feedback forms Completed project notebooks Cycle Exit Completed project (source, documents & all quality records) 7/31/2007 SE 652 2007_7_31_CMMI_Software_Quality. ppt 41