CTIS 494 Software Quality Assurance by Dr. Ozlem Albayrak 2007 Fall by Ozlem Albayrak
menu CTIS 494 Outline (1) Week Subject 17. 09 Introduction to Software Quality Assurance 24. 09 Software Inspections and Testing 01. 10 Capability Maturity Model 08. 10 ISO 9000 and SPICE (1 hour missing) 15. 10 Software Quality Metrics 22. 10 Seven Basic (B 7) Tools of Quality 29. 10 The Seven MP Tools 05. 11 MIDTERM (30%) by Ozlem Albayrak
CTIS 494 Outline (2) 12. 11 5 S for Intelligent Housekeeping in Software Development 19. 11 Software QFD and Voice of the Customer 26. 11 Risk Assessment and Failure Modes and Effects Analysis 03. 12 Quality Measures and Statistical Methods for Trustworthy Software 10. 12 Verification and Validation (PROJECT REPORTs Submission 15%) 17. 12 Testing and Evaluation (1 hour missing) 24. 12 Integration, Extension and Maintenance for Trustworthiness (PROJECT PRESENTATIONs 20%) 31. 12 Review (2 hours missing) 7. 12 FINAL (35 %) by Ozlem Albayrak
menu Today. . . Week Subject 17. 09 Introduction to Software Quality Assurance 24. 09 Software Inspections and Testing 01. 10 Capability Maturity Model 08. 10 ISO 9000 and SPICE (1 hour missing) 15. 10 Software Quality Metrics 22. 10 Seven Basic (B 7) Tools of Quality 29. 10 The Seven MP Tools 05. 11 MIDTERM (30%) by Ozlem Albayrak
Review l l l What is software inspection? Why do we test software? What are different dimensions of quality? What are 3 spheres of quality? What is quality assurance? What are quality characteristics? by Ozlem Albayrak
The Three Spheres of Quality Management Quality Assurance Quality Control by Ozlem Albayrak
ISO 9126 – Quality Characteristics l l l Functionality: extent to which functions are avaliable Reliability: extent to which software reliable Usability: users say easy to use Efficiency Maintainability: easy to maintain Portability: easy to transfer do different environments by Ozlem Albayrak
Software Quality Control l l Extensive inspections and testing Formal reviews Aim is to identify defects White box and black box tests Unit, function, subsystem, performance, integration, acceptance tests by Ozlem Albayrak
History of Quality l Shewhart – – Statistian at AT&T, 1920 s PDCA Model l l Plan Do Check Act by Ozlem Albayrak
History of Quality (2) l Deming – – – 1945 s, Japan 14 step programme Consistent purpose and direction in the organization is more important then everyone doing his best alone by Ozlem Albayrak
Software Quality Assurance Stephen Mc. Ilroy BT Retail Senior Systems Architect by Ozlem Albayrak
Software Quality Assurance l What is it? l Why do we need it? l How do we achieve it? by Ozlem Albayrak
What is Quality Assurance ? l Quality = – – – l l . . . meeting the customer’s requirements, . . . at the agreed cost, . . . within the agreed timescales. Quality = “Fitness for purpose” Quality = Customer satisfaction ! Quality Assurance is what we do to achieve this by Ozlem Albayrak
How do we achieve QA? l l By adopting a disciplined, professional approach - engineering is predictable and repeatable Quality Management System – – l Documented processes & Best Practice ISO 9001 & Tick. IT registration Continuous Improvement – – New methods (e. g. DSDM, UML) & tools Better & more focussed Training by Ozlem Albayrak
l The goal of software inspection and test is to reduce the expected cost of software failure over the life of a product. by Ozlem Albayrak
Popular Definitions of Software Quality l l l l Software that is developed right the first time Minimization of fixes between releases Measured loss imported to society after delivery User satisfaction Fitness for use Performance/ expectations Measurable properties of software Conformance to requirements by Ozlem Albayrak
Quantitative Definitions l l l Availability Reliability Defect rate Problems per month Mean time between failure Loss to society (Taguchi) by Ozlem Albayrak
What is a good software? l l l Written according to a coding standard Design must be high quality Succint and useful documentation Comprehensive suite of regression tests must accompany the software Ref: H. Erdogmus, IEEE Software, 2007 by Ozlem Albayrak
Software Inspections and Testing l l l Inspection plays a key role in building software Testing plays a key role in verifiying software There are different ways to software inspections: – – Formal Informal by Ozlem Albayrak
Formal Inspections l l Fagan inspection methodology Gilb methodology Roles Author Inspector Tester Moderator by Ozlem Albayrak
Informal Inspection l l Walkthrough of document or code by individual other than author Deliverable is reviewed informally by Ozlem Albayrak
Fagan Inspections l l l l Planning Overview Preparation Inspection Process improvement Re-work Follow-up by Ozlem Albayrak
Economic Benefits of Sofware Inspections l Positive impacts on: – Productivity Quality Time to market Customer satisfaction – IBM Space Shuttle %85 inspections, %15 testing – – – by Ozlem Albayrak
Benefits of Software Inspections l l l Good for educating people about standards and procedures Cost of defect increases as software is being developed Cost of quality estimation varies between 20%40% by Ozlem Albayrak
Software Inspection Methodology l l Fagan, Gilb Typical Roles Author – Inspector Also – Moderator – Tester – Reader – by Ozlem Albayrak
E-mail/Fax reviews l l No physical meeting reviews Depending on the organization this type of reviews may work by Ozlem Albayrak
Software Testing l Objective is – – l to verify the correctness of the software system and to identify defects that are present. Testing effort is often complicated by real world issues, such as late delivery of software. by Ozlem Albayrak
Steps in Test Plan l l l Review requirements Overall test plan and review System test plan and review Performance test plan and review Usability test plan and review Regression plan and review Test environment setup System testing and re-test defects Performance testing and re-test defects Usability testing Regression testing Test reporting by Ozlem Albayrak
Test Process l The quality of testing is dependent on the maturity of the test process and a good test process include: • Test planning and risk management • Dedicated test environment and test tools • Test case definition • Test automation • Formality in handover to test department by Ozlem Albayrak • Test execution • Test result analysis • Test reporting • Measurements of test effectiveness • Post mortem and test process improvement
Test Case l A Test Case should include: – – – Purpose of test case Setup required to execute the test case Inputs to test case The test procedure Expected outputs or results by Ozlem Albayrak
Test Management Tools Open source Test Management Tools: l http: //www. opensourcetesting. org/testmgt. php l http: //www. technologyevaluation. com/request/t esttool_softeval. asp? se=google_testtools&gcli d=CJvgv 8 Ca 3 o 4 CFQxa. Xgod. Iy-FPA by Ozlem Albayrak
CMMI and SEI l http: //www. sei. cmu. edu/cmmi/general/index. html l CMMI Main Page What Is CMMI? Models Adoption Training, Events, & Forums Performance Results Appraisals Frequently Asked Questions (FAQs) Background Contact Information l l l by Ozlem Albayrak
CMMI l Capability Maturity Model Integration by Ozlem Albayrak
SEI l Software Engineering Institute l Books @ Pearson by Ozlem Albayrak
Capability Maturity Model® Integration (CMMI®) Overview SM CMM Integration, SCAMPI Lead Appraiser, and SEI are service marks of Carnegie Mellon University. ® CMMI, Capability Maturity Model, and CMM are registered in the U. S. Patent and Trademark Office by Carnegie Mellon University. by Ozlem Albayrak
Objectives l. Provide an overview of CMMI by describing the following: – Background and motivation – Models – Representations – The bottom line – Benefits and improvements by Ozlem Albayrak
Software Engineering Institute (SEISM) l. Federally funded research and development center (FFRDC) established 1984 l. Awarded University to Carnegie Mellon l. Sponsored by the Office of the Under Secretary of Defense for Acquisition and Technology [OUSD (A&T)] l. Administered by Electronic Systems Center (ESC) by Ozlem Albayrak 1 -9
The State of the Practice l. Is – this the state of affairs in your organization? “I'd rather have it wrong than have it late. We can always fix it later. ” - – “The bottom line is schedule. My promotions and raises are based on meeting schedule first and foremost. ” - l. If – – A senior software manager (industry) A program manager (government) it is, are managers and practitioners unhappy with the status quo? Sufficiently unhappy to change things? Willing and able to attack the known problems? by Ozlem Albayrak
The Process Management Premise l. The quality of a system is highly influenced by the quality of the process used to acquire, develop, and maintain it. l. This premise implies a focus on processes as well as on products. – – This is a long-established premise in manufacturing (and is based on TQM principles as taught by Shewhart, Juran, Deming, and Humphrey). Belief in this premise is visible worldwide in quality movements in manufacturing and service industries (e. g. , ISO standards). by Ozlem Albayrak
Multiple Process Models Software CMM EIA 731 Systems Engr CMM IPD CMM People CMM Software Acq CMM Success of the Software CMM® caused development of other CMMs, but they • Have different structures, formats, terms, ways of measuring maturity • Cause confusion, especially when more than one are used • Are difficult to integrate into a combined improvement program • Are difficult to use in supplier selection Systems Security Engr CMM by Ozlem Albayrak
The Next Step Is CMM Integration. SM l. The – – CMM Integration Project was formed to build an initial set of integrated models improve best practices from source models based on lessons learned establish a framework to enable integration of future models create an associated set of appraisal and training products l. Collaborative – – – endeavor (over 100 people involved) Industry Government Software Engineering Institute (SEI) by Ozlem Albayrak
Enterprise-Wide Improvement l. CMMI enables organizations that want to pursue process improvement in multiple functional areas to do so with less additional investment for each additional function. – CMMI supports process integration and product improvement. – CMMI integrates multiple disciplines into one process-improvement framework. – CMMI provides a framework for introducing new disciplines as needs arise. by Ozlem Albayrak
Bodies of Knowledge Captured in CMMI Models l. An organization selects the bodies of knowledge most relevant to achieving its business objectives. Bodies of knowledge* available in CMMI models include – software engineering – systems engineering – integrated – supplier product and process development (IPPD) sourcing (SS) by Ozlem Albayrak
CMMI Models -SW CMMI d Stage tion senta Repre Source Models W CMMI-S ous Continu tion nta Represe • Capability Maturity Model for Software V 2, draft C (SW-CMM V 2 C) • EIA 731, System Engineering Capability Model (SECM) Software Engineering model • Integrated Product Development Capability Maturity Model, draft V 0. 98 (IPD-CMM) by Ozlem Albayrak – Meets the needs of software development and maintenance organizations – Meets the needs of information technology organizations – Benefits from best practices contributed from all three source models
Understanding CMMI Representations l. There – – are two types of representations in the CMMI models: staged continuous l. A representation allows an organization to pursue different improvement objectives. l. The organization and presentation of the data are different in each representation. However, the content is the same. by Ozlem Albayrak
Staged Representation l. Provides a proven sequence of improvements, each serving as a foundation for the next l. Permits comparisons across and among organizations by the use of maturity levels l. Provides an easy migration from the SW-CMM to CMMI l. Provides a single rating that summarizes appraisal results and allows comparisons among organizations Indicates maturity of an organization’s standard process -- to answer, “What is a good order for approaching improvement across the organization? ” by Ozlem Albayrak
Maturity Levels l. A maturity level is a well-defined evolutionary plateau of process improvement. l. There are five maturity levels. l. Each level is a layer in the foundation for continuous process improvement using a proven sequence of improvements, beginning with basic management practices and progressing through a predefined and proven path of successive levels. by Ozlem Albayrak
The Maturity Levels 5 4 3 2 1 Optimizing Focus on continuous process improvement Quantitatively Managed Process measured and controlled Defined Process characterized for the organization and is proactive Process characterized for projects and is often reactive Process unpredictable, poorly controlled, and reactive Managed Initial by Ozlem Albayrak
Maturity Levels Should Not Be Skipped l. Each maturity level provides a necessary foundation for effective implementation of processes at the next level. – – Higher level processes have less chance of success without the discipline provided by lower levels. The effect of innovation can be obscured in a noisy process. l. Higher maturity level processes may be performed by organizations at lower maturity levels, with the risk of not being consistently applied in a crisis. by Ozlem Albayrak
Continuous Representation l. Allows you to select the order of improvement that best meets your organization’s business objectives and mitigates your organization’s areas of risk l. Enables comparisons across and among organizations on a process-area-by-process-area basis l. Provides an easy migration from EIA 731 (and other models with a continuous representation) to CMMI Indicates improvement within a single process area -- to answer, “What is a good order for approaching improvement of this process area? ” by Ozlem Albayrak
Capability Levels l. A capability level is a well-defined evolutionary plateau describing the organization’s capability relative to a process area. l. There l. For are six capability levels 1 -5, there is an associated generic goal. l. Each level is a layer in the foundation for continuous process improvement. l. Thus, capability levels are cumulative, i. e. , a higher capability level includes the attributes of the lower levels. by Ozlem Albayrak
The Capability Levels 5 Optimizing 4 Quantitatively Managed 3 Defined 2 Managed 1 Performed 0 Incomplete by Ozlem Albayrak
Representing Capability Levels for a Single Process Area process area capability of an implemented process can be represented by a bar. Capability Level l. The This point represents a higher level of “maturity” than this point in a specific process area 3 2 1 0 Process Area n Process by Ozlem Albayrak
Relating Process Area Capability and Organizational Maturity l. Organizational maturity is the focus of the staged representation, whereas process area capability is the focus of the continuous representation. l. Organizational maturity and process area capability are similar concepts. l. The difference between them is that organizational maturity pertains to a set of process areas across an organization, while process area capability deals with a set of processes relating to a single process area or specific practice. by Ozlem Albayrak
Why Two Representations? l. The – – – representation of each source model was different Software CMM—Staged SECM—Continuous IPD CMM—Hybrid l. Combining different representations required deciding on a representation for CMMI models. l. Ultimately, the project decided to incorporate the benefits of both staged and continuous representations. by Ozlem Albayrak
Comparison of Representations Staged Continuous l. Process improvement is measured using maturity levels. l. Maturity level is the degree of process improvement across a predefined set of process areas. l. Organizational maturity pertains to the “maturity” of a set of processes across an organization by Ozlem Albayrak Process improvement is measured using capability levels. Capability level is the achievement of process improvement within an individual process area. Process area capability pertains to the “maturity” of a particular process across an organization.
Advantages of Each Representation l. Staged l. Provides – – a roadmap for implementing groups of process areas sequencing of implementation l. Familiar structure for those transitioning from the Software CMM l. Continuous l. Provides maximum flexibility for focusing on specific process areas according to business goals and objectives l. Familiar structure for those transitioning from EIA 731 by Ozlem Albayrak
CMMI in a Nutshell l. A CMMI model provides a structured view of process improvement across an organization l CMMI can help – set process improvement goals and priorities – provide guidance for quality processes – provide a yardstick for appraising current practices by Ozlem Albayrak
The Bottom Line l. Process improvement should be done to help the business — not for its own sake. “In God we trust, all others bring data. ” - W. Edwards Deming by Ozlem Albayrak
Categories of Process Improvement Benefits l. Process improvement benefits fall into one of eight general categories: 1. Improved schedule and budget predictability 2. Improved cycle time 3. Increased productivity 4. Improved quality (as measured by defects) 5. Increased customer satisfaction 6. Improved employee morale 7. Increased return on investment 8. Decreased cost of quality l. Benefits realized by organizations using the Software CMM are expected with CMMI. by Ozlem Albayrak
Improved Cycle Time Source: Software Engineering Div. , Hill AFB, Published in Crosstalk, May 1999 by Ozlem Albayrak
Increased Productivity and Quality l by Ozlem Albayrak
Percentage Improvements from Adopting Software CMM 40 35 30 25 20 15 10 5 0 39% 35% 19% Savings vs. cost of software process improvement (median) 5: 1 Productivity (increase) Time to market (reduction) Post-release defect reports (reduction) Annual Medians by Ozlem Albayrak
Benefits of Continuing Process Improvement l. SEI Software CMM Level 5: For the Right Reasons* – Defects are now nearly all found and fixed before testing begins. – Defects escaping into the field have been reduced from 11% to practically 0%. – Programs consistently reach customer satisfaction and performance targets. – Peer reviews increase total project costs by 4%, but reduced rework during testing by 31%. R. O. I. is 7. 75: 1. l* Reference: Yamamura and Wigle, Boeing Space and Transportation Systems, Crosstalk, Aug, 1997. by Ozlem Albayrak
CMM“I” – Improvement l. The CMMI Product Suite provides a foundation for enterprise-wide improvement and adds – new emphasis on products and services as well as process – emphasis on both process capability and organizational maturity – early emphasis on measurement and analysis l. The CMMI model improves upon Software CMM V 1. 1 and Software CMM V 2. 0 Draft C. by Ozlem Albayrak
CMM“I” – Integration l. Provides expanded model scope for integration – Integrated Product Management – Integrated Supplier Management – Decision Analysis and Resolution – “Relevant Stakeholder” planning and execution – Inclusion of the Integrated Product and Process Development body of knowledge by Ozlem Albayrak
CMMI Can Benefit You l. CMMI provides – Efficient, effective assessment and improvement across multiple process disciplines in an organization – Improvements to best practices incorporated from the Software CMM – A common, integrated vision of improvement for all elements of an organization – A means of representing new discipline-specific information in a standard, proven process-improvement context by Ozlem Albayrak
For More Information About CMMI l. Go to CMMI Web site: http: //www. sei. cmu. edu/cmmi http: //seir. sei. cmu. edu l. Contact SEI Customer Relations: Customer Relations Software Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213 -3890 FAX: (412) 268 -5800 customer-relations@sei. cmu. edu by Ozlem Albayrak
Скачать презентацию CTIS 494 Software Quality Assurance by Dr Ozlem
45585df9376093ca7d1ee7dc51fc964a.ppt