Software Quality Assurance Lecture 14 Dr R Mall

Software Quality Assurance (Lecture 14) Dr. R. Mall 1

Organization of this Lecture: z. Introduction Quality Engineering. z. Quality control and Quality Assurance z. ISO 9000 z. SEI CMM z. Summary 2

Introduction z. Traditional definition of quality: yfitness of purpose, xa quality product does exactly what the users want it to do. 3

Fitness of purpose z. For software products, yfitness of purpose: xsatisfaction of the requirements specified in SRS document. 4

Fitness of purpose z. A satisfactory definition of quality for many products: ya car, a table fan, a food mixer, microwave oven, etc. z. But, not satisfactory for software products. 5

Introduction z. Consider a software product: yfunctionally correct, xi. e. performs all functions as specified in the SRS document, ybut has an almost unusable user interface. xcannot be considered as a quality product. 6

Introduction z. Another example: ya product which does everything that users want. ybut has an almost incomprehensible and unmaintainable code. 7

Modern view of quality z. Associates several quality factors with a software product : y. Correctness y. Reliability y. Efficiency (includes efficiency of resource utilization) y. Portability y. Usability y. Reusability y. Maintainability 8

Correctness z. A software product is correct, yif different requirements as specified in the SRS document have been correctly implemented. y. Accuracy of results. 9

Portability z. A software product is said to be portable, yif it can be easily made to work in different operating systems, yin different machines, ywith other software products, etc. 10

Reusability z. A software product has good reusability, yif different modules of the product can easily be reused to develop new products. 11

Usability z. A software product has good usability, yif different categories of users (i. e. both expert and novice users) can easily invoke the functions of the product. 12

Maintainability z. A software product is maintainable, yif errors can be easily corrected as and when they show up, ynew functions can be easily added to the product, yfunctionalities of the product can be easily modified, etc. 13

Software Quality Management System z. Quality management system (or quality system): yprincipal methodology used by organizations to ensure that the products have desired quality. 14

Quality system z. A quality system consists of the following: y. Managerial Structure y. Individual Responsibilities. z. Responsibility of the organization as a whole. 15

Quality system z. Every quality conscious organization has an independent quality department: yperforms several quality system activities. yneeds support of top management. y. Without support at a high level in a company, xmany employees may not take the quality system seriously. 16

Quality System Activities: z. Auditing of projects z. Development of: ystandards, procedures, and guidelines, etc. z. Production of reports for the top management ysummarizing the effectiveness of the quality system in the organization. z Review of the quality system itself. 17

Quality system z. A good quality system must be well documented. y. Without a properly documented quality system, xapplication of quality procedures become ad hoc, xresults in large variations in the quality of the products delivered. 18

Quality system z. An undocumented quality system: ysends clear messages to the staff about the attitude of the organization towards quality assurance. z. International standards such as ISO 9000 provide: y guidance on how to organize a quality system. 19

Evolution of Quality Systems z. Quality systems have evolved: yover the last five decades. z. Prior to World War II, yway to produce quality products: xinspect the finished products xeliminate defective products. 20

Evolution of Quality Systems z. Since that time, yquality systems of organizations have undergone xfour stages of evolution. 21

Evolution of Quality Systems 22

Evolution of Quality Systems z. Initial product inspection method : ygave way to quality control (QC). z. Quality control: ynot only detect the defective products and eliminate them ybut also determine the causes behind the defects. 23

Quality control (QC) z. Quality control aims at correcting the causes of errors: ynot just rejecting defective products. z. Statistical quality control yquality of the output of the process is inferred using statistical methods yin stead of inspection or testing of all products 24

Quality control (QC) z. The next breakthrough, ydevelopment of quality assurance principles 25

Quality assurance z. Basic premise of modern quality assurance: yif an organization's processes are good and are followed rigorously, xthe products are bound to be of good quality. 26

Quality assurance z. All modern quality paradigms include: yguidance for recognizing, defining, analyzing, and improving the production process. 27

Total quality management (TQM) z. Advocates: ycontinuous process improvements through process measurements. 28

Business Process reengineering z. A term related to TQM. z. Process reengineering goes a step further than quality assurance: yaims at continuous process improvement. 29

Business Process reengineering Our focus is reengineering of the software process. Whereas BPR aims at reengineering the way business is carried out in any organization not just software development organizations. 30

Total quality management (TQM) z. TQM goes beyond documenting processes yoptimizes them through redesign. z. Over the years the quality paradigm has shifted: yfrom product assurance to process assurance. 31

ISO 9000 z. ISO (international Standards Organization): ya consortium of 63 countries established to formulate and foster standardization. z. ISO published its 9000 series of standards in 1987. 32

What is ISO 9000 Certification? z. ISO 9000 certification: yserves as a reference for contract between independent parties. z. The ISO 9000 standard: yspecifies guidelines for maintaining a quality system. 33

What is ISO 9000 Certification? z. ISO 9000 specifies: yguidelines for repeatable and high quality product development. y. Also addresses organizational aspects xresponsibilities, reporting, procedures, processes, and resources for implementing quality management. 34

ISO 9000 z. A set of guidelines for the production process. ynot directly concerned about the product it self. ya series of three standards: x. ISO 9001, ISO 9002, and ISO 9003. 35

ISO 9000 z. Based on the premise: yif a proper process is followed for production: xgood quality products are bound to follow. 36

ISO 9001: z. Applies to: yorganizations engaged in design, development, production, and servicing of goods. yapplicable to most software development organizations. 37

ISO 9002: z ISO 9002 applies to: yorganizations who do not design products: xbut are only involved in production. z Examples of this category of industries: ysteel or car manufacturing industries y buy the product and plant designs from external sources: xonly manufacture products. ynot applicable to software development organizations. 38

ISO 9003 z. ISO 9003 applies to: yorganizations involved only in installation and testing of the products. 39

ISO 9000 for Software Industry z. ISO 9000 is a generic standard: yapplicable to many industries, xstarting from a steel manufacturing industry to a service rendering company. z. Many clauses of ISO 9000 documents: yuse generic terminologies yvery difficult to interpret them in the context of software organizations. 40

Software vs. other industries z. Very difficult to interpret many clauses for software industry: ysoftware development is radically different from development of other products. 41

Software vs. other industries z. Software is intangible ytherefore difficult to control. x. It is difficult to control anything that we cannot see and feel. y. In contrast, in a car manufacturing unit: xwe can see a product being developed through stages such as fitting engine, fitting doors, etc. xone can accurately tell about the status of the product at any time. y. Software project management is an altogether different ball game. 42

Software vs. other industries z. During software development: ythe only raw material consumed is data. z. For any other product development: y. Lot of raw materials consumed xe. g. Steel industry consumes large volumes of iron ore, coal, limestone, etc. z. ISO 9000 standards have many clauses corresponding to raw material control. xnot relevant to software organizations. 43

Software vs. other industries z. Radical differences exist between software and other product development, ydifficult to interpret various clauses of the original ISO standard in the context of software industry. 44

ISO 9000 Part-3 z. ISO released a separate document called ISO 9000 part-3 in 1991 yto help interpret the ISO standard for software industry. z. At present, yofficial guidance is inadequate 45

Why Get ISO 9000 Certification? z. Several benefits: y. Confidence of customers in an organization increases xif organization qualified for ISO 9001 certification. x. This is especially true in the international market. 46

Why Get ISO 9000 Certification? z. Many international software development contracts insist: ydevelopment organization to have ISO 9000 certification. 47

Why Get ISO 9000 Certification? z. Requires: ya well-documented software production process to be in place. ycontributes to repeatable and higher quality software. z. Makes development process: yfocussed, efficient, and cost-effective 48

Why Get ISO 9000 Certification? z. Points out the weakness of an organizations: yrecommends remedial action. z. Sets the basic framework: yfor development of an optimal process and TQM. 49

How to Get ISO 9000 Certification? z. An organization intending to obtain ISO 9000 certification: yapplies to a ISO 9000 registrar for registration. z. ISO 9000 registration process consists of several stages. 50

How to Get ISO 9000 Certification? z. Application stage: y. Applies to a registrar for registration. z. Pre-assessment: ythe registrar makes a rough assessment of the organization. 51

How to Get ISO 9000 Certification? z. Document review and adequacy audit: yprocess and quality-related documents. ythe registrar reviews the documents ymakes suggestions for improvements. 52

How to Get ISO 9000 Certification? Compliance audit: the registrar checks whether the suggestions made by it during review have been complied. 53

How to Get ISO 9000 Certification? z. Registration: y. The registrar awards ISO 9000 certificate after successful completions of all previous phases. z. Continued surveillance: y The registrar continues monitoring the organization periodically. 54

ISO 9000 Certification z. An ISO certified organization ycan use the certificate for corporate advertizements ycannot use the certificate to advertize products. x. ISO 9000 certifies organization's process xnot any product of the organization. y. An organization using ISO certificate for product advertizements: xrisks withdrawal of the certificate. 55

Summary of ISO 9001 Requirements z. Management responsibility(4. 1): y. Management must have an effective quality policy. y. The responsibility and authority of all those work affects quality: x must be defined and documented. 56

Management responsibility(4. 1) z. Responsibility of the quality system. yindependent of the development process, ycan work in an unbiased manner. z. The effectiveness of the quality system: ymust be periodically by audited. 57

Quality system (4. 2) and contract reviews (4. 3): z. A quality system must be maintained and documented. z. Contract reviews (4. 3): y. Before entering into a contract, an organization must review the contract xensure that it is understood, xorganization has the capability for carrying out its obligations. 58

Design control (4. 4): z. The design process must be properly controlled, ythis includes controlling coding also. z. A good configuration control system must be in place. 59

Design control (4. 4): z. Design inputs must be verified as adequate. z. Design must be verified. z. Design output must be of required quality. z. Design changes must be controlled. 60

Document control (4. 5): z. Proper procedures for ydocument approval, issue and removal. z. Document changes must be controlled. yuse of some configuration management tools is necessary. 61

Purchasing (4. 6): z. Purchased material, including bought-in software: ymust be checked for conforming to requirements. 62

Purchaser Supplied Products (4. 7): z. Material supplied by a purchaser, y for example, xclient-provided software must be properly managed and checked. 63

Product Identification (4. 8): z. The product must be identifiable at all stages of the process. y. In software development context this means configuration management. 64

Process Control (4. 9) : z. The development must be properly managed. z. Quality requirements must be identified in a quality plan. 65

Inspection and Testing (4. 10) : z. In software terms this requires effective testing i. e. , yunit testing, integration testing and system testing. z. Test records must be maintained. 66

Inspection, measuring and test equipment(4. 11): z. If integration, measuring, and test equipments are used, ymust be properly maintained and calibrated. 67

Control of nonconforming product (4. 13) : z. In software terms, ykeeping untested or faulty software out of released product, xor other places whether it might cause damage. 68

Corrective Action (4. 14) : z. This is both about correcting errors when found, yinvestigating why they occurred yimproving the process to prevent further occurrences. z. If an error reoccurs despite the quality system, ythe system needs improvement. 69

Handling (4. 15) and Quality audits (4. 17): z. Handling (4. 15) Deals with: ystorage, packing, and delivery of the software product. z. Quality Audits (4. 17) : yquality system audit must be carried out to ensure its effectiveness. 70

Training (4. 18) : z. Training needs must be identified and met. z. Most items of ISO standard yare largely common sense. 71

Salient features of ISO 9001 requirements: z. All documents concerned with the development of a software product yshould be properly managed, authorized, and controlled. z. Proper plans should be prepared yprogress against these plans should be monitored. 72

Salient features of ISO 9001 requirements: z. Important documents independently checked and reviewed: y for effectiveness and correctness. z. The product should be tested : yagainst specification. z. Several organizational aspects: ye. g. , management reporting of the quality team. 73

Shortcomings of ISO 9001 Certification (1) z. ISO 9000 requires a production process to be adhered to: ybut does not guarantee the process to be of high quality. y. Does not give any guideline for defining an appropriate process. 74

Shortcomings of ISO 9001 Certification (2) z. ISO 9000 certification process ynot fool-proof yno international accredition agency exists. ylikely variations in the norms of awarding certificates: xamong different accredition agencies and among the registrars. 75

Shortcomings of ISO 9001 Certification (3) z. Organizations qualifying for ISO 9001 certification: ytend to downplay domain expertise. ytend to believe that since a good process is in place, xany engineer is as effective as any other engineer in doing any particular activity relating to software development. 76

Shortcomings of ISO 9001 Certification (4) z. In manufacturing industry yclear link between process quality and product quality yonce a process is calibrated: xcan be run again and again producing quality goods z. Software development is a creative process: yindividual skills and experience is significant 77

Shortcomings of ISO 9001 Certification (5) z. Many areas of software development are very specialized: yspecial expertize and experience (domain expertize) required. z. ISO 9001 ydoes not automatically lead to continuous process improvement, ydoes not automatically lead to TQM. 78

Shortcomings of ISO 9001 Certification (6) z ISO 9001 addresses mostly management aspects. z Techniques specific to software development have been ignored y. Configuration management y. Reviews y. Release builds y. Problem Notification system y. Intranets 79

SEI Capability Maturity Model z. Developed by Software Engineering Institute (SEI) of the Carnegie Mellon University, USA: yto assist the U. S. Department of Defense (Do. D) in software acquisition. y. The rationale was to include: xlikely contractor performance as a factor in contract awards. 80

SEI Capability Maturity Model z. Major Do. D contractors began CMMbased process improvement initiatives: yas they vied for Do. D contracts. z. SEI CMM helped organizations: y. Improve quality of software they developed y. Realize adoption of SEI CMM model had significant business benefits. z. Other organizations adopted CMM. 81

SEI Capability Maturity Model z. In simple words, y. CMM is a model for apprising the software process maturity of a contractor into different levels. y. Can be used to predict the most likely outcome to be expected from the next project that the organization undertakes. 82

SEI Capability Maturity Model z. Can be used in two ways: y. Capability evaluation y. Software process assessment. 83

Capability Evaluation z. Provides a way to assess the software process capability of an organization y. Helps in selecting a contractor y. Indicates the likely contractor performance 84

Software Process Assessment z. Used by an organization to assess its current process: y. Suggests ways to improve the process capability. y. This type of assessment is for purely internal use. 85

SEI Capability Maturity Model The SEI CMM classifies software development industries into: Five maturity levels. Stages are ordered so that improvements at one stage provide foundations for the next Based on the pioneering work of Philip Crosby 86

SEI Capability Maturity Model Optimizing (5) Managed (4) Defined (3) Repeatable (2) Initial (1) 87

Level 1: (Initial) z. Organization operates ywithout any formalized process or project plans z. An organization at this level is characterized by yad hoc and often chaotic activities. 88

Level 1: (Initial) z. Software production processes are not defined, ydifferent engineers follow their own process ydevelopment efforts become chaotic. y. The success of projects depend on individual efforts and heroics. 89

Level 2: (Repeatable) z. Basic project management practices ytracking cost, schedule, and functionality are followed. z. Size and cost estimation techniques yfunction point analysis, COCOMO, etc. used. z. Production process is ad hoc ynot formally defined yalso not documented. 90

Level 2: (Repeatable) z. Process used for different projects might vary between projects: yearlier success on projects with similar applications can be repeated. y. Opportunity to repeat process exist when a company produces a family of products. 91

Level 3: (Defined) z. Management and development activities: ydefined and documented. y. Common organization-wide understanding of activities, roles, and responsibilities. 92

Level 3: (Defined) z. The process though defined, yprocess and product qualities are not measured. z. ISO 9001 aims at achieving this level. 93

Level 4: (Managed) z. Quantitative quality goals for products are set. z. Software process and product quality are measured: y. The measured values are used to control the product quality. y. Results of measurement used to evaluate project performance xrather than improve process. 94

Level 4: (Managed) z. Organization sets quantitative quality goals z. World-wide about 100 organizations assessed at this level. 95

Level 5: (Optimizing) z. Statistics collected from process and product measurements are analyzed: ycontinuous process improvement based on the measurements. x. Known types of defects are prevented from recurring by tuning the process xlessons learned from specific projects incorporated into the process 96

Level 5: (Optimizing) z. Identify best software engineering practices and innovations: y tools, methods, or process are identified ytransferred throughout the organization z. World-wide about 50 organizations have been assessed at this level. 97

Key Process Areas z. Each level is associated with a key process area (KPA) identifies ywhere an organization at the previous level must focus to reach this level 98

Level 2 KPAs z. Software project planning y. Size, cost, schedule. yproject monitoring z. Configuration management z. Subcontract management 99

Level 3 KPAs z. Process definition and documentation z. Reviews z. Training program 100

Level 4 KPAs z. Quantitative measurements z. Process management 101

Level 5 KPAs z. Defect prevention z. Technology change management z. Process change management 102

Comparison between ISO 9001 and SEI CMM z. ISO 9001 awarded by an international standards body ycan be quoted in official documents and communications z. SEI CMM assessment is purely for internal use. 103

Comparison between ISO 9001 and SEI CMM z. SEI CMM was developed specifically for software industry: yaddresses many issues specific to software industry. z. SEI goes beyond quality assurance yaims for TQM y. ISO 9001 correspond to SEI level 3. 104

Comparison between ISO 9001 and SEI CMM z. SEI CMM provides a list of key areas yon which to focus to take an organization from one level to the other z. Provides a way for gradual quality improvements over several stages. ye. g trying to implement a defined process before a repeatable process: x counterproductive as managers are overwhelmed by schedule and budget pressure. 105

Remarks on Quality Model Usage z Highly systematic and measured approach to software development process suits certain circumstances ynegotiated software, safety-critical software, etc z What about small organizations? x. Typically handle applications such as internet, e-comm. xwithout an established product range, xwithout revenue base, experience on past projects, etc. x. CMM may be incompatible 106

Small Organizations z. Small organizations tend to believe: y. We are all competent people hired to do a job, we can’t afford training y. We all communicate with one another x. Osmosis works because we are so close y. We are all heroes x. We do what needs to be done x. Therefore rules do not apply to us 107

Small Organizations z. Often have problems: y. Undocumented requirements y. Inexperienced managers y. Documenting the product y. Resource allocation y. Training y. Peer reviews 108

Small Organizations z. A two week CMM-based appraisal is probably excessive: z. Small organizations need to operate more efficiently at lower levels of maturity y. Must first fluorish if eventually they are to mature 109

Personal Software Process (PSP) z. Based on the work of Humphrey z. PSP is a scaled down version of industrial software process ysuitable for individual use z. Even CMM assumes that engineers use effective personal practices 110

Personal Software Process (PSP) z. A process is the set of steps for doing a job z. The quality and productivity of an engineer ylargely determined by his process z. PSP is framework that yhelps software engineers to measure and improve the way they work. 111

Personal Software Process (PSP) z. Helps developing personal skills and methods y. Estimating and planning method y. Shows how to track performance against plans y. Provides a defined process xcan be fine tuned by individuals x. Recognizes that a process for individual use is different from that necessary for a team project. 112

Time Management z. Track the way you spend time y. Boring activities seem longer then actual y. Interesting activities seem short z. Record time for y. Designing y. Writing code y. Compiling y. Testing 113

Personal Software Process (PSP) Planning Design Code Compile Test Postmortem Logs Project plan summary 114

PSP-Planning z. Problem definition z. Estimate max, min, and total LOC z. Determine minutes/LOC z. Calculate max, min, and total development times z. Enter the plan data in project plan summary form zrecord the planned time in Log 115

PSP-Design z. Design the program z. Record the design in specified format z. Record the Design time in time recording log 116

PSP-Code z. Implement the design z. Use a standard format for code text z. Record the coding time in time recording log 117

PSP-Compile z. Compile the program z. Fix all the defects z. Record compile time in time recording log 118

PSP-Test/Postmortem z. Test y. Test the program y. Fix all the defects found y. Record testing time in time recording log z. Postmortem y. Complete project plan summary form with actual time and size data y. Record postmortem time in time record 119

Personal Software Process (PSP) PSP 3 PSP 2 PSP 1 PSP 0 Personal process evolution Personal quality management Design and code reviews Personal planning Time and schedule Personal measurement Basic size measures 120

Six Sigma z. Six sigma is a quantitative approach to eliminate defects y. Applicable to all types of industry - from manufacturing, product development, to service z. The statistical representation of Six Sigma quantitatively describes yhow a process is performing 121

Six Sigma z. To achieve six sigma ya process must not produce more than 3. 4 defects per million opportunities. y 5 Sigma -> 230 defects per million y 4 Sigma -> 6210 defects per million z. Six sigma methodologies y. DMAIC (Define, Measure, Analyze, Improve, Control) y. DMADV: (Define, Measure, Analyze, Design, Verify) 122

Six Sigma Methodologies z. The methodologies are implemented by Green belt and Black belt workers y. Supervised by Master black belt worker z. Pareto Chart: y. Simple bar chart to represent defect data y. Identify the problems that occurs with greatest frequency xor incur the highest cost 123

Summary z. Evolution of quality system: yproduct inspection yquality control yquality assurance ytotal quality management (TQM) z. Quality paradigm change: yfrom product to process 124

Summary z. ISO 9000: ybasic premise: xif a good process is followed xgood products are bound to follow yprovides guidelines for establishing a quality system. 125

Summary z. ISO 9000 yseries of three standards x 9001, 9002, and 9003 y 9001 is applicable to software industry 126

Summary z. SEI CMM ydeveloped specially for software industry yclassifies software organizations into five categories. x. According to the maturity of their development process. 127

Current Trends z. Many organizations have already tuned their process for y. Budget, y. Schedule, and y. Quality product. z. Competition is challenging them to: y. Reduce time for delivery y. Adopt Six-Sigma methodology 128