Software Engineering Software Quality Assurance Objectives 1

Software Engineering Software Quality Assurance

Objectives 1. To motivate for software quality assurance. 2. To provide a historical context for software quality assurance. 3. To consider the benefits of formal technical reviews. 4. To introduce statistical software quality assurance (SSQA).

Steps in Project Planning Project Scope Estimates Risks Schedule Control strategy Software Project Plan l Scope—understand the problem and the work that must be done. l Estimation—how much effort? how much time? l Risk—what can go wrong? how can we avoid it? what can we do about it? l Schedule—how do we allocate resources along the timeline? what are the milestones? l Control strategy—how do we control quality? how do we control change?

Defining Quality l One goal of software engineering is to produce high-quality software. But what is “software quality? ” l Software quality assurance (SQA) is an umbrella activity applied throughout the software process. l General engineering objective: reduce the “variation between samples. ” But how does this apply to software? l Quality of Design: Characteristics that designers specify for an item. Encompasses requirements specification, analysis and design. l Quality of Conformance: The degree to which the design specifications are followed during manufacturing. Covers implementation. l Alternatively: User satisfaction = compliant product + good quality + delivery within budget and schedule.

Mc. Call’s Triangle of Quality Maintainability Flexibility Testability Portability Reusability Interoperability PRODUCT REVISION PRODUCT TRANSITION PRODUCT OPERATION Correctness Usability Efficiency Reliability Integrity

Deriving Quality Metrics l Each of Mc. Call’s Quality Metrics combines different quality factors. l Examples: Correctness = Completeness + Consistency + Traceability Portability = Generality + Hardware Independence + Modularity + Self-Documentation + System Independence Maintainability = Conciseness + Consistency + Instrumentation + Modularity + Self-Documentation + Simplicity l “Mc. Call’s quality factors were proposed in the early 1970 s. They are as valid today as they were in that time. It’s likely that software built to conform to these factors will exhibit high quality well into the 21 st century, even if there are dramatic changes in technology. ”

Quality Concepts l Quality control: A series of inspections, reviews, tests to ensure that work products meet their requirements. Must include feedback. l Quality assurance: Analysis, auditing and reporting activities which communicate information about quality to management. l Cost of quality: all costs incurred in performing quality related activities. - Prevention Costs (quality planning, formal technical reviews, test equipment and training) - Appraisal costs (trying to gain insight into product condition = inprocess inspection, equipment calibration and maintenance, testing) - Failure costs (detect an error prior to shipping = rework, repair, failure mode analysis) - External failure costs (detect an error after shipping = complaint resolution, product return and replacement, help line support, warranty work)

Why SQA Activities Pay Off? Cost to find and fix a defect 100 60. 00 -100. 00 log scale 10 1 10. 00 0. 75 1. 00 1. 50 3. 00 design testsystem field anal. code test use

Software Quality Assurance l “Conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics. ” SQA Process Definition & Standards Formal Technical Reviews Analysis & Reporting Measurement Test Planning & Review

SQA Definition l Aspects of SQA: 1. Software requirements are the foundation from which quality is measured. Lack of conformance to requirements is lack of quality. 2. Specified standards must be followed. 3. A set of implicit requirements often goes unmentioned (e. g. ease of use). If software conforms to explicit requirements but fails to meet implicit requirements, software quality is suspect. l History: - Prior to 20 th. C province of the individual craftsman. - First formal quality assurance at Bell labs in 1916. - Software SQA parallels this (1950 s and 1960 s done by individual programmers, 1970 s military introduced structured SQA)

SQA Group l Prepares an SQA plan for the project. This must identify evaluations, audits, reviews, standards, procedures for error reporting and tracking, and feedback. l Participates in the development of the project process. l Reviews software engineering activities to verify compliance with the defined software process. l Audits designated software work products to verify compliance with those defined as part of the software process. l Ensures that deviations in work products are documented and handled according to set procedures. l Records any non-compliance and reports to senior management.

Formal Technical Reviews l Formal technical reviews can be up to 75% effective in uncovering design flaws (which cover 50 -65% of all errors) l “There is no particular reason why your friend and colleague cannot be your sternest critic” - Jerry Weinberg l Objectives: 1. To uncover errors in function, logic, or implementation for any representation of the software. 2. To verify that the software under review meets its requirements. 3. To ensure that the software has been represented according to predefined standards. 4. To achieve software that is developed in a uniform manner. 5. To make projects more manageable.

A Case for Reviews l Defect amplification: a model for the generation and detection of errors during a sequence of tasks. Errors from previous step Defects Errors passed through Amplified errors 1: x Newly generated errors Detection Percent efficiency for error detection Errors passed to next step l Example: - Each test step uncovers 50% of errors but no reviews conducted in earlier stages. 10 errors in analysis lead to 12 latent defects. Assume amplification of 1: 1. 5. - Formal technical reviews with 50 -70% efficiency in analysis design and coding reduce latent defects to 3.

What Are Reviews? l What reviews are: - a meeting conducted by technical people for technical people - a technical assessment of a work product created during the software engineering process - a software quality assurance mechanism - a training ground l What reviews are not: - A project budget summary - A scheduling assessment - An overall progress report - A mechanism for reprisal or political intrigue

The Players review leader standards bearer (SQA) producer maintenance oracle reviewer recorder user rep

Conducting the Review 1. be prepared—evaluate product before the review 2. review the product, not the producer 3. keep your tone mild, ask questions instead of making accusations 4. stick to the review agenda 5. raise issues, don't resolve them 6. avoid discussions of style—stick to technical correctness 7. schedule reviews as project tasks 8. record and report all review results

Metrics Derived from Reviews l l l l Inspection time per page of documentation Inspection time per KLOC or FP Inspection effort per KLOC or FP Errors uncovered per reviewer hour Errors uncovered per preparation hour Errors uncovered per SE Task (e. g. design) Number of minor errors (e. g. typos) Number of major errors (e. g. non-conformance to design) l Number of errors found during preparation

Statistical SQA Product & Process • collect information on all defects • find the causes of the defects • move to provide fixes for the process measurement. . . an understanding of how to improve quality. . .

Aspects of Quality l Software Reliability: - Important aspect of overall quality - The probability of failure free operation of a computer program in a specified environment for a specified time. - Where failure is non-conformance to specification. - Example: program X is estimated to have a reliability of 0. 96 over 8 elapsed processing hours. - Mean Time Between Failure (MTBF) = Mean Time To Failure (MTTF) + Mean Time To Repair (MTTR). - Regarded by some as better than num. defects. l Software Availability: - Probability that a program is operating according to requirements at a given point in time. - Availability = [MTTF / (MTTF + MTTR)] * 100%.

Further Aspects of Quality l Software Safety: - Important in safety critical systems - Unlike software reliability, failure leads to a hazard or mishap. - Use techniques from Risk Analysis. Hazards are identified and categorized according to probability and criticality. - Example: Cruise control in a car. - Hazards: (1) Causes uncontrolled acceleration, (2) does not respond to depression of break pedal, (3) does not engage when switch is activated, (4) slowly loses or gains speed.

Mistake Proofing l Poka-Yoke (Mistake Proofing) developed by Shigeo Shingo at Toyota in the 1960 s. l A quality assurance technique applicable to software. Automatic error detection and prevention. l Devices: - Prevention of potential quality problems before they occur. - Rapid detection of quality problems if they are introduced. l Everyday examples: seatbelt warning signal (detection) or Car unable to start when in gear (prevention). l Characteristics: - It is simple and cheap. Otherwise it will not be cost effective. - It is part of the process. Integrated into the process. - It is located near the process task where the mistakes occur. Allows rapid feedback and error correction.

ISO 9000 Quality Standards l A quality assurance system is defined as the organizational structure, responsibilities, processes and resources for implementing quality management. l Satisfy customers expectations by meeting their specifications. l ISO 9000: - A generic description of quality assurance activities applicable to any business - Adopted by many countries - Often required to supply goods and services to the government - Compliance scrutinized by third party auditors and subject to semiannual review l ISO 9001: - Standards that apply to Software Engineering