Software Testing and Quality Assurance Theory and Practice

Software Testing and Quality Assurance Theory and Practice Chapter 17 Software Quality Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 1

Outline of the Chapter n n n Five Views of Software Quality Mc. Call’s Quality Factors and Criteria The ISO 9126 Quality Characteristics The ISO 9000: 2000 Software Quality Standard Summary Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 2

Five Views of Software Quality n n n Transcendental view User view Manufacturing view Product view Value-based view Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 3

Five Views of Software Quality n Transcendental view – Quality is something that can be recognized through experience, but not defined in some tractable form. – A good quality object stands out, and it is easily recognized. n User view – Quality concerns the extent to which a product meets user needs and expectations. – Is a product fit for use? – This view is highly personalized. • A product is of good quality if it satisfies a large number of users. • It is useful to identify the product attributes which the users consider to be important. – This view may encompass many subject elements, such as usability, reliability, and efficiency. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 4

Five Views of Software Quality n Manufacturing view – This view has its genesis in the manufacturing industry – auto and electronics. – Key idea: Does a product satisfy the requirements? • Any deviation from the requirements is seen as reducing the quality of the product. – The concept of process plays a key role. – Products are manufactured “right the first time” so that the cost is reduced • Development cost • Maintenance cost – Conformance to requirements leads to uniformity in products. – Some argue that such uniformity does not guarantee quality. – Product quality can be incrementally improved by improving the process. • The CMM and ISO 9001 models are based on the manufacturing view. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 5

Five Views of Software Quality n Product view – Hypothesis: If a product is manufactured with good internal properties, then it will have good external properties. – One can explore the causal relationship between internal properties and external qualities. – Example: Modularity enables testability. n Value-based view – This represents the merger of two concepts: excellence and worth. – Quality is a measure of excellence, and value is a measure of worth. – Central idea • How much a customer is willing to pay for a certain level of quality. • Quality is meaningless if a product does not make economic sense. • The value-based view makes a trade-off between cost and quality. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 6

Five Views of Software Quality n Measuring quality – Reasons for developing a quantitative view of quality – Measurement of user’s view – Measurement of manufacturing view n Reasons for developing a quantitative view of quality – Measurement allows us to establish baselines for qualities. – Measurement is key to process improvement. – The needs for improvements can be investigated after performing measurements. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 7

Five Views of Software Quality n Measurement of user’s view – Functionalities can be easily measured. • What functionality test cases have passed? • Measure of delivered functionality = ratio of # of passed test cases to the total number of test cases designed to verify the functionalities. – Apply Gilb’s technique. • The quality concept is broken down into component parts until each can be stated in terms of directly measurable qualities. • Example: Usability can be broken down into – Learnability – Understandability – Operability Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 8

Five Views of Software Quality n Measurement of manufacturer’s view – Manufacturers are interested in defect count and rework cost. – Defect count: The total number of defects detected during development and operation. • It is a measure of the quality of the work produced. • One can analyze the defects as follows. – For each defect, identify the development phase in which it was introduced. – Categorize the defects, say, based on modules. – Normalize defect count by product size. ØDefect density: Number of defects per 1000 lines of code. – Separate the defects found during operation from those found during development. – Rework cost: How much does it cost to fix the known defects? • Development rework cost: This is the rework cost incurred before a product is released. This is a measure of development efficiency. • Operation rework cost: This is the rework cost incurred when a Software Testing and QA This Practice (Chapter 17: Software Quality) product is in operation. Theory andis a measure of the delivered © Naik & Tripathy quality. 9

Mc. Call’s Quality Factors and Criteria n Quality Factors – Mc. Call, Richards, and Walters studied the concept of software quality in terms of two key concepts as follows: • quality factors, and • quality criteria. – A quality factor represents the behavioral characteristic of a system. • Examples: correctness, reliability, efficiency, testability, portability, … – A quality criterion is an attribute of a quality factor that is related to software development. • Example: – Modularity is an attribute of the architecture of a software system. – A highly modular software allows designers to put cohesive components in one module, thereby increasing the maintainability of the system. – Mc. Call et al. identified 11 quality factors (Table 17. 1. ) Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 10

Mc. Call’s Quality Factors and Criteria Table 17. 1: Mc. Call’s quality factors [10]. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 11

Mc. Call’s Quality Factors and Criteria n The 11 quality factors defined in Table 17. 1 have been grouped into three broad categories (See Table 17. 2. ) – Product operation – Product revision – Product transition Table 17. 2: Categorization of Mc. Call’s quality factors [10]. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 12

Mc. Call’s Quality Factors and Criteria n Quality Criteria – A quality criterion is an attribute of a quality factor that is related to software development. – Example: • Modularity is an attribute of the architecture of a software system. • A highly modular software allows designers to put cohesive components in one module, thereby increasing the maintainability of the system. – In Table 17. 3, we have listed 23 quality criteria. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 13

Mc. Call’s Quality Factors and Criteria Table 17. 3: Mc. Call’s quality criteria [10]. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 14

Mc. Call’s Quality Factors and Criteria n Relationship Between Quality Factors and Quality Criteria – Each quality factor is positively influenced by a set of quality criteria, and the same quality criterion impacts a number of quality factors. • Example: Simplicity impacts reliability, usability, and testability. – If an effort is made to improve one quality factor, another quality factor may be degraded. • Portable code may be less efficient. – Some quality factors positively impact others. • An effort to improve the correctness of a system will increase its reliability. – See Figure 17. 1. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 15

Mc. Call’s Quality Factors and Criteria Figure 17. 1: Relation between quality factors and quality criteria [10]. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 16

The ISO 9126 Quality Characteristics n The ISO 9126 document is the product of an international effort. – ISO: International Organization for Standardization n n The document defines six broad quality characteristics as shown in Table 17. 4. The document includes an example quality model (Figure 17. 2) that further decomposes the quality characteristics. – Figure 17. 2 is just an example, and not a universal one. – The 20 subcharacteristics of Figure 17. 2 have been defined in Table 17. 5. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 17

The ISO 9126 Quality Characteristics Table 17. 4: ISO 9126 quality characteristics. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 18

The ISO 9126 Quality Characteristics Figure 17. 2: The ISO 9126 sample quality model refines the standard’s features into subcharacteristics [9]. (©[1996] IEEE) Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 19

The ISO 9126 Quality Characteristics Table 17. 5: Quality subcharacteristics of ISO 9126. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 20

The ISO 9126 Quality Characteristics n Mc. Call’s quality model vs. ISO 9126 model – What is called quality factor in Mc. Call’s model is called a quality subcharacteristic in the ISO 9126 model. – The following quality factors/characteristics are found in both the models. • reliability, usability, efficiency, maintainability, and portability – Differences between the two models • The ISO 9126 model emphasizes on characteristics visible to the users, whereas the Mc. Call model considers internal qualities as well. • In Mc. Call’s model, one quality criterion can impact many quality factors, whereas in the ISO 9126 model, one subcharacteristic impacts exactly one quality characteristic. • A high-level quality factor, such as testability, in the Mc. Call’s model is a low-level subcharacteristic of maintainability in the ISO 9126 model. n Concerns – There is no consensus about what high-level quality factors are most important at the top level. Mc. Call, et al. define 11 high-level quality factors, whereas there are only six in the ISO 9126 document. – There is no consensus regarding what is a top-level quality factor/ characteristic and what is a more concrete quality criterion/ subcharacteristic. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 21

The ISO 9000: 2000 Software Quality Standard n n n The international organization ISO has developed a series of standards for quality assurance and quality management, collectively known as the ISO 9000. The ISO 9000 standards are reviewed and updated once every 58 years. The standards released in the year 2000 are known as ISO 9000: 2000. There are three components of the ISO 9000: 2000 standard. – ISO 9000: Fundamentals and vocabulary – ISO 9001: Requirements – ISO 9004: Guidelines for performance improvements n Note: ISO 9002 and ISO 9003 were parts of ISO 9000: 1994, but these are no more parts of ISO 9000: 2000. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 22

The ISO 9000: 2000 Software Quality Standard n ISO 9000: 2000 Fundamentals: – This is based on eight principles. • Principle 1: Customer focus • Principle 2: Leadership • Principle 3: Involvement of people • Principle 4: Process approach • Principle 5: System approach to management • Principle 6: Continual improvement • Principle 7: Factual approach to decision making • Principle 8: Mutually beneficial supplier relationships Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 23

The ISO 9000: 2000 Software Quality Standard n ISO 9001: 2000 Requirements – The five major parts of this document are as follows. • Part 4. Systemic requirements • Part 5. Management requirements • Part 6. Resource requirements • Part 7. Realization requirements • Part 8. Remedial requirements Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 24

The ISO 9000: 2000 Software Quality Standard n ISO 9001: 2000 Requirements – Part 4. Systemic requirements (partial) • Document the organizational policies and goals. Publish a vision document. • Document all quality processes and their interrelationship. • Implement a mechanism to approve documents before those are distributed. – Part 5: Management requirements (partial) • Generate an awareness for quality to meet a variety of requirements, such as customer, regulatory, and statutory. • Focus on customers by identifying and meeting their requirements in order to satisfy them. • Develop a quality policy to meet the customer’s needs. • Clearly define individual responsibilities and authorities concerning the implementation of quality policies. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 25

The ISO 9000: 2000 Software Quality Standard n ISO 9001: 2000 Requirements – Part 6. Resource requirements (partial) • Identify and provide resources required to support the organizational quality policy in order to realize the quality objectives. • Allocate quality personnel resource to projects. • Put in place a mechanism to enhance the quality level of personnel. – Part 7: Realization requirements (partial) • Develop a plan to realize a product from its requirements. • Interact with customers to gather their requirements. Classify those requirements. • Review the requirements. • Follow a defined purchasing process by evaluating potential suppliers based on a number of factors, such as ability to meet requirements and price. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 26

The ISO 9000: 2000 Software Quality Standard n ISO 9001: 2000 Requirements – Part 8. Remedial requirements (partial) • Measure and track the customer’s satisfaction level. • Perform internal audit. – Example: Find out whether or not personnel with adequate education, experience, and skill have been assigned to a project. • Monitor processes by using a set of key performance indicators. Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 27

Summary • Five views of quality – – – • Transcendental view User view Manufacturing view Product view Value-based view Measure quality for three reasons – Baseline – Quality improvement based on cost – Know the present level for future planning • • Global initiatives for quality control – ISO 9126 – ISO 9000: 2000 • ISO 9126 – Six quality characteristics – Twenty quality subcharacteristics • ISO 9000: 2000 – ISO 9000: Fundamentals – ISO 9001: Requirements – ISO 9004: Performance improvements Measuring the manufacturer’s view – Defect count – Rework cost • Mc. Call’s quality factors – Quality factors (11): 3 classes (operation, revision, transition) – Quality criteria (23) Software Testing and QA Theory and Practice (Chapter 17: Software Quality) © Naik & Tripathy 28