SOFTWARE QUALITY ASSURANCE Song Guo, Shan Bai CSC 8350 Advanced Software Engineering Spring, 2008 Instructor: Dr Xiaolin Hu
outline l Concept of SQA and standards l Compare SQA process methods l A new approach in SMSE (small and medium software enterprises ) l Conclusion
Quality Is 99% good enough ? No, · Accuracy shall be sufficient to support mission · System shall provide real-time response · System shall make efficient use of resources
Software Quality Assurance l What is “quality”? l IEEE Glossary: Degree to which a system, component, or process meets l l l (1) specified requirements, (2) customer or user needs or expectations ISO: the totality of features and characteristics of a product or service that bear on its ability to satisfy specified or implied needs
Software Quality Assurance l An alternate view of Quality: l l l is not absolute is multidimensional, can be difficult to quantify has aspects that are not easy to measure assessment is subject to constraints (e. g. , cost) is about acceptable compromises criteria are not independent, can conflict
Software Quality Assurance l Quality Criteria include: l l l correctness efficiency flexibility integrity interoperability maintainability portability reliability reusability testability usability
What is Software Quality Assurance (SQA)? l “Set of systematic activities providing evidence of the ability of the software process to produce a software product that is fit to use” l G. Schulmeyer and J. Mc. Manus, Software Quality Handbook, Prentice Hall, 1998.
What is SQA? l l Monitoring processes and products throughout the software development lifecycle to ensure the quality of the delivered product(s) Monitoring the processes l l Provides management with objective feedback regarding process compliance to approved plans, procedures, standards, and analyses Monitoring the products l l Focus on the quality of product within each phase of the SDLC l e. g. , requirements, test plan, architecture, etc. Objective: identify and remove defects throughout the lifecycle, as early as possible
Relation between terms Software Quality Engineering Quality Development Requirements engineering Implementation Design Quality Assurance Software quality assurance Verification & validation Quality management
SQA VS SVV l SQA is not the only checking activity in a project. Whereas SQA checks procedures against plans and output products against standards, Software Verification and Validation (SVV) checks output products against input products.
Standards Differing views of quality standards: • taking a systems view (that good management systems yield high quality); • taking an analytical view (that good measurement frameworks yield high quality). Examples: ISO 9000 -3 Quality Management and Quality Assurance Standards Guidelines for the application of 9001 to the development, supply, installation and maintenance of computer software IEEE Std 1061 -1992 Standard for Software Quality Metrics Methodology ISO 9126 is the standard for evaluating software quality.
IEEE Std 730 -2002 (Revision of IEEE Std 730 -1998) IEEE Standard for Software Quality Assurance Plans
Required SQAP Sections l l l l Purpose Reference Documents Management Documentation Standards, practices, conventions, and metrics Test Problem Reporting and corrective action Tools, techniques, and methodologies l l l l Media control Supplier control Records collection, maintenance, and retention Training Risk management Glossary SQAP change procedure and history
4. 4 Documentation (section 4 of the SQAP) l l l l 4. 4. 2. 1 Software requirements description (SRD) 4. 4. 2. 2 Software design description (SDD) 4. 4. 2. 3 Verification and validation plans 4. 4. 2. 4 Verification results report and validation results report 4. 4. 2. 5 User documentation 4. 4. 2. 6 Software configuration management plan (SCMP) 4. 4. 3 Other documentation l a) Development process plan l b) Software development standards description l c) Software engineering methods/procedures/tools description l d) Software project management plan (see IEEE Std 1058™-1998 [B 13]) l e) Maintenance plan (see IEEE Std 1219™-1998 [B 15]) l f) Software safety plans (see IEEE Std 1228™-1994 [B 16]) l g) Software integration plan
4. 5 Standards, practices, conventions, and metrics (section 5 of the SQAP) l 4. 5. 1 Purpose l 4. 5. 2 Content l The subjects covered shall include the basic technical, design, and programming activities involved, such as documentation, variable and module naming, programming, inspection, and testing. As a minimum, the following information shall be provided l l l a) Documentation standards b) Design standards c) Coding standards d) Commentary standards e) Testing standards and practices f) Selected software quality assurance product and process metrics
4. 6 Software reviews (section 6 of the SQAP) l l 4. 6. 1 Purpose 4. 6. 2 Minimum requirements l 4. 6. 2. 1 Software specifications review (SSR) l 4. 6. 2. 2 Architecture design review (ADR) l 4. 6. 2. 3 Detailed design review (DDR) l 4. 6. 2. 4 Verification and validation plan review l 4. 6. 2. 5 Functional audit l 4. 6. 2. 6 Physical audit l 4. 6. 2. 7 In-process audits l a) Code versus design documentation l b) Interface specifications (hardware and software) l c) Design implementations versus functional requirements l d) Functional requirements versus test descriptions l 4. 6. 2. 8 Managerial reviews l 4. 6. 2. 9 Software configuration management plan review (SCMPR) l 4. 6. 2. 10 Post-implementation review l 4. 6. 3 Other reviews and audits
Software Capability Maturity Model SEI-CMM
SEI-CMM l Classifies software organization according to its capability l CMM identifies 5 different maturity levels 1. INITIAL The software development is run informally and depends on the competence of some persons 2. REPEATABLE There is common system for project management and control 3. DEFINED There is a common system for the software engineering activities 4. MANAGED The Software development process is stable and gives a consistent product quality. Measurements are used to keep the process and product under control 5. OPTIMIZING The software development process contains its own improvement process l l l
Software Quality Assurance - Prepare and review SQA plan for project - Form QA group for project (or common across organization) - Involve QA personnel in project plan preparation and review - Plan for and perform work-product audits - QA review of project progress and activities Quality Assurance Activities • Auditing • Monitoring • Self-Checking • Problem Solving (Process Methodology Improvement)
Auditing Periodic audits are planned and executed by the independent audit team to ensure compliance to procedures and to improve the process methodologies by evaluating the quality system.
Self-Checking Each team member has to ensure that they are aware of the procedures and adhered in practice as a self-checking concept.
Problem Solving ( Process Methodology Improvement ) The data collected during the Auditing and Monitoring processes are analyzed for finding out deficiencies and abnormalities in the process. The results of the analysis are used to improve the processes.
QUALITY AUDIT A systematic and independent examination to determine whether quality activities and related results comply with planned arrangements and whether these arrangements are implemented effectively and are suitable to achieve objectives. AUDITS · Adequacy Audit · Compliance Audit
Conduct Record Quality System Audits Prepare Review Report Correctiv e Actions
outline l Concept of SQA and standards l Compare SQA process methods l A new approach in SMSE (small and medium software enterprises ) l Conclusion
MSF : Microsoft Solution Framework l l l Focussing on delivering high quality software, the major principles of the framework include iteratively adding functionality and the “teamofpeers” approach without a dominating project leader. Both principles improve not only the quality of software but also the way it is built. Microsoft supplies a couple of templates for artefacts recommended in the MSF. Since the MSF does not include a huge set of recommended tools (which will change with the next release of Visual Studio) it can be easily integrated into existing development environments. MSF was developed based on the large experience Microsoft has gathered over the decades in engineering standard software products.
Content MSFv 4 “Core” Application Development MSF v 4 CMMI MSF for Agile Software Development Infrastructure Agile MSF v 3 Infrastructure MSF for CMMI® Process Improvement Discipline Family Product
MSF: Team model Advocacy Groupings Was: Role Clusters New: Architecture Responsibilities pulled from Proj Mgmt and Dev Release Operations Was: Release Mgmt More functional areas in each advocacy group
MSF overall
Iterative Approach Risk Knowledge Solution Completion Minimize risks by breaking large projects into multiple versions Time
RUP: Rational Unified Process l l l The most commonly used software development process in the industry. The main advantages are the support through Rational Software. RUP delivers a well structured frame work, divided into phases and workflows, allowing easy navigation within the framework. Additional concepts of artifacts and workers are easy to understand facilitate resource planning and work structuring. The process includes comprehensive quality assurance measures. Minimal standards as requirements engineering and iterative software development are included as well as testing, configuration management and collaboration with the customer during the overall process. Additional reviews form an important part of the process, continuously monitoring quality and progress.
RUP Late ’ 60 s Ericsson Approach Objectory Process ‘ 87 –’ 96 ‘ 97 –’ 98 ‘ 99 –’ 05 The Unified Process IBM Rational Unified Process Good Software
RUP
RUP Iteration Model RUP meets almost all requirements to reach CMM levels 2 and 3.
XP: Extreme Programming l For smaller projects, XP offers a good approach to achieve high software quality. l The tight involvement of the customer, the main focus on testing and the approach to reduce design efforts support this goal. l However, XP may lead to organizational problems when applied in large projects. For most projects, the tight integration of the customer during the development is hard to achieve.
Planning game System Metaphor Simple design pair programming Coding standards Test-driven Refastening Continuous integration Small releases On-site customer Collective ownership 40 -hour week Communication Simplicity Feedback Courage XP is compatible to CMM as well
Quality Support Practices for RUP, XP and MSF
Compare with RUP, XP, MSF: Quality is formally defined as an objective. CCM can be easily integrated into the framework to accomplish a maximum of quality assurance. The MSF provides methods regarding requirements and architecture as both are key elements used during the overall process. RUP strongly verifies quality throughout the process. A well-defined review procedure is triggered at the end of iterations. Customer requirements are very well-handled. The process is also very architecture driven. The first candidate architecture is already created during Inception phase. XP: performs poorly regarding quality assurance methods only. focuses on small projects. simple and fast development. the customer is on site and involved in the iteration planning process strengthens quality control from the customer site. The short iterations force the project team to develop functional releases at the end of each iteration to pass acceptance testing.
Creating Organic Software Maturity Attitudes (COSMA) Selected Principles and Activities for Software Maturity in Small and Medium Software Enterprises
Outline l l Introduction Characteristics of SMSE(small and medium software enterprises) l l SM Principles (SMP) for SMSE SM activities (SMA)
Introduction l l l “big approaches” cannot simply be scaled down or customized to smaller company sizes The problem often arises from size, growth and granularity: - no quality management (QM) department: high expenses low return - techniques of quality control (tests, reviews, audits, etc. ) will often be established by individual effort - upon particular project histories The terms software maturity (SM) and software quality assurance (SQA) are used synonymously here.
a scenario based approach l l SM principles (SMP) represent concepts on a general level. SMP target the project strategies and the engineering teams’ general mindset. SMP consider and respect major risks and chances concerning SM in SMSE. SMP form a guideline structure for lower level SM activities (SMA) can be regarded as a sort of ‘software quality tool box’, specially customized to the characteristics of SMSE. SMA are used and applied according to best fit (project type, problem type, method) and to available resources (time, costs, human resources). This approach helps SMSE to choose their individual way to enhance software maturity.
Where are SMP & SMA from l l Empirical assessment of successful software development practices in SMSE through case studies, quantitative investigations or from qualitative interviews with experienced software engineers and consultants. Customizing and optimizing known and approved software quality assurance methods to the needs of small companies in an appropriate and “SMSE conscious” way.
Characteristics of SMSE l l l Local customer scope Local developer scope Limited resources (budget, skill) Small to medium project size Specialized domains Unspecialized developers
Extremely customer driven l l l Many SMSE are specialized in a small domain serving a specific group of customers. Specialization enables more precise software than “the big” could or would provide. Strong ties and “family-like” can influence strategy. Development tends to become “customer driven” vs. “market driven”. In case of limited resources or of tight decisions, a customer’s milestone will win over a quality guideline.
Strongly key people driven l l Compared to large software companies the absolute number of software engineers who work in an SMSE is low. SMSE depend on the individual performance of single engineers. No quality behavior can be established without the “inner” permission of the key developers in the company. While in bigger enterprises “the faster ones may eat the slower ones”, mutual acceptance is a must here.
Strongly key people driven l While management cannot “enforce” a certain behavior easily to its key engineers, this structural disadvantage of SMSE turns out to be a ”silent” quality enhancer: small teams, little fluctuation and similar teams in a series of projects lead to reduced chance of misunderstanding and inconsistency, denser internal communication, a better picture of the overall project in team members’ mind.
Limited resources (budget&skills) l l Resources in SE practice are always limited. Three major limitations directly affect software maturity: - SE specialists: usability, security, web design, database monitoring, transaction management, etc, even software quality management. - Financial playgrounds and ranges: SM mostly is not at the core of the business. SM is one of the first victims if resources become narrow. - Corporate infrastructure: SMSE lack huge laboratories, specialized equipment or dedicated departments like e. g. a software quality assurance department. Therefore SM methods for SMSE should not rely on or organize themselves on the basis of such infrastructural prerequisites.
Small and medium-small sized projects l l l Applying a large scale SM approach to a rather small project (e. g. 2 developers 3 months) can be quite ridiculous. Imagine you spend 2 person months (pemos) for analysis, 3 pemos for design, coding and testing. Finally take 1 pemo for integration into actual production. Who will make this small project spend 1. 5 pemos (25%!) in order to meet a proper quality guideline? In small sized software projects it is easier for an individual engineer has a rather complete picture of the whole project. This overview will help the engineer to do a better job with respect of quality factors.
Domain driven software engineering l l l A lot of SMSE are specialized in only one application domain. Domain experts rather than software engineering experts. Developers’ communication with customers and users is highly efficient. With respect to SM methodology this makes the community of SMSE a type of “tribal community”. General approaches for enhancing software maturity in SMSE must be aware of this diversity. They must not stress on strict standardization or streamlined structures.
Generalists as developers l l Software engineers in SMSE are often not specialized in one engineering task. A specialized task like quality assurance will be performed by people who are also in charge of other software engineering responsibilities.
SM Principles (SMP) for SMSE l SM principles affect software development on the level of developers’ mindset and general development behavior. They target the engineers’ attitude of mind, the department’s software development culture and the project environment needed to produce high quality software. - Observations from a local sample: accepted and applied SM Principles - Bottom-up SQA - (Bottom-up) evolutionary improvement - Extra focus on “vital few” - Extra focus on early project phases - User involvement
Bottom-up SQA l l Bottom-up SQA is more important in a SMSE because no QA department will “catch the chaos”. Software quality assurance cannot be imposed from above on an unreceptive programming staff. It must be a grass roots movement.
(Bottom-up) evolutionary improvement l l l Evolutionary improvement deals with consciously collecting experiences. To achieve evolutionary, adaptation advantages and weaknesses of the current SQA plan have to be identified and tuned. After a series of projects and various steps of customization the activities for software maturity will fit to the SMSE’s special needs.
Extra focus on “vital few” l l l Very often 20% of the inputs cause 80% of results. In a lot of cases 80% of quality can be achieved with < 30% of the resources which would be needed for a 99+% quality level. If a level of 80% is sufficient, this approach will save lots of resources.
Extra focus on early project phases l l The longer an error or inconsistency remains in a software system, the higher is the cost of error correction. Excellent performance from the beginning can shorten total development time.
User involvement l l software quality is its “fitness for use” Serious and strategic user involvement results in the following benefits: - Clear definition of user requirements - Detection of conflicting goals - Establishment of a common system view - Strange ‘surprises’ for the users can be avoided - Creates trust between users and developers
SM activities l l Software maturity activities deal with software development issues, rules and behaviors on a rather practical level. A sort of ‘behavior tool box’ (BTB) selected from SQA literature. - acquirements and specialties of the projects - appropriateness for participating engineers - given resources (budget, time)
Criteria for The SMA BTB l l l No-specialist: no (permanent) software quality assurance specialists are needed to perform the action Resource appropriateness: Costs (time, money) for the activities is adequate to the capabilities of SMSE Flexibility: flexible enough to be easily adapted to different scenarios Coverage: target the most common quality problems Locality: target defined trouble Dose: SMA have to cause visible results, even when applied with a minor or middle dose.
SMA l l l l l Educate engineers Set quality goals Anchor QA in project Prototyping (vertically and horizontally) Configuration Management Use templates Review important documents Use software tools (appropriately) Test software independently Analyze post mortem
Relation between SMA and SAP
Presentation l l a 2 -dimensional grid horizontal axes are: - Organizational level: Activity doesn’t influence software development directly but helps creating a good corporate environment for producing high quality software. - Constructive level: Activity avoids the introduction of errors into the system during development. - Analytical level: Activity helps identifying errors in the system or in other development products. l vertical axes are: - Person level: Activity influences the individuals and engineers who work on the software. - Process level: Activity targets the process of development. - Product level: Activity directly affects a certain development product.
Educate engineers l l Software Quality starts with the individual engineer more important than in larger companies: on QA departments cost time&money, and may not be directly productive: right measure is important Personal Software Process (PSP): a popular program to increase the individual engineer’s skills
Set quality goals l l l “set the appropriate goals” and “set them clearly” Difficult: quality goals also depend on the application domain and on the specialization of the software company Quality goals can have different origins: impossible to cater all at the same time
Anchor QA in project l l l Bottom-up SQA can only succeed, if it becomes a way of everyday work for the single developer in the team. Customers and project managers have to be part of that behavior pattern, too. Anchoring QA in a software project means directing the attitude of people involved towards integrating SMA and SMP into daily development.
Prototyping (vertically and horizontally) l l l Vertical prototypes reduce technical misunderstandings and risks Horizontal prototypes support completeness and resource estimation: Interface sketches Candidates for prototyping have to be carefully selected and must not become the central process model
Configuration Management l l l Software configuration management (SCM) deals with organizing all documents and products being created during software development. SCM is done by specialists in larger companies. Simple management operation or more effectively by tools in SMSE: CONTINUUS™, PVCS™, IDE integrated solutions
Use templates l Templates (document and code standards) help to increase quality by avoiding errors: - They guarantee that required data will be filled in rather complete in a document. - They increase data consistency. - They speed up routine work by avoiding similar work to be done several times. - They help assess the status or quality in a project via checklists.
Review important documents l Reviews can be simple and extremely effective: - they can be applied to all kinds of artifacts and documents, even to processes or people. - they help to find a wide range of error types. - they can be performed very early and any time. - they list problems including description, nature and location of the error.
Use software tools (appropriately) l Successful software tools increase the productivity and decrease the error rate, but has preconditions: - tools cannot replace a software development process. - “A fool with a tool is still a fool“: the use of the tool has to be guarded and guided. - engineers can use tools effectively only if they were taught. - tools should be part of a framework.
Test software independently l Testing in actual practice sometimes shows major drawbacks: - major testing loads in practice take place late within the software life cycle. - engineers who test their own code are very bad testers.
Analyze post mortem l l Customer orientation makes analysis of projects after delivery important beyond the collection of the SMSE’s internal technical/organizational experiences. The best time to analyze the success of a project is some time after the end of the project. (e. g. Xerox™ analyzed project success 60 days and 90 days after projects’ end).
Thanks ! Q&A
Скачать презентацию SOFTWARE QUALITY ASSURANCE Song Guo Shan Bai CSC
93dcd8a228bc4c287a59e3a7de9135d6.ppt