Software reliabilty Aim of Software quality assurance

Software reliabilty • Aim of Software quality assurance (SQA) is to build quality s/w products in a repeatable manner • A Repeatable S/w Development organization: S/w development is person independent • A Non-Repeatable S/w Development organization: S/w development is person dependent

Software reliabilty • • • Denotes trustworthiness or dependability Having large number of defects makes the s/w unreliable 10% instructions are core in the program the rest are non-core Where the errors are located is of importance The reliability increases due to fixing a single bug depends on where the bug is located in the code • Perceived reliability of a s/w is highly observer dependent • The reliability of a product keeps changing as errors are detected and fixed.

H/w vs S/w reliability Failure rate Wear out Burn in Useful life Time Obsolete Testing Time

Reliability Metrics • Rate of occurrence of failure (ROCOF) • Mean time to failure time: Average time between two successive failures, observed over a large number of failures. Record failure data for n failures. Let the failures occur at time instants t 1, t 2, …, tn. Then, MTTF = ∑ni=1 (ti+1 - ti)/(n-1) {only run-time is taken} • Mean time to repair (MTTR): Average time to track the errors causing failures and then to fix them • Mean time between failures: MTBF = MTTF + MTTR, If MTBF is 300 hrs it means once a failure occurs the next failure will occur only after 300 hrs. It is real time not execution time. • Probability of failure: likelihood of the system failing when a service request is made • Availability: how likely will the system be available for use over a period of time

Classification of failures • Transient: occur for only some i/p values for a function • Permanent: occur for all i/p values for a function • Recoverable: system recovers with or without operator intervention • Unrecoverable: system need to be restarted • Cosmetic: cause minor irritation and do not lead to incorrect results

What is quality? • • Quality, simplistically, means that a product should meet its specification. This is problematical for software systems – There is a tension between customer quality requirements (efficiency, reliability, etc. ) and developer quality requirements (maintainability, reusability, etc. ); – Some quality requirements are difficult to specify in an unambiguous way; – Software specifications are usually incomplete and often inconsistent.

The quality compromise • We cannot wait for specifications to improve before paying attention to quality management. • We must put quality management procedures into place to improve quality in spite of imperfect specification.

Scope of quality management • Quality management is particularly important for large, complex systems. The quality documentation is a record of progress and supports continuity of development as the development team changes. • For smaller systems, quality management needs less documentation and should focus on establishing a quality culture.

Quality management activities • Quality assurance – Establish organisational procedures and standards for quality. • Quality planning – Select applicable procedures and standards for a particular project and modify these as required. • Quality control – Ensure that procedures and standards are followed by the software development team. • Quality management should be separate from project management to ensure independence.

Quality management and software development

S/w Quality: External characteristics • Correctness: Degree to which system is free from faults in specification, design and implementation • Usability: The ease with which users can learn and use a system • Efficiency: Minimal use of system resources incl. memory and execution time • Reliability: The ability of a system to perform whenever required without/with few failures • Integrity: Prevention of unauthorized or improper use (allowed permissible users and data) • Adaptability: Usability in other applications than the original one • Accuracy: Degree of ‘quantitative’ correctness • Robustness: Functioning of system in presence of invalid inputs, stressful environment

S/w Quality: Internal characteristics • Maintainability: Ease of modifying software for changing/adding capabilities, improving performance, correcting defects • Flexibility: Extent of modifying system for other uses/environments • Portability: Ease of modifying system for operating in different environment • Reusability: Extent of using parts in other systems • Readability: Ease of reading and understanding source code • Testability: Support for unit- and system-testing • Understandability: Ease of comprehension of a system’s organization at high and low levels.

Defintions of s/w quality • Oxford Dictionary: the standard of something when compared to other things like it (a wine of excellent quality); of high quality, general excellence • Crosby: zero defects • Juran: Fitness for Purpose • ISO standard: The totality of features and characteristics of a product or service that bear on its ability to satisfy specified or implied needs • Dept. of Defense (USA): The degree to which the attributes of the software enable it to perform its intended end use

How can we achieve software quality? • By making sure everyone understands the qualities that must be embedded in the product • By determining “up front” how to achieve each quality – eg. requirements, implementation, design, standards • By establishing and enforcing a process that is focused on relevant qualities – eg. testing, inspection, documentation, measurement, formal methods, procedures • Using a quality model – high level qualities are associated with low level qualities – address the low level qualities to achieve the high level qualities

Quality assurance and standards • Standards are the key to effective quality management. • They may be international, organizational or project standards. • Product standards define characteristics that all components should exhibit e. g. a common programming style. • Process standards define how the software process should be enacted.

Importance of standards • Encapsulation of best practice- avoids repetition of past mistakes. • They are a framework for quality assurance processes - they involve checking compliance to standards. • They provide continuity - new staff can understand the organisation by understanding the standards that are used.

ISO 9000 • • • An international set of standards for quality management. Applicable to a range of organisations from manufacturing to service industries. Quality standards and procedures should be documented in an organisational quality manual. An external body may certify that an organisation’s quality manual conforms to ISO 9000 standards. Some customers require suppliers to be ISO 9000 certified although the need for flexibility here is increasingly recognised.

ISO 9000 certification • ISO 9001 applicable to organisations which design, develop and maintain products. ISO 9001 is a generic model of the quality process that must be instantiated for each organisation using the standard. Applicable to s/w development organization. • ISO 9002 applies to those organizations which do not design products but are only involved in production. Eg. steel, , car manufacturing industries. • ISO 9003 applies to organizations involved only in installation and testing of the products.

ISO 9001

ISO 9000 and quality management

Features of ISO 9001 • Documents should be properly managed, authorized and controlled • Proper plans should be prepared & then progress against these plans should be monitored. • Important documents should be checked & reviewed for effectiveness & correctness • Product should be tested against specification • Organizational aspects should be addressed such as quality team reporting to management.

Demerits of ISO 9000 • Requires a s/w production process to be adhered to but does not guarantee the product to be of high quality • Certification process is not foolproof and no international accredition agency exists • Organizations getting ISO 9000 certification often tend to downplay domain expertise. • Does not automatically lead to continuous process improvement.

Documentation standards • • Particularly important - documents are the tangible manifestation of the software. Documentation process standards – Concerned with how documents should be developed, validated and maintained. • Document standards – Concerned with document contents, structure, and appearance. • Document interchange standards – Concerned with the compatibility of electronic documents.

Documentation process

Document standards • Document identification standards – How documents are uniquely identified. • Document structure standards – Standard structure for project documents. • Document presentation standards – Define fonts and styles, use of logos, etc. • Document update standards – Define how changes from previous versions are reflected in a document.

Quality planning • A quality plan sets out the desired product qualities and how these are assessed and defines the most significant quality attributes. • The quality plan should define the quality assessment process. • It should set out which organisational standards should be applied and, where necessary, define new standards to be used.

Quality plans • Quality plan structure – – – Product introduction; Product plans; Process descriptions; Quality goals; Risks and risk management. • Quality plans should be short, succinct documents – If they are too long, no-one will read them.

Quality control • This involves checking the software development process to ensure that procedures and standards are being followed. • There are two approaches to quality control – Quality reviews; – Automated software assessment and software measurement.

Quality reviews • • • This is the principal method of validating the quality of a process or of a product. A group examines part or all of a process or system and its documentation to find potential problems. There are different types of review with different objectives – Inspections for defect removal (product); – Reviews for progress assessment (product and process); – Quality reviews (product and standards).

Types of review

Quality reviews • • • A group of people carefully examine part or all of a software system and its associated documentation. Code, designs, specifications, test plans, standards, etc. can all be reviewed. Software or documents may be 'signed off' at a review which signifies that progress to the next development stage has been approved by management.

Review functions • Quality function - they are part of the general quality management process. • Project management function - they provide information for project managers. • Training and communication function product knowledge is passed between development team members.

Quality reviews • The objective is the discovery of system defects and inconsistencies. • Any documents produced in the process may be reviewed. • Review teams should be relatively small and reviews should be fairly short. • Records should always be maintained of quality reviews.

Review results • Comments made during the review should be classified – – – • No action. No change to the software or documentation is required; Refer for repair. Designer or programmer should correct an identified fault; Reconsider overall design. The problem identified in the review impacts other parts of the design. Some overall judgement must be made about the most cost-effective way of solving the problem; Requirements and specification errors may have to be referred to the client.

What is Software Maintenance? • IEEE 91 Definition: – the process of modifying the software system or component after delivery to correct faults, improve performance or other attributes, or adapt to a change in the environment. • SM is concerned with modifying software once it is delivered to a customer • Major economic importance: 40 – 90% of the total lifecycle costs

What is Software Maintenance? • Four categories: 1. Perfective maintenance: changes required as a result of user requests (a. k. a. evolutive maintenance) 2. Adaptive maintenance: changes needed as a consequence of operated system, hardware, or DBMS changes 3. Corrective maintenance: the identification and removal of faults in the software 4. Preventative maintenance: changes made to software to make it more maintainable • The majority of SM is concerned with evolution deriving from user requested changes.

What is Software Maintenance? • What is meant by SM for organization: – a major economic importance – a substantial applications backlog • Starting for late 1960 s, SM started to be recognized as a significant activity

Software Maintenance: The Root Problem • The root problem is complexity. • Sometimes complexity arises because a system is migrated from hardware to software in order to gain the additional functionality found in software. • The combination of scale and application complexity mean that it is infeasible for one person alone to understand the complete software system.

Software Maintenance: Evolution • The important requirement of software maintenance for the client is that changes are accomplished quickly & cost effectively. – Reliability should not degrade. – Maintainability should not degrade. • Maintenance that becomes increasingly more expensive and difficult becomes known as a legacy system. – The legacy system may still be of essential importance to the organization.

Problems of SM 1. Alignment with organizational objectives. – SM is resources consuming and it has no clear quantifiable benefit for the organization. 2. Process issues – SM requires a number of additional activities not found in initial development. Impact analysis and regression tests on the software changes are crucial issues. 3. Technical issues – How to construct software that it is easy to comprehend is a major issue and the technology to do this is still not available.

Problems of SM • Domino Effect (a. k. a Ripple Effect) – When a change is made to the code, there may be substantial consequential changes, not only in the code itself, but within documentation, design, and test suites. • SM usually has a lower status compared with software development. • Management has trouble assessing a software product to determine how easy it is to change: – This leaves little incentive for initial development projects to construct software that is easy to evolve.

Organizational Aspect of SM • SM is much closer to a service and be related to quality – As opposed to initial software development which is product-oriented. • SM requires financial investment – SM is a prime candidate for funding reduction and even elimination. – SM needs to be expressed in terms of ROI. • The trend for SM to be outsourced • Work has been undertaken in applying predictive cost modeling to software maintenance – Based on the COCOMO techniques.

IEEE Draft Standard for SM [IEEE 94] • Represents many elements of good practice in SM: – – Accepting a stream of change requests & error reports Implementing the changes Testing Forming new software releases • IEEE draft model comprises four key stages: – Help desk: the problem is received, a preliminary analysis undertaken, and if the problem is sensible, it is accepted. – Analysis: a managerial and technical analysis of the problem is undertaken to investigate and cost alternative solutions. – Implementation: the chosen solution is implemented and tested. – Release: the change (along with others) is released to the customer.

IEEE Draft Standard for SM [IEEE 94] • Analysis phase – Feasibility analysis • Assesses the impact of the modification, investigates alternative solutions, assesses short and long term costs, and computes the benefit value of making the change. – Detailed analysis • Determines firm requirements of the modification, identifies the software involved, and requires a test strategy and an implementation plan to be produced. • The standard requires that: – All infected components shall be identified and brought in to the scope of the change; Unit test, integration test, user-oriented functional acceptance tests and regression test strategy be provided.

Technical Aspects of SM • 1. 2. 3. 4. Impact Analysis – helps to determine the cost of making a change Translate the problem into software terms to decide if it is viable or be rejected Determine the origin of the change and suggest solutions All solutions be investigated to determine they are applied to all software components. Make a decision on the best implementation route or to make no change.

Technical Aspects of SM • The Ripple Effect problem: – Ripple Effect propagation is a phenomenon by which changes made to a software component along the software life cycle have a tendency to be felt in other components. – Ripple effects cannot determine statically, and dynamic analysis must be used. • Impact Analysis is needed: – To ensure that the change has been correctly and consistently bounded; to identify all objects impacted by changes in the primary sector.

Technical Aspects of SM • Traceability – A degree to which a relationship can be established between two or more products of the development process, especially products having a predecessor successor or master subordinate relationship to one another. (IEEE 91) – Traceability provides semantic links for impact analysis. – Some types of traceability links are very hard to determine.

Characteristics of s/w evolution • Lehman’s 1 st law: A s/w product must change continually or become progressively less useful • Lehman’s 2 nd law: The structure of a program tends to degrade as more and more maintenance is carried out on it • Lehman’s 3 rd law: Over a program’s lifetime, its rate of development is approximately constant

Legacy Systems • Legacy problems – a legacy system is typically: – – – very old and large has been heavily modified based on the old technology documentation not be available none of the original members of the software development team may still be around – often support very large quantities of live data – the software is often at the core of the business and replacing it would be a huge expense. • Dealing with legacy system is very hard and there are some solutions.

Legacy Systems • Solutions for the legacy system: – Carry on as now, possibly subcontracting the maintenance – Replace software with a package – Re-implement from scratch – Discard software and discontinue – Freeze maintenance and phase in new system – Encapsulate the old system and call as a sever to the new – Reverse Engineer the legacy system and develop a new software suite. the most fruitful solution!

Legacy Systems – Reverse Engineering • Definition: Reversing Engineering – The process of analyzing a subject system to identify the system’s components and their inter-relationships, and to create representations of the system in another form or at higher levels of abstraction. ( by Chikofsky & Cross) • Two types of Reverse Engineering: 1. Redocumentation: the creation or revision of a semantically equivalent representation within the same relative abstraction layer; 2. Design Recovery: involves identifying meaningful higher level abstractions beyond those obtained directly by examining the system itself. • The main motivation is to provide help in program comprehension

Legacy Systems – Reverse Engineering • Two approaches: – Slicing: • A static analysis technique in which only those source code statements which can affect a nominated variable are displayed. – Hypertext form of documentation: • The maintainer attaches and manage notes to the source code for the “hot spots” (Foster and Munro)

Legacy Systems – Reverse Engineering • Reasons to use Reverse Engineering: – 26 decision criteria in three categories for reverse engineering: (by Bennett) • • Management criteria Quality criteria Technical criteria Two Important Points: – Many legacy systems represent years of accumulated experience, and this experience may now no longer be represented anywhere else. – It is not obvious that a legacy system does actually have a high level, coherent representation.

Reengineering • Reverse Engineering • Forward engineering

Maintenance cost • Boehm in 1981 proposed a maintenance cost estimation in terms of KLOC. Here the annual change traffic (ACT) ACT = (KLOCadded + KLOCdeleted) / KLOCtotal Maintenance cost = ACT x Development cost

Conclusion • Research Questions: – Promising trends and challenges – How do we change software quickly, reliably, and safely? – How easily can new software be changed? – Management and technical solutions are needed to address the problems of legacy systems. • There are 3 level approach to considering SM – in terms of the impact on – the organization – the process – technology

Conclusion • To well improve the best and average practice in the SM, we should adopt a standard maintenance process model, along with formal process assessment and improvement. • Coping with legacy system could be a headache, but there are still several practical techniques for addressing such system. • There are still major research issues of strategic industrial importance to be solved. • There are no “Silver Bullets” to solve the problems in SM, and the progressive and incremental improvement of practices is likely to be more successful.

Software reuse • In most engineering disciplines, systems are designed by composing existing components that have been used in other systems. • Software engineering has been more focused on original development but it is now recognised that to achieve better software, more quickly and at lower cost, we need to adopt a design process that is based on systematic software reuse.

Reuse-based software engineering • • • Application system reuse – The whole of an application system may be reused either by incorporating it without change into other systems (COTS reuse) or by developing application families. Component reuse – Components of an application from sub-systems to single objects may be reused. Object and function reuse – Software components that implement a single welldefined object or function may be reused.

Reuse benefits 1

Reuse benefits 2

Reuse problems 1

Reuse problems 2