CIS 573 Computer Aided Verification Carl A Gunter

CIS 573 Computer Aided Verification Carl A. Gunter Fall 1998 Part 2

How is Software Built? Learn by building. l Knowledge captured in standards. l Many to choose from! l We will examine PSS-05 -0, the software engineering standard of the European Space Agency. l

References C. Mazza, J. Fairclough, B. Melton, D. de Pablo, A. Scheffer, and R. Stevens. Software Engineering Standards. Prentice Hall, 1994. l Available online from course web page. l C. Mazza, J. Fairclough, B. Melton, D. de Pablo, A. Scheffer, and R. Stevens. Software Engineering Guide. Prentice Hall, 1996. l

Classes of Software Custom l `Shrink Wrapped’ l Most critical software follows custom design standards; these will be our principle concern. l

Standards for the Software Industry Computer software standards are less advanced than computer hardware standards at the present time. l Successful development of the software industry depends on (and is currently driven by) progress in software standardization. l

Standard Defined A standard is a document, established by concensus and approved by a recognized body, that provides for common and repeated use, rules, guidelines, or characteristics for activities or their results aimed at an achievement of the optimum degree of order in a given context. Note---Standards should be based on the consolidated results of science, technology, and experience aimed at the promotion of optimum community benefits. ISO/IEC Guide 2

Types of Standards l l l De jure: standards created by a body empowered by concensus to formulate standards. De facto: products that have gained market acceptance and are recognized as the product against which others will be compared. Regulatory: standards that have been accepted by the government and have a legal force behind them and governmental ability to force acceptance and use.

Institutional Standards Internal standards: procedures internal to a company or government body. l Procurement standards: standards of an institution concerning its contractual relations with others. l

International Standards Organizations ISO--International Organization for Standardization l IEC---International Electrotechnical Commission l ITU---International Telecommunicati on Union l

Information Technology Committees l Each of these international organizations has a committee concerned with information technology. » Joint Technical Committee on Information Technology (JTC 1). Joint committee of ISO and IEC. » ITU Telecommunications (ITU-T, formerly CCITT)

Examples of Interface Standards ISO/IEC 8652: 1995, Information technology --- Programming languages -- Ada. l ISO/IEC 9794 -8, Information technology --- Open Systems Interconnection --The Directory: Authentication framework. Better known as ITU-T Rec. X. 509 l

Examples of Quality Control Standards ISO 9001: 1994, Quality Systems --Model for Quality Assurance in Design, Development, Production, Installation and Servicing. l ISO 9000 -3: 1991, Quality Management and Quality Assurance Standards --Part 3: Guidelines for the Application of ISO 9001 to the Development, Supply and Maintenance of Software. l

Other Standards l l ANSI/IEEE Std 754 -1985, Binary Floating. Point Arithmetic. IETF RFC 793, Internet Protocol. J. Postel, 1981. IEFT RFC 2068, Hypertext Transfer Protocol -HTTP/1. 1. R. Fielding, J. Gettys, J. Mogul, H. Frystyk, T. Berners-Lee. January 1997. ANEP: Active Network Encapsulation Protocol, S. Alexander, C. A. Gunter, A. D. Keromytis, G. Minden, D. Wetherall, B. Braden, A. W. Jackson.

People in Standards Preparation Secretariat: non-volunteer portion of the standards group responsible for administering the organization. l Working Groups: volunteers (typically industry and government representatives) engaged in standards development. l Example: the American National Standards Institute (ANSI) is the secretariat for JTC 1. l

ISO Organization l Example: ISO standards working group formal specification languages. » Technical Committee JTC 1 » Sub-Committee SC 22: Programming Languages, Their Environments and System Software Interfaces. » Working Group WG 19: Formal Specification Languages.

ANSI Organization l Example: ANSI standards technical committee for specification languages. » Accredited Standards Committee (ASC) NCITS (formerly X 3): Information Technology. » Technical Committee Class J: Languages. » Technical Committee J 21: Formal Description Techniques.

The Documentary Hypothesis Amid a wash of paper, a small number of documents become the critical pivots around which every project's management revolves. These are the manager's chief personal tools. Brooks

PSS-05 -0 l l l The European Space Agency is a consortium formed by European governments for the development of a commercial space program. PSS-05 -0 is their standard for software engineering. It was developed out of experience from the US Apollo missions. Our course reading is a generalized version of the standard from which space-specific material has been removed. Hereafter `ESA' refers to PSS-05 -0 and this

Use of ESA It is intended for software projects involving 5 to 50 man-years of effort. l It defines no specific contractual boundaries. l It is now used by a variety of companies and government organizations. l It is a project standard, not an organizational standard. l

Structure of ESA Product Standards: software to be defined, implemented, operated, and maintained. l Procedure Standards: procedures which are used to manage a software project. Appendices: summaries, tables, forms and checklists of mandatory practices. l

Product Standards User Requirements l Software Requirements l Architectural Design l Detailed Design and Production l Transfer l Operations and Maintenance l

Procedure Standards l People » Software Project Management » Quality Assurance l Software » Configuration Management » Verification and Validation

Classes of Standard Practices Mandatory: These must be followed, without exception, in all software projects. l Recommended: These are strongly recommended; if not followed, a justification must be given. l Guideline: No justification is required if not followed. l

Indicating Priorities l l These priorities are indicated by the verbs `shall', `should', and `may'. Examples from ESA: » ``The products of a software development project shall be delivered in a timely manner and be fit for their purpose. '’ » ``Procedures for handling new requirements should be established at the beginning of the project. '’ » ``Software requirements may require the construction of prototypes to clarify or verify them. '’ l Mandatory practices are numbered in an

Those Acronyms! The ESA standard is unreadable without knowing the meaning of several dozen acronyms. l Fortunately they are listed on a page near the end. l

Examples l l ``Up to the end of the TR phase, the formal review meeting is a UR/R, SR/R, AD/R or DD/R, depending on the document. In the OM phase the formal review is conducted by the SRB. Templates for RID, DCR and DSS forms are provided in Appendix E. '’ ``During the DD phase, the TR phase section of the SQAP shall be produced (SQAP/TR). The SQAP/DD shall cover in detail all the quality assurance activities to be carried out in the DD phase. '’

ESA Contents l Product Standards » User Requirements » Software Requirements » Architectural Design » Detailed Design and Production » Transfer l Process Standards » Software Project Management » Quality Assurance » Configuration Management » Verification and Validation

ESA Life Cycle A life cycle model structures project activities into phases and defines the activities in the phases. l A life cycle approach is a combination of the basic phases of the life cycle model. l SLC 03: All software projects shall have a life cycle approach which includes the basic phases shown in Figure 1. 1. l

Life Cycle Phases, 1 of 3 UR (User Requirements) The problem definition phase: determine the needs of the users by interviewing or surveying them. Ask them questions or obtain reactions to a prototype. l SR (Software Requirements) The analysis phase: describe what the software has to do, and not how to do it. l

Life Cycle Phases, 2 of 3 AD (Architectural Design) Define the structure of the software: determine components, their inputs and outputs, and the control flow between them. Select a programming language. l DD (Detailed Design and Production) Use the selected programming language to divide the components into modules and/or classes; produce implementations of components; do l

Life Cycle Phases, 3 of 3 TR (Transfer) Build deliverables and carry out provisional acceptance (validation) tests with end-user. l OM (Operations and Maintenance) Place software into practical use. Correct bugs during warranty period. l

The Waterfall Model The waterfall model (`waterfall approach' in ESA terminology) was the earliest software `process model'. l The choice of phases differs in various standards and organizations. l The model has been widely criticized as too monolithic because communication backwards between phases is often needed. l It seems best-suited to solving welll

The Waterfall Approach

Plan to Throw One Away For most projects, the first system built is barely usable: too slow, too big, too hard to use, or all three. Plan to throw one away; you will, anyhow. Brooks

Alternatives to the Waterfall Approach Incremental Approach: There is a staged delivery of components or functions. l Evolutionary Approach: Multiple releases are provided. This approach is well-suited to prototyping. l

Incremental Delivery Approach

Evolutionary Design Approach The “DEV” box is equivalent to the UR, SR, AD, DD, and TR phases.

Prototyping Prototypes usually implement high risk functional, performance or user interface requirements and l usually ignore quality, reliability, maintainability and safety requirements. l

Spiral Model Another model that emphasizes risk identification as a central aspect of its approach is based on a `non-linear' view of the software lifecycle. l The `spiral model' is due to Barry Boehm (1988). l It has a widely-recognized picture. l

ESA Contents l Product Standards » User Requirements » Software Requirements » Architectural Design » Detailed Design and Production » Transfer l Process Standards » Software Project Management » Quality Assurance » Configuration Management » Verification and Validation

Activities Capture of User Requirements: Involve criticism and experience with existing software. Use interviews and surveys. l Determination of Operational Environment: Account for the real world in which the software is to operate. l Specification of User Requirements: Write down the user requirements. l Technical review of User Requirements Specification. l

Questions about the ESA Standard Who are the `users' of the software? l Is it assumed that hardware requirements are already known? l

Classification of User Requirements Capabilities needed by users to solve a problem or achieve an objective. l Constraints placed by users on how the problem is to be solved or the objective achieved. l

Attributes of User Requirements Identifier l Need l Priority l Stability l Source l Clarity l Verifiability l

ESA Contents l Product Standards » User Requirements » Software Requirements » Architectural Design » Detailed Design and Production » Transfer l Process Standards » Software Project Management » Quality Assurance » Configuration Management » Verification and Validation

Activities Construction of Logical Model. l Specification of software requirements. l Technical review. l

Software Requirements Phase SR 02 The developer shall construct an implementation independent model of what is needed by the user. l SR 03 A recognized method for software requirements analysis shall be adopted an applied consistently in the SR phase. l SR 08 Each software requirement shall be verifiable. l

Classification, part 1 Functional l Performance l Interface l Operational l Resource l Verification l Acceptance testing l

Classification, part 2 Documentation l Security l Portability l Quality l Reliability l Maintainability l Safety l

Challenge of Specifying Software The difficulty of specifying what software is intended to do is a dirty little secret of the software industry. l This problem pervades experience with both technical and commercial aspects of software. l One would like to see something like an architect's blueprints for software, but such a successful formalism has not yet emerged. l

Pictures can be Misleading

Recognized Specification Languages The CS literature describes a wide variety of languages for specifing software requirements. Here a few that are ISO standards or DIS's: l Telecommunications languages: SDL, Lotos, Estelle. l `Model-based' languages: VDM, Z. l Lotos and Estelle are derived from process algebras. We will look at one such algebra. l

CSP l l Process Algebras are algebraic formalisms for describing concurrency, communication, nondeterminism, and resources. Communicating Sequential Processes (CSP) is one of the best-known process algebras. It was introduced by Tony Hoare. Another well-known process algebra, due to Robin Milner, is Calculus of Communicating Systems (CCS). There are now many process algebras under study. We will look at a specific problem

Dining Philosophers The problem was introduced by Edsgar Dijkstra; it epitomizes a collection of issues important to concurrent computation. l We will use the problem to illustrate formal specification (using CSP) and rigorous proof of safety from deadlock. l Let us assume that there are 5 philosophers and 5 forks. We describe the problem informally at first. l

Dining Philosophers, Described Informally Actions the philosophers can take: sit down, get left fork, get right fork, put down left fork, put down right fork, stand up. l They lead a simple life: thinking and eating. l They do not communicate directly with one another. l They interact only through their contention over forks! l

Dining Philosophers in Deadlock What if all of the philosophers pick up the forks on their left at the same time? l No right forks will be available so all 5 philosophers will wait. l The result is deadlock, a situation in which progress cannot be made because of interlocking resource contention. l

Rigor l How can we treat this problem rigorously so we will be able to characterize it precisely and recognize when it has been solved? » Code it in C. Too many extraneous details. » Describe it mathematically. But, what kind of mathematics? l We require a suitable applied mathematics of computing.

ESA Contents l Product Standards » User Requirements » Software Requirements » Architectural Design » Detailed Design and Production » Transfer l Process Standards » Software Project Management » Quality Assurance » Configuration Management » Verification and Validation

Activities Construction of the Physical Model. l Specification of the Architectural Design. l Selection of Programming Languages. l Technical review and walkthrough. l

Major AD Approaches Functional Decomposition l Jackson Development System l Object-Oriented Design l

Product Standards User Requirements l Software Requirements l Architectural Design l Detailed Design and Production l Transfer l Operations and Maintenance l

ESA Contents l Product Standards » User Requirements » Software Requirements » Architectural Design » Detailed Design and Production » Transfer l Process Standards » Software Project Management » Quality Assurance » Configuration Management » Verification and Validation

Activities Detailed Design l Production l » Coding » Integration » Testing l Technical review, inspection, walkthrough.

Testing l l Unit tests verify the design and implementation of all components from the lowest level defined in the DD up to the lowest level defined in the AD. Integration tests verify that major components interface correctly. System tests verify compliance with system objectives as stated in the SRD. Acceptance tests verify compliance with system objectives as stated in the URD.

Production of Test Documentation

System Tests l System tests should include such activities as: » passing data into the system, correctly processing and outputing it; » practice for acceptance tests; » stress tests; » preliminary estimation of reliability and maintainability; » verification of the Software User Manual.

ESA Contents l Product Standards » User Requirements » Software Requirements » Architectural Design » Detailed Design and Production » Transfer l Process Standards » Software Project Management » Quality Assurance » Configuration Management » Verification and Validation

Activities l Installation. l Acceptance testing. l Provisional acceptance.

Product Standards User Requirements l Software Requirements l Architectural Design l Detailed Design and Production l Transfer l Operations and Maintenance l

ESA Contents l Product Standards » User Requirements » Software Requirements » Architectural Design » Detailed Design and Production » Transfer l Process Standards » Software Project Management » Quality Assurance » Configuration Management » Verification and Validation

Activities l Final acceptance l Maintenance

ESA on Warranties The early part of the OM phase should include a warranty period in which the developer should retain responsibility for correcting errors. The end of the warranty period is marked by final acceptance. ESA at page 48.

ESA Contents l Product Standards » User Requirements » Software Requirements » Architectural Design » Detailed Design and Production » Transfer l Process Standards » Software Project Management » Quality Assurance » Configuration Management » Verification and Validation

ESA Contents l Product Standards » User Requirements » Software Requirements » Architectural Design » Detailed Design and Production » Transfer l Process Standards » Software Project Management » Quality Assurance » Configuration Management » Verification and Validation

Activities Organizing l Setting objectives and priorities l Managing risk l Choosing tools and methods (`technical management') l Planning, scheduling, and budgeting l Reporting l

Project Management Documents SPMP The Software Project Management Plan is the controlling document for the management of the project. l PHD The Project History Document explains deviations between estimates and actuals. l

ESA Potential Risks Quality and stability of user requirements l Level of definition and stability of external interfaces l Adequacy and availability of resources l Availability and quality of tools l Staff training and experience l Definition of responsibilities l Short time scales l

Topics to be Covered Software metrics will be discussed in two parts: cost and quality. l Material will be drawn from SEI maturity model literature. l Organization, staffing, and tracking will be discussed only indirectly. l Our topic in this section is cost estimation. l

Organizational Standards l l The ESA standard is a project standard. It describes the approach to be taken for an individual project without direct reference to other projects. Organizational standards include criteria relating projects. They may be applied to whole companies or to specific teams. The ISO 9000 series and the SEI process maturity levels are examples. The goal is to provide sellers a way to convince buyers of quality. (Or, to shift to a

ISO 9000 Computer Software

Process Maturity Levels The SEI framework is based on the idea that organizations have achieved a level of maturity that can be determined by an assessment. l The assessed level of maturity aids the organization in determining its priorities for improvement. l Maturity is founded on the concept of statistical control. l

Process Maturity Levels

The First Two Maturity Levels l l Initial Until the process is under statistical control, orderly progress in process improvement is not possible. While there are many degrees of statistical control, the first step is to achieve rudimentary predictability of schedules and costs. Repeatable The organization has achieved a stable process with a repeatable level of statistical control by initiating rigorous project management of commitments, costs, schedules, and changes.

Cost Estimation Effective cost estimation is essential for project success in a commercial context. l Cost estimation is difficult for software, but this doesn't make it less necessary. It is assigned as a managment responsibility in the ESA standard. l Cost estimation is generally attempted after the software specification phase. l SPM 06: An estimate of the total project cost shall be included in the SPMP/AD. l

The Mythical Man-Month l l l More programming projects have gone awry for lack of calendar time than for all other causes combined. Our estimating techniques, built around costaccounting, confuse effort and progress. The man-month is a fallacious and dangerous myth, for it implies that men and months are interchangeable. Partitioning a task among multiple people occasions extra communication effort--training and intercommunication.

Size Estimation Start with as detailed a product structure as is technically possible. l Precisely define the standard of measurement. l Estimate the size of each product element. l Sum these elements to produce a total estimate. l Apply appropriate contingencies. l

Work Breakdown Structures A WBS is a semi-formal way to break down a goal hierarchically into simpler work elements. l The breakdown generally has two parts: product structure and process structure. l Size estimates are made for each of the product elements. l

WBS for Compiler

Size Measures l There are two popular measures » LOC: Lines of Code based on source text. » Function Points: Weighted measure of functional requirements.

Counting Lines of Code For given code, what should we count? l Executable lines plus data definitions l Executable lines, data definitions, and comments l Physical lines on an input screen l Logical delimiters (eg. semicolons) l

Counting Lines of Code, Continued What body of code should be counted? l Only new lines l New and changed lines l New, changed, and reused lines l All delivered lines plus temporary scaffolding l All delivered lines, temporary scaffolding, and support code l

Problems with LOC l l There is no semantic content to code size. In many cases the metricis misleading and may do harm. Searching through a library to find reusable code takes time, but gratuitous use of library code may bloat target code size. Good coding involves finding abstractions that reduce code size. Benefits may lie in maintenance and quality. Tools may aid solving a problem quickly while increasing or decreasing code size.

Function Points l l l The function point method is based on experimental results from business applications. It is based on the counting items and assigning weights: Adjustment of 25% on the total is at the discretion of the estimator.

Wideband Delphi Estimation Technique l l l l Given a measure, a procedure is needed for obtaining an estimate. A group of experts is given the specifications. They meet to discuss the product. They anonymously complete estimation forms. The results are tabulated and returned. Estimates of others are kept anonymous. They meet again to discuss their results. Cycle continues until estimates converge adequately.

Delphi Estimation Form Sample

Contingency Factors

Size Estimation

Productivity Estimation Programmer productivity shows wide variation. l Multi-project data may offer a reasonable basis for estimation. l There is wide variation in productivity based on the nature of the product. l Productivity of staff are influenced by many factors; cause and effect may be difficult to separate. l

Relative Productivity

Relative Productivity, Cont.

Environment Productivity Figure

Sample Calculation Figure

Scheduling Differences Definitions of project phases l Software methods used l Product types being developed l Programmer skill levels l Degrees of urgency l

TRW Scheduling Data Figure

Martin Marietta Scheduling Data Figure

ESA Contents l Product Standards » User Requirements » Software Requirements » Architectural Design » Detailed Design and Production » Transfer l Process Standards » Software Project Management » Quality Assurance » Configuration Management » Verification and Validation

Software Quality Assurance (SQA) From ANSI/IEEE Std 730 -1989: SQA is “a planned and systematic pattern of all actions necessary to provide adequate confidence that the item or product conforms to established technical requirements”. l ESA quality assurance activity is the process of verifying that the ESA standard is being applied. l

Activities Management l Documentation l Standards, practices, conventions, and metrics l Reviews and audits l Testing activities l Problem reporting and corrective action l

Activities, Continued Tools, techniques and methods l Code and media control l Supplier control l Training l Risk management l

ESA Contents l Product Standards » User Requirements » Software Requirements » Architectural Design » Detailed Design and Production » Transfer l Process Standards » Software Project Management » Quality Assurance » Configuration Management » Verification and Validation

Definition and Standards l l “Configuration management is the art of identifying, organizing, and controlling modifications to the software being built by a programming team. ” (Babich) Configuration management is different from maintenance. MIL-STD-943 is the Do. D CM standard. The ESA CM standard is similar to the ANSI/IEEE Std 828 -1990.

ESA Requirement SCM 01 All software items, for example documentation, source code, object or relocatable code, executable code, files, tools, test software and data, shall be subjected to configuration management procedures.

Activities Identification: name and identify items. l Item storage: control libraries. l Change control: evaluate proposed changes and coordinate implementation of approved changes. l Status accounting: track and report configuration items status. l Release: transfer appropriate items to user. l

SCM 23 -25 l To ensure security and control of the software, at a minimum, the following software libraries shall be implemented for storing all the deliverable components (e. g. documentation, source and executable code, test files, command procedures): » SCM 23 Development (or Dynamic) library; » SCM 24 Master (or Controlled) library; » SCM 25 Static (or Archive) library.

Libraries

SCM 30 Software Problem Report (SPR). l Software Review Board (SRB) assigns the SPR for analysis. l Suggested changes are reported in a Software Change Request (SCR). l The SRB reviews the SCR and assigns responsibility for implementation if appropriate. l Software Modification Report (SMR). l

AST Tool Set Activities l l l Build: generate executables, libraries, documentation, and so on from source files. Test: test, debug, and improve the source. Install: place generated files in public directories. Package: collect source or gnerated files for use on other systems. Bootstrap: build and install on other systems, possibly in the absence of AST tools. Version Management: manage multiple versions of software.

ESA Contents l Product Standards » User Requirements » Software Requirements » Architectural Design » Detailed Design and Production » Transfer l Process Standards » Software Project Management » Quality Assurance » Configuration Management » Verification and Validation

Activities Reviews. l Tracing. l Testing. l Formal proof. l

Reviews l Review standards in ESA are very similar to those in IEEE/ANSI Std 10281928. This latter standard describes: » Technical reviews » Software inspection » Walkthroughs » Management reviews » Audits

Verification by Reading Product Project Testing Technical Review Simulation Software Inspection Formal Verification Management Review Walkthrough Audit From the IEEE Standard for Software Reviews and Audits

ESA Perspective Technical reviews are recommended at the UR, SR, AD, and DD phases. l Walkthroughs are recommended for the SR, AD, and DD phases. l Inspections are recommended for the SR, AD, and DD phases. l Audits are required as part of the SQA procedure. l

Technical Review Objective is to evaluate a software element and provide management with evidence that: l The element conforms to its specification. l The development (or maintenance) is being done according to plans, standards, and guidelines for the project. l Changes to the element are properly implemented and affect only those l

Software Inspection l l l Objective is to detect and identify software element defects. This is a peer examination that: Verifies that the software element(s) satisfies its specifications Verifies that the software element(s) conforms to applicable standards Identifies deviation from standards and specifications Collects software engineering data Does not examine stylistic issues

Software Inspection, Continued A software inspection is conducted by a group of 3 to 6 peers of the author(s) of the software. l Roles: moderator, reader, recorder, inspector(s), author. The moderator cannot be the author. l Reader leads the inspection team through the software. l

Walkthroughs The objective of a walkthrough is to evaluate a software element. l The scope of the evaluation is more general than that of a technical review. l Better implementation approaches are considered. l The walkthrough also includes exchanges of techniques and style variations, and education of the participants. l

Walkthroughs, Continued The author makes an overview presentation of the software. l This is discussed with his audience. l The author then `walks through' the software in detail. l

Traceability SVV 01 Forwards traceability requires that each input to a phase shall be traceable to an output of that phase. l SVV 02 Backwards traceability requires that each output of a phase shall be traceable to an input to that phase. l

Testing ``The amount of time spent in testing software is frequently underestimated. '' (ESA on page 75) l Testing classes l » Unit: DD test of DD » Integration: DD test of AD » System: DD test of SR » Acceptance: TR test of UR » Regression: Change at all phases

Testing Activities Planning the general approach and allocating resources. l Detailing the general approach for various kinds of tests in a test design. l Defining the inputs, predicted results and execution conditions in a test case specification. l Stating the sequence of actions to be carried out by test personnel in a test procedure. l

Formal Proof The challenge of proving software correct has led to unrealistic expectations, controversy, and productive bursts of research activity for more than two decades. l For this discussion, let us distinguish between: l » Rigorous proof to the standards of a mathematician, and » Formal proof using logic.

Rigor and Formality l Two controversial claims: » Formality provides automated support for the creation of rigorous arguments and enforcement for standards of rigor. » Rigorous specifications are not difficult to formalize.

Formal Specification and Proof Specification (formally or rigorously) of correctness criteria, and l Proof (formally or rigorously) that the criteria are met. l

Limits of Formal Specification A precise specification is not guaranteed to be correct from the UR perspective. l However, precision can aid validation. l