Technical Reviews A Presentation By Manuel Richey for

Technical Reviews A Presentation By Manuel Richey for EECS 814 November 17 th, 2005

Overview Of This Presentation n n There are numerous types of technical reviews, and numerous methods for implementing reviews. This presentation will briefly describe several types and techniques of reviews It will go into greater detail on one type and method for technical reviews Finally, we will perform a technical review in class.

Purpose of Reviews n n n Oversight To create buy-in or a sense of ownership To disseminate or transfer knowledge To improve the quality of a document or work product Other purposes?

Types Of Reviews n n n Customer Reviews Management Reviews Peer Reviews Personal Reviews Other Types of Reviews?

Customer Reviews n n Often specified as milestones in a contract (e. g. preliminary and critical design reviews). Formal documentation is submitted prior to the review. A long walkthrough/review is conducted with the customer. Often leads up to customer sign-off of a milestone.

Management Reviews n n n Each company has their own format for these. Typically a power point presentation to Management or technical leadership followed by Q&A session The usual objective is to either approve a project or monitor project status Inputs are typically project plans and schedules or status reports. Management team makes decisions and charts or approves a course of action to ensure progress, and properly allocate resources.

Technical Peer Review n Structured encounter in which a group of technical personnel analyze a work product with the following primary objectives: improve the original quality of the work product n improve the quality of the review process n

Why Hold Technical Peer Reviews? n n n Software Development is a very error-prone process. Early detection of defects is cost effective, and peer reviews find errors early. Peer reviews find many of the same errors as testing, but earlier and with less effort. They serve to educate the participants and provide training They raise a team’s core competence by setting standards of excellence

A Generic Review Process n Plan Review: assess readiness of work product, assign team members, send out announcement & review package. n Detect Defects: each reviewer looks for defects in work product n n n Collect Defects: in a meeting or via email, etc. Correct Defects: author corrects work product Follow Up: verify rework, document review

Methods For Technical Reviews n n n n Ad-Hoc Personal Walkthrough Fagan Style Inspection Asynchronous Review N-Fold Inspection Many Other Techniques

Ad-Hoc Reviews n n n Provide no process or instructions on how to detect defects Defect detection depends on inspector’s skill and experience Still valuable for: n n Enforcing standards Project status evaluations Improved communications Training and knowledge dissemination

Personal Reviews n n n An integral part of the “Personal Software Process” by Watts Humphrey. Involves only the author of a work. Employs checklists and metrics (if following PSP). For PSP, code review occurs before first compile. May be performed by the author prior to a Formal technical review.

Walkthroughs n n n A meeting in which the author presents a work product in a sequential manner and clarifies the product as necessary. No preparation by meeting attendees. May be held as part of another type of review. n As an example, a walkthrough may be held for a work product prior to distributing the review packet to the reviewers for a Fagan style software inspection.

Fagan Style Software Inspections n n A method of technical review that involves a meeting based process and specific roles. Process: Reviewers detect defects separately, but hold a meeting to collect, classify and discuss defects. Defined roles: Moderator, Author, Presenter, Recorder, etc. We will examine this technique in detail.

Asynchronous Inspections n n n No Meeting, so review can be distributed in space and time Doesn’t involve author Process is as follows: n n n Moderator sends out material via email Individual reviewers create list of defects Defect lists are circulated to all inspectors and discussed via email Individual reviewers update defect list and send to Moderator compiles final defect list, sends it to author and follows up – eliminates group approval

N-Fold Inspections n n Several independent teams inspect the same work product using traditional inspection methods. Of course, many teams find overlapping defects, but unique defects are typically found by each team. The Moderator collects faults from the independent teams and composes the final defect list. This is an expensive process used when high reliability is desired.

Fagan Inspection Process A six step review process Author submits work products for review n Moderator assesses the product’s readiness, assigns review team, and announces the review n Reviewers prepare for review n Reviewers hold review meeting n Author corrects defects n Moderator verifies rework and closes review n

Standards and Checklists n Standards: n n Rules for requirements/design/coding that all work products must adhere to Typically either project or company specific Improve software maintenance and quality Checklists: n n A list of questions for the inspectors to answer while reading the document. Should be less than a page long Should be derived from most common past defects Should be periodically updated

Inspection Package n n Work-Product to be inspected (line numbered if possible) Supporting documentation (requirements or work-product from which the workproduct to be inspected was derived) Checklists and Standards are available Inspection meeting notice (Often sent by email)

Fagan Inspection Roles n n n Producer/Author – creates the product being reviewed answers questions. Moderator – prepares review package, moderates the meeting, verifies rework. Presenter/Reader – presents product during meeting Recorder/Scribe – records defects during meeting Reviewer – everyone is a reviewer, but you may have reviewers that don’t have another role.

Reviewer’s Responsibilities n n n Responsible for objectively inspecting the work-product Responsible for tracking the amount of time spent preparing for the inspection meeting Actively participate in inspection meeting by providing defects found during examination of the work product

Producer’s Responsibilities n n n Provides required reference material for the inspection Finds defects Provides clarification Answers questions Modifies the inspected work-product to correct defects

Moderators Responsibilities n n n n Ensures entry criteria are met Distributes the inspection package to review team Ensures that all reviewers are prepared prior to the inspection meeting Facilitates the inspection meeting Also participates in review as a reviewer Assures that all items logged at the meeting are dispositioned Collects the data and completes the inspection record

Presenter’s Responsibilities n n n Presents the product in logical fashion paraphrased at a suitable rate Typical review rates are 100 -200 LOC/hour (or 10 -12 pages/hour for documents) Can vary significantly due to following factors n n language comments and readability type of software structure of software

Recorder’s Responsibilities n n Completes the defect log Defects should be classified based on team consensus by: Severity (Major, Minor) n type n class n n n Should use techniques to minimize defect logging time Not a secretary

Review Meeting n n n Reviewers fill out and sign inspection form indicating time spent reviewing product. Reviewers collectively decide if they are ready for the review to be held. Presenter progresses through review product eliciting defects as he progresses. Recorder records defects on defect form. Reviewers collectively disposition review as “Accept As Is”, “Accept with Corrections” or “Rereview”.

Review Comprehension Methods n Front to Back n n n Bottom Up n n n Start at the lowest level routines and work up Used when code is new to inspector Top Down n Start at the front of a document or top of a code module, and proceed to the end in sequential order Use with documents, or if already familiar with the code design Start at the main SW entry points and review those, then review the routines they call Used when inspector is familiar with code Integrated n Use both Top-Down and Bottom-up approaches as appropriate.

What Makes A Good Reviewer n n n A good reviewer is thorough Is prepared (most peer review postponements are due to lack of team preparation) Reviews the product, not the producer Raises issues, doesn’t resolve them Doesn’t give the author the benefit of the doubt

What Makes A Good Moderator n n n Encourages individuals to prepare and participate Controls Meeting (starts on time, keeps focus on agenda, eliminates problem solving, etc. ) Nurtures inexperienced reviewers Is sensitive to Author Feels ownership in quality of product

How To Select Reviewers n n Participants are typically selected by the Moderator. Important criteria for selecting participants: ability to detect defects (expertise) n knowledge of source documents n need to understand work-product (recipients of work-product) n motivation, and other personal qualities n

Review Metrics n n Help measure the effectiveness of reviews Aid in continuous process improvement Provide feedback to management Typical metrics are: n n n Average preparation effort per unit of material (typically LOC, KLOC or pages) Average examination effort per unit of material Average explanation rate per unit of material Average number of defects and major defects found per unit of material Average hours per defect and per major defect Percentage of re-inspections

Industry Experience n Aetna Insurance Company: n n Bell-Northern Research: n n Est. inspection savings (1993): $21, 454, 000 IBM n n Inspection cost: 1 hour per defect. Testing cost: 2 -4 hours per defect. Post-release cost: 33 hours per defect. Hewlett-Packard n n FTR found 82% of errors, 25% cost reduction. Reported 83% defect detection through inspections AT&T n Reported 92% defect detection through inspections

Personal Case Study The KDR 510 A VHF aviation radio and modem. n A Real-time, embedded, safety critical, DSP system. n Won the editors choice award from Flying Magazine. n Formal peer reviews were main QA activity. n

Quality Data for the KDR 510 n KDR 510 reviews detected many errors. n n n 72% of SW requirements defects 90. 7% of SW design defects 90. 6% of SW coding defects Total review time was approx 5% of total project time. Only 23% of total project time was spent in integration & test. Only one error escaped into the field.

The Real Reasons For Holding Reviews n Reviews improve schedule performance Reviews Req R Design R No Reviews Req n Design Code R Code Test Reviews reduce rework. n Rework accounts for 44% of dev. cost!

Tools For Technical Reviews n Various tools for different inspection methods. n n n ICICLE – for inspection of C & C++ programs Scrutiny & Inspe. Q for specific inspection processes ASSIST –supports generic inspection process For larger list, see: http: //www 2. ics. hawaii. edu/~johnson/FTR/ Home grown tools n n Typically built with Access Database. Reviewer enters defects offline into database. Eliminates recorder and reader roles. Gives author time to consider defects before meeting

Some References n n n A great website for Formal Technical Reviews: http: //www 2. ics. hawaii. edu/~johnson/FTR/ A Discipline for Software Engineering, Watts S. Humphrey, Addison. Wesley, January, 1995. M. E. Fagan, Design and code inspections to reduce errors in program development, IBM Systems Journal, Vol 15, No 3, 1976, 182 -211 G. M. Schneider, J. Martin, W. T. TSAI, An Experimental Study of Fault Detection In User Requirements Documents, ACM Transactions On SW Engineering & Methodology, Vol 2, No 2, April 1992, 188 -204. A. Porter, H. Siy, C. Toman, and L. Votta, An Experiment to Assess the Cost-Benefits of Code Inspections in Large Scale Software Development, IEEE Transactions On Software Engineering, VOL. 23, NO. 6, JUNE 1997, 329 -346

Questions?

Review Workshop n n n Objective: Allow everyone to take a role in a Fagan style code review Combine results to create an N-Fold inspection. Break into teams of 4. Handouts: Source Code Files, Supplementary Materiel, Review Forms. Schedule: 20 minutes to prepare for review, 20 minutes for review, 10 minute break for everyone but moderators. 5 minutes to summarize results.

Discussion on Review Workshop n n n Results from N-Fold inspection. What did you learn from this code review? Was it effective? How long would it have taken to detect some of these defects by testing? Other comments or conclusions?

Conclusion n n Reviews compliment software testing. Reviews are cost effective techniques for improving the quality of a developed product. They pay for themselves. Reviews improve the maintainability of a developed product. One size doesn’t fit all - An organization’s size, culture and industry should be considered in deciding on the methods to use for reviews.