Defect Density Estimation through Verification and Validation Mark

Defect Density Estimation through Verification and Validation Mark Sherriff and Laurie Williams North Carolina State University HCSS ’ 06 April 19, 2006

Agenda • Background – Motivation and Hypothesis – Software Certificates and Software Certificate Management Systems (SCMS) • Dev. COP (Defect Estimation with V&V Certificates on Programming) – Research Goal and Methodology – Dev. COP Certificates – The Dev. COP SCMS Eclipse plug-in • Limitations • Current Status • Questions 2

Motivation • Software Reliability – Often not estimated until development is complete – Actual reliability not known until system is shipped to customers • Corrective action is more expensive later in the process • If defect density could be estimated in-process… – Steps could be taken to address issues early – More economical, could improve development effort 3

Hypothesis • Defect density estimation can be based upon the history of verification and validation techniques that have been performed on the project. Questions that need to be answered: • What is the best way to record V&V techniques? • How do I build a model that can predict defect density with V&V information? 4

Recording V&V Techniques • Software Certificates – A record of a verification and validation (V&V) practice employed by developers and used to support traceability between code and evidence of the V&V technique used • Software certificates could be any type of record: – Logs from test case runs – Reports from code inspections – Details on pair programming assignments • However, these are all different forms of V&V information and maintaining this information could be expensive. 5

Recording V&V Techniques • Software Certificate Management Systems – Provides an interface and infrastructure to automatically create, maintain, and analyze software certificates – Benefits: • Software maintenance • Analysis of V&V technique effectiveness • Reference in future projects • Current research: – OGI/PSU: Programatica, a SCMS for Haskell 6

Parametric Modeling • Method by which dependant variables are related to one or more independent variables with regard to previous data • In Software Engineering… – Purpose is to provide an estimated answer to a software development question earlier in the development lifecycle • Famous SE parametric models: COCOMO 81 and COCOMO II 7

Parametric Modeling • Software Testing Reliability Early Warning – Java and Haskell versions – Uses a suite of metrics gathered on static code to provide a reliability estimate – The model is calibrated to a particular organization using a regression equation – STREW is a good option because using operational profiles for a similar prediction can be expensive to create and maintain • If a reliability estimate can be created from testing and static metrics, could it be improved if we added other verification and validation information to the model? 8

Dev. COP • Research hypothesis: Defect density estimation can be based upon the history of V&V techniques that have been performed on the project. • Methodology: Build a parametric model using software certificate information and previous project defect data to create a prediction model for future projects. 9

Dev. COP Certificates • Records: – identifying information for the function it is associated with including its name, signature, class, and file location; – identifying information for the developer that created it; – the type of V&V technique used; – a hash of the function’s abstract syntax tree (AST); and – a significance weight. 10

Dev. COP Certificates • Types of Certificates – Manual • Includes all manual techniques, such as pair programming and code inspections – Automated Static Analysis • Includes all techniques that can be run automatically on uncompiled code – Dynamic • Includes all techiques performed automatically at run time, such as black box testing – Formal • Includes all formal methods, such as lambda calculus and proofs 11

Dev. COP SCMS Eclipse Plug-in • Currently supports manual V&V techniques and jcoverage certificates • Provides different methods for examining and managing certificate data • Demo 12

Building the Model 13

Limitations • Granularity of Certificates – Method level, not class or line of code • Composition of Certificates – Not much is known about how one V&V technique adds to another • Defect Severity – All defects are treated equally 14

Potential Side Effects • Retrospective Causal Analysis – Once certificates are recorded on a project and bugs are reported, developers can use certificate and defect information to evaluate the efficacy of their V&V practices. • Building certificate information in with compiled code base – If certificate information could travel with compiled code, it could be referenced at runtime so that other systems could evaluate whether it was to work with that system. 15

Thank you! • Questions? Queries? Quandries? Contact Information: Mark Sherriff mark. sherriff@ncsu. edu Dev. COP Project: http: //agile. csc. ncsu. edu/devcop/ 16