IV V Facility NASA Software IV V Facility Metrics Data

IV&V Facility NASA Software IV&V Facility Metrics Data Program Mike Chapman chapman@ivv. nasa. gov 304 -367 -8341 Pat Callis Patrick. E. Callis@nasa. gov 304 -367 -8309 Ken Mc. Gill Kenneth. G. Mc. Gill@nasa. gov 304 -367 -8300 June 5, 2003 1

IV&V Facility The Metrics Data Program Goal ØEstablish a centralized repository that provides consistent, fully-involved software product data across multiple domains: • To improve the effectiveness of software assurance • To improve the effectiveness of software research • To improve the ability of projects to predict software errors early in the lifecycle 2

Data, Data Everywhere: The Data Drought IV&V Facility There is very little fully-involved software product data… ØError data associated to smallest functional unit ØRequirements traced through the design to the smallest functional unit …available to those who need it. 3

IV&V Facility SARP PI Survey To what degree has a lack of software defect data from actual NASA projects impacted your SARP funded research? - Greatly (9). Lack of software defect data from actual NASA project has seriously hampered my research - Moderately (8). It would be nice to have more (or some) real project data, but I have found non-NASA data sources or other workarounds. - Not significantly (5). My project has not been impacted because I either have all the data I need or I don't need software defect data Note: The totals are in parentheses. 4

The Lack of Data from a Project Perspective IV&V Facility ØError data associated to smallest functional unit • There is little value in this activity for the project ØRequirements traced through the design to the smallest functional unit • Projects only need to trace requirements to the executable (CSC level or above) ØFully-involved data made available to the research community • Vulnerability of the program • Proprietary issues 5

The Quest for the Holy Grail IV&V Facility Regardless of the development model the following is needed: Requirements Traceability Design Code (Many to one issue. ) Problem Reports dev, test, user, maintenance Association to the smallest functional unit. (smallest functional unit) 6

Recruitment of Project Data IV&V Facility ØExisting repository data Identifier Language LOC Domain Error Data JM-1 C 315 K Real-Time 8 years KC C++ ~750 K Data System 5 years CM-1 C 20 K Instrument 2 years CM-4 C ~40 K Flight N/A ØError Data Enhancement Program 7

Error Data Enhancement Program IV&V Facility The goal of the enhancement effort is to successfully recruit projects to work with the MDP to provide fullyinvolved software product data – the Holy Grail. The MDP team will provide: ØRequirements analysis and traceability support ØConfiguration Management Support (Error Tracking) ØMetrics generation and analysis ØDatabase and web support ØMachine learning analysis 8

MDP Repository IV&V Facility Other Project Data Project Error Data MDP Web Site Clear. Quest Interface Queries Sanitized Data Firewall Stored Metrics and Error Data Clear. Quest Sanitized Data Project Security Project 1…………………. Project n Project Metrics and Error Data 9

Benefits IV&V Facility ØAgency benefits: • The improved ability to predict error early in the lifecycle • The improved ability to assess the quality of the software ØThe research community benefits: • Availability of quality error and metric data • Availability of a support team for data needs ØParticipating projects benefit: • Additional metrics analysis • Additional error analysis • Problem tracking tool • Other support such as requirements traceability 10

Where to Find More Information IV&V Facility mdp. ivv. nasa. gov 11

Site Metrics IV&V Facility Web Site activity for 3 months: Ø 596 hits Ø 46 accounts Ø 146 logins Ø 85 downloads of data 12

Special Data Requests IV&V Facility ØFive time stamps of KC-2 data ØSanitized activity fields of JM-1 data ØError Reports from CSCI of KC Note: Five papers have been written from the repository data so far. 13

Current Data Request IV&V Facility ØCM-1 requirements data and associated errors - JPL ØCM-1 data including metrics – Dolores Wallace SATC ØKC semantic metrics generation – Letha Etzkorn UA – Huntsville ØJM-1 time stamp data (five sets) – Tim Menzies WVU 14

A Study IV&V Facility Metric Combination % of Errors % of Code Probability of Failure (%) l >= 0. 047015525 50% 34% 71% loc >= 9. 197 AND l >= 0. 066 30% 20% 37% Metric Combination Probability of Module Error Probability of Failure (%) % of Severity One Errors v(g)>45. 9 & e>=1340053 80% 0% 50% (8/16) v(g)>45. 9 & loc>= 150 73% 0% 33% (23/69) • v(G) – cyclomatic complexity – independent linear paths • ev(G) – essential complexity – unstructured constructs • e – programming effort – mental effort • l – (2/u 1)*(u 2/N 2)-program level – the level at which a program can be understood 15

IV&V Facility Background Slides on Metrics 16

Mc. Cabe Metrics IV&V Facility Halstead: programmers read code. Too many “words” error Mccabe: paths between “words” Twisted paths error v(G): cyclomatic complexity = # path(ish)s = edges-nodes+1 m = # one entry/one exit sub-graphs ev(G): essential complexity = v(G) – m iv(G): design complexity (reflects complexity of calls to other modules) 17

Halstead Metrics IV&V Facility µ = µ 1+ µ 2 N = length = N 1+N 2 V = volume = N*log 2(µ) V’ = (2+ µ 2’)*log 2(2+ µ 2’) L D L’ E T = level = V’/V = difficulty = 1/L = 1/D = effort = V/L’ = time = E/18 e. g. 2+2+3 N 1 = 3 N 2 = 2 µ 1 = 2 µ 2 = 2 µ 1’ = 2(ish) µ 2’ = #input parameters µ 1 N 2 µ 2 N 1 Could be found via simple tokenizers 18

Operators IV&V Facility ! % & * + , - . / ; < > ? ^ | ~ = >= <= == != >> << += -= *= /= %= &= ^= |= >>= <<= && || ++ -- -> return size of enum struct delete continue new break union if (…) … else switch (……) default: case : goto do … while (…) …do for (… ; …; …) this-> [ ] { } ( ) in any other cases not covered … ‘? ’ … ‘ ‘: … 19

Operands IV&V Facility ØVariables and identifiers ØConstants (numeric literal/string) ØFunction names when used during calls 20

Error Metrics IV&V Facility Module LOC ER ED* 6 mth 1 yr 2 yr CSC 1/mod 1 59 6 101. 7 3 4 5 CSC 2/mod 1 190 6 31. 58 2 4 5 CSC 1/mod 9 40 5 125 1 2 3 CSC 7/mod 27 47 3 63. 83 CSC 4/mod 10 287 2 6. 9 1 2 2 CSC 6/mod 51 482 10 20. 8 1 4 *ED = ER/KLOC 21

OO Metrics IV&V Facility ØNumber of Children (NOC) – number of sub-classes ØDepth – level of class in the class hierarchy ØResponse for Class (RFC) – number of local methods plus the number of methods called by local methods (>100) ØWeighted Methods per Class (WMC) – sum of the complexities of the methods (>100) ØCoupling Between Object Classes (CBO) – dependency on classes outside the class hierarchy (>5) ØLack of Cohesion of Methods (LOCM) – the use of local instance variable by local methods 22

ARM Metrics IV&V Facility ØWeak Phrases (adequate, be able to) – clauses that cause uncertainty ØIncomplete (TBD, TBR) – Words and phrases that indicate the spec may not be fully developed ØOptions (can, optionally) – Words that give the developer latitude ØImperatives (shall, may, will, should) – Words that are explicit ØContinuances (below, as follows, and) Extensive use of continuances can indicate complex requirements. ØDirectives (for example, figure, table) – examples or illustrations ØLines of text (size) ØDocument structure (levels) 23

IV&V Facility Program Background Slides 24

Problem Report Fields IV&V Facility Error Identifier: (Alpha-numeric) Headline: (text- short description) Submitted-on: (Date yymmdd) Severity: (1 thru 5) Status: (NVTCDARMB) System Mode: (operations versus test) Request type: (Problem or enhancement) Problem-type: (requirements, design, source code, COTS, documentation, hardware, etc) Problem Mode: (PR or Action Item) Assigned-to: CCB Approval: (Date) Impacts-csci: (high level design element) Impacts-csc: CSC Impacts-class/file: class/file Impacts-method/function/module: Method/function/module Impacts – Requirement Impacts – design element Resolution: source code, COTS, documentation, not a bug, unreproducible Problem: (text) Analysis: (text) Resolution: (text) Closed-on: (Date) 25

IV&V Facility Project Non-Specific - Universal Error Identifier: (Alpha-numeric)* Submitted-on: (yymmdd) Severity: (1 thru 5) Status: (NVTCDARMB) Mode: (operations versus test) Request type: (Problem or enhancement) Problem-type: (requirements, design, source code, COTS, documentation, etc) Impacts-csci: (high level software design element)* Documents: (What documents are affected? ) CCB Approval: (Date) Resolution: source code, COTS, documentation, not a bug, unreproducible Verified-on: (Date) Closed-on: (Date) * May need sanitized. 26

Project Non-specific - Expanded IV&V Facility Impact data: Impacts-csc: CSC* Impacts-class/file: class/file* Impacts-method/function/module: Method/function/module* Recommend-change: source code Process data: How-found: (e. g. , Acceptance Test) When-found: (e. g. , Acceptance Testing) Analysis-due: 020322 Assigned-Eval-on: 020322 Assigned-Implement-on: 020323 Implement-due: 020325 Fix-date: 020325 Fixed-on: (date) In-Test-on: 020325 Test-name: (Numeric id)* Test-system: (hardware) Verify-date: Merge-build-id: Deferred-on: Build-name: (Alpha-numeric identifier) Patch-rel-name: Patch-rel-date: (date) Automated history entries: (Alpha-numeric) Costing Data: Cost: (high, medium, low) Est-fix-hours: Est-fix-date: (date) Est-Num-SLOC: Rev-fix-time: Rev-fix-date: (date) SLOC-Type: SLOC-count: Fix-hours: Miscellaneous Data: Operating-system: Priority: (High, Medium, Low) Enhancement: (Y or N) Workaround: Y Iteration: (Version bug identified in) * May need sanitized 27

Project Specific - Universal IV&V Facility Headline: (text – short problem description) Problem: (text – expanded problem description) Analysis: (text) Resolution: (text) Closure: (text) Submitter-id: Assigned-to: Closer-id: Question: Can Project Specific data be sanitized? 28

Project Specific - Universal IV&V Facility Headline: (text – short problem description) Problem: (text – expanded problem description) Analysis: (text) Resolution: (text) Closure: (text) Submitter-id: Assigned-to: Closer-id: Question: Can Project Specific data be sanitized? 29