Identifying and Using a Project s Key Subprocess Metrics

Identifying and Using a Project’s Key Subprocess Metrics Jeff S. Holmes BTS Fort Worth

Everyone Loves a Hero

Heroes Come Through! Firemen – Saves Baby in Burning House l Policemen – Catches Bad Guy l Athlete – Hits Game Winning Homerun l

Software Engineering Heroes! All night coding! l Debugging over the weekend! l l THIS SHOULD NOT BE THE NORM!

How To Minimize “Fire Drills”? l Preventative ¡ Proper Actions wiring can prevent fires. ¡ Locking your can prevent theft. ¡ Don’t get behind in the ball game. ¡ Understand project status earlier.

Metrics, metrics, everywhere…

But What is Really Important? n Customer Wants – – – n n Functionality Zero Defects On Time What software metrics map to these? How can we optimize these outputs?

BTS Fort Worth Approach n n n Selected DMAIC to Improve Process Identified Project with Two Years of Data Performed Statistical Analysis Conducted Pilot Currently in “Control” Phase

DMAIC : Define Identify “what is important” n BTS FW Monitors n ¨ Productivity (KLOC/Hour)* ¨ Quality (Post Release Defects/KLOC) ¨ Schedule Adherence n These are BTS FW “Big X’s”

DMAIC : Measure Requirements Resources n Software Development Life Cycle The “Simple” View Perfect Software!

DMAIC : More Details Requirements Resources Requirements Perfect Requirements Resources Design Perfect Models Resources Perfect Code Resources Perfect Design Perfect Models Perfect Code Test Perfect Software!

DMAIC : Subprocess Identification n BTS FW Identified Following Subprocesses ¨ Planning Phase ¨ Requirements Phase ¨ Design Phase ¨ Code Phase ¨ Test Phase ¨ Release Phase ¨ Code Inspections

DMAIC : Measured Data n BTS FW Uses Following Data: ¨ # Requirements ¨ # Developers on the project (Resources) ¨ % Time in Planning ¨ % Time in Requirements ¨ % Time in Design ¨ % Time in Code ¨ % Time in Test ¨ % Time in Release ¨ Requirements Churn ¨ Actual Size (KLOC) ¨ Avg Defect Detection Rate (DDR) in Code Inspection

DMAIC : Data Sources n BTS FW Data ¨ DOORs for Requirements ¨ Project Plans # Developers and ¨ Clear. Case for Code Size ¨ Inspection Database for DDR % Times

DMAIC : BTS Subprocess Metrics Resources Time in Phase Planning Req Count Time in Phase Requirements Req Churn Time in Phase Design Req Churn KLOC Time in Phase Code DDR Code Inspections Test Release Time in Phase

DMAIC : BTS Subprocess Metrics Resources Time in Phase Planning Req Count Time in Phase Requirements Productivity (KLOC/Hour) ? Req Churn Time in Phase Design Quality (PR Defects/KLOC) ? ? Schedule Adherence Req Churn KLOC Time in Phase Code ? ? DDR Code Inspections Test Release Time in Phase

DMAIC : BTS FW Analysis l Project Data l Used Step-wise Linear Regression to • 8 Releases since 2002 • Similar work • “Stable” team • Identify statistically significant factors • Develop prediction formulas for “Big X’s”

CAUTION !! The following slides contain statistics that could be hazardous to your health! Persons who suffer from narcolepsy or “statisticitis” should consider leaving the room.

DMAIC : Stepwise Linear Regression l l Describes the relationship between one 'predicted' variable and 'predictor' variables Goal – get the simplest equation with the best predictive power for • Productivity – KLOC/Hour • Quality – Post Release Defect/KLOC

DMAIC : Standard Least Squares Model accounts for 99. 82% of variance.

DMAIC : Significant Effects < 0. 05 is significant l Most significant effects • % Time in Req • Average DDR • Interaction between % Time in Code and Average DDR

DMAIC : Standard Least Squares Model accounts for 90. 62% of variance.

DMAIC : Significant Effects < 0. 05 is significant l Most significant effects • % Time in Requirements • Interaction between % Time in Requirements and Requirements Churn

DMAIC : Statistically Significant l l l # Requirements # Developers on the project % Time in Planning % Time in Requirements % Time in Design % Time in Code % Time in Test % Time in Release Requirements Churn Actual Size (KLOC) Avg Defect Detection Rate (DDR) in Code Inspection

DMAIC : Key Subprocess Metrics Resources Time in Phase Planning Req Count Time in Phase Requirements Productivity (KLOC/Hour) ? Req Churn Time in Phase Design Quality (PR Defects/KLOC) ? Schedule Adherence ? Req Churn KLOC Time in Phase Code ? ? DDR Code Inspections Test Release Time in Phase

DMAIC : Variation Analysis l l Prediction formulas generated to identify: • Good and bad variance • Most significant factors NOTE: Prediction formula uses all effects from the models, not just the significant ones. Formula added in Percent Planning.

DMAIC : Factor Weighting Metric 10% Deviation in Each Subprocess Area % Plan 10% 0 0 % Req 0 10% 0 0 0 % Code 0 0 10% 0 0 Req Churn 0 0 0 10% 0 Avg DDR 0 0 10% Productivity 0 3% 31% 0 9% Quality 14% 40% 0 6% 0 Cycle Time 0 3% 24% 0 9%

DMAIC : Factors’ Effects Metric LOC/Hr Effect PR Defect/KLOC Effect % Planning Time N/A More time = Less defects/KLOC % Req Time More Time = Less KLOC/Hr Less time = More defects/KLOC % Code Time More Time = Less KLOC/Hr N/A Code Insp. DDR Higher DDR = Less KLOC/Hr N/A 2 SR Req. Churn N/A Less Churn = Less defects/KLOC

DMAIC : BTS FW Limits Metric Average Std Dev Lower Limit Upper Limit % Planning Time 18 % 8% 10 % 26 % % Req Time 15 % 8% 7% 23 % % Code Time 21 % 12 % 11 % 33 % Code Insp. DDR 1. 89 . 5 1. 39 2 SR Req. Churn 1. 76 1. 89 0 3. 65 Green limit indicates direction a metric can deviate from the average and have desired results. Red indicates direction of undesired results.

Subprocess Metrics Notes l l Initial Data Left Much to be “Desired” Despite Poor Data, the Analysis Identified: • Which Metrics and Processes Are Significant • Prediction Formulas Based on Project’s Data • Insight into Factors’ Effects • Limits for Monitoring the Factors

DMAIC : Pilot Confirmation l l Used prediction formulas on other projects Compared project actuals vs. predicted. Used historical data from 5 projects. Unable to compare predicted quality versus actual. (Predicted LOC/Hr) • l These projects have not been in field long enough for CRUD to stabilize. Interesting results found on predicted LOC/Hr.

DMAIC : Predicted vs. Actual LOC/Hr • Projects A, B, and C projects had huge deviations. • Projects D and E were within 20%.

DMAIC : Improve Performance? l l l So what? How do you use this information? Does Project Management have confidence in this analysis?

DMAIC : Applying Analysis l l More emphasis on statistically significant activities Resulting in l Increased Productivity l l l On-Time Delivery Desired Functionality Delivered Improved Quality

DMAIC : Agile Processes l BTS FW Adopted Agile Practices l Iterative Development l l l Pair Programming l l Prioritizes Requirements Negates Requirements Churn Optimizes Coding and Inspection Time Minimal Documentation l Moves effort from non-statistical activities.

DMAIC : Agile Pilot Results l Productivity l l l Inspection Defect Detection Rate l l l 0. 00291 KLOC/Hr 20% improvement from 0. 002399 1. 18 Defects/Hr Detected 48% improvement from 0. 8 Quality l 0 Post Release Defects!

DMAIC : Agile Pilot Results l Customer Wants l l l Functionality – All functionality delivered Zero Defects – No customer found defects On Time – Product delivered 6 months early!

DMAIC : Agile Monitoring l l l Monitor Iterations Not Phases Refactoring Subprocess Monitoring Two Agile projects in-work now

DMAIC : Agile Monitoring l l l LOC per week Defects caught per week by inspection Defects caught per week by test Time spent per week Ratio of new work to correction work.

DMAIC : Agile Metrics

Summary n n Save your “heroes” for real crises. Understand subprocesses Monitor subprocesses Seek to optimize key subprocesses

Recommendations n n n Examine current project data, it could prove to be very valuable! Improve data capture on important data. Use the data as a guideline, but experience can never be discounted.

THANK YOU! Jeff S. Holmes Principal Staff Software Engineer Motorola Six Sigma Black Belt Fort Worth BTS Development Team Fort Worth, TX 817 -245 -7053 J. Holmes@Motorola. com