University of Southern California Center for Systems and

University of Southern California Center for Systems and Software Engineering Reducing Estimation Uncertainty with Continuous Assessment Framework Pongtip Aroonvatanaporn Annual Research Review March 8, 2011

University of Southern California Center for Systems and Software Engineering Outline • • • Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending the framework 03/08/2011 © USC-CSSE 2

University of Southern California Center for Systems and Software Engineering The Cone of Uncertainty Also applies to project estimation accuracy 03/08/2011 Inexperienced teams Experienced teams © USC-CSSE 3

University of Southern California Center for Systems and Software Engineering Definition • Inexperience – Inexperienced in general – Experienced, but in a new domain – Anything that is new with little knowledge or experience 03/08/2011 © USC-CSSE 4

University of Southern California Center for Systems and Software Engineering The Problem • Experienced teams can produce better estimates – – Use “yesterday’s weather” Past projects of comparable size Past data of team’s productivity Knowledge of accumulated problems and solutions • Inexperienced teams do not have this luxury No tools or data that monitors project’s progression within the cone of uncertainty 03/08/2011 © USC-CSSE 5

University of Southern California Center for Systems and Software Engineering The Problem • Imprecise project scoping – Overestimate vs. underestimate • Manual assessments are tedious – Complex and discouraging • Project estimation not revisited – Insufficient data to perform predictions – Project’s uncertainties not adjusted • Limitations in software cost estimation – Models cannot fully compensate for lack of knowledge and understanding 03/08/2011 © USC-CSSE 6

University of Southern California Center for Systems and Software Engineering The Goal • Develop a framework to address mentioned issues • Help unprecedented projects track project progression • Reduce the uncertainties in estimation – Achieve eventual convergence of estimate and actual Must be quick and easy to use 03/08/2011 © USC-CSSE 7

University of Southern California Center for Systems and Software Engineering Benefits • Improve project planning and management – Resources and goals • Improved product quality control • Actual project progress tracking – Better understanding of project status – Actual progress reports 03/08/2011 © USC-CSSE 8

University of Southern California Center for Systems and Software Engineering Outline • • • Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending the framework 03/08/2011 © USC-CSSE 9

University of Southern California Center for Systems and Software Engineering Estimation Model Integration of the Unified Code Count tool and COCOMO II estimation model Adjusted with REVL 03/08/2011 © USC-CSSE 10

University of Southern California Center for Systems and Software Engineering Outline • • • Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending the framework 03/08/2011 © USC-CSSE 11

University of Southern California Center for Systems and Software Engineering Experiment Setup • Performed simulation on 2 projects from USC software engineering course • Project similarities – – – 03/08/2011 Real-client SAIV: 24 -weeks Architected agile process, 8 -member team Size, type, and complexities Product • E-services • Web content management system • JSP, My. SQL, Tomcat © USC-CSSE 12

University of Southern California Center for Systems and Software Engineering Obtaining Data • Source code files retrieved from Subversion server • Simulation of assessment done weekly • Both teams were closely involved – Provide estimation of module completion – Rationale 03/08/2011 © USC-CSSE 13

University of Southern California Center for Systems and Software Engineering Outline • • • Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending the framework 03/08/2011 © USC-CSSE 14

University of Southern California Center for Systems and Software Engineering Results Initial estimate ~50% ~18% Initial estimate Adjusted estimate Accumulated effort 03/08/2011 Accumulated effort © USC-CSSE 15

University of Southern California Center for Systems and Software Engineering Results • Project progress reaches 100% – Reflects reality • Estimation errors reduced to 0% 03/08/2011 © USC-CSSE 16

University of Southern California Center for Systems and Software Engineering Outline • • • Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending the framework 03/08/2011 © USC-CSSE 17

University of Southern California Center for Systems and Software Engineering Conclusion • Both teams demonstrated the same phenomenon – Gaps in estimation errors decrease – Representation of “cone of uncertainty” • Estimation framework reflects the reality of project’s progress • Assessment process was quick and simple – Requires few inputs – Little analysis needed • Assessment framework help inexperienced team improve project tracking and estimation 03/08/2011 © USC-CSSE 18

University of Southern California Center for Systems and Software Engineering Future Work • Tool development currently in progress • Determine the frequencies of assessments required – The sweet spot • Observe prediction accuracies – Experiment on projects of larger scale – Experiment on projects of different types • Use concept of value-based – Apply weights to calculation of each software module based on priorities and criticalities – How to adjust COCOMO parameters 03/08/2011 © USC-CSSE 19

University of Southern California Center for Systems and Software Engineering Outline • • • Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending to the Framework 03/08/2011 © USC-CSSE 20

University of Southern California Center for Systems and Software Engineering Problem and Motivation • No effective way to improve COCOMO estimation accuracy – How to adjust the parameters? – Often done by guessing, not accurate, not realistic • Need an approach to assess team performance – Good performance is essential in achieving successful project – Aid in stabilization and synchronization of the team 03/08/2011 © USC-CSSE 21

University of Southern California Center for Systems and Software Engineering Approach • Develop an assessment framework to aid COCOMO estimation • Determine the criteria for adjusting COCOMO parameters • Develop a survey-based approach to implement the framework – Questions correlate to COCOMO cost drivers and scale factors 03/08/2011 © USC-CSSE 22

University of Southern California Center for Systems and Software Engineering Survey Question Development • Team members assess and evaluate their own team – operational concept engineering, requirements gathering, business analysis, collaboration, etc. • Identify strengths, weaknesses, issues. – Things done right and things done wrong • Survey questions derived based on these assessment data Team’s assessment performed by each team member 03/08/2011 Strengths, weaknesses, issues, etc. © USC-CSSE Survey questions 23

University of Southern California Center for Systems and Software Engineering Question COCOMO • How to determine the effects? – Which COCOMO parameters are affected by the question? – How much? • Two approaches: – Finding correlation – Expert advice 03/08/2011 © USC-CSSE 24

University of Southern California Center for Systems and Software Engineering Correlation Survey Team Performance Issues filed by IV&Vers Correlation Survey questions Compare Team grades Estimated effort from COCOMO Client satisfaction Actual effort reported by team 03/08/2011 © USC-CSSE 25

University of Southern California Center for Systems and Software Engineering Expert Advice • Survey experts for opinions • Provide ratings for weights on effects – Relationship between question and COCOMO parameters – Some questions affect COCOMO drivers more than others • Ratings can be averaged to determine actual weights 03/08/2011 © USC-CSSE 26

University of Southern California Center for Systems and Software Engineering Expert Advice Subject Question Product Platform Personnel Project RELY DATA DOCU CPLX RUSE TIME STOR PVOL ACAP APEX PCAP PLEX LTEX PCON TOOL SCED SITE Do you prototype regularly to verify the validity of requirements among all the Prototyping stakeholders? 0. 5 NA 0. 7 0. 4 0. 3 NA NA NA 0. 8 0. 4 0. 2 NA 0. 2 0. 4 NA Do the prototypes address the critical usecases, GUI issues, and risks and evolve over Prototyping time as requirements change? 0. 9 NA 0. 2 0. 4 0. 7 NA NA 0. 2 0. 8 0. 4 0. 2 0. 4 NA NA NA 0. 5 Business process Do you understand the current and proposed business process or workflow? This includes the current system that is being utilized and what needs to be improved 0. 3 NA NA 0. 8 0. 4 NA NA NA 0. 7 Relationship between question/answer and COCOMO parameter it affects 03/08/2011 © USC-CSSE 27

University of Southern California Center for Systems and Software Engineering Benefit • Improved process to adjusting project estimates – Adjustments made based on data, not guesses • Faster convergence of “Cone of Uncertainty” – Less range of possible product outcomes – More accurate project tracking 03/08/2011 © USC-CSSE 28

University of Southern California Center for Systems and Software Engineering Tool Support • Utilize IBM Jazz Server technology – Focus on distributed environment support – Provides services for user management, team management, project management • Integration of Unified Code Count tool and COCOMO model – Automatic code counting – Effort calculation – COCOMO parameter adjustments 03/08/2011 © USC-CSSE 29

University of Southern California Center for Systems and Software Engineering Conclusion • A semi-automated framework to reduce estimation uncertainty throughout project life cycle • Integration of Code Count and COCOMO – Helps track project progress – Shows improved estimates • Survey-based assessment – Aid in team synchronization and stabilization – Automatically adjust COCOMO parameters to reflect reality of project 03/08/2011 © USC-CSSE 30

University of Southern California Center for Systems and Software Engineering References 1. 2. 3. 4. 5. 6. 7. Boehm, B. , Abts, C. , Brown, A. W. , Chulani, S. , Clark, B. K. , Horowitz, E. , Madachy, R. , Reifer, D. J. , and Steece, B. Software Cost Estimation with COCOMO II, Prentice-Hall, 2000. Cohn, M. Agile Estimating and Planning, Prentice-Hall, 2005 De. Marco, T. Controlling Software Projects: Management, Measurement, and Estimation, Yourdon Press, 1982 Fleming, Q. W. and Koppelman, J. M. Earned Value Project Management, 2 nd edition, Project Management Institute, 2000 Jorgensen, M. and Boehm, B. “Software Development Effort Estimation: Formal Models or Expert Judgment? ” IEEE Software, March-April 2009, pp. 14 -19 Nguyen, V. , Deeds-Rubin, S. , Tan, T. , and Boehm, B. "A SLOC Counting Standard, " COCOMO II Forum 2007 Stutzke, R. D. Estimating Software-Intensive Systems, Pearson Education, Inc, 2005. 03/08/2011 © USC-CSSE 38

University of Southern California Center for Systems and Software Engineering Related Work • Software estimation methods – Estimating Software-Intensive Systems [Stutzke, 2005] – Expert-judgement vs. parametric-model [Jorgensen, 2007] – Agile estimation [Cohn, 2005] • Software estimation uncertainty – PERT sizing methods [Nguyen, 2007] – Wideband Delphi estimate distributions [Boehm, 2000] • Software project tracking methods – Controlling Software Projects [De. Marco, 1982] – Earned Value Management [Fleming, 2000] 03/08/2011 © USC-CSSE 39