BSPIN Agile Practice Benchmarking Report 2012 BSPIN Agile

BSPIN Agile Practice Benchmarking Report 2012 BSPIN Agile Practices Benchmarking Report October 2012 Highlights

BSPIN Agile Practice Benchmarking Report 2012 Introduction • Agile Practice Benchmarking: BSPIN conducted a study during 2011 -2012 to benchmark the Agile practices to provide insights on the effective ways of adopting Agile in small to large organizations. Ø 12 Organizations presented their Agile experience across 3 Panel discussions Ø Agile Practice benchmarking survey was conducted in July 2012 • Designing of Questionnaire, Consolidation, Analysis of feedback was done in coordination with Agile coach and Senior Consultant from Improvalue Consulting, Bangalore (www. improvalue. com). • Objectives : – Explore new areas like: • Domains where Agile has been adopted. Agile Practices in use. • Benefits and Barriers encountered by organizations on their Agile journey • Metrics Reported (Project and Product) • Coexistence/Compatibility with other standards (CMMi) and practices (Lean) • Best Practices – Focus is on IMPLEMENTATION aspects of AGILE • Outcome: – Learn from industry experience – Enable informed decisions 2

BSPIN Agile Practice Benchmarking Report 2012 Agenda • • • Demographics - Agile Spread Usage of Agile Practices Estimation Techniques Automation, Tools Agile Metrics Agile – Coexistence with other Methodologies/Models Benefits and Barriers Challenges Faced, Best Practices Conclusions/Recommendations 3

BSPIN Agile Practice Benchmarking Report 2012 Demographics • 94 responses from 75 Organizations • More participation from Small and Medium Organizations (compared to other surveys) • Product and Service Organizations covered equally • 98% of Large Organizations and 77% of Small an d Medium Organizations have external quality certifications • More than 50% of the respondents have been using Agile for more than 2 years • Panel discussions – 12 Organizations shared their Agile Experience ( 6 Large and 6 Small-Medium) 4

BSPIN Agile Practice Benchmarking Report 2012 Support to Agile How Agile adoption is supported? • This section was answered by 72% of the organizations • As the organization gains more experience on Agile, Agile becomes a part of QMS • Only 6% of the organization indicated “Self Management by teams” with no other support. Training Methods used: • This was answered by 90% of respondents • About 40% of the organization had more than one method of training • About 20% of the organization indicated only “In -house” with no other support. 5

BSPIN Agile Practice Benchmarking Report 2012 Agile Spread • SCRUM is the mostly adopted methodology • Increase in usage of KANBAN – Many Organizations evolved from manufacturing background where LEAN is adopted • Agile experience of the organization has no major effect on the methodology chosen • Nearly 70% of the organizations are using 1 or 2 methods • Of the 7% using more than 4 methodologies, 3% are small organizations 6

BSPIN Agile Practice Benchmarking Report 2012 Agile Spread Others Domains Reported: • OS, Middleware, Kernel • Platform development • Control Systems • Product Integration • Media and Publication, Education • Travel, Energy and Storage • Agile has been used in almost all domains including safety critical domains like Pharma, Avionics and other Embedded Systems. • Telecom domain which is mostly hardware dependent has reported highest percentage of Agile implementation. • More than 75% of the respondents have reported the implementation of ‘Distributed Agile’ • Agile has been used in most of the project types – including Testing and Systems integration ABREVIATIONS (1) BFS – Banking and Financial Services (2) IA – Industrial Automation (3) A, A, A – Avionics, Aviation, Aerospace (4) SI – Systems Integration 7

BSPIN Agile Practice Benchmarking Report 2012 Usage of Agile Practices 82% have release cycles <= 4 weeks • Core Agile (Scrum) Practices have been well adopted. • Engineering Practices usage (Refactoring, Test Driven Development) needs to be strengthened. • 48% of the respondents use 2 to 3 weeks release cycles. 8

BSPIN Agile Practice Benchmarking Report 2012 Agile Practices Vs Agile Experience • Core Agile practices are embraced right from the beginning • There is a significant increase in the usage of Open Work Areas (compared to other surveys) • Some of Scrum practices take more time to stabilize • Organizations can have increased rigor early in the Agile adoption for better benefits. • The Engineering practices strengthen over time • Refactoring and TDD need more focus for effective Agile adoption 9

BSPIN Agile Practice Benchmarking Report 2012 Agile Estimation Techniques • About 53% of the respondents use "User Stories" estimation at proposal/bid stage • About 80% of the respondents are using one or more of the following estimation techniques: i. User Stories estimation ii. WBS iii. Guesstimation Usage of traditional estimation techniques at proposal/bid stage and User Story Estimation/ WBS at iteration stage seems to be in vogue • Organizations are taking more than 2(two) years to move away from Guesstimation to WBS which needs attention. 10

BSPIN Agile Practice Benchmarking Report 2012 Extent of Automation • Only 76% of the respondents answered this section • Automation of unit testing is low, inline with low adoption of TDD • Automation of code reviews is low, as evident from low adoption rate of Refactoring • Automation done in later stages of testing and quality compromisedmissing out the essence of Agile • The automation level increases with Agile experience of the organization • The size of the organization had no impact on the level of automation Organizations have shared multiple Tools : For Agile Project Management and Engineering. 11

BSPIN Agile Practice Benchmarking Report 2012 Tools Used Requirements Code Reviews • Caliber RM • Check. Style • DOORS • Eclipse plugins • Greenhopper • Fx. Cop • JIRA • IBM App. Scan • RALLY • • Rational Requisite Pro Rational Software Analyzer • Red Mine • Scrum. Works • Sharepoint • TFS • Version. One Testing Project Management Defect Tracking HP Quality Center • Bugzilla • Collab. Net • • • HP Quick Test Professional Team. Forge Green Hopper • Clearquest • • JIRA • • Load. Runner HP Quality Center HP-Project and Portfolio Management • • JIRA • MS VS JIRA • Rational Clear. Quest • Mingle • Rally TFS • Scrum Works • • SONAR • Selenium • Style. Cop • Silk • TFS • VSTS • TFS • Toad • Winrunner • Version One • x. Test • XPlanner • x. Unit Not a complete list. Widely used tools are reported here. • 12

BSPIN Agile Practice Benchmarking Report 2012 Traditional Practices with Agile • 56% of the respondents reported: – Upfront collection and finalization of requirements – Upfront architecture • This practice was predominantly observed in Product companies when compared to Service Providers 13

BSPIN Agile Practice Benchmarking Report 2012 Management Metrics Other Metrics Reported: Schedule: Schedule Variance, Sprint Stretch Factor Effort: Effort Variance, Scope: Commitment Accuracy, (De)Scoping Index, Requirements Stability, Scope burn down Quality: Testing: Product Metrics, Review Efficiency, Defect Turnaround Time, Defect Resolution Productivity, Defect Removal effectiveness Test Yield, Test case pass percentage • 90% of the respondents have answered this question • The usage of metrics increases with Agile experience to measure effectiveness Recommend BSPIN to conduct ‘Agile data Benchmarking’ to obtain industry benchmark 14

BSPIN Agile Practice Benchmarking Report 2012 Engineering Metrics • • Measuring Code coverage is key for quality – needs more attention There seems to be a steady improvement as the organization gains more experience on Agile • Measuring Code complexity is key for refactoring – needs more focus • • Monitoring Product Metrics is fairly good except for the Code related activities However, Code Complexity and Code Coverage metrics needs more focus 15

BSPIN Agile Practice Benchmarking Report 2012 Agile - CMMi Coexitence What does the Survey Report? Reasons cited for “Yes”: : i. Agile does not throw away the processes. It is a mature way of looking how efficient the process ii. CMMi prescriptions can be applied to any methodology. Reasons cited for “No” and “not Sure” : i. Different focus (People centric Vs Process centric) ii. Agile is project management, engineering framework and CMMi is a process improvement framework iii. It is compatible till L 3 except IPM, OPD • This question was answered by 72% of the respondents • 68% of the respondents think that Agile and CMMi can coexist. Of these 60% are from organizations that have ISO or CMMi certification. • 72% of the respondents reported that their Agile projects were also being audited (as per the corresponding certification process) 16

BSPIN Agile Practice Benchmarking Report 2012 CMMi on Agile What does CMMi v 1. 3 say? To help those who use Agile methods to interpret CMMI practices in their environments, notes have been added to selected process areas. These notes are added, usually in the introductory notes, to the following process areas in CMMI-DEV: CM, PI, PMC, PPQA, RD, REQM, RSKM, TS, and VER. . Even if you are not using an Agile approach, you might still find value in these notes. . Some general background and motivation for the guidance given on Agile development approaches are found in the SEI technical note CMMI or Agile: Why Not Embrace Both! [Glazer 2008]. Reference: Section-5: Interpreting CMMI When Using Agile Approaches – CMMI-DEV, V 1. 3 by SEI Possible reason for higher maturity organizations to agree that Agile and CMMi are compatible Practitioner Groups of CMMi and Agile can integrate the practices to adopt the best of both 17

BSPIN Agile Practice Benchmarking Report 2012 Coexistence with other models Integrating LEAN with Agile • 33% of the respondents reported using LEAN techniques with Agile • Most organizations (26%) have adopted KANBAN Integrating Agile with other Life Cycle Models • 46% of the respondents reported using Agile best practices in other LCMs • Agile practices predominantly used: 18

BSPIN Agile Practice Benchmarking Report 2012 Benefits of Agile adoption Key Benefits: • Better management of Requirement and Priority changes, Risks • Improved time-to-market and customer satisfaction 19

BSPIN Agile Practice Benchmarking Report 2012 Benefits Perceived – Role wise • Ability to handle risks and improved customer satisfaction were rated high by Agile team and Agile Coach • Managing changing priorities and better project visibility were rated high by senior management and process professionals • Benefits perceived by organizations increase with Agile experience, inline with practices adopted and metrics collected/analyzed 20

BSPIN Agile Practice Benchmarking Report 2012 Barriers to Agile adoption These observations are in line with the world wide survey reports by other organizations Agile Coaches/Consultants and Senior Management to focus on methodical transition to AGILE 21

BSPIN Agile Practice Benchmarking Report 2012 Barriers Perceived – Role wise • Barriers reported by Senior Management and Process Professional are much higher than Agile Coach/Consultant and Agile Team • The size and Agile experience of the organization had no significant impact on the barriers 22

BSPIN Agile Practice Benchmarking Report 2012 Benefits and Barriers - Impact • Time-to-market • IT and Business Collaboration • Handle risks earlier/ better • Manage changing priorities • Better project visibility • Customer/End-user satisfaction Customer Management • Agile cannot handle Maintenance • Lack of skilled resources • Agile produces Insufficient documentation • Lack of coordinated transition to agile • Lack of focus among PM/PMO Groups • Resistance to change • Manage changing requirements • Better Team-morale • Better collaboration Team • Project technical complexity / size • Lack of onsite - offshore collaboration • Lack of tools/ Automation for short iterations 23

BSPIN Agile Practice Benchmarking Report 2012 Challenges Expressed Challenges Faced Possible Solution • Fixed Price Model is a challenge, T&M have worked well. • Can think of Hybrid model based on the scope clarity. • Collection of Traditional Metrics in Agile Projects. • Redefine/Customize the traditional metrics. • Moving away from Command & Control; Self Organizing Teams • Handholding by Agile coach. Mandatory Agile skill training. • Scope creep and sign-offs. • Stakeholders to come together and decide. • Compensation / Appraisal related issues due to no individual Heroes • Policies need to be flexible to have Team benefits • Communication/ Collaboration challenges between the distributed sub-teams. • Address the dependency issues in Daily stand-up meetings. Effective Scrum of Scrums • Infrastructure needs for Continuous integration. Teams Not exposed to usage of tools and automation. • Senior management to focus on infrastructure needs. • Frequent release cycles in sprints thus not enough time for testing. Very less focus on Non functional Test. Some legacy application testing cannot be fully automated. • Teams to look into non-functional aspects during planning meetings. Detail planning of testing effort for manual testing • No predefined performance benchmarks. • Benchmark within the organization. Participate in external Forums. • User story articulation by different stakeholders. • Breakdown stories for ease of development. Help customers articulate User Stories and Product Backlog. • Focus on Maintainability and code quality. • Usage of Tools • Not all teams will be empowered or trained for ‘Test Driven Development’. Need External Support. • Enhance People Agile skills; especially on Engineering side.

BSPIN Agile Practice Benchmarking Report 2012 Best Practices Reported • Record the retrospection and use some Agile tools for tracking them. • Stake holder involvement in defining test scenarios. • More encouragement from management to adopt Test Driven and Behaviour Driven Development. • Invest on tools and infrastructure to run automated regression suite on every check in • Representative at onsite for each offshore team: Overlap of the key people from offshore with onsite team. • Slightly heavy onsite team for better coordination. • Good build plan. Report invocation from Build. • Teams based on functionality rather than skills • Buddy Testing from developer machine to save the deployment time. • All the team members who are part of particular sprint should be part of the scope and estimation. • Measurement of code complexity for quantification. • Use of Agile training from time to time, participation in Agile conferences and Forums.

BSPIN Agile Practice Benchmarking Report 2012 Conclusions and Recommendations • Agile as a methodology is well established - more than 50% of respondents have used it for past 2 years and included it in QMS. Scrum is the most widely used methodology followed by Kanban • Agile usage is spread across multiple domains and project types; Over 75% of the respondents have confirmed using Distributed Agile • While Management practices (predominantly from SCRUM) have been well adopted, the adoption of engineering practices (predominantly from XP) needs more rigor • Usage of metrics and automation level increased with Agile experience • Coexistence of Agile with other industry standards (like ISO and CMMi) has been agreed upon NEXT STEPS • Data Benchmarking of Agile projects • Practitioner Groups of CMMi and Agile can integrate the practices to adopt the best of both Bibliography • CMMi for Development, Version 1. 3 by SEI • “ 2011 State of Agile Development Survey Results” by Version. One • “Agile adoption in India Survey Report – 2011” by Thought. Works • “Agile India 2012 Survey Results” by Valtech

BSPIN Agile Practice Benchmarking Report 2012 Discussions THANK YOU 27