Fast Flexible Software Development Michael A Cusumano

2

Different Cultural Orientations EUROPE: Software as a Science – Formal Methods, Object-Oriented Design JAPAN: Software as Production – Software Factories, Zero-Defects INDIA: Software as a Service – Infosys, Tata, Wipro, Satym, Cognizant, Patni The USA: Software as a Business – Windows, Office, Navigator, $$$$

Problems in Software Development • Similar problems recurring since the 1960 s • 1969 NATO Report on Software Engineering: Documented problems in – requirements, design vs. coding separation – estimates, monitoring progress, communication – productivity (26: 1), metrics, reliability (bugs) – hardware dependencies, reuse – maintenance costs • Sound familiar? ?

Proposed Solutions • Many attempts – IBM-style software engineering (1960 s, 1970 s) – Japanese “software factories” (1970 s, 1980 s – stable teams, standard process & tools, reuse) – SEI Capabilities Maturity Model (1980 s to present) – “Iterative” and “Agile” methods (US, Europe) • No one process perfect for all projects – Variations: business models, customer requirements, application domain, competition, pace of change, local culture? ? , etc. • Common Theme: How balance quality, features & design flexibility, with cost & speed

Different Process Philosophies • Waterfall-style (sequential, “Stage-gate”) versus • Iterative-style (agile, incremental) – – – – Spiral Rapid Prototyping Synch-and-Stabilize (Microsoft, PC makers) IBM Rational’s Unified Process & Toolkit HP’s Evo Process (short cycles of mini-waterfalls) Extreme Programming (XP), SCRUM, AGILE Many other variations at companies

Traditional Waterfall Model (One development cycle) Requirements Functional design Detailed module design Module construction Integration/system test Module rework (debug) Re-test, debugging Product release

Frequent Waterfall Process Result Requirements Functional design Detailed module design Module construction Integration/system test If modules change a lot, integration fails and you can experience an infinite defect loop.

Reality: Spectrum of Approaches Process Choices for Different Projects Early/Often High Rapid Prototyping User Feedback During Project Uncertainty in Requirements Traditional Waterfall, Older “Factory-like” Approaches XP Agile or Iterative Methods Incremental Late/Occasional Low 1 Adapted from Bill Crandall (HP) # of Cycles, Releases Many

Iterative Style (“Synch & Stabilize”) Waterfall-Style 10

International Comparisons (2003 IEEE Software, “Software Dev. Worldwide”) • Survey: Completed in 2002 -2003, with Alan Mac. Cormack (HBS), Chris Kemerer (Pittsburgh), and Bill Crandall (HP) • Objective: Determine usage of iterative (Synch-&-Stabilize) versus Waterfall-ish techniques, with performance comparisons – 118 projects plus 30 from HP-Agilent for pilot survey • Participants – India: Motorola MEI, Infosys, Tata, Patni – Japan: Hitachi, NEC, IBM Japan, NTT Data, SRA, Matsushita, Omron, Fuji Xerox, Olympus – US: IBM, HP, Agilent, Microsoft, Siebel, AT&T, Fidelity, Merrill Lynch, Lockheed Martin, TRW, Micron Tech – Europe: Siemens, Nokia, Business Objects

“Conventional” Good Practices India Japan USA Europe etc Total Number of Projects 24 27 31 22 104 Architectural Specs % 83% 70% 55% 73% 69% Functional Specs % 96% 93% 74% 82% 86% Detailed Design % 100% 85% 32% 68% 69% Code Generators -- Yes 63% 41% 52% 55% 52% Design Reviews -- Yes 100% 77% 88% Code Reviews -- Yes 96% 74% 71% 82% 80%

“Newer” Iterative Practices India Japan USA Europe etc Total No. of Projects 24 27 31 22 104 Subcycles -- Yes 79% 44% 55% 86% 64% Beta tests -- Yes 67% 77% 82% 73% Pair Testing -- Yes 54% 44% 35% 32% 41% Pair Programmer -- Yes 58% 22% 36% 27% 35% Daily Builds at project start 17% 22% 36% 9% 22% In the middle 13% 26% 29% 27% 24% At the end 29% 37% 36% 41% 36% 92% 96% 71% 77% 84% Regression test each build

“Crude” Output Comparisons India Japan Europe etc. TOTAL 24 Projects USA 27 31 22 104 LOC/ Month median 209 469 cf. 389 in 1990 270 cf. 245 in 1990 436 374 Bugs/ 1000 LOC median . 263 . 020 cf. . 20 in 1990 . 400 cf. . 80 in 1990 . 225 . 150

Observations from Global Survey • Most projects (64%) not pure waterfall; 36% were! • Mix of “conventional” and “iterative” common -- use of functional specs, design & code reviews, but with subcycles, regression tests on frequent builds • Customer-reported defects improved -- over past decade in US and Japan; LOC “productivity” may have improved a little • Japanese still report best quality & productivity -- but what does this mean? Preoccupation with “zero defects”? Need more lines of code to write same functionality per day as US & Indian programmers? • Indian projects strong in process and quality -- but not as strong as their dominance of CMM Level 5 suggests? ?

Hewlett Packard Pilot Study (2003 IEEE Software, “Tradeoffs” article) • Managers – When use iterative or waterfall? – Survey: 35 responses, 29 projects with complete data – Median – 170 K LOC, with 70 K new code; 9 -person team, 14 month projects – 59% applications, 38% systems, 28% embedded • 74% of variation in defects explained by early prototypes, design reviews, and integration/regression testing on builds – Median project: 40% of functionality complete when first prototype released and 35. 6 defects per million (. 04/1000) LOC, reported by customers in 12 months after release, and 18 LOC person day (360/month)

Multivariate Regression Analysis Some striking results, compared to median: • Releasing prototype earlier with 20% of functionality 27% reduction in defect rate • Integration/regression testing at each code check-in 36% reduction in defect rate • Design reviews • 55% reduction in defect rate • Releasing prototype with 20% of functionality 35% rise in LOC output/programmer • Daily builds • 93% rise in LOC output/programmer

Observations from HP Survey • Best “nominal” quality from traditional “waterfall” (fewer cycles & late changes = less bugs, of course!!) • Best balance of quality, flexibility, cost & speed from combining conventional & iterative practices • BUT: Differences in quality between waterfall & iterative disappear if use a bundle of techniques: – Short development subcycles (subprojects/milestones) – Early prototypes to get customer feedback – Frequent builds to incorporate feedback, changes – Design/code reviews (check quality continuously) – Regression tests on each build (check for errors, late changes, integration problems)

Where Iterative May Work Better • Fast-paced product markets where technologies, competition, and product features may have to change a lot during a project. • Product or custom projects where customers place very high emphasis on leading-edge features. • Custom systems where customers place high emphasis on responsiveness to their input during a project. • Product or custom projects that require experimentation as in lots of short design, build, and test cycles. • Other?

Where Waterfall May Work Better • Extremely high-reliability systems (product or custom projects), where functions are very well understood and no changes in requirements during a project are desired • Embedded products with hardware constraints that cannot be easily changed • Contract software where client requires a detailed proposal upfront and not possible to limit scope/phases • Complex projects (product or custom) where parts of design & coding are outsourced, off-shored, or done in multiple sites, AND there are weak mechanisms to synchronize and manage distributed teams

Some Comments on Good Practices • • • High-Level Process Innovation & Design Architecture Team Management Project Management Testing & QA

High-Level Process Strategy • “You can’t do anything that’s complex unless you have structure. ” • For creative people, that structure should be as subtle as possible.

Dave Maritz Test Mgr, Windows 95 Everything that I do here I learned in the military. . You can't do anything that's complex unless you have structure. . And what you have to do is make that structure as unseen as possible and build up this image for all these prima donnas to think that they can do what they like. Who cares if a guy walks around without shoes all day? Who cares if the guy has got his teddy bear in his office? I don't care. . [But if] somebody hasn't checked in his code by five o'clock, then that guy knows that I am going to get into his office. ” From Microsoft Secrets

High-Level Process cont’d • Structure: – Process must fit the product & market/customer • Nature of the Product or Service – Mission critical vs. not; new vs. derivative – Infrastructure vs. applications, embedded, etc. – Packaged vs. custom, or multi-system solution • Nature of the Market or Customer: – Speed & innovation for leading-edge markets – Quality & support for enterprise & mass mkts

Innovation & Design Strategy • Being “slightly out of control” can stimulate innovative thinking and creativity. • But too few controls chaos or “rocket science” • Iterative a middle approach: Vision statements, outlines of functional specs, prototypes, early beta releases, multi-version release mentality, etc. • Late design changes can be good: help improve the product -- respond fast to user feedback, competitors’ moves, or unforeseen changes in the market & technology.

Architecture Strategy • Definition: – How to divide a product into subsystems and modules/components, and with what interfaces • Strategy: – degree of “modular” vs. “integral” – degree of “open” vs. “closed” (proprietary) – consider/beware of “Open but not Open”! • Modular & “horizontal” (feature-oriented) – de-couples small development teams – facilitates adding or cutting functionality – facilitates reuse of components

Architecture Strategy Tradeoff: • Investment in architecture for the future versus new features for the present • Try to design architectures that will last. – Rush-jobs “spaghetti” & hard to modify • Or, plan to evolve architectures incrementally

Bob Lisbonne, Netscape VP Client Development When our teams grew beyond a certain point, they began to resemble a 200 -person threelegged race … That’s why the componentization or the modularization of the product is so key, so that ultimately we can get back to lots of small teams each doing their own thing… and not getting caught up in one another’s efforts. From Competing on Internet Time

Team Management Strategy • Small teams of great people work best • Large teams can work like small teams (within limits – not Longhorn!) • What you need: – One strong person in charge! – Map project architecture to product architecture feature & subsystem teams – Keep feature teams small 3 -8 people. – Overlap functional responsibilities (vision/scope, requirements, QA) – Focus everyone on shipping the product!

Chris Peters, VP Microsoft Office Everybody in a business unit has exactly the same job description and that is to ship products. Your job is not to write code, your job is not to test, your job is not to write specs. Your job is to ship products. . When you wake up in the morning and you come to work, you say what is the focus? Are we trying to ship? Are we trying to write code? The answer is we are trying to ship. . You're trying not to write code. If we could make all this money by not writing code, we'd do it. From Microsoft Secrets

Microsoft Project Team Structure

Project Management Strategy • Avoid sequential “waterfall” schedules (usually). • Divide long projects into multiple sub-projects or milestones, of a few weeks or months duration. • Evolve requirements incrementally: do spec design, development & testing concurrently. • Impose a few rigid rules to force frequent synchronizations and periodic stabilizations. Synchronize-and-Stabilize!

Project Management cont’d • Prioritize Rigorously: most important features first • • Schedule “backwards” – people & time Let engineers schedule their own tasks Managers keep historical data on estimates Prototypes and early beta releases = rapid feedback on designs and quality • Use historical data to schedule buffer time (for projects, not individuals) for changes & unknowns

Synch-and-Stabilize Process

Synch-Up and Debug Daily • Microsoft Secrets: Few rules, but “military-like” discipline to force coordination & communication – – – Developers can check in when they want Each project must build daily Check-in at set times; can’t leave until build is OK If you break the build, you must fix your code now! Penalties for those who break the build • Minimize build overhead through build team and automated tools Objectives: – Synching-up: 5 -20 minutes; Quick test: 30 minutes – Total Check-in: 5 -60 minutes

Testing and QA Strategy • Try to build-in quality continuously – design and code reviews – gates or check points – continuous customer feedback, etc. • But continually integrate and test – Frequent builds (daily, weekly) – Especially check late design changes • Test what? – unit/feature testing, of course – system integration testing -- from early on!

Testing and QA cont’d • Automate to test & retest design changes quickly and frequently. • But automation never eliminates the need for people. Someone has to write and rewrite (update) the automated tests. • Also need human testers to probe real user behavior

Testing and QA cont’d • Post-mortems: what went well, what went poorly, and what the team should do next time. • “Eat your own dog food”: first-hand feedback on products as quickly as possible. • A few quantifiable metrics: to control and improve quality as well as monitor key product and process characteristics. • Early beta releases: provide feedback on design as well as quality.

Concluding Comments • No one “best” software development process – But “waterfall” less responsive to change – But using a “bundle” of practices eliminates differences in quality between waterfall and iterative! • Process should depend on business, context, and strategy – Type of software, customer requirements, team experience and culture, contract needs, etc. – Product vs. custom, mass-market vs. niche, individual vs. enterprise, leading-edge vs. follower, etc.

Main References • The Business of Software by M. Cusumano (Free Press/Simon & Schuster, 2004) • Microsoft Secrets by M. Cusumano and R. Selby (Free Press/Simon & Schuster, 1995 and 1998) • Competing on Internet Time by M. Cusumano and D. Yoffie (Free Press/Simon & Schuster, 1998) • Michael Cusumano, Alan Mac. Cormack, Chris Kemerer, and Bill Crandall, “Software Development Worldwide: The State of the Practice”, IEEE Software, November-December 2003. (International Comparisons) • Alan Mac. Cormack, Chris Kemerer, Michael Cusumano, and Bill Crandall, “Trade-offs between Productivity and Quality in Selecting Software Development Practices”, IEEE Software, September-October 2003. (HP Survey)