Development Process Metrics A look at development process

Development Process Metrics A look at development process metrics with view to managing the development of secure software. Dr. Michael Van. Hilst 22 March 2007 Secure Systems Research Group - FAU

Development Process Metrics A look at development process metrics with view to managing the development of secure software. Dr. Michael Van. Hilst 22 March 2007 Secure Systems Research Group - FAU

ISO-15504 (level 2), CMM (level 4) Measure how well the process is being followed (using GQM). 1. Identify goals (i. e. reliable software products) 2. Analyze causal chains in the process 3. Identify process steps that contribute to the goals 4. Define “measurements” that assess if identified steps are followed. 5. Measurement can use indicators (presence of documents) or survey questions (“How committed was management in the review meeting? ”) 6. Measure and hold assessment meetings every 2 months W. S. Humphrey. A Discipline for Software Engineering. Addison-Wesley, January 1995. Jarvinen, J. , Hamann, D. , and van Solingen, R. , “On integrating assessment and measurement: towards continuous assessment of software engineering processes”, IEEE Software Metrics Symposium, 1999. Birk, A. , Derks, P. , Hamann, D. , Hirvensalo, J. , Oivo, M. , Rodenbach, E. , van Solingen, R. , and Taramaa, J. , “Applications of measurement in product-focused process improvement: a comparative industrial case study”, IEEE Software Metrics Symposium, 1998. Secure Systems Research Group - FAU

Personal Software Process • Requires developers to record lots of notes on how they spend time. • Developed by Watts Humphrey at SEI, promoted as part of Team Software Process within CMMI. • Experience shows dedication slacks off over time. • Studies by Johnson, et al. , find self reports inaccurate, easily skewed. W. S. Humphrey. A Discipline for Software Engineering. Addison-Wesley, January 1995. Anne M. Disney and Philip M. Johnson, “Investigating Data Quality Problems in the PSP (Experience Paper)”, ACM SIGSOFT 1998 Philip M. Johnson, Hongbing Kou, Joy Agustin, Christopher Chan, Carleton Moore, Jitender Miglani, Shenyan Zhen, and William E. J. Doane, “Beyond the Personal Software Process: Metrics collection and analysis for the differently disciplined. ”, IEEE ICSE 2003 Secure Systems Research Group - FAU

Causal Analysis (CMM level 5) Dumke, R. R. ; Blazey, M. ; Hegewald, H. ; Reitz, D. ; Richter, K. : Causalities in Software Process Measurement and Improvement. International Conference on Software Process and Product Measurement (MENSURA 2006), November, 6 -8, 2006 Several other papers on process discovery, inference, or compliance. Secure Systems Research Group - FAU

Low Level Non-Invasive Metrics Instrument IDE, CM, Office Tools, OS Reports which app/file in foreground Used to look for indicators of lowered productivity. Johnson example looked at contributing factors in build failures. Johnson, P. M. , Kou, H. , Paulding, M. , Zhang, Q. , Kagawa, A. , and Yamashita, T, Improving software development management through software project telemetry, IEEE Software, July 2005 Andrea Janes, Marco Scotto, Alberto Silliti, and Giancarlo Succi, “A Perspectrive on Non Invasive Software Management”, IEEE Instrumentation and Measurement Technology Conference (IMTC), 2006 Secure Systems Research Group - FAU

Siemens Measurement System • Four measurement categories: Time / Size / Quality / Cost. – Size metric undecided (IFPUG 4. 2, COSMIC FFP V 2. 2) • Four selected indicators are – Schedule / Budget compliance – Cost of Defect Correction – Feature changes – Defect Detection Profile • An annual “Measurement Excellence Day” helps to keep participants in the SMS loyal to the program. Secure Systems Research Group - FAU

Bias in the Data • Measures conformance to plans/predictions – Time spent – Code written (LOC, FP) – Defects found / fixed • Analysis always blames the developer – Developer productivity (slow, error prone) – Process conformance (followed process? ) • Won’t drive process change or innovation Secure Systems Research Group - FAU

Shell Information Technology • Five metrics are collected: (Tom Dekker) – Project Delivery Rate, – Speed of Delivery, – Defect Density, – Time Schedule, – Effort Cost. • The target is: 90% of the projects should be within schedule, budget, and within planned scope. • Revised estimation based on industry benchmarks proved much better than the original values based on Shell’s own history-based data. They were less biased than their own. • (Less biased, closer to business goals) Secure Systems Research Group - FAU

Typical Effort vs. Calendar Day develop start Secure Systems Research Group - FAU test … alpha release fix beta release

Waste & Delay in Typical Effort develop … fault caused test … fix lag fault fixed fault found production requirements freeze Secure Systems Research Group - FAU waste alpha release beta release

Toyota Product Development System Design defects out of the process test & develop A product with many defects at first release will still have many defects at final release start Secure Systems Research Group - FAU release

IS Security Design Methods • Much literature on requirements for security • Very little on the development process Richard Baskerville, Information systems security design methods: implications for information systems development, ACM Computing Surveys, 25, 4 (December 1993) Pages: 375 – 414 Chung, L. , Nixon, B. Dealing with non-functional requirements: three experimental studies of a process-oriented approach. Proceedings of ICSE’ 95, pp. 25– 37 Eloff, MM, von Solms, SH, ‘Information Security Management : An Approach to combine Process Certification, and product Certification’, Computers and Security, Vol 19, pp 698 – 709 T. Tryfonas, E. Kiountouzis, A. Poulymenakou, Embedding security practices in contemporary information systems development approaches, Information Management & Computer Security, Oct 2001 9, 4 183 -197 Secure Systems Research Group - FAU

A Toyota Security Method? • A product with many security flaws at first release will still have many flaws at final release • We get what we measure. If we don’t measure security we won’t get security. So how do we measure security? • Is there an idiot-proof or test-first equivalent for driving security flaws out in the development process? • Suggestion from Nelly: use automated code analysis for security anti-patterns, unsafe library calls, … Mylopolous has paper on process for transforming insecure code to secure cope. Secure Systems Research Group - FAU

Secure Systems Research Group - FAU