Injection of realistic software faults for experimental software

Injection of realistic software faults for experimental software risk assessment Henrique Madeira Critical Software, SA and University of Coimbra, DEI-CISUC Coimbra, Portugal Universidade de Coimbra NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

What’s the point of injecting software faults? • Software faults are most probably the major cause of computer system outages • Goals: u Experimental risk assessment in component-based software development u Dependability evaluation of COTS components u Robustness testing u Fault tolerance layer evaluation u Dependability benchmarking Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Two possible injection points 1. Injection of interface faults in software components (classical robustness testing) Input SW component under test Output Interface faults 2. Injection of realistic software faults inside software components (new approach) Input Target SW component Output Software faults Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Example of results obtained with interface faults (robustness testing) Robustness failure in RTEMS 4. 5. 0 Excerpt of application code: requested. Size 1 = 4294967295; return. Status = rtems_region_get_segment (region. Id, requested. Size 1, option, timeout, ptsegment 1); Result: Memory exception at fffffffc (illegal address) Unexpected trap (0 x 09) at address 0 x 0200 aaac Data access exception at 0 xfffffffc This software fault was discovered automatically by the Xception robustness testing tool! Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Example of results obtained with injection of software faults What happens if a software bug in a device driver becomes active? • • Software faults are injected in a device driver using the G-SWFIT technique. The device is heavily used by programs running during test. Availability Wors t Best Windows NT Henrique Madeira Feedback Wors t Best Windows 2000 Stability Wors t Best Windows XP NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Component-based software development • Vision: development of systems using pre-fabricated components. Reuse custom components or buy software components available from software manufactures (Commercial-Off-The-Shelf: COTS). • Potential advantages: u u Reduce development effort since the components are already developed, tested, and matured by execution in different contexts Improve system quality Achieve of shorter time-to-market Improve management of increased complexity of software • Trend → use general-purpose COTS components and develop domain specific components. Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Some potential problems • COTS u In general, functionality description is not fully provided. No guarantee of adequate testing. u COTS must be assessed in relation to their intended use. u u The source code is normally not available (makes it impossible white box verification & validation of COTS). • Reuse of custom components in a different context may expose components faults. Using COTS (or reusing custom components) represent a risk! How to assess (and reduce) that risk? Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Case-study: I-don’t-care-about software architecture diagram Software components Different sizes Different levels of granularity Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Question 1 This is a COTS! What’s the risk of using it in my system? Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Question 2 This is custom component previously built! What’s the risk of reusing it in my system? Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Question 3 This is a new custom component! What’s the risk of using it without further testing? Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Experimental risk assessment Example of question: What’s the risk of using Component 3 in my system? Risk = prob. of bug * prob. of bug activation * impact of bug activation Software complexity metrics Henrique Madeira Injection of software faults NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Again, two possible injection points Injection of interface faults Injection of realistic SW faults Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Why injection or real software faults? Injection of SW faults • Error propagation through non conventional channels is a reality. • Faults injected inside components are more representative. Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

How to inject software faults? • Use G-SWFIT (ISSRE 2002, DSN 2003, DSN 2004) u u Injects the top N most common software faults. This top N is based on field data (our study + ODC data from IBM) and corresponds to ~65% of the bugs found in field data. Injects faults in executable code. Largely independent on the programming language, compiler, etc that have generated the executable code. • G-SWFIT is now a reasonably mature technique. Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

G-SWFIT Generic software fault injection technique Target executable code Low-level code mutation engine 01 X 1 10001 00100 1 01011 00010 01001 Library of software fault injection operators Low level mutated versions . . . 01011 0 X 01 00100 1 01011 0001 X 01001 01011 00010 0 X 001 Emulate common programmer mistakes The technique can be applied to binary files prior to execution or to in-memory running processes Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Experimental risk assessment (again) Example of question: What’s the risk of using Component 3 in my system? Risk = prob. of bug * prob. of bug activation * impact of bug activation Software complexity metrics Henrique Madeira Injection of software faults NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Estimation of the probability of residual bugs Target code • Many studies indicate that fault probability correlates with the software module complexity • Metrics of software complexity base on: • Static feature of the code; • Dynamic features; • Possible information on the development process (type of tests, etc); • . . . Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Estimation of bug activation probability and bug impact Target code Software faults • Test campaigns to evaluate the activation probability and the impact of software faults (bugs) inside the component in the rest of the system. • Use software metrics to choose the modules to inject faults and define trigger locations accordingly. Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005

Conclusions and current work on experimental risk assessment • Experimental software risk assessment seems to be viable. • Risk is a multi-dimensional measure. Many software risks can be assessed, depending on the property I’m interested in. • Current work: u Improve the G-SWFIT technique: – – – u u u Improving current tool. Expansion of the mutation operator library Construction of a field-usable tool for software fault emulation in Java environments Study of software metrics and available tools. Composeability measures. Real case-studies to demonstrate the methodology. Henrique Madeira NASA IV&V Facility, Fairmont, WV, USA, August 8, 2005