Black-box Testing for Evolving COTS-Based Software Ye Wu

Black-box Testing for Evolving COTS-Based Software Ye Wu Information and Software Engineering Department George Mason University IWICSS © Dr. Ye Wu

Agenda n n IWICSS Introduction Challenges Methodology Conclusion © Dr. Ye Wu 2

What is COTS? n n IWICSS COTS – Commercial-Off-The-Shelf COTS describes ready-made products that can easily be obtained. © Dr. Ye Wu 3

Main characteristics n n n IWICSS It exists a priori Used without source code modification It is available to the general public It can be bought (or leased or licensed). Supported and evolved by vendor © Dr. Ye Wu 4

Challenges Facing Evolving COTS-based Software n n IWICSS What should COTS-users do if COTS products are changed? COTS-user transparent changes Changes that affect COTS product users, but do not affect user system architecture Changes that affects COTS user architecture © Dr. Ye Wu 5

COTS-user transparent changes n n Mainly corrective maintenance and a small part of the changes in perfective and adaptive maintenance activities No change is required for COTS users IWICSS © Dr. Ye Wu 6

COTS-user aware changes n n IWICSS Poor design in the previous version Technology changes Changes of standards Sometimes a strategic move to lock in customers © Dr. Ye Wu 7

COTS-user aware changes n n IWICSS Changes that affect COTS users, but do not affect COTS users’ architecture Changes that affect COTS users’ architecture © Dr. Ye Wu 8

Black-box Testing Strategies n n n IWICSS Random Testing Partition Testing Boundary value Testing Decision Table Testing …… © Dr. Ye Wu 9

Black-box Testing for Evolving COTS-based Software n n IWICSS Information about COTS products to perform adequate black-box testing is not always available Customization needs to be done on some COTS products © Dr. Ye Wu 10

COTS Product Modifications n n n IWICSS Generalization Modifications Specialization Modifications Reconstruction Modifications © Dr. Ye Wu 11

Black-box Testing Methods for Components Partition Testing of Generalization Modification: IWICSS © Dr. Ye Wu 12

Black-box Testing Methods for Components Partition Testing for Specialization modification: IWICSS © Dr. Ye Wu 13

Black-box Testing Methods for Components Partition Testing for Reconstruction modification: IWICSS © Dr. Ye Wu 14

Initial Results • Total 48 partitions – old version • 17 changed or added partitions – new version • 34 faulty versions (each with one inserted fault) • 21 faults identified • 13 faults unidentified • 5 faults can be detected during Unit-testing • 8 faults are caused by boundary value or inaccurate partition IWICSS © Dr. Ye Wu 15

Testing Framework for Evolving COTS-based Software Normal Interfaces Client Testing Interfaces Test Case Repository Component COTS Developer Delivered Version Test Case Analyzer Specification Old Version new Version IWICSS © Dr. Ye Wu 16

Conclusion and Future Work n n n IWICSS Adopting evolving COTS products can be risky if without adequate testing Black-box testing can be adopted, but need to be adequately adjusted Automated test framework is required © Dr. Ye Wu 17