Top Coder Component-Based Software Engineering CBSE Quality Assurance

Top. Coder Component-Based Software Engineering (CBSE) Quality Assurance (QA) Sean Campion [scamp] Presented To: 27 June 2007 Top. Coder Open, Las Vegas 19 March 2018 CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] Agenda • CBSE Quality Assurance Specific Challenges • scamp@home • Top. Coder

[ ] CBSE Quality Assurance Specific Challenges The following are common QA problems cited against the use of reusable components. I will discuss each issue, and what Top. Coder does to address the issue. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] CBSE Quality Assurance Specific Challenges Issue: Lack of access to commercial component source code. Top. Coder: We provide both the source code and all design documentation for every component that we produce. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] CBSE Quality Assurance Specific Challenges Issue: Components are not readily testable. Top. Coder: All of our components are required to have unit tests of stress, failure, and accuracy. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] CBSE Quality Assurance Specific Challenges Issue: Difficult to use and interpret tests from component vendors Top. Coder: Our documentation provides examples on how to run the tests; we also provide the test results in the form of logs. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] CBSE Quality Assurance Specific Challenges Issue: Concern that components were not developed for a specific operating environment; and that developing a component test bed for all possible environments is too costly and time consuming. Top. Coder: We are developing scamp@home, a software system that will take advantage of participants diverse environments to test components. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] scamp@home - Distributes component tests to participating computers. - Takes advantage of diverse environments, hardware, character sets, and so on. - Massively parallel - Can be setup at customer sites to start validating existing components during architecture phase on actual target environment. - Customers can use to test their own components. - Ideal for regression-testing when component is updated. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] CONFIDENTIAL Copyright 2007, Top. Coder, Inc. scamp@home

[ ] Top. Coder Process from a QA Perspective Specification Phase - Mid and end-point peer reviews of the application specification, conducted by 2 Project Managers and 1 Architect. - High-level test cases, tied to application level functionality, developed. - Customer review and sign-off on application specification. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] Top. Coder Process from a QA Perspective Architecture Phase - Mid and end-point peer reviews of the architecture design, conducted by 1 Project Manager and 2 Architects. - High-level test cases expanded into detailed test cases and scenarios. - Customer review and sign-off on architecture design. - Future: Existing components identified for reuse can be validated using scamp@home, including on the customer target systems if desired. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] Top. Coder Process from a QA Perspective Component Production Phase - Design competitions; designers know they are competing against each other. - Automated submission screening tool. - Manual screening phase. - Review Board. - Scorecard-based evaluation. - Final fixes. - Winning designer required to support the development phase. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] Top. Coder Process from a QA Perspective Component Production Phase - Development competitions; developers know they are competing against each other. - Automated submission screening tool. - Manual screening phase. - Review Board. - Scorecard-based evaluation. - Final fixes. - Unit tests: stress, failure, accuracy - Coverage Tool: minimum of 85% line coverage from unit tests. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] Top. Coder Process from a QA Perspective Assembly Phase - Assembly competitions; teams know they are competing against each other. - Manual screening phase. - Review Board. - Scorecard-based evaluation. - Final fixes. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] Top. Coder Process from a QA Perspective Assembly Phase - Testing competitions; teams know they are competing against each other. - Produces automated tests that will be run against the assembly competition submissions. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] Top. Coder Process from a QA Perspective Certification Phase - Replicate the customer’s target environment on a TC server. - Deploy and rigorously test the winning assembly submission. - Run automated tests against the submission. - Run manual tests against the submission. - Early customer access to the TC testing environment for immediate feedback. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] Top. Coder Process from a QA Perspective User Acceptance Testing Phase - Customer has option of performing their own User Acceptance Testing. - Top. Coder will directly support any UAT. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] Top. Coder Process from a QA Perspective Deployment Phase - Top. Coder will support the deployed application for 90 days. - Customer receives all source code, including component source code, and all design documentation from all phases. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] Metrics Here’s how Top. Coder compares to industry standards: Industry Defects per

[ ] Wrap-Up - Quality Assurance addressed at every stage of the process in one form or another. - Top. Coder continuously looking to improve QA through processes and tools. - scamp@home addresses a primary concern of reusable components. - Quality software distinguishes Top. Coder from other software. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.

[ ] References - “Testing and Quality Assurance for Component-Based Software”, Jerry Zeyu Gao, H. -S. Jacob Tsao, & Ye Wu; Artech House, Boston, 2003. - “Risk Management in Component-Based Development: A Separation of Concerns Perspective”, Awais Rashid, Gerald Kotonya, Computing Department, Lancaster University, UK. - “The Trouble With Testing Components”, Elaine J. Weyuker, in “Component Based Software Engineering”, Chapter 28, Addison-Wesley, 2001. - “CBS Testing Requirements and Test Case Process Documentation Revisited”, Nasib S. Gill, Pradeep Tomar, Department of Computer Science and Applications, Haryana India, March 2007. From ACM SIGSOFT Software Engineering Notes, Volume 32 Number 2. - “The QSM Software Alamanac: Application Development Series”, QSM Inc. , 2006. CONFIDENTIAL Copyright 2007, Top. Coder, Inc.