Tools for Automating Verification and Validation checking Are

Tools for Automating Verification and Validation: checking: Are we building the right system? Verification: checking: Are we building the system right? • • • Model dependent size dependent platform dependent Purpose: improve the S/W development process • • - avoid delays, reworks less error-prone more standard better defined more predictable faster (time-to-market) better managed process assist in better understanding product (helping to manage complexity of product)

Automating V & V in the S/W Development Process Must be at SEI CMM Level 3 or higher. (- process is defined) [pre-condition for use of tools] e. g. Reqts. High Level Design BUT only automate where necessary Detailed Design Release & Support Codes Inspections & Unit Test Regression Performance. . . Build & System Test Level 4 CMM - metrics to help in process management

taking the process view for V & V tool assessment, 1) Where in the process does the tool apply? 2) What is the entry criteria to the use of the tool(set)? 3) What is the exit criteria (for stopping the use of the tool? )

Ed Kit’s Tool Classifications Tools for reviews and inspections - assist in reviews, walkthroughs and inspections of requirements, functional design, internal design, and code. - Complexity analysis - “ 20% of code causes 80%of the problems” (Pare to principle) [not good predicates of bad code, easy to collect $] - Code comprehension - help to understand unfamiliar code (take an unbiased look) - Syntax and semantic analysis - extensive error checking to find errors a compiler would miss - flag potential defects Tools for Test Planning - define the scope, approach, resources and schedule of testing activities - tools required are: - templates for test plan documentation - test schedule and staffing estimates - complexity analyzer # of issues R 8 R 3 R 1 R 2 R 4 R 10 R 5 R 6 R 7 R 9

Kit’s Tool Classifications (2) Tools for Test Design and Development • detailing the overall test approach specified in the test plan for software features or combinations of features, and identifying and prioritizing the associated test cases. • Not much tool support • types of tools required: – – test data generator requirements-based test design tool capture/playback coverage analysis Test Execution and Evaluation Tools • process of executing test cases and evaluating the results – includes selecting test cases for execution, setting up the environment, running the selected tests, recording the execution activities, analyzing potential product failures, and measuring the effectiveness of the effort. • Types of tools required: – – – capture/playback coverage analysis memory testing simulators and performance regression testing - necessary

Kit’s Tool Classifications (3) Software Testing Support Tools • tools that lend overall support to the overall test process • when these tools are of poor quality or non-existent the professional tester suffers • types of tools required: – Problem management • defect tracking tools, etc. , used to record, track, and assist with the management of defects and enhancements throughout the lifecycle • Configuration management – manages, controls and coordinates changes to documents and other artifacts Questions before tool acquisition • • How do the tools fit into and support our test process? Do we know how to plan and design tests? Who will be responsible for making sure we get the proper training on our new tool? Who will promote and support tool use within the organization on an ongoing basis?

Beizer’s Test Tool Classifications • Fundamental Tools – – coverage certifiers flowgraph generators and analyzers (e. g. MSC generator) metrics support instrumentation support (probe insertion & monitoring) • Test Execution Automation – capture/replay • • • test editor smart comparator local and distributed single and multi-thread exception reports – stubs and drivers • • test case initialization input simulation outcome comparison path instrumentation and verification reporting passage to next test debugging support – test environments • support for testing in a CASE environment

Beizer’s Tool Classifications (2) • Test Design Automation – – – structural generators data-flow test generators functional generators FSM (EFXM) based generators random test data generators Test Result Analysis Automation

Load and Performance Testing Current Listings Benchmark Factory Chariot CYRANO Test ITF Load Runner Product Family Pegasus PLATINUM Final Exam Internet Load Test pre Vue-C/S pre Vue-X QALoad SQA Load. Test Teleprocessing Network Simulator Test. Master Web. Load Silk Performer

Non-GUI Drivers and Test Suite Managers Current Listings Acqua QES/EZ Acqua. Navio SNA Development Test Facility Ada. TEST SQA Manager ANVL SQA Suite ARTT STAR Bench. Works STEPMaster Cantata STW/Regression Cob. Retest STW/Regression/MSW CTB TALC 2000 Deja. Gnu TBGEN ESIM Testing Environment TCS for Embedded Software Test. Director Internetworking Test Facility Test. Expert MYNAH Test. Mate OTF - An Object Testing Framework Test. Plan Pilot TMS Pre Vue TMX QA Director TETware QES/Architect VPRO-G

Test Evaluation Tools Current Listings Branch validator C-Cover Code. TEST Cover. Test CTC++ Deep. Cover for Java Deep. Cover for C/C++ GCT Hindsight/TCA Hindsight/TPA Insure++ LDRA Testbed Logiscope Mc. Cabe Visual Testing Toolset Object. Coverage Panorama-C/C++ pf 77 Pure. Coverage safec Sof. Inst STW/Coverage TCMON TCAT for Java Test. Center Vision. Soft/TEST

GUI Test Drivers Current Listings ATF (Automated Test Facility) B-Tree Validor Family CAPBAK/X, CAPBAK/MSW CYRANO TEST Evaluator Ferret QARun Panorama-2 PLATINUM’s Final Exam C/S-Test PLATINUM Final Exam Internet Test QA Partner QC/Replay QES/EZ for GUI Replay. Xt RSW Web Analyst SET (Solution Evaluation Tool) Smalltalk Test Mentor Vermont High Test Plus SQA Robot Visual Test 4. 0 STW/Web Win. Runner Test Edge Win Vue Test Now WITT for OS/2, WITT/Windows, X/WITT Test. Runner xrc - X Remote Control Test. Web XRunner

Marick’s Corner Site Hosted by ST Labs http: //www. stlabs. com/marick/faqs/t-design. htm CALIBER: www. tbi. com