Software Testing: A Selection of Papers by Elaine Weyuker Peter Greenwood for EEE 599 Apr 98 EEE 599 - Weyuker Presentation
Elaine Weyuker • Ph. D (CS) Rutgers U ‘ 77 • Software Testing Researcher at AT&T Bell Labs, Murray Hill, NJ • CS Professor Courant Institute of NYU • 30+ papers, primarily on Software Testing Apr 98 EEE 599 - Weyuker Presentation 2
The Papers • Axiomatizing Software Test Data Adequacy (TSE Dec 86) • The Automatic Generation of Load Test Suites and the Assessment of the Resulting Software (TSE Sep 95, with Avitzer) • Reliability Testing of Rule-Based Systems (Software Jun 96, with Avritzer and Ros) Apr 98 EEE 599 - Weyuker Presentation 3
Goal of Presentation • highlight some of Weyuker’s prolific work in software testing • contrast the general and theoretical character of her early work with the specific and practical nature of her current work Apr 98 EEE 599 - Weyuker Presentation 4
Axiomizing Software Test Data Adequacy (Dec 86) • How much testing is enough to determine if a program is acceptable? – very important question – relatively little research activity (1986) – no industry standards • test until testing time expired • test until all cases in current test set pass Apr 98 EEE 599 - Weyuker Presentation 5
When Should Testing Stop? • Terminate when an adequacy criterion is met – a measure of how well the testing process has been performed – relates a test set to the program, the specification, or both – could relate a test set to the program’s operational profile Apr 98 EEE 599 - Weyuker Presentation 6
Constraints of this Approach • deals only with dynamic testing (running the program on test cases) • does not involve any static testing (analysis of code, structure, etc. ) Apr 98 EEE 599 - Weyuker Presentation 7
Constraints (con’t) • Weyuker’s paper deals only with program-based criteria – ignores specifications – most adequacy criteria are of this type – easier to automate Apr 98 EEE 599 - Weyuker Presentation 8
Weyuker on Testing “The purpose of testing is to uncover errors, not to certify correctness. ” Apr 98 EEE 599 - Weyuker Presentation 9
The Axioms (1 & 2) • Axiom 1: (Applicability) For every program, there exists a finite adequate test set. • Axiom 2: (Nonexhaustive Applicability) There is a program P and a test set T such that P is adequately tested by T, and T is not an exhaustive test set. Apr 98 EEE 599 - Weyuker Presentation 10
Axioms 3 & 4 • Axiom 3: (Monotonicity) If T is adequate for P, and T is a subset of T’, then T’ is adequate for P. • Axiom 4: (Inadequate Empty Set) The empty set is not adequate for any program. Apr 98 EEE 599 - Weyuker Presentation 11
Axioms 5 & 6 • Axiom 5: (Anti-extensionality) There are programs P and Q such that P is equivalent to Q, T is adequate for P, but T is not adequate for Q. • Axiom 6: (General Multiple Change) There are programs P and Q which are the same shape, and a test set T such that T is adequate for P, but T is not adequate for Q. Apr 98 EEE 599 - Weyuker Presentation 12
Axiom 7 • Axiom 7: (Anti-decomposition) There exists a program P and a component Q such that T is adequate for P, T’ is the set of vectors of values that variables can assume on entrance to Q for some t of T, and T’ is not adequate for Q. Apr 98 EEE 599 - Weyuker Presentation 13
Axiom 8 • Axiom 8: (Anti-composition) There exist programs P and Q such that T is adequate for P and P(T) is adequate for Q, but P is not adequate for P; Q. Apr 98 EEE 599 - Weyuker Presentation 14
Summary of Axiomatizing • axioms are applied to 5 adequacy criteria – path, branch, statement, mutations, modified size • path, branch, statement fail application and antidecompositon • branch and statement also fail anticomposition Apr 98 EEE 599 - Weyuker Presentation 15
Summary of Axiomatizing Apr 98 EEE 599 - Weyuker Presentation 16
Criticism • Zweben & Gourlay, On the Adequacy of Weyuker’s Test Data Adequacy Axioms (IEEE TSE Apr 89) – “The properties proposed by Weyuker are neither fundamental enough nor precise enough to serve as a useful formal system for discussing adequacy or comparing criteria” Apr 98 EEE 599 - Weyuker Presentation 17
More Criticism • Professor Norman Fenton, keynote address to 2 nd QUALMS User Group Meeting, South Bank University, Jul 92 – “Elaine Weyuker … has presented a list of axioms … Measurement Theory reveals that some of these axioms are contradictory. ” Apr 98 EEE 599 - Weyuker Presentation 18
Dr. Weyuker’s Current Work Apr 98 EEE 599 - Weyuker Presentation 19
Automatic Generation of Load Test Suites (Sep 95) • develops 3 algorithms for automatic test case generation • applies them to 5 real industrial software systems • assess the results Apr 98 EEE 599 - Weyuker Presentation 20
Failure Types • Hard failure – complete loss of traffic ( ie cable break, power failure, router failure) • Soft failure – degraded performance (ie activity burst) – important that this does not occur – telecoms designed with excess capacity – avoid permanent soft failure Apr 98 EEE 599 - Weyuker Presentation 21
Goals • a smoothly degrading system • graceful recovery Apr 98 EEE 599 - Weyuker Presentation 22
Load Testing Strategies • system functionality has already been tested • assess system performance under a known load • black-box oriented • based on system operational profile Apr 98 EEE 599 - Weyuker Presentation 23
Typical Telecom Operational Profile • number of different types of call • average service demands and probability distribution of each call type • average external arrival rate • routing probabilities for each node • this information is usually readily available Apr 98 EEE 599 - Weyuker Presentation 24
Purpose of Algorithms • tests resource allocation mechanisms associated with software systems • system behavior under load is not measured by functionality testing • each variable vector is a distinct system state Apr 98 EEE 599 - Weyuker Presentation 25
Basis of Algorithms • too many states to exhaustively test • calculate probability of system being in each state (from operational profile) • heuristically determine some threshold • generate a test case for those states where P(S) > Apr 98 EEE 599 - Weyuker Presentation 26
The Algorithms • Deterministic State Testing (DST) – generate a test case for each state S where P(S) > • DST with State Transition Checking (TDST) – as for DST, but also generate test cases for every S’ reachable from S Apr 98 EEE 599 - Weyuker Presentation 27
The Algorithms (con’t) • DST with Length N Cycles (CDST) – as for DST, but also generate test cases for every S’ within a cycle of length N from S with a probability > Apr 98 EEE 599 - Weyuker Presentation 28
Target Systems • Project 1: • Project 2: • Project 3: • Project 4: • Project 5: Apr 98 telephone call rerouter 800 number call traffic monitor routing automator transaction processing system network interface software EEE 599 - Weyuker Presentation 29
Testing Results • automated test generation routines were developed for all five projects • automated test results were presented for projects 1 & 2 Apr 98 EEE 599 - Weyuker Presentation 30
Project 1 Results • telephone call rerouter software • test results accurately predicted system performance • testing revealed a serious architectural problem Apr 98 EEE 599 - Weyuker Presentation 31
Project 2 Results • 800 number call traffic monitor • led to better requirements analysis • used automated test case generation exclusively, replacing hand-crafting • more test cases with more variety • uncovered four potentially severe faults • minimal additional cost Apr 98 EEE 599 - Weyuker Presentation 32
Reliability Testing of Rule. Based Systems (Sep 96) • applies same automatic test case generation algorithm to an industrial network surveillance system • system was a reimplementation, so a large body of operational data was available from the original system • system designed to run 24 hours/day; 365 days/year Apr 98 EEE 599 - Weyuker Presentation 33
Approach • system has very large state space • 355 days of operational data from 93 switches were used to identify 857 states entered • 0. 999 coverage required testing 843 states Apr 98 EEE 599 - Weyuker Presentation 34
State Testing Requirements Apr 98 EEE 599 - Weyuker Presentation 35
Findings • historical data is useful for selecting test cases based on state probabilities • selection algorithm must consider – reliability requirements – size of the (limited) state space – comprehensiveness of individual test cases Apr 98 EEE 599 - Weyuker Presentation 36
Summary • change of focus from theory to application • extension from telecom to general application • accomplished, respected, and prolific researcher in software testing and reliability Apr 98 EEE 599 - Weyuker Presentation 37
Скачать презентацию Software Testing A Selection of Papers by Elaine
36f19bcc17c32af9f45ded9c69630285.ppt