An Exploration of Errors in Webbased Applications in

An Exploration of Errors in Webbased Applications in the Context of Web-based Application Testing Ph. D Proposal Kinga Dobolyi May 2009 1

The shopping cart 2

The shopping cart 3

The shopping cart 4

What is going on • Problem: faults in web-based applications cause losses of revenue, and they are hard to test • Approach: study errors in web-based applications in the context of web testing • Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing 5

Outline • • • Introduction and motivation Thesis statement Background Goals and approaches Preliminary work Expected contributions 6

Motivation • Testing of web-based applications in particular deserves further examination due to economic considerations: – Monetary throughput: Backbone of ecommerce and communications businesses – Customers: low customer loyalty – Development: Companies are choosing not to test due to resource constraints 7

Motivation: e-commerce • Internet usage: 73% of people in the US in 2008 – Browsers are dominant application – $204 billion in Internet retail sales annually • Global online B 2 B transactions total several $trillions annually • One hour of downtime at Amazon. com cost $1. 5 million dollars • 70% of major online sites exhibit user-visible failures 8

Motivation: customers • Customer loyalty is notoriously low – Determined by the usability of the application [Offutt 2002] – Freedom and options 9

Motivation: customers • Lesson learned: web-based applications need to be well-designed and well tested • Are they? 10

Motivation: development • Technology challenges: – Heterogeneous, opaque components – Dynamic page content generation – Persistent state operated upon by concurrent, global users around the clock 11

Motivation: development • Web-based applications are often not tested – Enormous pressure to change • Short delivery times, high developer turnover rates, and quickly evolving user needs – No formal process model 12

Motivation: summary • Problem: faults in web-based applications cause losses of revenue, and they are hard to test • Approach: study errors in web-based applications in the context of web application testing • Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing 13

Thesis statement • Hypothesis: web-based applications have special properties that can be exploited to build tools and models that improve the current state of web application testing and development: – Tend to fail in predictable and similar ways – Human centric definition of acceptability 14

Thesis statement • Problem: faults in web-based applications cause losses of revenue, and they are hard to test • Approach: study errors in web-based applications in the context of web testing • Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing 15

Background: testing techniques • Non-functional (static) validation – Server load testing – Link testing – HTML/spelling validation • Modeling approaches • Capture-replay – User session-based testing 16

Background: oracles • Oracles (oracle-comparator) 1 < <P>The same table could be indented. 2 < <TABLE border="1"> 3 --4 > <p>The same table could be indented. </p> 5 > <table border="1" summary=""> – False positives from diff-like tools – Want precise comparators 17

Background: automation • Automation – Test case generation: Veri. Web, PHP – Test case replay • URL + post-data – Failure detection 18

Background: metrics • How do we measure success? – Code coverage – Fault seeding • Human • Automatic – Cost • How do we know these are indicative of the real world? 19

Background: fault definition • Defining an error: – “the inability to obtain and deliver information, such as documents or computational results, requested by web users. ” [Ma & Tian 2003] • Fault taxonomies – Figure from [Marchetto et al 2007] 20

Background: challenges • Functional validation remains a challenge – Regression testing should be more precise and automatic • We do not know if test suite efficacy metrics are indicative of the real world – We should examine the severity of uncovered faults 21

Goals and approaches • Problem: faults in web-based applications cause losses of revenue, and they are hard to test • Approach: study errors in web-based applications in the context of web testing • Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing 22

Goals and approaches • I propose to: – Model errors in web-based applications • Identify them more accurately • Automate the oracle-comparator process – Make web testing more cost-effective • Devise a model of fault severity that will guide test case design, selection, and prioritization • Validate or refute the current underlying assumption that all faults are equally severe in fault -based testing 23

Goals and approaches: Goals • Reduce the cost of regression testing web-based applications – Use special structure of web-based application output to precisely identify errors • Automate web-based application regression testing – Unrelated web-based applications tend to fail in similar ways • Understand customer-perceived severities of web application errors. 24

Goals and approaches: Goals • Formally ground the current state of industrial practice – Validate or refute fault injection as a standard for measuring web application test suite quality • Understand how to avoid high-severity faults during web application design and development • Reduce the cost of regression testing web applications by exposing high-severity faults – Test case design, selection, and prioritization (test suite reduction) 25

Goals and approaches: Outline 26

Goals and approaches: Step 1 – oracle-comparator • Construct a precise oracle-comparator that uses the tree-structured nature of XML/HTML output and other features – Model errors on a per-project basis – Semantic distance metric to reduce false positives 27

Goals and approaches: Step 2 – automated oracle-comparator • Exploit the similar way in which web applications fail to avoid the need for human annotations in training a precise oracle-comparator – Train a precise oracle-comparator on data from other, unrelated web applications – Use fault injection to improve the results when necessary 28

Goals and approaches: Step 3 – human study • Conduct a human study of real-world fault severity to identify a model of fault severity – Severities different than self-reported in bug repositories – Screenshots of current-next idiom – Also survey developers 29

Goals and approaches: Step 4 – fault seeding validation • Compare the severities of real-world faults to seeded faults using human data (validate fault seeding) – The severities of seeded errors have uniform distributions? – The severity distribution of seeded errors matches the distribution of real-world errors, according to the results of the survey from Step 3? 30

Goals and approaches: Step 5 – software engineering guidelines • Identify underlying technologies and methodologies that correlate with highseverity faults – As an alternative to testing – Tie high severity errors to underlying code, components, programming languages, and software engineering practices 31

Goals and approaches: Step 6 – testing techniques • Identify testing techniques to maximize return on investment by targeting highseverity faults – Introduce a new metric for the (web application) test suite reduction research community 32

Preliminary Work: Step 1 • Step 1: Construct a precise oracle-comparator using tree structured XML/HTML output and other features – Multiple versions of 10 open-source benchmarks – 7154 pairs of oracletestcase output, 919 of which labeled as “possibly an error” 33

Preliminary Work: Step 1 • Evaluation: F-measure (precision and recall) using our model 34

Preliminary Work: Step 1 • Longitudinal study to measure effort saved – Calculate cost of looking : cost of missing – Low ratio means we are saving effort 35

Preliminary Work: Step 2 • Step 2: Exploit similarities in web application failures to avoid human annotations when training a precise oracle -comparator – Same setup as Step 1 – Use existing, annotated pairs of test-oracle output from unrelated applications to train a comparator for the application at test 36

Preliminary Work: Step 2 • Evaluation: measure precision and recall 37

Preliminary Work: Step 2 • Use fault seeding to introduce projectspecific faults into the training data set 38

Preliminary Work: Step 3 • Step 3: Model real-world fault severity based on a human study. – Collect 400 real-world faults • Evaluation: have subjects use the model to classify faults 39

Preliminary Work: Step 4 • Step 4: Compare the severities of realworld faults to seeded faults using human data. – Test same human subjects with 200 + 200 manually-injected and automatically-injected faults – Conduct a survey of developers for fault severity 40

Preliminary Work: Step 5 • Step 5: Identify technologies and methodologies that correlate with high-severity faults • Do high severities correlate with: – – – Programming Language Level in three-tier architecture COTS component User error (usability issue) Type of error (business logic, resource allocation, syntax error, etc) – Fault taxonomies (existing) – Surface features of visible output: white screens, stack traces, misspellings, formatting errors • Evaluation: developer survey (time permitting) 41

Preliminary Work: Step 6 • Step 6: Identify testing techniques to target high-severity faults – Targets testing – Assign a testcase a severity rating a priori • Evaluation: compare the severity of faults exposed with our technique versus other test suite reduction approaches 42

Expected Contributions • Problem: faults in web-based applications cause losses of revenue, and they are hard to test • Approach: study errors in web-based applications in the context of web testing • Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing 43

Expected Contributions • Reduce the cost of regression testing by constructing a precise oracle-comparator • Develop a model of customer-perceived severities of web application faults • Validate or refute fault injection as a standard for measuring web application test suite quality • Propose new software engineering guidelines for web application development and testing 44

Questions? 45

Original Contributions • Fault Severity Model – Severity has not been studied in this domain, and customers are an integral part of these applications – Providing a new metric to the research community – Validate/refute fault seeding • Precise-oracle comparators – First to use different versions of benchmarks – Can be completely automated – XML and HTML 46

Expected Impact • Fault Severity Model – Can be applied to testing techniques in this field to make them financially feasible for developers – Change the way in which test suite efficacy is measured – Potentially impact web site design as usability issues may become more evident • Precise oracle-comparator – Automation makes it much more feasible for adoption than existing techniques – Potentially allow companies to conduct regression testing if they were not testing beforehand 47

Timeline • Steps 1 and 2: precise comparators – Completed, 2 papers under submission • Steps 3 and 4: human study – Data collection completed, analysis under way for submission of 1 paper – Expected completion by September • Step 5: software engineering guidelines – Expected completion by October – Expected 0. 5 paper • Step 6: testing according to fault model – Expected completion by February – Expected 1 paper 48