An Industrial Case Study of Bypass Testing on

An Industrial Case Study of Bypass Testing on Web Applications Jeff Offutt Software Engineering George Mason University Fairfax, VA USA www. cs. gmu. edu/~offutt/ offutt@gmu. edu Expanded version of a talk given at the first International Conference on Software Testing, Verification and Validation Joint research Dr. Ye Wu, Xiaochen Du, Hong Huang, Vasileios Papadimitriou, Qingxiang Wang and Joann J. Ordille of Avaya Labs Research Based on papers in ISSRE 2004 and ICST 2008

Outline of Talk • Motivation • Applying bypass testing – early results • Automating bypass testing • Real-world examples • Industrial case study • Conclusions and future work ICST 2008 © Jeff Offutt 2

Web Application Input Validation Check data Sensitive Data Bad Data • Corrupts data base • Crashes server • Security violations Client Malicious Data Can “bypass” data checking Server

Deploying Software • Bundled : Pre-installed on computer • Shrink-wrap : Bought and installed by end-users • Contract : Purchaser pays developer to develop and install, usually for a fixed price • Embedded : Installed on a hardware device, usually with no direct communication with user • Web : Executed across the Internet through HTTP © Jeff Offutt, 2004 4

Problem Parameters • HTTP is a stateless protocol – Each request is independent of previous request • • Servers have little information about where a request comes from Web site software is extremely loosely coupled – Coupled through the Internet – separated by space – Coupled to diverse hardware devices – Written in diverse software languages ICST 2008 © Jeff Offutt 5

Bypass Testing • “bypass” client-side constraint enforcement • Bypass testing constructs tests to intentionally violate constraints : – Eases test automation – Validates input validation – Checks robustness – Evaluates security ICST 2008 © Jeff Offutt 6

Simple Example Web Page User Name: Age: Version to purchase: Small Large $150 ICST 2008 Medium $250 $500 © Jeff Offutt 7

Proper Behavior Invalid data, please correct … User Name: Alan<Turing Username should be plain text only. Age: 500 Age should be between 18 and 150. Version to purchase: Small Large $150 ICST 2008 Medium $250 $500 © Jeff Offutt 8

<FORM > Abbreviated HTML <INPUT Type=“text” Name=“username” Size=20> <INPUT Type=“text” Name=“age” Size=3 Maxlength=3> <P> Version to purchase: … <INPUT Type=“radio” Name=“version” Value=“ 150” Checked> <INPUT Type=“radio” Name=“version” Value=“ 250”> <INPUT Type=“radio” Name=“version” Value=“ 500”> <INPUT Type="submit" on. Click="return check. Info(this. form)"> <INPUT Type=“hidden” is. Logged. In=“no”> </FORM> ICST 2008 © Jeff Offutt 9

Bypass Behavior • Extremely loose coupling … • combined with the stateless protocol … • allows users to easily bypass client-side checking : Users can save and modify the HTML ICST 2008 © Jeff Offutt 10

Saved & Modified HTML <FORM > <INPUT Type=“text” Name=“username” Size=20> <INPUT Type=“text” Name=“age” Size=3 Maxlength=3> <P> Version to purchase: … Allows an input with arbitrary age, no checking, cost=$25 … ‘<‘ can crash an XML parser <INPUT Type=“radio” Name=“version” Value=“ 150”> Text fields can have SQL statements <INPUT Type=“radio” Name=“version” Value=“ 250”> <INPUT Type=“radio” Name=“version” Value=“ 500” Checked> 25 <INPUT Type="submit" on. Click="return check. Info(this. form)"> <INPUT Type=“hidden” is. Logged. In= yes > “no” </FORM> ICST 2008 © Jeff Offutt 11

SQL Injection User Name: turing’ OR ‘ 1’=‘ 1 Password: enigma’ OR ‘ 1’=‘ 1 Original SQL: SELECT username FROM adminuser WHERE username='turing' AND password ='enigma' “injected” SQL: SELECT username FROM adminuser WHERE username='turing‘ OR ‘ 1’ = ‘ 1’ AND password ='enigma‘ OR ‘ 1’ = ‘ 1’ © Jeff Offutt, 2004 12

Applying Bypass Testing Validating input data on the client is like asking your opponent to hold your shield in a sword fight • Analyze HTML to extract each form element • Model constraints imposed by HTML and Java. Script • Rules for data generation : – From client-side constraints – Typical security violations – Common input mistakes ICST 2008 © Jeff Offutt 13

Example Client-Side Constraint Rules • Violate size restrictions on strings • Introduce values not included in static choices – Radio boxes – Select (drop-down) lists • • Violate hard-coded values Use values that Java. Scripts flag as errors Change “transfer mode” (get, post, …) Change destination URLs © Jeff Offutt, 2004 14

Example Server-Side Constraint Rules • Data type conversion • Data format validation • Inter-field constraint validation • Inter-request data fields (cookies, hidden) © Jeff Offutt, 2004 15

Example Security Violation Rules Potential Illegal Character Symbol Empty String Commas , Single and double quotes ’ or ” Tag symbols < and > Directory paths . . / Strings starting with forward slash / Strings starting with a period . Ampersands & Control character NIL, newline Characters with high bit set 254 and 255 Script symbols <javascript> or <vbscript> © Jeff Offutt, 2004 16

First Example Cyber. Chair • Cyber. Chair : Web-based conference management – – • www. cyberchair. org ICSE , ICST, ISSRE, ICSM, … Bypass testing found 5 types of faults 1. 2. 3. 4. 5. Submission without authentication Unsafe use of hidden form field Disclosing information (program crashes) Lack of validation of file type Allows papers of negative length © Jeff Offutt, 2004 17

Automating Bypass Testing • Autobypass : A web application that accepts a URL and generates input data for the HTML form fields – Also accepts any needed login data – MS thesis by Vasileios Papadimitriou • Built on top of Http. Unit – Parses HMTL pages – Identifies forms and their fields – Creates bypass test cases – Submits test cases to the application’s server ICST 2008 © Jeff Offutt 18

Types of Client Input Validation • Client side input validation is performed by HTML form controls, their attributes, and client side scripts that access DOM • Validation types are categorized as HTML and scripting – HTML supports syntactic validation – Client scripting can perform both syntactic and semantic validation HTML Constraints • Length (max input characters) • Value (preset values) • Transfer Mode (GET or POST) • Field Element (preset fields) • Target URL (links with values) ICST 2008 Scripting Constraints • Data Type (e. g. integer check) • Data Format (e. g. ZIP code format) • Data Value (e. g. age value range) • Inter-Value (e. g. credit # + exp. date) • Invalid Characters (e. g. <, . . /, &) © Jeff Offutt 19

Example Interface: yahoo registration form Preset Transfer Mode in form definition (HTML) Preset Values (HTML) Limited Length (HTML) URL with preset Values (HTML) Inter Value validation (script) Data Value, Type, & Format validation (script) ICST 2008 © Jeff Offutt Preset No of Fields (HTML) 20

Test Value Selection • Challenge: – How to automatically provide effective test values? • Semantic Domain Problem (SDP) – Values within the application domain are needed – Enumeration of all possible test values is inefficient • Possible Solutions – – Random Values (ineffective – lots of junk) Automatically generated values (very hard) Taking values from session log files (feasible but incomplete) Tester input (feasible) • Auto. Bypass uses an input domain created by parsing the interface and tester input ICST 2008 © Jeff Offutt 21

Auto. Bypass • Auto. Bypass steps (the big picture) Parse Interface Set Default Values Generate Test Cases & Run Tests Review Results • All HTML violation rules are used to generate test cases • First version of Auto. Bypass does NOT automatically violate scripting validation, but : – Auto. Bypass behaves as a browser with scripts disabled – Tester can provide test inputs that will bypass scripting validation. ICST 2008 © Jeff Offutt 22

Real-World Examples atutor. ca nytimes. com Atalker Us-markets demo. joomla. or mutex. gmu. edu Poll, Users Login form php. My. Admin Main page, Set Theme, SQL Query, DB Stats yahoo. com Notepad, Composer, Search reminder, Weather Search barnesandnoble. com bankofamerica. com ATM locator, Site search comcast. com Service availability ecost. com Detail submit, Shopping cart control google. com Froogle, Language tools Pure Cart manager, black-box pageflakes. com Book search/results testing Submit Request Info, New user Registration means amazon. com myspace. com source (or dispatch, Item permission)wellsfargolife. com no needed ! Events & Music brainbench. com Search ICST 2008 Handle buy © Jeff Offutt Quote search 23

Classifying Output Responses • (V) Valid Responses : invalid inputs are adequately processed by the server (V 1) Server acknowledges the invalid request and provides an explicit message regarding the violation (V 2) Server produces a generic error message (V 3) Server apparently ignores the invalid request and produces an appropriate response (V 4) Server apparently ignores the request completely • (F) Faults & Failures : invalid inputs that cause abnormal server behavior (typically caught by web server when application fails to handle the error) • (E) Exposure : invalid input is not recognized by the server and abnormal software behavior is exposed to the users ICST 2008 © Jeff Offutt 24

Research to Practice Naw - I already don’t plow as good as I know how. . . They’re teaching a new way of plowing over at the Grange tonight - you going? “Knowing is not enough, we must apply. Willing is not enough, we must do. ” Goethe ICST 2008 © Jeff Offutt 26

Industrial Case Study • Inventions from scientists are slow to move into industrial practice • We wanted to investigate whether the obstacles are : – Technical difficulties of applying to industrial use – Social barriers – Business constraints • Tried to technology transition bypass testing to the research arm of a software company ICST 2008 © Jeff Offutt 27

Avaya’s NPP Technology • Avaya Labs Research creates research prototypes of software systems, then turns successful prototypes over to product groups • NPP : Notification Preference Portal – Users specify how and when they should be contacted – Types include phone, email and SMS – Contacts an be made in parallel or sequentially • Used to notify users of events • This study was part of system testing – NPP is now in production ICST 2008 © Jeff Offutt 28

NPP Design and Implementation • NPP is a highly user interactive web application • Uses many screens • Javascript is used on the client to : – Validate inputs – Dynamically modify screen by manipulating DOM – Encode input data into XML before sending to the server • The second two uses necessitated changes to how bypass testing was applied ICST 2008 © Jeff Offutt 29

NPP Bypass Tests • Bypass testing analyzes HTML (statically) and generates inputs that violate input constraints • The extensive modification of the HTML DOM meant HTML could not be analyzed statically • Instead, the input requirements of the server software were identified, and tests encoded in XML – A special-purpose tool was written to convert XML tests into Html. Unit tests ICST 2008 © Jeff Offutt 30

NPP Testing Results • Six NPP screens were tested • Tests are invalid inputs – exceptions are expected • Effects on back-end were not checked – Failure analysis just based on response screens Web Screen Tests Failing Tests Unique Failures Points of Contact 42 23 12 Time Profile 53 23 23 Notification Profile 34 12 6 Notification Filter 26 16 7 5 1 1 24 17 14 184 92 63 Change PIN Create Account TOTAL ICST 2008 © Jeff Offutt 31

Types of Faults • Invalid data saved into the database – These lead to more visible failures later – Example : An invalid password was accepted, but the account could not subsequently be used – Example : Subsequent messages could not be sent to invalid contacts • No response at all – Probably a software component failed – Database or web server sometimes crashed • Exposure errors – Internal exception message sent in the response screen ICST 2008 © Jeff Offutt 32

Conclusions • Bypass testing worked very well in an industrial context – There is no technical obstacle to adoption – Source is not needed • Even hand generation of tests was quite cheap in comparison with other methods – There is no valid business barrier • Most problems are unlikely with non-malicious users and a correct implementation – But client-side validation is notoriously error-prone We conclude the primary obstacle is social ICST 2008 © Jeff Offutt 33

Future Work • A major observability problem with web application testing is detecting invalid database values – A comprehensive valid data model could allow database auditors to be developed • Javascript needs to be fully parsed analyzed – Implement scripting violation rules • Widen the scope of testing from a form/ to a site – Test sequence of events – Application level Input Domain • Explore possibilities for automated response evaluation • Ajax allows client-server messages to be sent asynchronously through message passing – This introduces more controllability and observability problems ICST 2008 © Jeff Offutt 34