Automated Whitebox Fuzz Testing Network and Distributed System

Automated Whitebox Fuzz Testing Network and Distributed System Security (NDSS) 2008 by Patrice Godefroid, Michael Y. Levin, and David Molnar Present by Diego Velasquez

Acknowledgments q q Figures are copy from the paper. Some slides were taken from the original presentation presented by the authors 2

Outline q Summary q q q q Review q q Goals Motivations Methods Experiments Results Conclusions Strengths Weakness Extensions Reference 3

Goals q q q Propose a novel methodology that performs efficiently fuzz testing. Introduce a new search algorithm for systematic test generation. Outcast their system SAGE (Scalable, Automated, Guided Execution) 4

Methods q Fuzz testing inserts random data to input of applications in order to find defects of a software system. Heavily used in Security testing. q q Pros: Cost effective and can find most of known bugs Cons: It has some limitations depending on some types of branches, for example on project 2 in order to find bug # 10 we need to execute the if statement below. if(address ==613 && value >= 128 && value<255)//Bug #7 printf("BUG 10 TRIGGERED); q Has (1 in 5000) * (128 in 2^32) in order to be executed if we know that is only 5000 addresses and value is a random 32 -bit input 5

Methods Cont. q Whitebox Fuzz Testing q Combine fuzz testing with dynamic test generation [2] q Run the code with some initial input q Collect constraints on inputs with symbolic execution q Generate new constraints q Solve constraints with constraint solver q Synthesize new inputs 6

Methods Cont. q The Search Algorithm figure 1 from [1] q q Black box will do poorly in this case Dynamic test could do better 7

Methods Cont. q Dynamic Approach q q q Input ‘good’ as example Collect constrain from trace Create a new path constraint Figure 2 from [1] 8

Methods Cont. q Limitations of Dynamic Testing q Path Explosion q q q Path doesn’t scale to large in realistic programs. Can be corrected by modifying the search algorithm. Imperfect Symbolic Execution q q Could be imprecise due to Complex program statements (arithmetic, pointer manipulation) Calls to OS have to be expensive in order to be precise 9

Methods Cont. q New Generation Search Algorithm Figure 3 and figure 4 from [1] q q A type of Bread First Search with heuristic to get more input test cases. Scores return the number of new test cases covered. 10

Methods Cont. q Summary of Generation Search Algorithm q q q Push input to the list Run&Check(input) check bugs in that input Traverse the list by selecting from the list base in score Expanded child paths and adding to the childlist Traverse childlist Run&Check, assigned score and add to list Expand Execution q q q Generates Path constrain Attempt to expand path constraints and save them Input. bound is used to limit the backtracking of each sub-search above the branch. 11

Experiments q q Can test any file-reading program running on Windows by treating bytes read from files as symbolic input. Another key novelty of SAGE is that it performs symbolic execution of program traces at the x 86 binary level FIGURE FROM [2] 12

Experiments Cont. q Sage advantages q Not source-based, SAGE is a machine-code-based, so it can run different languages. q Expensive to build at the beginning, but less expensive over time q Test after shipping, q q Since is based in symbolic execution on binary code, SAGE can detects bugs after the production phase Not source is needed like in another systems q SAGE doesn’t even need specific data types or structures not easy visible in machine code 13

Experiments Cont. q q MS 07 -017: Vulnerabilities in Graphics Device Interface (GDI) Could Allow Remote Code Execution. Test in different Apps such as image processors, media players, file decoders. [2] Many bugs found rated as “security critical, severity 1, priority 1”[2] Now used by several teams regularly as part of QA process. [2] 14

Experiments Cont. q More in MS 07 -017, figure below is from [2] left is input right is crashing test case RIFF. . . ACONLIST B. . . INFOINAM. . 3 D Blue Alternat e v 1. 1. . IART. . . . . 1996. . anih$. . . . . rate. . . . . seq. . . . . LIST. . framic on. . . 15 RIFF. . . ACONB B. . . INFOINAM. . 3 D Blue Alternat e v 1. 1. . IART. . . . . 1996. . anih$. . . . . rate. . . . . seq. . . . . anih. . framic on. . . Only 1 in 232 chance at random!

Results q Statistics from 10 hour searches on seven test applications, each seeded with a well formed input file. 16

Results q q Focused on the Media 1 and Media 2 parsers. Ran a SAGE search for the Media 1 parser with five “well-formed” media files, and five bogus files. Figure 7 from [1] 17

Results q Compared with Depth-First Search Method q DFS runs for 10 hours for Media 2 with wff-2 and wff-3, didn’t find anything GS found 15 crashes q Symbolic Execution is slow q Well formed input are better than Bogus files q Non-determinism in Coverage Results. q The heuristic method didn’t have too much impact q Divergences are common 18

Results q Most bugs found are “shallow” Figure from [2] 19

Conclusions q Blackbox vs. Whitebox Fuzzing q Cost/precision tradeoffs q q q Blackbox is lightweight, easy and fast, but poor coverage Whitebox is smarter, but complex and slower Recent “semi-whitebox” approaches q q Which is more effective at finding bugs? It depends… q q q Less smart but more lightweight: Flayer (taint-flow analysis, may generate false alarms), Bunny-the-fuzzer (taint-flow, source-based, heuristics to fuzz based on input usage), autodafe, etc. Many apps are so buggy, any form of fuzzing finds bugs! Once low-hanging bugs are gone, fuzzing must become smarter: use whitebox and/or user-provided guidance (grammars, etc. ) Bottom-line: in practice, use both! *Slide From [2] 20

Strengths q Novel approach to do fuzz testing q q Applied as a black box q q q Not source code was needed symbolic execution of program at the x 86 binary level Shows results comparing previous results q q Introduced new search algorithm that use codecoverage maximizing heuristic Test large applications previously tested found more bugs. Introduced a full system and applied the novel ideas in this paper 21

Weakness q The results were non-determinism q q Only focus in specific areas q q q Same input, program and idea different results. X 86 windows applications File manipulation applications Well formed input still some type of regular fuzzing testing SAGE needs help from different tools In my opinion the paper extends too much in the implementation of SAGE, and the system could of be too specific to Microsoft 22

Extensions q Make SAGE more general Easy to implement to another architectures q Use for another types of applications q Linux based applications q q Better way to create input files q q May be used of grammar Make the system deterministic q Having different results make me think that it could not be reliable. 23

Reference q q q [1] P Godefroid, MY Levin, D Molnar, Automated Whitebox Fuzz Testing, NDSS, 2008. [2] Original presentation slides www. truststc. org/pubs/366/15%20%20 Molnar. ppt [3] Wikipedia Fuzz testing http: //en. wikipedia. org/wiki/Fuzz_testing. 24

q Questions, Comments or Suggestions? 25