Automated Tools for Software Reliability Suhabe Bugrara suhabe@stanford. edu Stanford University
Problem • • 80% of development cost on identifying and correcting defects Software errors cost US economy $60 billion annually (0. 6% of GDP)
Manual Testing • • Traditional approach to quality assurance Expensive Time consuming Not systematic Difficult to quantify effectiveness of test suite Cannot make any guarantees about reliability Insufficient for safety critical systems
Automated Tools • • Programs to find defects in programs Automated Systematic Easy to quantify effectiveness Provide guarantees about reliability Sometimes expensive (for now…) Sometimes time consuming (for now…)
Program Analyzers Complete Undecidable Sound • Reports all errors • Reports no false alarms Decidable Unsound • May not report all errors • Reports no false alarms Incomplete Decidable • Reports all errors • May report false alarms Decidable • May not report all errors • May report false alarms
Static Driver Verifier • • • Program analyzer for API usage rules Developed by Microsoft Research Applied to device drivers in Windows Sound: reports all possible errors Incomplete: may report false alarms
SDV: Overview 1. 2. 3. 4. 5. 6. 7. Write API usage rule specification Instrument program with usage checks Abstract program Check abstraction for errors If error found, see if error is false alarm If false alarm, refine abstraction If not false alarm, report error as bug
API Usage Rules • Ex. locks are alternatingly acquired and released
API Usage Rules • Ex. locks are alternatingly acquired and released • Expressed as finite state machine – States = {locked, unlocked, error} – Transitions = {acquire(), release()}
API Usage Rules • Ex. locks are alternatingly acquired and released • Expressed as finite state machine – States = {locked, unlocked, error} – Transitions = {acquire(), release()} unlocked release(); acquire(); release(); error locked acquire();
state { enum { Unlocked=0; Locked=1} state = Unlocked; } Ke. Acquire. Spin. Lock. return { if (state == Locked) error(); else state = Locked; } Ke. Release. Spin. Lock. return { if (!(state == Locked)) error(); else state = Unlocked; }
enum {Unlocked=0, Locked=1} state = Unlocked void Ke. Acquire. Spin. Lock_return() { if (state == Locked) error(); else state = Locked; } void Ke. Release. Spin. Lock_return() { if (!(state == Locked)) error(); else state = Unlocked; }
1: void example() { 2: do { 3: Ke. Acquire. Spin. Lock(); 4: 5: n. Packets. Old = n. Packets; 6: req = dev. Ext->WLHV 7: if (req && req->status) { 8: dev. Ext->WLHV = req->Next 9: Ke. Release. Spin. Lock(); 10: 11: irp = req->irp; 12: if (req->status > 0) { 13: irp->Io. S. Status = SUCCCESS; 14: irp->Io. S. Info = req->Status; 15: } else { 16: irp->Io. S. Status = FAIL; 17: irp->Io. S. Info = req->Status; 18: } 19: Smart. Dev. Free. Block(req); 20: Io. Complete. Request(irp); 21: n. Packets++; 22: } 23: } while (n. Packets!=n. Packets. Old); 24: Ke. Release. Spin. Lock(); 25: 26: }
enum {Unlocked=0, Locked=1} state = Unlocked void Ke. Acquire. Spin. Lock_return() { if (state == Locked) error(); else state = Locked; } void Ke. Release. Spin. Lock_return() { if (!(state == Locked)) error(); else state = Unlocked; }
1: void example() { Program 2: do { 3: Ke. Acquire. Spin. Lock(); 4: Ke. Acquire. Spin. Lock_return(); 5: n. Packets. Old = n. Packets; 6: req = dev. Ext->WLHV 7: if (req && req->status) { 8: dev. Ext->WLHV = req->Next 9: Ke. Release. Spin. Lock(); 10: Ke. Release. Spin. Lock_return(); 11: irp = req->irp; 12: if (req->status > 0) { 13: irp->Io. S. Status = SUCCCESS; 14: irp->Io. S. Info = req->Status; 15: } else { 16: irp->Io. S. Status = FAIL; 17: irp->Io. S. Info = req->Status; 18: } 19: Smart. Dev. Free. Block(req); 20: Io. Complete. Request(irp); 21: n. Packets++; 22: } 23: } while (n. Packets!=n. Packets. Old); 24: Ke. Release. Spin. Lock(); 25: Ke. Release. Spin. Lock_return(); 26: } A
SDV: Abstraction • Construct abstraction B of original program A – Over-approximates reachability • If error() is reachable in A, then it is also reachable in B – This characteristic makes SDV sound • If error() is reachable in B, then it may not be reachable in A – This characteristic makes SDV incomplete • Check abstraction B for any errors
Reachable States real bug! Abstraction B error Original A Sound: If A has error, then B has error
false alarm! Reachable States Abstraction B error Original A Incomplete: If B has error, then A may not have error
bool b 1; Abstract state == Locked with b 1 = false; void Ke. Acquire. Spin. Lock_return() { if (b 1) error(); else b 1 = true; } void Ke. Release. Spin. Lock_return() { if (!(b 1)) error(); else b 1 = false; }
1: void example() { 2: do { 3: ; 4: Ke. Acquire. Spin. Lock_return(); 5: ; 6: ; 7: if (Sdv. Make. Choice()) { 8: ; 9: ; 10: Ke. Release. Spin. Lock_return(); 11: ; 12: if (Sdv. Make. Choice()) { 13: ; 14: ; 15: } else { 16: ; 17: ; 18: } 19: ; 20: ; 21: ; 22: } 23: } while (Sdv. Make. Choice()); 24: ; 25: Ke. Release. Spin. Lock_return(); 26: } Program B
1: void example() { 2: do { 3: ; 4: Ke. Acquire. Spin. Lock_return(); 5: ; 6: ; 7: if (Sdv. Make. Choice()) { 8: ; 9: ; 10: Ke. Release. Spin. Lock_return(); 11: ; 12: if (Sdv. Make. Choice()) { 13: ; 14: ; 15: } else { 16: ; 17: ; 18: } 19: ; 20: ; 21: ; 22: } Error trace 23: } while (Sdv. Make. Choice()); found! 24: ; 25: Ke. Release. Spin. Lock_return(); 26: }
1: void example() { 2: do { 3: Ke. Acquire. Spin. Lock(); 4: Ke. Acquire. Spin. Lock_return(); 5: n. Packets. Old = n. Packets; 6: req = dev. Ext->WLHV 7: if (req && req->status) { 8: dev. Ext->WLHV = req->Next 9: Ke. Release. Spin. Lock(); 10: Ke. Release. Spin. Lock_return(); 11: irp = req->irp; 12: if (req->status > 0) { 13: irp->Io. S. Status = SUCCCESS; 14: irp->Io. S. Info = req->Status; 15: } else { 16: irp->Io. S. Status = FAIL; 17: irp->Io. S. Info = req->Status; 18: } 19: Smart. Dev. Free. Block(req); 20: Io. Complete. Request(irp); 21: n. Packets++; 22: } But, no bug 23: } while (n. Packets!=n. Packets. Old); original 24: Ke. Release. Spin. Lock(); 25: Ke. Release. Spin. Lock_return(); program! 26: } in
1: void example() { Program 2: do { 3: ; 4: Ke. Acquire. Spin. Lock_return(); 5: b 2 = false; 6: ; 7: if (Sdv. Make. Choice()) { 8: ; 9: ; 10: Ke. Release. Spin. Lock_return(); 11: ; 12: if (Sdv. Make. Choice()) { 13: ; 14: ; 15: } else { 16: ; 17: ; 18: } 19: ; 20: ; 21: b 2 = !b 2 ? true : Sdv. Make. Choice(); 22: } 23: } while (b 2); 24: ; 25: Ke. Release. Spin. Lock_return(); 26: } C
Reachable States error Abstraction B Refined C Original false alarm no longer reported! A
SDV: Summary 1. 2. 3. 4. 5. 6. 7. Write API usage rule specification Instrument program with usage checks Abstract program Check abstraction for errors If error found, see if error is false alarm If false alarm, refine abstraction If not false alarm, report error as bug
Soundness • Assume memory safety – No buffer/integer overflows – Safe memory management – No null pointer dereferences • Oversimplified harness – Use stubs to model calls into OS procedures – Stubs may not represent all behavior
Research Challenges in Verification • • Eliminate assumption of memory safety Eliminate false alarms Scale to the entire operating system Verify more complicated properties – prove consistency of file system data structures
Program Analyzers Complete Undecidable Sound • Reports all errors • Reports no false alarms Decidable Unsound • May not report all errors • Reports no false alarms Incomplete Decidable • Reports all errors • May report false alarms Decidable • May not report all errors • May report false alarms
EXE • Automatically generate test cases that explore important program paths • Developed by Dawson Engler’s group • Bug finding tool • Unsound: may not report all errors • Complete: never reports false alarms
int bad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; }
int bad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; }
int bad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; } (x >= INT_MIN) && (x <= INT_MAX) && (x < 0) && (ret = -x) find a solution using an automatic constraint solver… x = -1
int bad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; } (x >= INT_MIN) && (x <= INT_MAX) && (x >= 0) && (x = 12345678) && (ret = -x) find a solution using an automatic constraint solver… x = 12345678
int bad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; } (x >= INT_MIN) && (x <= INT_MAX) && (x >= 0) && (x != 12345678) && (ret = x) find a solution using an automatic constraint solver… x=4
int bad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; } EXE automatically generated test cases for each path… x = -1 x = 12345678 x = 4
int bad_abs (int x) { if (x < 0) return –x; if (x == 12345678) return –x; return x; }
1: int 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: } symbolic_bad_abs (int x) { add_constraints(x >= INT_MIN, x <= INT_MAX); ret = new symbol; if (fork() == child) { add_constraints(x < 0, ret = -x); return ret; //(x >= INT_MIN) && (x <= INT_MAX) && (x < 0) && (ret = -x) } else add_constraints(x >= 0); if (fork() == child) { add_constraints(x = 12345678, ret = -x); return ret; //(x >= INT_MIN) && (x <= INT_MAX) && (x >= 0) && (x = 12345678) // && (ret = -x) } else add_constraints(x != 12345678); add_constraints(ret = x); return ret; //(x >= INT_MIN) && (x <= INT_MAX) && (x >= 0) && (x != 12345678) && (ret = x)
1: int 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: } main (void) { unsigned i, t, a[4] = { 1, 3, 5, 2}; make_symbolic(&i); if (i >= 4) exit(0); char *p = (char *) a + i * 4; *p = *p – 1; t = a[*p]; t = t / a[i]; if (t == 2) assert(i == 1); else assert(i == 3);
Review • Why does SDV produce false alarms and EXE doesn’t? • Why use SDV, then?
Saturn • • • Large-scale program verification Developed by Alex Aiken’s group Sound: reports all errors Incomplete: may report false alarms Gives guarantees of reliability on systems as large as the Linux kernel with over 6. 2 million lines of code
Program Analyzers Complete Undecidable Sound • Reports all errors • Reports no false alarms Decidable Unsound • May not report all errors • Reports no false alarms Incomplete Decidable • Reports all errors • May report false alarms Decidable • May not report all errors • May report false alarms
Unchecked User Pointer Dereferences • Security property of operating systems • Two types of pointers in operating systems – kernel pointer: pointer created by the operating system – user pointer: pointer created by a user application and passed to the operating system via an entry point such as a system call • Must check that a user points into userspace before dereferencing it
Unchecked User Pointer Dereferences 1: static ssize_t read_port(…, char * __user buf, …) { 2: unsigned long i = *ppos; 3: char * __user tmp = buf; 4: 5: if (!access_ok(. . , buf, . . . )) //check 6: return -EFAULT; 7: 8: while (count-- > 0 && i < 65536) { 9: if (__put_user(inb(i), tmp) < 0) //deref 10: return -EFAULT; 11: i++; 12: tmp++; 13: } 14: 15: *ppos = i; 16: return tmp-buf; 17: }
Security Vulnerability • Malicious user could – Take control of the operating system – Overwrite kernel data structures – Read sensitive data out of kernel memory – Crash machine by corrupting data
Verifying the Security Property • Eliminate the need for annotations • Eliminate false positives • Provide guarantee that no security vulnerabilities of this kind are present
Security Verifier • Design a sound and incomplete verifier to prove statically that no unchecked user pointer dereferences exist
Security Verifier • • • Compute set of facts at each program point States = { user, checked, error } Facts are pairs of locations and states – (*v, user) signifies that v is a user pointer • Verify that program never in error state
Security Verifier • • • Pointer is in user state if created by user application Pointer is in checked state if access_ok applied Pointer is in error state if dereferenced when 1. Pointer is in user state, AND 2. Pointer is NOT in checked state
Example 1: 2: int sys_call (int *u, int cmd) { //u is user pointer int x; 3: 4: if (cmd == 1) { 5: if (!access_ok(u)) { 6: //check u return –ERR; 7: } 8: } 9: … 10: if (cmd == 1) 11: 12: x = *u; … //dereference u
One Possible Approach 1: 2: int sys_call (int *u, int cmd) { int x; 3: 4: (*u, user) if (cmd == 1) { 5: if (!access_ok(u)) { 6: (*u, user) return –ERR; 7: } 8: } 9: … 10: if (cmd == 1) 11: 12: (*u, user) x = *u; … (*u, user) (*u, checked) (*u, user) lost precision! (*u, user) (*u, error) emit warning! …, but, procedure does not contain any vulnerabilities!
Path Sensitivity • Ability to reason about branch correlations • Important for reducing false positive rate • Programs use substantial amount of branch correlation in practice
Example 1: 2: int sys_call (int *u, int cmd) { //u is user pointer int x; 3: 4: if (cmd == 1) { 5: if (!access_ok(u)) { 6: //check u return –ERR; 7: } 8: } 9: … 10: if (cmd == 1) 11: 12: x = *u; … //dereference u
Path Sensitivity 1: 2: int sys_call (int *u, int cmd) { //u is user pointer int x; 3: 4: if (cmd == 1) { 5: if (!access_ok(u)) { 6: //check u return –ERR; 7: } 8: } 9: … 10: if (cmd == 1) 11: 12: x = *u; //dereference u … Valid Path
Path Sensitivity 1: 2: int sys_call (int *u, int cmd) { //u is user pointer int x; 3: 4: if (cmd == 1) { 5: if (!access_ok(u)) { 6: //check u return –ERR; 7: } 8: } 9: … 10: if (cmd == 1) 11: 12: x = *u; //dereference u … Valid Path
Path Sensitivity 1: 2: int sys_call (int *u, int cmd) { //u is user pointer int x; 3: 4: if (cmd == 1) { 5: if (!access_ok(u)) { 6: //check u return –ERR; 7: } 8: } 9: … 10: if (cmd == 1) 11: 12: x = *u; //dereference u … Valid Path
Path Sensitivity 1: 2: int sys_call (int *u, int cmd) { //u is user pointer int x; 3: 4: if (cmd == 1) { 5: if (!access_ok(u)) { 6: //check u return –ERR; 7: } 8: } 9: … 10: if (cmd == 1) 11: 12: x = *u; //dereference u … Invalid Path!
Path Sensitive Analysis 1: 2: int sys_call (int *u, int cmd) { int x; 3: 4: (*u, user) true if (cmd == 1) { 5: if (!access_ok(u)) { 6: (*u, user) true return –ERR; 7: } 8: } 9: … 10: if (cmd == 1) 11: 12: (*u, user) true x = *u; … (*u, user) true (*u, checked) cmd == 1 (*u, error) cmd == 1 && !(cmd == 1) && true false
Design of Saturn Security Verifier • Generate summary of behavior for each procedure with respect to calling context • Apply summary of callee at call site in caller • Repeatedly generate and apply summaries until a fixed point is reached
Experimental Setup • Implemented verifier for unchecked user pointer dereferences • Applied verifier to Linux 2. 6. 17. 1 built for x 86 architecture • 6. 4 million lines of code • Analyzed in 6 hours over 50 node cluster
Results • • • 91, 543 procedures 154 (. 17%) of procedures time out 627 system call parameters 867, 544 dereferences 15, 452 (1. 8%) of dereferences time out
Results • Verified automatically – 620 out of 627 system call arguments (99%) – 851, 914 out of 852, 092 dereferences (99. 96%) • Warnings – 7 warnings on system call arguments – 278 warnings on dereferences – 20 annotations required to verify
Saturn: Other Analyses • Null pointer dereferences bug finder – Found hundreds of bugs in systems code – Isil Dillig, Thomas Dillig, and Alex Aiken. Static Error Detection Using Semantic Inconsistency Inference, PLDI 2007 • • • Buffer overflow Safe casting Integer overflow Locking Safe memory management
Other Tools • • • BLAST CQual Metal Daikon Vault ESPX MOPS DART • • • CSSV Alloy e. Xplode Chord TVLA CCured Clouseau STe. P Prefix • Prefast • Failure Oblivious Computing
References • • A. Aiken et al. An Overview of the Saturn Project. PASTE 2007 T. Ball et al. Thorough Static Analysis of Device Drivers. Euro. Sys 2006 C. Cadar et al. EXE: Automatically Generating Inputs of Death. CCS 2006 C. Cadar et al. Execution Generated Test Cases: How to Make Systems Code Crash Itself. SPIN 2006 B. Hackett et al. Modular Checking for Buffer Overflows in the Large. ICSE 2006. J. Yang et al. Automatically generating malicious disks using symbolic execution. IEEE Security and Privacy 2006 Software Errors Cost U. S. Economy $59. 5 Billion Annually. NIST 2002. http: //www. nist. gov/public_affairs/releases/n 02 -10. htm
Скачать презентацию Automated Tools for Software Reliability Suhabe Bugrara suhabe stanford
3c7aff6738fa01c332ab08c5888d289d.ppt