Techniques for Validating the Security Quality of Infrastructure

Techniques for Validating the Security Quality of Infrastructure Software John D. Mc. Gregor johnmc@cs. clemson. edu

Outline Motivation n Proposed strategy n Detailed actions n Conclusion n

Theme – Life Cycle Threats n There is not a means for automated testing of large software, both static and mobile code, to detect, identify malicious code, sleeper codes, and exploitable vulnerabilities and to determine and understand the potential impact on the lifecycle of the codes. Current testing approaches are largely manual rather than automated. CSIIR Workshop Themes document

A Recommended Strategy n n Software security vulnerabilities are often caused by defective specification, design, and implementation. …require that software be designed with security at the very heart of the design process… Establish a security verification and validation program to evaluate different software development processes and practices for effectiveness in producing secure software. Certify those processes demonstrated to be effective for producing secure software. Security Across the Software Development Lifecycle Task Force

Recommended Practices n n n Statistical testing - Usage based testing permits valid statistical estimation of quality with respect to all the executions not tested and tends to find any remaining high-failure-rate defects early. Production testing - Two strategies are testing security functionality with standard functional testing techniques, and risk-based security testing based on attack patterns and threat models. A good security test plan (with traceability back to requirements) uses both strategies. Process models - organizations can use the goals and attributes defined in process models as high-level guides for defining and improving their management and engineering processes in the ways they feel are most appropriate for them. Security Across the Software Development Lifecycle Task Force

SEI The security of a software-intensive system is directly related to the quality of its software 1. n n 1 Over 90% of software security incidents are caused by attackers exploiting known software defects. Analysis of 45 e-business applications showed that 70% of security defects were design defects. Experienced and capable software engineers inject, on average, one defect every nine lines of code. A one million line of code systems typically contains 1, 000 -5, 000 defects when shipped. http: //www. sei. cmu. edu/tsp-security. html

A Final Source § “One of the key things that developers can do to help secure their systems is to write code that can withstand attack and use security features properly. ” § Defend Your Code with Top Ten Security Tips Every Developer Must Know § 8 out of 10 tips are directly programming issues http: //msdn. microsoft. com/security/securecode/default. aspx ? pull=/msdnmag/issues/02/09/securitytips/default. aspx

Basic Assumptions Testing early is cheaper than testing later. n System test of a system with j modules and k experiments (iterations or different implementations for a given interface) results in the following cost of testing Where ct is the cost per test. § Module-level tests with j modules and k experiments results in Baldwin and Clark. Design Rules The Power of Modularity, volume 1, MIT Press, 2000.

Basic assumptions - 2 Though estimates vary, the cost of removing defect increases dramatically later in the life cycle. Design IBM 1 Development Test 60 1 Remus Code 1. 5 1 Boehm Design Inspection 15 to 40 20 Acceptance Production 100 30 to 70 40 to 1000 82 Ackerman 1 2 to 10 Russell 1 2 to 4 30

Premise n n n Our premise is that poorly written software will contain more vulnerabilities than well written software where the security quality attribute is a design driver. Current views of security often take a defensive approach. Some of the security infrastructure even adds to the security risk due to the complexity it adds to the product. We propose an offensive approach in which security is a key design driver and a priority throughout the development process.

Context Domain Expert User: Frequency: See Also: Use Case Modeling Systems Engineeri ng Class Name Responsibility Super Class Collaborator Requirements model inspection Domain Analysis Client Analysis model inspection Application Analysis Incremental Integration and System Testing Class Specification Class Design Class Derivation Class Reuse Use Case Name: User: Frequency: See Also: Architectural Design Analysis model inspection Refinement Model Testing Class Delivery Architecture inspection lle r Application Use Cases Implementation ntr o A chain of quality should be threaded through the entire process so that validation is most effective and efficient. Co n Use Case Name: Use: Requirements View Class Development Cluster design inspection

A Proposed Strategy n n n Develop method engineering tactics and guidelines that enhance the security quality of the software through improved processes. Structure architecture evaluation techniques to focus on security by searching for static security patterns. Discover and capture test patterns that correspond to dynamic security patterns. Develop focused test techniques to effectively explore security test patterns while reducing the test suite size. Create a defect model for security that can be used to predict types and number of security vulnerabilities in scientific codes.

Action – Develop method engineering techniques n n n 1 Method engineers create custom-made processes to help a project achieve specific goals. The goal of being “secure” needs to be operationalized so that these engineers can assemble methods in ways that enhance the security of the software product built using their processes. This task would involve extending the Software Process Engineering Meta. Model 1 (SPEM) standard to define security-specific constructs. The model would be automated using existing tools such as Meta. Edit+. Software Process Engineering Metamodel, version 1. 1, Object Management Group (OMG), 2005.

Action – Develop method engineering techniques - 2 n n n The security-oriented method fragments would be suitable to integrate into processes defined in the context of the SPEM. Process audits could evaluate the strength of the security aspect of the process, once it is explicitly embedded in the process, just as other qualities are validated. Deliverables: A process definition guide that would show to design security-centric development process fragments, an assembly guide, and an example process.

Action – Create architecture security analysis n n n Architects have techniques that they apply to an architecture in order to improve the behavior of the architecture with respect to classes of security threats 1. Assuring system survivability requires showing that the system architecture is adequately resilient to likely patterns of attack 2. One approach to architecture design is to identify quality attributes for the architecture and make design decisions that enhance the desirable qualities and degrade the less desirable ones. Architecture evaluation techniques such as the architecture trade-off analysis method (ATAM) can be used to focus architecture evaluations on the security quality through security-specific scenarios. One aspect of this task would be to develop a set of architecture-level security scenarios that would guide the evaluation of an architecture for the security quality. Security and Survivability Reasoning Frameworks and Architectural Design Tactics. CMU/SEI-2004 -TN-022 2 Architectural Refinement for the Design of Survivable Systems CMU/SEI-2001 -TN-008 1

Action – Create architecture security analysis - 2 n n This evaluation can be automated if the architects use a formal architecture description language (ADL). Our current implementation uses the Architectural Analysis and Design Language (AADL) from the Society of Automotive Engineers (SAE) and Eclipse plug-ins. A second part of the task could be to identify standard mechanisms for representing security in an architecture description so that security patterns could be automatically recognized. Deliverables: A set of validation scenarios and specific architecture tactics for recognizing security vulnerabilities at the architecture level

Current implementation as Eclipse plug-in for AADL

Action – Create architecture security analysis - 3 This analysis could also be conducted on pre/post-test basis n Use an architecture re-construction technique such as OAR 1 to extract the “as built” architecture from the implementation. n Compare to the “as designed” architecture to determine if changes have occurred n 1 Options Analysis for Reengineering (OAR): A Method for Mining Legacy Assets CMU/SEI-2001 -TN-013

Action – Discover test patterns for security n n n Test patterns are solutions to testing problems in context. They often correspond to design patterns. Test patterns are language independent Content of a test pattern 1 Problem - Description of pattern to be tested Context - Special testing conditions Forces - What types of faults are we looking for? ¨ Solution - Test case selection strategy that tests the interactions among the components that implement the pattern ¨ Example – A sample implementation ¨ ¨ ¨ 1 John Mc. Gregor and David Sykes Practical Guide to Testing Object-Oriented Software, Addison-Wesley, 2001.

Action – Discover test patterns for security - 2 n Problem: The synchronous communication between two objects is modified to be asynchronous by adding a callback object. How should this be tested? Client Callback Server

Action – Discover test patterns for security - 3 n Context ¨ n Forces ¨ ¨ n An intermediate object forwards messages Possible to intermingle successive messages/responses Temporal considerations must be added to the test case Solution Construct tests that exercise the callback in a variety of states Construct multiple clients, submit multiple requests through multiple callback objects ¨ Build test cases that submit a second message prior to receiving the response from the first message ¨ ¨

Action – Discover test patterns for security - 4 n n In this task, security vulnerabilities and standard designs to correct them would be studied. The standard designs would lead to a catalog of specific test patterns. Users of the test patterns could implement them in whatever language was in use. Deliverables: test pattern catalog

Task force identified security patterns 49 security patterns have been identified. Here a few. 1. Make the Client Invisible 2. Target Programs That Write to Privileged OS Resources 3. Use a User-Supplied Configuration File to Run Commands That Elevate Privilege 4. Make Use of Configuration File Search Paths 5. Direct Access to Executable Files 6. Embedding Scripts within Scripts 7. Leverage Executable Code in Nonexecutable Files 8. Argument Injection 9. Command Delimiters 10. Multiple Parsers and Double Escapes 11. User-Supplied Variable Passed to File System Calls 12. Postfix NULL Terminator 13. Postfix, Null Terminate, and Backslash 14. Relative Path Traversal 15. Client-Controlled Environment Variables 16. User-Supplied Global Variables (DEBUG=1, PHP Globals, and So Forth) 17. Session ID, Resource ID, and Blind Trust 18. Analog In-Band Switching Signals (aka “Blue Boxing”) 19. Attack Pattern Fragment: Manipulating Terminal Devices 20. Simple Script Injection Security Across the Software Development Lifecycle Task Force

Action – Develop focused test techniques - 1 n n n Operational profiles establish a relative frequency of operations for a particular type of user or application. Test cases are chosen with the same relative frequency to mimic actual use or to mimic attack scenarios. This narrows the range of values to cover in test. Operation Relative frequency Op 1 . 2 User 1 Op 2 . 6 Op 3 . 2 Op 1 . 4 User 2 Op 2 . 3 Op 3 . 3 User

Action – Develop focused test techniques - 2 n n n Testing all possible values even once, much less all possible combinations of values, is usually impossible. Combinatorial test designs can systematically sample test values to ensure the maximum coverage with minimum test cases. Orthogonal array testing is one combinatorial approach.

OATS tool - 1 n Three variables, each with three possible values, would require 27 test cases for all combinations Database coverage. driver JDBC 1. 0 JDBC 2. 0 ODBC 1. 0 browser Netsca pe IE 5. 0 Firefox # of web servers 5 50 200

OATS tool - 2 n OATS reduces this to 9 test cases, testing pair-wise but remains approximately 96% as effective as “all combinations” coverage at finding defects. 1 JDBC 1. 0 Netscape 200 2 JDBC 1. 0 IE 5. 0 50 3 JDBC 1. 0 Fox. Fire 5 4 JDBC 2. 0 Netscape 50 5 JDBC 2. 0 IE 5. 0 5 6 JDBC 2. 0 Fox. Fire 200 7 ODBC 1. 0 Netscape 5 8 ODBC 1. 0 IE 5. 0 200 9 ODBC 1. 0 Fox. Fire 50

Action – Develop focused test techniques - 3 Pair-wise value combinations can reduce the size of the test suite dramatically n These techniques have been applied in domains such as telecommunications. n In this task we would develop an industrial strength tool that allows the design of combinatorial test suites. n

Action – Develop focused test techniques - 4 n n n This technique is used in conjunction with traditional functional and structural strategies for test case selection. Functional tests ensure that all specified functions are correct and structural tests ensure that all feasible paths are valid. The result is confidence that the software does everything it is supposed to and nothing that it Is not supposed to. Combinatorial techniques increase the coverage of large software products without significant increases in test effort. This approach allows for risk-based test selections. Deliverables: software tool that uses combinatorial techniques and risk-based techniques to define effective test cases.

Action – Develop a security defect model for scientific codes If you just found the 73 rd defect in your 50, 000 LOC program, do you feel good about it? n With a validated defect model you would at least know how to feel. n You would know approximately how many defects to expect and what types of defects to look for. n

Action – Develop a security defect model for scientific codes - 2 n Using the relative Security frequency and defect type estimated size, estimates can be Static queue made of the number sizes of defects and the test HTTP cookies case selection Argument process be more injection precisely directed. Total Relative frequency per KLOC 60% 30% 100%

Action – Develop a security defect model for scientific codes - 3 n n In this task we would develop a defect model framework for scientific codes using historic data. The framework would be packaged with a specialization method for tailoring the framework to a specific development context. This would allow us to provide statistical estimates of the number of defects remaining in a software product and to provide a confidence interval on statistics such as reliability. Deliverables: defect model

Conclusion Use Case Name: Use: Requirements Domain Expert User: Frequency: See Also: Use Case Modeling Systems Engineeri ng Class Name Responsibility Super Class Collaborator Requirements model inspection Domain Analysis Client Analysis model inspection Application Analysis Incremental Integration and System Testing Class Specification Class Design Class Derivation Class Reuse Use Case Name: User: Frequency: See Also: Architectural Design Analysis model inspection Refinement Model Testing Class Delivery Architecture inspection lle r Application Use Cases Implementation ntr o n No single action or algorithm will produce secure software. There must be a chain of quality activities. The outlined strategy places security quality gates at several places in the software product development life cycle. Co n View Class Development Cluster design inspection

Conclusion - 2 n Implementing any part of this strategy will improve the security of the software products produced. n Develop method engineering tactics and guidelines that enhance the security quality of the software through improved processes. Structure architecture evaluation techniques to focus on security by searching for static security patterns. Discover and capture test patterns that correspond to dynamic security patterns. Instrument combinatorial test techniques to effectively explore security test patterns while reducing the test suite size. Create a defect model for security that can be used to predict types and number of security vulnerabilities in a given product. n n