University of Southern California Center for Systems and

University of Southern California Center for Systems and Software Engineering Value Driven Security Threat Modeling for Off The Shelf Software Systems Yue Chen USC Center for Systems and Software Engineering Emails: {yuec, boehm, lshep}@usc. edu Feb 14, 2007 CSSE Annual Research Review Los Angeles, U. S. A. ©USC-CSSE 1

University of Southern California Center for Systems and Software Engineering Trends: Increasing Concerns on COTS Vulnerability Increasing Trend of COTS Based Applications • Increasing COTS usage Data source: [Boehm et al 2003][Standish] • Increasing number of COTS vulnerabilities published Data source: [CERT Statistics] ©USC-CSSE 2 Introduction

University of Southern California Center for Systems and Software Engineering Prioritizing COTS Vulnerability is Important • Significant Loss: CSI/FBI 2006 Survey reported a total loss of $52, 494, 290 caused by security incidents from 313 organizations • Limited resources for security: 47% spend equal or less than 2% in IT security Data source: CSI/FBI Computer Crime and Security Survey 2006 • Challenge: How to spend limited amount of security resources smartly? ©USC-CSSE 3 Introduction

University of Southern California Center for Systems and Software Engineering Current Practices • System context neutral, static vulnerability rankings are recommended by authority organizations such as Bug. Traq/Symantec, CERT, ISS, NIST, Microsoft, SANS • Coarse granularity of rankings: three ~ four levels of rankings (e. g. Critical, Important, Moderate, Low) are used to differentiate 23, 620 known COTS vulnerabilities • An Real-life Story – USC Ticketing Office System (Source: USC-ISD) • Decisions made based on [Butler, 2002] – Best knowledge – Individual experience – Ad hoc ©USC-CSSE 4 Introduction

University of Southern California Center for Systems and Software Engineering Proposal: Threat Modeling framework based on Attack Path analysis (T-MAP) A novel threat modeling framework for COTS based systems that is sensitive to system stakeholder value context, dynamic, and tool-automated Current Approaches: COTS Vulnerability Rankings Value Neutral Assessment T- MAP: COTS Vulnerability Value Neutral Assessment Scenario Evaluation COTS Vulnerability Rankings Evaluation criteria based on stakeholder value propositions ©USC-CSSE 5 Introduction

University of Southern California Center for Systems and Software Engineering Nature of The Problem Attacking Paths Permitted Ports Unblocked vulnerabilities Vulnerabilities impacting confidentiality, availability, integrity Blocked vulnerabilities Firewall Wrapper e. g. Windows Server 2003 e. g. SQL Server 2000 Software Applications, COTS e. g. Web Server e. g. IIS 6. 0 e. g. CRM Server IT Infrastructure e. g. Regulatory Productivity Org. Values ©USC-CSSE Reputation T-MAP 6 Framework

University of Southern California Center for Systems and Software Engineering Castle Defense Analog Measure the security of a castle by the value of treasures in the castle, the number of holes on the walls, as well as the size of the holes. ©USC-CSSE T-MAP 7 Framework

University of Southern California Center for Systems and Software Engineering T-MAP Framework • Step 1: Identify key stakeholders and value propositions (the treasures in the castle); • Step 2: Establish a set of security evaluation criteria based on stakeholder value propositions; • Step 3: Use tool to enumerate and analyze attack paths based on a comprehensive COTS vulnerability database containing 23, 620 vulnerability information (the holes); • Step 4: Evaluate the severity of each scenario in terms of numeric ratings against the evaluation criteria established in Step 2 (the size of the holes); • Step 5: The security threat of each vulnerability is quantified with the total severity ratings of all attack paths that are relevant to this vulnerability; • Step 6: System total threat is quantified with the total severity ratings of all attack paths; [Note] Step 3 to 6 are tool automated by the Tiramisu Tool ©USC-CSSE T-MAP 8 Framework

University of Southern California Center for Systems and Software Engineering Step 1 -2: Evaluate Security against Criteria based on Stakeholder/Values • Evaluate the severity of security hazard scenarios against stakeholder/value impacts • Involves both qualitative and quantitative criteria • Technical approach: Figure of Merits and Analytical Hierarchy Process (AHP) – A convenient tool to determine the priority of alternatives in terms of weight through pair-wise comparison – Invented by Dr. Saaty at Business School of Univ. of Pennsylvania in 1970 s [Saaty, 1980] – Recommended by Dr. Bodin, Dr. Gordon et al for security investment evaluation [Bodin et al, 2005] ©USC-CSSE T-MAP 9 Framework

University of Southern California Center for Systems and Software Engineering Using AHP Determine the Weights – An Example §Example (from USC ISD Server X Case Study) Weights derived through AHP pair-wise comparisons Possible Breach Scenarios Value Centric Criteria ©USC-CSSE T-MAP 10 Framework

University of Southern California Center for Systems and Software Engineering Step 3 -4: Enumerate Attack Scenarios • Enumerate the scenarios how an attacker can compromise stakeholder values through COTS system vulnerabilities • Establish Structured Attack Graph based on a comprehensive COTS vulnerability database involves 23, 620 known vulnerabilities reside in 31, 713 COTS software ©USC-CSSE T-MAP 11 Framework

University of Southern California Center for Systems and Software Engineering Step 5: Vulnerability Technical Severity Evaluation • Vulnerability Technical Severity Attributes – Impact on confidentiality, integrity and/or availability – Remotely exploitable – Require valid user account on victim host – Needs user activities Based on an emerging national standard [CVSS] by NIST ©USC-CSSE T-MAP 12 Framework

University of Southern California Center for Systems and Software Engineering Step 6 T-MAP Severity Rating System • Severity Weight of Attack Path P: • Overall Security Threat Score of COTS System G: • Threat. Key of elements in Attack Graph: • Effectiveness of Security Practice: ©USC-CSSE T-MAP 13 Framework

University of Southern California Center for Systems and Software Engineering Security Investment Effectiveness Estimation • How much security threats can be avoided by implementing Firewall, Software hardening (patching), user account control, or file system encryption? • Results as well depends on the total value of the protected system * Case study results estimated by professional security manager at USC-ITS T-MAP 14 ©USC-CSSE Framework

University of Southern California Center for Systems and Software Engineering Tool Implementation – Project Code: Tiramisu How it works? High-level software architecture ©USC-CSSE T-MAP 15 Implementation

University of Southern California Center for Systems and Software Engineering Security Manager’s Ranking vs. Tiramisu Ranking ©USC-CSSE T-MAP 16 Framework

University of Southern California Center for Systems and Software Engineering Security Economics in Patching • Prioritize COTS Based System vulnerabilities under business context – “ 20% percent of vulnerabilities causes 80% of the security risks”, T-MAP tells what are the 20% • Rational: Prioritize vulnerabilities with its Threat. Key; • Example screenshot: ©USC-CSSE 17

University of Southern California Center for Systems and Software Engineering COTS Security Economics – Finding Sweet Spots • Sweet spot to invest in • Economic curve of security patching • Also driven by the total (from USC Server X case study) value of system (from USC Server X case study) Sweet spots to invest ©USC-CSSE 18

University of Southern California Center for Systems and Software Engineering Limitations • Only sensitive to known COTS Vulnerabilities – Empirical study by Arora shows that the average attacks per host per day jumped from 0. 31 to 5. 45 after vulnerability get published • Not sensitive to nuance in local system configurations – Disabled services – Services running on different privileges, etc. • Only cover “one-step-attacks” that exploiting COTS vulnerabilities • Depends on comprehensive vulnerability database – Our database: 23, 260 vulnerability published from 1999 -2006 that resides in 31, 313 COTS software • Cannot effectively address attacks such as Phishing ©USC-CSSE T-MAP 19 Framework

University of Southern California Center for Systems and Software Engineering Hypothesis #1 *: For given COTS based system whose confidentiality, integrity and availability have different priorities to the stakeholder values, the accuracy of T-MAP results, measured by the Inaccuracy, the ratio of the number of clashes between vulnerability priorities and stakeholder value priorities and the total number of comparisons, will not make any difference comparing to the existing leading approaches (say, the average level of at least three leading approaches, if rating data available). Clash definition: for two technically identical vulnerabilities VA and VB, given the stakeholder value impact severity of VA is higher than VB, if a prioritizing system assign VA less or equal priorities as VB, it is counted as a clash for this prioritizing system. * Hypothesis #1 is a null hypothesis. We aim at disprove them through case Clash Counting Illustration studies and tool demos. Vuln. VA ©USC-CSSE VB - - - X - - Research 20 Hypothesis

University of Southern California Center for Systems and Software Engineering Inaccuracy Comparison • Inaccuracy *: the ratio of the number of clashes between vulnerability priorities and stakeholder value priorities and the total number of comparisons * Inaccuracy is a numerical value between 0~1, the larger value indicates the less accuracy For the USC ISD case study, the security manager prioritized 8 vulnerabilities, which translates to 28 comparisons, where 28 = 1+2+… + 7 # of Clashes Inaccuracy T-MAP 2 0. 071 CVSS 6 0. 214 ISS 9 0. 321 ©USC-CSSE Hypothesis 21 Validation

University of Southern California Center for Systems and Software Engineering Contributions • A framework to model CBS security – A COTS security evaluation framework that captures stakeholder value propositions – Distill the potential impacts of thousands of vulnerabilities into management friendly numbers at a high-level – Tool automated • Has the potential to pro-actively attack security in early life-cycle, instead of taking whatever as is reactively after system built ©USC-CSSE Contribution & 22 Next Steps

University of Southern California Center for Systems and Software Engineering Discussions and Future Work ©USC-CSSE 23

University of Southern California Center for Systems and Software Engineering Thanks! ©USC-CSSE 24