Integrating Software into PRA Presented by C Smidts

Integrating Software into PRA Presented by C. Smidts Center for Reliability Engineering University of Maryland OSMA 2003 Center for Reliability Engineering 1

Integrating Software into PRA Probabilistic Risk Assessment (PRA) is a technique to assess the probability of failure or success of a mission. Current PRA neglects the contributions of software to the risk of the mission. The objective of our research is to extend current PRA methodology to integrate software in the risk assessment process. Center for Reliability Engineering 2

What We Have Done to Date • Built a Software Failure Mode Taxonomy • Failure Modes’ Quantification: generic high-level data - Public Literature - Expert Opinion • Collaborated with JSC through Ms. Alice Lee - Validate Our Methodology - Collect Data • Developed a Test-Based Methodology for Integrating Software Into PRA Center for Reliability Engineering 3

What We Are Planning to Do in the Future • Investigate Scalability Issues of the Test-based Approach • Continue the validation of our methodology with JSC • Apply the approach to JSC system • Revise the methodology based on NASA system • Develop an Analytical Approach • Apply the Analytical Approach to JSC system • Revise the Analytical Approach based on NASA system Center for Reliability Engineering 4

Integrating Software into PRA: A Test-based Approach Presented by C. Smidts, B. Li, M. Li Center for Reliability Engineering University of Maryland OSMA 2003 Center for Reliability Engineering 5

Integrating Software into PRA - Approach • We are working on an approach to integrate software into PRA. - Step 1: Identify events/components controlled/supported by software in MLD, accident scenarios, fault trees. - Step 2: Specify the functions involved - Step 3: Model software functions in ESDs/ETs and Fault Trees - Step 4: Construct the input tree - Step 5: Quantify the input tree - Step 6: Develop and perform software safety tests Center for Reliability Engineering 6

Example System An exit system in a building is used as the example in this case study. The exit system includes an emergency exit system and the PACS system. The Emergency exit system includes an emergency exit door and a marked egress router. It provides an escape route for personnel located inside the building during emergency situations. The PACS system is a simplified version of an automated personal entry/exit access system used to provide privileged physical access to rooms /buildings, etc. Personal ID and PIN are needed to access this system. Center for Reliability Engineering 7

Integrating software into PRA - Approach Step 1: Identify events/components • Identify events/components controlled/supported by software in MLD, accident scenarios, fault trees. • For all such events, create/expand contributors to account for software. • Verify that no neglected “events” may now have become possible due to software. Center for Reliability Engineering 8

MLD Center for Reliability Engineering 9

MLD Center for Reliability Engineering 10

Accident Description • Fire is the initiating event • Response systems: Emergency system and PACS system • End State: Loss of life Center for Reliability Engineering 11

Center for Reliability Engineering 12

Integrating software into PRA - Approach Step 2: Specify the functions involved • Not all software functions are involved in accident scenarios, i. e, not all software functions are involved in particular scenarios/fault trees or even in the entire realm of possible scenarios/fault trees. • To identify the specific functions involved in a scenario, determine the specific input to/output from the software – this will describe one function. • A list of possible functions can be found in the requirements. • Match the input/output combinations of these functions to the risk model Center for Reliability Engineering 13

Integrating software into PRA – Approach PACS Functional Decomposition Center for Reliability Engineering 14

Integrating software into PRA - Approach Actions and their inputs and outputs Center for Reliability Engineering 15

Integrating software into PRA - Approach Step 3: Modeling software function in ESDs/ETs and Fault trees • In the ESDs/ETs, the function of interest is modeled as Center for Reliability Engineering 16

Integrating software into PRA - Approach Step 3: Modeling software function in ESDs/ETs and Fault trees • In the fault tree, the function of interest is modeled as Center for Reliability Engineering 17

Integrating software into PRA - ESD Center for Reliability Engineering 18

Integrating software into PRA - Approach Step 4: Input Tree • Build the input tree for the particular function involved • The input tree is a decomposition of the space of possibilities • The input tree is mostly generic for a function. But may VARY due to context. (i. e. probabilities of basic events may vary, certain events may conflict with the rest of the scenario conditions. ) Center for Reliability Engineering 19

Integrating software into PRA - Approach Step 4: Input Tree Center for Reliability Engineering 20

Input Fault Tree for SW 1 Center for Reliability Engineering 21

Input Fault Tree Center for Reliability Engineering 22

Integrating software into PRA - Approach Step 5: Quantify the input tree Center for Reliability Engineering 23

Integrating software into PRA - Approach Step 6: Develop and perform software safety tests • These tests’ unique objective is to answer the questions contained in the model, i. e. in the MLD, accident scenarios and fault tree. • The test is completely automated using Test Generation/test execution tools (Test. Master/Win. Runner). • The process is as follows: - Build a finite State Machine model of the software by following the software functional decomposition derived from the risk model and the software requirements. - Derive the test profile and output conditions to be quantified from the risk model - Define and run the test cases according to the following test strategy - Analysis consists in computing the probabilities of the different outcomes based on the test data. Center for Reliability Engineering 24

Test. Master Model Center for Reliability Engineering 25

Test Script Example win_activate ("mmount-76. umd. edu - CRT"); start_time 1= get_time(); type ("1<k. Return>"); Check_Message(Message_b, 1); type ("0<k. Return>"); Check_Message(Message_c, 1); type ("155721495<k. Return>"); Check_Message(Message_b, 1); type ("0<k. Return>"); Check_Message(Message_c, 1); type("Gayyard. Lupie. N<k. Return>"); Check_Message(Message_b, 1); type("1<k. Return>"); end_time 1=get_time(); report_msg("Cardtime is "&(end_time 1 -start_time 1)"Seconds"); start_time 2= get_time(); Check_Message(Message_d, 1); wait( 9); type("4"); Check_Message(Message_e, 1); wait(3); type("5"); Check_Message(Message_f, 1); wait(1); type("1"); Check_Message(Message_g, 1); wait(3); type("9"); end_time 2=get_time(); report_msg("PINtime is "&(end_time 2 -start_time 2)"Seconds"); Case_Judge(Message_a, 1); Center for Reliability Engineering 26

Test Profile for PACS Center for Reliability Engineering 27

Failure modes application • Test Case Selection 1. Sample from the profile/input tree to see whether we have a “Normal” or an “Abnormal Input”. 2. If it is a normal input, select randomly from the “Normal Input” domain. 3. If it is an abnormal input, randomly select the failure mode according to the profile/input tree. 4. Then randomly select the “base”value from the “Normal Input” domain and mutate this “base” value using the rules given below: Center for Reliability Engineering 28

Testing Results 200 cases have been tested for SW 1 and SW 2. 19 cases failed. Failed cases classification SW 1 fails in only one case (58). Therefore, the point-estimate probability of Card failure is 1/200=0. 005. 18 cases failed for SW 2. Therefore, the unsafe probability (gate closed) is 18/199 =0. 09. Center for Reliability Engineering 29

Testing Results Card time and Probability Center for Reliability Engineering 30

Testing Results PIN time and Probability Center for Reliability Engineering 31

Test Cases Coverage Input Failure Modes Coverage (SW 1) Input Failure Modes Coverage (SW 2) Center for Reliability Engineering 32

ESD Center for Reliability Engineering 33

Future Work • Represent hardware-related input failure modes in test model • Quantification of input fault tree based on field data • Output failure modes/Support failure modes • Sensitivity Analysis • Scalability - Test case generation - Test case execution - Number of test cases for each software component Center for Reliability Engineering 34