Assertion–Based Requirements Validation September 8, 2008 Patrick Theeke (PMP, CSQE) John Ryan cmg@ivv. nasa. gov IV&V Facility
Assertion–Based Requirements Validation Objectives IV&V Facility • Describe the process of using executable models for requirements validation • Discuss experiences and characterize outcomes from a pilot project employing the technique • Identify lessons learned to date and possible next steps 3/16/2018 2
Assertion–Based Requirements Validation Agenda IV&V Facility • Briefly describe Model-Based Requirements Validation • Define the process for Assertion-Based Requirements Validation • Present experiences of pilot project • Present observations • Discuss additional details via panel 3/16/2018 3
Requirements Validation Goals of Requirements Validation IV&V Facility • Assure that the documented requirements fully specify the capabilities and characteristics necessary for the system being defined to meet its goals • In particular, NASA IVV is interested in critical capabilities • Requirements defects include – Missing – Superfluous – Ambiguous – Incomplete – Inconsistent – Unverifiable – Incorrect 3/16/2018 4
Requirements Validation Approach IV&V Facility • Build a standard of reference for comparison against the documented requirements of the system of interest • This standard is the System Reference Model (SRM), which represents IVV’s understanding of the system of interest SRM Behaviors SRM Correct? Requirement s Complete? Valid? 3/16/2018 Validated Requireme nts Correct, Complete, Consistent, Unambiguous, and Verifiable Requirements Software Requirements SRM Correct? Requirement s in Scope? Valid? 5
Requirements Validation SRM components IV&V Facility • Behaviors – Use cases and UML diagrams (e. g. activity diagrams) • Structure – Placeholder for named physical and logical components of the system – Identify component roles and events important to understanding the system • Independent Modeling Assertions (IMA) – Formalized IMA extracted from the other SRM components – Collection of unit tests to validate the IMAs • System Scenarios – Scenarios that describe various sequences of events • Nominal sequences (what the system is supposed to do) • Unwanted behaviors (what the system is not supposed to do) • Detection and response to adverse conditions (how the systems responds to adverse conditions) 3/16/2018 6
Assertion–Based Requirements Validation Pilot Assertion Statechart Definition IV&V Facility • Each Statechart Assertion is a formal specification of a single requirement. • A statechart assertion is fundamentally a monitoring device that observes system behavior and determines whether that behavior is valid • Observed behavior is valid when it matches the behavior specification coded into the assertion, and invalid when it violates the specification • An assertion is run against observable behavior, typically supplied by some executable artifact running under a test scenario 3/16/2018 7
Requirements Validation Correlation-Based Requirements Validation IV&V Facility • Capture independent understanding of system behavior or characteristic in a SRM • Correlate project requirements with SRM behaviors • Analyze results to identify – Ambiguous, incorrect, incomplete, inconsistent, or non-verifiable requirements that correspond to SRM behaviors – Requirements without matching SRM behavior – SRM behavior without matching requirements SRM Correlatio ns with SRM Analyze Results Findings Project Requirements 3/16/2018 8
Assertion–Based Requirements Validation Motivation IV&V Facility • Formalization of natural language requirements reveal errors in the specification • Leveraging formal assertions for requirements validation increases the utility of formalized requirements – Formalization of requirements is a necessary step in leveraging the SRM for verification • Utilizing assertions more closely mimics the highly parallel threads of execution of real time, reactive systems, including timing and synchronization, allowing more in-depth examination of complex behavioral interactions • Discovering requirements and representing them as assertions is more readily done early in the modeling process while the SRM is under development – so we might as well use them! 3/16/2018 9
Assertion–Based Requirements Validation Dependencies IV&V Facility • Assertion Based Validation Depends on: – SRM was built to the appropriate level of depth and details – The requirements were traced to the SRM – The set of system scenarios provided an appropriate coverage of the system behaviors – The naming convention for the assertion events match the events of the formalized unit tests, system scenarios, and assertion state charts 3/16/2018 10
Assertion–Based Requirements Validation Process Assertion-Based Requirements Validation IV&V Facility • Formally capture developer’s documented requirements as Project Based Assertions (PBA) • Exercise PBAs and IMAs using validated System Scenarios • Analyze results Develop & Validate IMAs SRM Develop System Scenarios Correlatio ns with SRM Develop & Validate PBAs Execute Scenarios in Context of PBAs Analyze Results Findings Project Requirements 3/16/2018 11
Assertion–Based Requirements Validation Process Develop & Validate Project–Based Assertions IV&V Facility • Translate the documented requirements from the development project into a formal representation that is conducive to comparison with the SRM • Requires detailed understanding of the requirements • Exposes errors in the requirements specifications Identify Relevant Documented requirements Interpret Requirement s Formalize Requirement s as PBAs Validate PBAs SRM Correlatio ns with SRM Develop & Validate PBAs Execute Scenarios in Context of PBAs Analyze Results Findings Project Requirements 3/16/2018 12
Assertion–Based Requirements Validation Process Execute scenarios in Context of PBAs IV&V Facility • We are interested in how the PBAs react to the system scenarios – Which system scenarios ended as expected? – Which system scenarios ended differently than expected? – Which PBAs were not exercised? Include PBAs in executable environment Execute system scenarios Collect results for further analysis SRM Correlatio ns with SRM Develop & Validate PBAs Execute Scenarios in Context of PBAs Analyze Results Findings Project Requirements 3/16/2018 13
Assertion–Based Requirements Validation Process Analyze Results IV&V Facility • Examine each assertion failure for: – Errors in constructing the SRM (system scenarios, IMAs, etc. ) – Errors in constructing the PBA(s) – Errors in a requirement • Fix the errors in the SRM • Rerun test until no SRM or PBA construction errors are discovered. Therefore all unexpected results indicate findings Ensure system scenario set is correct and complete Ensure PBA is correct Ensure PBA set is complete Ensure that PBA is in scope SRM Correlatio ns with SRM Develop & Validate PBAs Execute Scenarios in Context of PBAs Analyze Results Findings Project Requirements 3/16/2018 14
Assertion–Based Requirements Validation Pilot Develop and Validate IMAs IV&V Facility • Examine behavior in UML activity diagrams – Begin with preconditions, triggers, constraints – Assert important sequences that must occur in order (under all conditions or adverse conditions) – Constraints on looping (will continue to try until …) – Important executions if a critical behavior is unavailable – Any possible/impossible executions that will occur during offnominal events – Look at the whole picture and observe modeling issues as well Develop & Validate IMAs SRM Develop System Scenarios Identify Candidate IMAs 3/16/2018 Project Requirements Analyze Correlatio Candidate ns IMAs SRM with Represent Candidate IMAs in & Develop Validate Natural PBAs Language Formalize IMAs as Validate Execute assertion IMAs Analyze Scenarios in state Results Context of PBAs charts Findings 15
Assertion–Based Requirements Validation Pilot Identify candidate IMAs IV&V Facility -Actively Damp Nutation [After burn maneuver, need to stabilize damping before selecting Idle Mode to downlink data] Select Damping Mode Determine Attitude Configure Damping Filter Estimate Nutation Angle IF Angle > target: Then compute damping torque prior to controlling fired thruster burn IF Angle < target: continue process Estimate target Angle Check if < target Track time passing Once stablized for minimal time, GNC subsytem can Select Idle Mode 3/16/2018 16
Assertion–Based Requirements Validation Pilot Analyze candidate IMAs IV&V Facility • Analysis Note: The nutation angle must be less than or equal to target nutation angle for the minimum amount of stable time to maintain an acceptable and stable nutation angle. [P <= R and T 1 >= T 2 (Tmin)] mus Q 1. The nutation angle t (/can? ) be less than or equal to target nutation angle for the minimum amount of stable time to select Idle Mode. Domain Answer. Actually both Ground and Fault Protection Mode can select Idle Mode – at any time even if the above is not true ([P <=1 >= T 2 T(Tmin)]). R and Assumption: If there was a satellite timeout occurring, and if the fault protection mode has not been entered (due to # attempts or timeout), then the above is true. Q 1 b. The Reference Orientation (GN&C sub system) must maintain a nutat ion angle is less than or equal to the target nutation angle for a minimal that amount of stable time to (See select Idle Mode. Figure 16 and 20) Q 2. What occurs if the goal is met (nutation angle stabilized and maintainable) and something causes the ang exceed the nutation angle again? le to Domain Answer. The issue is monitored and the active damp nutation behavior is called – again. 3/16/2018 17
Assertion–Based Requirements Validation Pilot Analyze candidate IMAs IV&V Facility • Analyze each candidate assertion and create good and bad scenarios to represent the requirements • Use the diagrams in the SRM to derive scenarios or unit tests that will cause the requirement to succeed and fail • Use the scenarios to challenge the assertion against – – – redundancy of events improper sequencing Unknowns misuse of system boundaries Dependencies and constraints (i. e time) • Determine whether or not the sequencing, repetition, looping, constraints, and concurrence always (“must”) occur this way or conditionally throughout the entire SRM 3/16/2018 18
Assertion–Based Requirements Validation Pilot Analyze candidate IMAs IV&V Facility Good and Bad Scenarios for Q 1 B: The Reference Orientation must maintain a nutation angle that is less than or equal to the target nutation angle for a minimal amount of stable time to select Idle Mode. Good Scenario 3/16/2018 ac ce ed T arg et Tim h Ta Nu rge e p tat ion as t N Se se uta An d = lec tio gl t Id n 4 An e le gle Mo de inable angle Re Ex h T Tim ac Re Ex ce ed T arg e arg t Nu e p tat et ion a Ex N ce ssed uta An tio e gle n Re d Ta = 2 An ac rge gle h t Tim Ta N e p rget uta Tim ass Nu tion tat A e p ed ion ngl = as Se se 4 An e d = lec gle t Id 3 le Mo de Timer Tmin is the minimal amount of stable time required to reach a mainta Tmin = 5. state and allow ability to select Idle Mode. Bad Scenario 19
Assertion–Based Requirements Validation Pilot Represent the candidate IMAs as NLR IV&V Facility • Natural Language Requirement for Q 1 b – The Reference Orientation (GN&C sub system) shall achieve a stable nutation angle that is less than or equal to an acceptable target nutation angle for a minimal amount of time prior to selecting Idle Mode. 3/16/2018 20
Assertion–Based Requirements Validation Pilot Formalize each Natural Language Requirement 3/16/2018 IV&V Facility 21
Assertion–Based Requirements Validation Pilot Validate each formalized IMA IV&V Facility • Validate the IMA is unambiguous, correct, and consistent with the SRM behavior and scenarios. – Validate each model-based formalized requirement has been modeled correctly to represent the intended requirement • Create formalized unit tests for the scenarios public void test. Damp. Nutation_IMA 001_sc 001(){ // this is testing a good scenario for RO selecting Idle Mode print. State(IMA 001); IMA 001. reach. Target. Nutation. Angle(); print. State(IMA 001); IMA 001. incr. Sim. Time(1); print. State(IMA 001); IMA 001. exceed. Target. Nutation. Angle(); print. State(IMA 001); IMA 001. reach. Target. Nutation. Angle(); print. State(IMA 001); IMA 001. incr. Sim. Time(5); print. State(IMA 001); IMA 001. incr. Sim. Time(1); print. State(IMA 001); IMA 001. ROselect. Idle. Mode(); print. State(IMA 001); assert. True(IMA 001. is. Success()); System. out. println("Completed test. Damp. Nutation_IMA 001_sc 001"); System. out. println("-------------"); System. out. println(); } public void test. Damp. Nutation_IMA 001_sc 002(){ // this is testing a b scenario for RO selecting Idle Mode print. State(IMA 001); IMA 001. reach. Target. Nutation. Angle(); print. State(IMA 001); IMA 001. incr. Sim. Time(1); print. State(IMA 001); IMA 001. exceed. Target. Nutation. Angle(); print. State(IMA 001); IMA 001. reach. Target. Nutation. Angle(); print. State(IMA 001); IMA 001. incr. Sim. Time(4); print. State(IMA 001); IMA 001. ROselect. Idle. Mode(); print. State(IMA 001); assert. False(IMA 001. is. Success()); System. out. println("Completedtest. Damp. Nutation_IMA 001_sc 002"); } • Validate the IMA against the formalized unit tests 3/16/2018 22
Assertion–Based Requirements Validation Pilot Develop & Validate Project–Based Assertions IV&V Facility • Translate the documented requirements from the development project into a formal representation that is conducive to comparison with the SRM • Requires detailed understanding of the requirements • Exposes errors in the requirements specifications Identify Relevant Documented requirements Interpret Develop Requirement & Validate IMAs s Formalize Requirement s as PBAs Validate PBAs Develop System Scenarios SRM Correlatio ns with SRM Develop & Validate PBAs Execute Scenarios in Context of PBAs Analyze Results Findings Project Requirements 3/16/2018 23
Assertion–Based Requirements Validation Pilot Develop & Validate Project–Based Assertions 3/16/2018 IV&V Facility 24
Assertion–Based Requirements Validation Pilot Validate each formalized PBA IV&V Facility Validate the PBA is unambiguous, correct, and consistent. Validate each project-based formalized requirement has been modeled correctly to represent the intended requirement Create formalized unit tests for the scenarios public void test. Torque. Vector_IMA 001(){ print. State(pba 001); pba 001. begin. JOIPhase(); print. State(pba 001); pba 001. SRUUpdate(); print. State(pba 001); pba 001. incr. Sim. Time(4); pba 001. begin. Fire. Thrusters(); print. State(pba 001); pba 001. end. Fire. Thrusters(); print. State(pba 001); assert. True(pba 001. is. Success()); System. out. println("Completed test. Torque. Vector_pba 001"); System. out. println("-------------"); System. out. println(); } 3/16/2018 25
Assertion–Based Requirements Validation Pilot Add system scenarios to SRM IV&V Facility • Derive system scenarios – Nominal, expected behavioral sequences – Behavioral sequences that should not be permitted – Detection and responses to adverse conditions Develop & Validate IMAs SRM 3/16/2018 Impleme nt system scenario s Validate system scenario s Develop System Scenarios Correlatio ns with SRM Project Requirements Identify events Define system scenario s Develop & Validate PBAs Execute Scenarios in Context of PBAs Analyze Results Findings 26
Assertion–Based Requirements Validation Pilot Identify Events IV&V Facility • Identify events in the activity diagrams – Activities model the control of execution of one or more actions. Activities do not model sequences of event occurrences. – Activities do not often model timing constraints – Action names and control flow guards are defined in natural language and thus prone to ambiguity 3/16/2018 27
Assertion–Based Requirements Validation Pilot Define System Scenarios IV&V Facility • “Good” and “Bad” Scenarios – Used the semantics of UML sequence diagrams to define a scenario, by modeling sequence of event occurrences (i. e. action execution start/finish events) – Used guard conditions in activity diagrams that should not hold true in the context of the goal as means of creating “bad scenarios” 3/16/2018 28
Assertion–Based Requirements Validation Pilot Implement System Scenarios IV&V Facility • Scenarios are implemented in J-Unit and reference events in assertions 3/16/2018 29
Assertion–Based Requirements Validation Pilot Validate System Scenarios IV&V Facility • Use the IMAs to ensure that the scenarios return the expected results – Q 1 scenarios complete without errors, as expected – Q 2 scenarios do not cause any assertions to “pop” – Q 3 scenarios result in detection of an adverse condition and respond in a correct way, as expected • Scenarios represent a test oracle 3/16/2018 30
Assertion–Based Requirements Validation Pilot Execute scenarios in Context of PBAs IV&V Facility • Interested in how the PBAs react to the system scenarios – Which system scenarios ended as expected? – Which system scenarios ended differently than expected? – Which PBAs were not exercised? Include PBAs in executable environment Execute system Develop & scenarios Validate IMAs Collect results for further analysis Develop System Scenarios SRM Correlatio ns with SRM Develop & Validate PBAs Execute Scenarios in Context of PBAs Analyze Results Findings Project Requirements 3/16/2018 31
Assertion–Based Requirements Validation Process Analyze Results IV&V Facility • Examine each assertion failure for: – Errors in constructing the SRM (system scenarios, IMAs, etc. ) – Errors in constructing the PBA(s) – Errors in a requirement • Fix the errors in the SRM • Rerun test until no SRM or PBA construction errors are discovered. Therefore all unexpected results indicate findings Ensure system scenario set is correct and complete Ensure PBA is correct Ensure PBA set is Develop & Validate IMAs complete Ensure that PBA is in scope Develop System Scenarios SRM Correlatio ns with SRM Develop & Validate PBAs Execute Scenarios in Context of PBAs Analyze Results Findings Project Requirements 3/16/2018 32
Assertion–Based Requirements Validation Pilot Observations IV&V Facility • Developing IMAs helps identify defects and deficiencies in the SRM. Doing this early in the SRM development process yields early benefits • The process of examining the documented requirements helps to identify omissions in the correlation process • The depth of understanding required to create IMAs and PBAs results in more rapid understanding of the system of interest • Assertions best represent requirements of specific behaviors • When creating PBAs for high level requirements such as “The system shall have the capability to do X”, these behavioral details need to be supplied from domain knowledge and analysis 3/16/2018 33
Assertion–Based Requirements Validation Pilot Observations IV&V Facility • It was easier than expected to: – Discover early questions and concerns that could lead to good assertions – Develop a high level understanding of what the system is supposed to do – Create higher level system scenarios that describe what the system is supposed to do – Develop Formal Assertion State Charts from Natural Language – Observe the lack of model depth, details – Discover possible missing behavioral IMAs – Write unit tests for IMAs and PBAs (although easier for IMAs if the SRM provides more information) – Write formalized Junit Tests – Test assertions against the System Scenarios 3/16/2018 34
Assertion–Based Requirements Validation Pilot Observations IV&V Facility • It was harder than expected to: – Vet good/worthwhile NL (informal assertions) from the list of answered questions – Create System Scenarios with the lack of model depth and details – Create System Scenarios that describe what the system is not supposed to do, and what the system is supposed to do under adverse conditions – Correlate system scenario events, IMA events, and PBA events without a data dictionary or complete Domain Model – Discover worthwhile PBAs from documented requirements that define and monitor a single event, and/or provide limited details and constraints – Discover matching IMAs and PBAs when there are limited requirement details – Discover matching IMAs and PBAs when the model is not developed to needed level of coverage/detail – Validate Requirements when the model is not developed to the needed level of coverage/detail – Set up the test rig for the first time 3/16/2018 35
Assertion–Based Requirements Validation Recommendations IV&V Facility • Develop IMAs early in the SRM development process • Consider developing IMAs as part of the process for reviewing activity diagrams • Continue refining techniques for using assertionbased methods for requirements validation 3/16/2018 36
Assertion–Based Requirements Validation Summary IV&V Facility • Model–Based Requirements Validation is part of the SRM Validation process – The purposes for using assertions include: • Have ability to test conflicts and validate complex requirements • Provides an advantage of using formal requirements over natural language • Sets groundwork for automated testing – Use assertions to validate requirements that correspond to SRM behaviors and IMAs as unambiguous, correct, complete, consistent, and verifiable. – Use assertions to ensure that the right behaviors have been defined and the behaviors are of high quality. The right behaviors are those that adequately describe: • What the system is supposed to do • What the system in not supposed to do • What the system is supposed to do under adverse conditions Assertion–based validation leverages our independent modeling to find issues earlier in the life–cycle 3/16/2018 37
Assertion–Based Requirements Validation Panel Discussion • • 3/16/2018 IV&V Facility Michael Facemire John Ryan Bill Stanton Patrick Theeke 38
References IV&V Facility • D. Drusinsky, “From AD to Assertions”; informal presentation, 2008. • D. Drusinsky, J. B. Michael, and M. Shing, “A Framework for Computer-aided Validation”; Naval Postgraduate School, NPSCS-07 -008, 08/2007. • K. Hurley, and D. Kranz, “Executable Reference Model”; informal presentation, 2008. • Juno IVV Team, “Juno Level-4 Flight Software Requirements Validation Report”; NASA IV&V, 2008. • D. Kranz, and J. Ryan, “Juno and Assertion Based Validation”; NASA IV&V, white paper, 2008. • NASA IV&V Facility, “System Level Procedure 09 -1”; 2007. • S. Raque, “SRM Example Overview”; informal presentation , 2008. • P. Theeke, “A process architecture for reference model based validation”; NASA IV&V, white paper, 2007. • P. Theeke, “System Reference Models and Executable Assertions for Requirements Validation”; NASA IV&V, white paper, 2007. 3/16/2018 39
Скачать презентацию Assertion Based Requirements Validation September 8 2008 Patrick Theeke
5324e04e89dcbbdb969d3e08ba0a7565.ppt