New Development Techniques New Challenges for Verification and

New Development Techniques: New Challenges for Verification and Validation Mats Heimdahl Critical Systems Research Group Department of Computer Science and Engineering University of Minnesota 4 -192 EE/CS; 200 Union Street SE Minneapolis, MN 55455 1

Domain of Concern http: //www. cs. umn. edu/crisys 2

How we Develop Software Concept Formation System Test http: //www. cs. umn. edu/crisys Test Requirements Specification System Integration Test Design Integration Unit Test Implementation Analysis Object Code 3

Validation and Verification Concept Formation http: //www. cs. umn. edu/crisys Requirements Specification System Design Verification: Are we building the thing right? Integration Implementation Validation: Are we building the right thing? 4

Model-Based Development Properties http: //www. cs. umn. edu/crisys Visualization Analysis Testing Specification Model Code 5 Prototyping

Model-Based Development Tools • Commercial Products u http: //www. cs. umn. edu/crisys u u u Esterel Studio and SCADE Studio from Esterel Technologies Spec. TRM from Safeware Engineering Rhapsody from I-Logix Simulink and Stateflow from Mathworks Inc. Rose Real-Time from Rational Etc. 6

Research Tools (many): RSML-e and Nimbus Simulations of environment http: //www. cs. umn. edu/crisys RSML-e Formal Models (~20 running concurrently) 7

How we Will Develop Software Concept Formation Requirements http: //www. cs. umn. edu/crisys Analysi s Properties System Specification/Model Syste m Test Integration Specification Test Implementation 8

FGS/FMS Mode Logic RSML-e and Nimbus Simulations of environment http: //www. cs. umn. edu/crisys RSML-e Formal Models (~20 running concurrently) 9

Sample RSML-e Specification http: //www. cs. umn. edu/crisys 10

Capture Requirements as Shalls http: //www. cs. umn. edu/crisys 11

Translated All the Shalls into SMV Properties http: //www. cs. umn. edu/crisys 12

Early Validation of Requirements Using Model-Checking (Nu. SMV) http: //www. cs. umn. edu/crisys • • • Prove Over 300+ Properties in Less Than an Hour Found Several Errors in Our Models Using Model-Checking Substantially Revised the Shalls to Correct Errors 13

Early Validation of Requirements Using Theorem Proving (PVS) http: //www. cs. umn. edu/crisys • • • Proved Several Hundred Properties Using PVS More Time Consuming than Model-Checking Use When Model-Checking Won’t Work 14

Model-Based Development Examples http: //www. cs. umn. edu/crisys 15

A Simplified Development Model Requirements and Specification http: //www. cs. umn. edu/crisys System Test Code Unit Test Time 16

Ongoing Research RSML-e, SCR, Spec. TRM, Statecharts, Esterel, SCADE, Simulink, Etc. CMU, SRI, Stanford, UC Berkley, VERIMAG, NASA, Etc. Minnesota, Pennsylvania, George Mason, NRL, NASA Ames, Etc. RSML-e, SCR, Spec. TRM, Statecharts, Esterel, SCADE, Simulink, Etc. –UML Properties http: //www. cs. umn. edu/crisys Visualization Analysis Testing Prototyping Specification Model Code RSML-e, SCR, Spec. TRM, Statecharts, Esterel, SCADE, Simulink, Etc. –UML Proof carrying code, Provably correct compilers, Test for correctness 17

Problems… Tested enough? Trust the results? Can we trust execution environment? Properties http: //www. cs. umn. edu/crisys Visualization Analysis Testing Prototyping Are the languages usable—syntax and semantics? Can they play nice together? Specification Model Code Can we really trust the code? 18

Benefits of Modeling Fewer “Bugs” http: //www. cs. umn. edu/crisys Savings Time 19

Code Generation Fewer “Bugs” http: //www. cs. umn. edu/crisys Coding effort greatly reduced Savings Time 20

Qualified Code Generation (theory) http: //www. cs. umn. edu/crisys Unit testing moved here. Unit testing eliminated for generated code Time Savings 21

Code Generation Concerns Concept Formation Is our model “right”? Requirements Properties http: //www. cs. umn. edu/crisys Specification/Model Can we trust the execution environment? Can we trust our analysis tools? Can we trust our properties? System Integration Can we trust the code generator? Implementation 22

“Correct” Code Generation • Provably correct compilers u • http: //www. cs. umn. edu/crisys • Proof carrying code u Generate Specification/Model Total correctness required Base all specification testing on the generated code u • Very hard (and often not convincing) Loose the benefits of working at the specification level Output Specification Based Tests Generate test suites from specification u u Implementation Compare specification behavior with generated code to better trust your specification testing Unit testing is now not eliminated, but completely automated 23 Output

Specification Testing • Certify the execution • When have we tested environment u Too costly and probably impossible enough? u http: //www. cs. umn. edu/crisys • Specification based n testing u n Any discrepancy and either n n n the code generator is wrong, or the execution environment is wrong, or the target platform is faulty Specification coverage criteria What is adequate coverage? Criteria for measurement are not good for generation – Technically covering the specification, but with useless tests u Do we reveal faults n Tradeoff between the impossible and the inadequate 24

Proof Techniques (theory) Reduced testing since properties proved correct in specification stage http: //www. cs. umn. edu/crisys Proofs performed here Time Savings 25

Verification Trust We need properties (requirements)!!! Concept Formation Often lost in the modeling “frenzy” Requirements http: //www. cs. umn. edu/crisys Specification/Model Properties How System do we trust our proofs? Integration Implementation Proof validity in production environment? 26

Proof Techniques • Certify analysis tools u • http: //www. cs. umn. edu/crisys u Technically feasible, but is the redundancy “trustworthy”? ? Cost… Automation is key u • Too costly and probably impossible Use redundant proof paths u • Trusted Translators ? Translation RSML-e Translation Model Checker Translation State Exploration Must keep analysis cost under control Generate test suites from specification u Theorem Prover Low cost since it is already done for the code generator Many languages and Trusted many analysis Translators? techniques 27

Proof Techniques (worst case) Added burden that cannot be leveraged later http: //www. cs. umn. edu/crisys Most analysis is not easy, nor cheap! Time Savings 28

Regression Verification Large Evolving Model 100 s, if not 1000 s, of properties http: //www. cs. umn. edu/crisys Iterated Weekly? Daily? Hourly? Analysis Result • Abstraction cost amortized • Impact of change on abstraction • Approximate techniques in day-to-day activities 29

Can We Achieve the Goal? Redundant proof process (PVS, SMV, Prover, SAL, …) http: //www. cs. umn. edu/crisys Specification testing Test case generation Verifiable code generator ? Yes! Abbreviated system testing augmented with generated tests ? ? ? Automated unit testing (to MC/DC? )—to check code generator and specification execution environment ? Time Savings 30

Perfection is Not Necessary ≥ Missed Faults http: //www. cs. umn. edu/crisys • We only need to be better than what we are now… u How do we demonstrate this? n Empirical studies are of great importance 31

Education of Regulatory Agencies • Regulatory agencies are very conservative And rightly so… u Avionics software is very good u http: //www. cs. umn. edu/crisys • We need to understand regulatory and • industry concerns to get our techniques into practice We need to have convincing evidence that our techniques work and are effective 32

New Challenges for V&V • Validate models u u http: //www. cs. umn. edu/crisys • u Validate tools u • u We will rely a lot on tools for model validation, can we trust them? Creative use of testing necessary Verify and Validate generated code u u • The models must satisfy the “real” requirements Validate the properties used in analysis Model testing crucial to success u Can we trust that the translation was correct? Test automation crucial Includes output to analysis tools Adapt to the various modeling notations u u Models will not come in one language Translation between notations and tools 33

Discussion http: //www. cs. umn. edu/crisys 34