Research Heaven West Virginia IV V Facility SA

Research Heaven, West Virginia IV&V Facility SA @ WV (software assurance research at West Virginia) Kenneth Mc. Gill NASA IV&V Facility Research Lead 304. 367. 8300 Kenneth. Mc. Gill@ivv. nasa. gov Dr. Tim Menzies Ph. D. (WVU) Software Engineering Research Chair tim@menzies, com 1

IV&V Facility Why, what is software assurance? Research Heaven, West Virginia • Why software assurance? –bad software can kill good hardware. –E. g. ARIANE 5: (and many others) Ariane 5 • Software errors in inertial reference system • Floating point conversion overflow • Definition: Before bad software After bad software – Planned and systematic set of activities – Ensures that software processes and products conform to requirements, standards, and procedures. • Goals: – Confidence that SW will do what is needed when it’s needed. 2

OSMA Software Assurance Research Heaven, Research Program West Virginia IV&V Facility • • Office of Safety & Mission Assurance (Code Q- OSMA) Five million per year Applied software assurance research Focus: – Software, not hardware – SW Assurance – NASA-wide applicability • Externally valid results; i. e. useful for MANY projects • Organization: – Managed from IV&V Facility – Delegated Program Manager: Dr. Linda Rosenberg, GSFC 3

Many projects IV&V Facility • • Horn of plenty • Research Heaven, West Virginia Mega: highest-level perspective – e. g. project planning tools like ASK-PETE [Kurtz] Macro: – e. g. understanding faults [Sigal, Lutz & Mikulski] Micro: – e. g. source code browsing [Suder] Applied to basic: – Applied: • (e. g. ) MATT/RATT [Henry]: support large scale runs of MATLAB – Basic (not many of these) • e. g. Fractal analysis of time series data [Shereshevsky] Many, many more – Too numerous to list – Samples follow – See rest of SAS! 4

Many more projects! IV&V Facility Research Heaven, West Virginia Ratio FY 02/FY 01 Total proposals: NASA centers: Industry: University: 2. 2 1. 5 26 3. 7 Good news! • More good proposals than we can fund Bad news! • same as the good 5

A survey of 44 FY 01 CSIPs IV&V Facility project 1 2 3 4 5 6 AATT ISS Space Shuttle ST 5 CHIPS 10 11 12 2 1 1 Aura 9 7 CLCS CM 2 DSMS CMMI EOSDIS 13 8 14 to 44 Research Heaven, West Virginia FAMS 1 GLAST 1 HSM 4 1 HST 1 Mars 07 1 Mars 08 1 1 75% with no claim for project connections PCS Space Station Starlight Stereo 1 SWIFT 1 X-38 1 4 3 2 2 5 2 2 2 1 1 (but don’t forget theory) 1 Need more transitions! 1 1 1 0 6

IV&V Facility Action plan- restructure CSIPS: more transitions! Research Heaven, West Virginia • New (year 1) – Fund many • Renewed (year 2) – Continue funding the promising new projects – Recommended: letter of endorsement from NASA project manager • Transition (year 3) – Select a few projects – Aim: tools in the hands of project folks – Required: project manager involvement • Reality check: – Transition needs time – Data drought 7

Long transition cycles IV&V Facility • Mars atmosphere Research Heaven, West Virginia Pecheur & practical formal methods – In-Situ Propellant Production project – Taught developers: • Livingstone model-based diagnosis CH 4 + O 2 • model-checking tools • developed by Reid Simmons, oxidizer (CMU) – Technology to be applied to the Intelligent Vehicle Health Maintenance (IVMS) for 2 nd generation shuttles on-board CO 2 + 2 H 2 —> fuel • Lutz, Mikulski & ODC-based analysis of defects – Deep-space NASA missions – Found 8 clusters of recurring defects – Proposed and validated 5 explanations of the clusters – Explanations changes to NASA practices – ODC being evaluated by JPL’s defect management tool team Charles Pecheur RIACS, ASE, ARC Robyn Lutz JPL, CS-Iowa State (no photo) Carmen Mikulski JPL 8

IV&V Facility The data drought Research Heaven, West Virginia Gasp… need data… 9

IV&V Facility End the drought: bootstrap off other systems Research Heaven, West Virginia • Find the enterprise-wide management information system • Insert data collection hooks – E. g. JPL adding ODC to their defect tracking system – WVU SIAT sanitizer 10

IV&V Facility End the drought: Contractors as researchers • Buy N licenses of a defect tracking tool (e. g. Clearquest) • Give away to projects Research Heaven, West Virginia take me to your data – In exchange for their data • Build and maintain a central repository for that data – With a web-based query interface • Data for all active data 11 repository

End the drought: Contractors as researchers (2) IV&V Facility 4 Mark Suder Titan, IV&V See also: experience 1 action • Titan’s new ROI project reflection 2 abstraction Research Heaven, West Virginia 3 } 4 Hypertext power browser for source code 1’ Use it. 2’ For high-severity errors, recall what SIAT queries lead to finding those errors 3’ Assess each such “power queries” Reject the less useful ones 4’ Procedures manual for super SIAT or new search options in interface SIAT-1 } • Any contractor proposing an NRA • Galaxy Global’s metric project SIAT 2 12

IV&V Facility End the drought: raid old/existing projects Research Heaven, West Virginia • Cancelled projects with public-domain software – E. g. X-34 • Or other open source NASA projects – E. g. GSFC’s ITOS: – real-time control and monitoring system during development, test, and on-orbit operations, – UNIX, Solaris, Free. BSD, Linux, PC – Free!! – NASA project connections: • • Triana, Swift, HESSI, ULDB, SMEX, Formation Flying Testbed, Spartan 13

End the drought: synergy groups IV&V Facility Research Heaven, West Virginia • N researchers – Same task – Different technologies • Share found data • E. g. IV&V business case workers • E. g. monthly fault teleconferences – JPL: • Lutz, Nikora – Uni. Kentucky: • Hayes – Uni. Maryland: • Smidts – WV: • Chapman (Galaxy Global) & Menzies (WVU) 14

IV&V Facility End the drought: Tandem experiments Research Heaven, West Virginia • “Technique X finds errors” – So? • Industrial defect detection capability rates: – TR(min, mean, max) – TR(0. 35, 0. 50, 0. 65) – Assumes manual “Fagan inspections” • Is “X” better than a manual 1976 technique? • Need “tandem experiments” to check • I. e. do it twice – Once by the researchers – Once by IV&V contractors (baseline) fictional data 15

IV&V Facility Alternatively: End your own drought Research Heaven, West Virginia • Our duty, our goal: – Work the data problem (e. g. see above) – Goal of CI project year 1: build bridges – But the more workers, the better • Myth: there is a “data truck” parked at IV&V – full of goodies, just for you • Reality: Access negotiation takes time – With contractors, within NASA • We actively assist: – Each connection is a joy to behold, an occasion to celebration – We don’t celebrate much • Bottom line: – We chase data for dozens of projects – Researchers have more time, more focus on their particular data needs • Ken’s law: – $$$ chases researchers who chase projects – CI year 2, year 3: needs a project connection 16

IV&V Facility • Alternatively (2), accept the drought and sieve the dust Research Heaven, West Virginia The DUST project: – Assumes a few key options control the rest • • Methodology: The answer my – Simulate across range of options friend, is blowin’ – Data dust clouds in the wind – Too many options: what leads to what? – Summarize via machine learning – Condense dust cloud – Improve mean, reduce variance Case studies: – JPL requirements engineering: • Feather/JPL [Re 02] – Project planning: Each dot = 1 random project plan But wait: the times they are changing • DART- Raque/ IVV; Chaing/UBC; • IV&V costing: Marinaro/IVV, Smith/WVU • general: Raffo, et. al/PSU [Ase 02] – An analysis of pair programming: Smith/WVU – Better predictors for: • testability: Cukic/WVU, Owen/WVU [Issre 02, Ase 02] • faults: di. Stefano/WVU, Mc. Gill/IVV; Chapman/GG • reuse : di. Stefano/WVU [Tools. With. AI 02] 17

Other WVU SA research IV&V Facility Testing & formal methods Bojan Cukic Hany Ammar Katerina Goseva Popstojanova UML (sequence diagrams, state charts) Software Specs & design UML simulations (early life cycle) Architectural Static (SIAT, descriptions Mccabe, entrophy) Code analysis (iv&v, operational Dynamic (testing, usage) runtime monitoring) Fault, failure data on components, connectors Research Heaven, West Virginia Bayesian approach to reliability Architectural metrics Risk assessment & dynamic UML Reliability & operational profile errors collaborator Goal: accurate, stable, risk assessment early in the lifecycle Metrics(complexity, coupling, entropy ) Failure data from testing Severity of failures 18

More WVU research (FY 02 UIs) IV&V Facility Architectural metrics Ammar Risk assessment & dynamic UML Cukic Intelligent flight controllers Testing & formal methods Bayesian approach to reliability Goseva. Popstojanova cccccc jc X 34 jj ITOS w F 15 SIAT X 38 c SE research chair FY 03 proposals = 2. 2*FY 02 interns Menzies new renewed ISS hub controller, “Dryden application” c Fractal study of resource dynamics Reliability & operational profile errors Research Heaven, West Virginia c = conference w = workshop j = journal DUST j, ccccccc, w “JPL deep space mission” DART “KC-2” IVV cost models 19

IV&V Facility • Function Point Metrics for Safety-Critical Software Thesis: – Traditional function-point cost estimation – Incorrect for safety-critical software • Design Diversity, add eight more > 1 way to skin a cat – >1 way to realize a safety critical function: – NCP= N-copy programming – NVP= N-Version Programming , – NSCP= N Self-Checking Programming, – … – With, without redundancy, • Research Heaven, West Virginia Method: – explore them all! Design Diversity, add one more Data Diversity H 2 and C 2 : effort & cost, redundant system H 1 and C 1: effort & cost, non-redundant Afzel system Noore 20

IV&V Facility Pre-disaster warnings [Cukic, Shereshevsky] Research Heaven, West Virginia Can we defer a maintenance cycle and keep doing science for a while longer? Mark Shereshevsky Bojan Cukic } Early warning Time for graceful shutdown Crash ARTS II 21

IV&V Facility Intelligent flight controllers [Napolitano, Cukic] (and menzies) Research Heaven, West Virginia Marcello Napolitano Bojan Cukic (Mechanical and (CSEE) ) Aerospace Lifecycle opportunities for V&V of neural network based adaptive control systems. 22

The road ahead: applied & theoretical research IV&V Facility Research Heaven, West Virginia Need both CSIPs: applied research USIPs: applied + theoretical research To boldly go… 23