Recovery Oriented Programming Olga Brukman and Shlomi Dolev

Recovery Oriented Programming Olga Brukman and Shlomi Dolev Ben-Gurion University Beer-Sheva Israel

Towards Correct Software • Software should respects its specifications – Safety, Liveness • Atomic power station – Safety: the atomic station shouldn't explode – Liveness: the atomic station should produce some electricity 2 Atomic power station

Recovery Oriented Design • Software performs substantially in accordance with specifications for a period of 90 days. . . (IEEE Computer, October 2006) • How to cope with such software? ! – Recovery Oriented Computing [PBB'02]! • Recovery actions – Reboot, wait, reschedule – Non-intrusive: avoid rewriting the program (possibly new other bugs) 3

Recovery Oriented Programming • Specifications Composer (Project Manager) – Invariants and predicates • important properties on program IO – Recovery actions 4 • Programmer • Best-effort implementation • Using same IO variables as specifier • Still: bugs and unexpected states

Recovery Oriented Programming: Assumptions • Self-stabilizing processor • Self-stabilizing OS • Infrastructure for robust monitoring and recovery • Processes exist and execute their code 5

Recovery Oriented Programming: Assumptions • Not immediately Byzantine – eventual Byzantine program Long enough to do sufficient job

Our Framework External Monitor Subsystems hierarchy event-driven monitoring Recovery tuples External Monitor event-driven monitoring Pre-compiler External Monitor Code event-driven monitoring System is able to recover from any state 7 Subsystem External Monitor

Generated Code: One Process External Monitor Recovery tuples event-driven monitoring Code event-driven monitoring

Generated Code: Subsystems hierarchy Recovery tuples External Monitor event-driven monitoring External Monitor Code 9 event-driven monitoring Subsystem External Monitor

Our Framework: Transforming Recovery Tuples into Code External Monitor Subsystems hierarchy Recovery tuples event-driven monitoring External Monitor event-driven monitoring Pre-compiler External Monitor Code event-driven monitoring Subsystem External Monitor 10

Safety Recovery Tuple 1 process PRED: x!=7 RA: this. restart() . . . x=a; . . . 11 Pre-compiler temp_x=a; if temp_x!=7 x=temp_x; else this. restart();

Safety Recovery Tuple in the Scope of Stabilization: External Monitoring 1 process if !(ps. x!=7) ps. restart(); PRED: x!=7 RA: this. restart(); . . . x=a; . . . 12 Pre-compiler No more x=. . . temp_x=a; if temp_x!=7 x=temp_x; else this. restart(); . . .

Liveness Recovery Tuple 1 process INV: eventually x+y=15 RA: this. restart() HTR: history={} Pre-compiler x=x+2; . . . y=y+5; . . . history=history▪this. state(); History= [. . . {. . , x=1, y=2, . . }, {. . , x=3, y=7, . . }, . . . ] 13 x=x+2; if (x+y==15) this. history={}; . . . y=y+5; if (x+y==15) this. history={}; if loop in history and CPU(this) ps. restart();

Generated Monitoring Code for Subsystem sub: p 1, p 2 History= [. . . distributed snapshot(sub), . . . ] External monitor for sub Recovery Tuples External Monitor Pre-compiler Code for p 1 Code for p 2 14 External Monitor event-driven monitoring

Generic Correctness Theorem • In the program produced by the precompiler every rsf (restart supporting fair)-execution E has a suffix in which the program respects its specification function – A rsf-execution is the execution in which system is trusted to behave according to its specifications after restart. 15

Generic Correctness Proof • Assumption: Processes and external monitors are scheduled fairly due to presence of self-stabilizing software platform • Safety: process either reaches monitoring section in its code or its external monitor makes scheduled check – Subsystem: external monitor makes scheduled check 16

Generic Correctness Proof Cont. • Liveness: the process (subsystem) external monitor makes scheduled check of the history log • Corrupted history: – If causes (unnecessary) recovery - trimmed – New correct records are eventually accumulated and reflect the real state of system 17

Related Work: Perfect Software • Formal specification languages – ASM [GRS'04], IO Automata [L'96], NURPL [CKB'84] – Gradually and manually translated into fully verified program • Model checking – Doesn't scale • Specification embedding programming languages – SRC (Software Cost Reduction) language [RLHL'06] – Programmer bugs 18

Related Work: Programming Tools • Design By Contract – Eiffel, i. Contract for Java – Checking invariants on an object state, pre/post-conditions on object methods, recovery by predefined recovery action – Partial monitoring of liveness, based on timeout – Monitoring of safety outside of stabilization scope • Exceptions 19

Related Work: Online Recovery • Recovery blocks (N-programming) [RX 94] • ROC [PBB 02], Java MOP[CR'05], Kinesthetics e. Xtreme [KPGV'03], "On Modeling and Tolerating Incorrect Software" [AT'03] • Monitoring/correcting layer that alternates the failed component behaviour 20

Related Work: Online Recovery • Assumption of monitoring/correcting layer stability – ROC [PBB 02], Java MOP[CR'05], Kinesthetics e. Xtreme [KPGV'03] • Intrusive correcting actions – Empty program: correcting actions define the program • "On Modelling and Tolerating Incorrect Software" [AT'03] 21

Conclusions • Recovery Oriented Programming paradigm for a programming language • Full monitoring of safety and liveness properties in the scope of stabilization • Formal correctness proof scheme for the resulting code 22