FAUST Fail-Aware Untrusted Storage Alex Shraer Technion Israel

FAUST: Fail-Aware Untrusted Storage Alex Shraer, Technion, Israel Joint work with: Christian Cachin IBM Zurich Idit Keidar Technion

Agenda • Motivation • Defining a Fail-Aware Untrusted Service • FAUST: Fail-Aware Untrusted STorage • Conclusions

Once upon a time… strong consistency (linearizability/atomicity) Don't be Evilstrong liveness (wait-freedom) (Dobe. E) Inc. Dobe. E Online. Docs DSN paper Alice Bob Jlduiy baios kjshh jcouhc lj ljxc jcjjzxkcj hyah dg dgf gh hnbbv hjy v fg djjiaj djsjd , lmcgg nhycgygb cnjik d ji khcgtu jnn mckpkk, c jjkpackm kj ihgdiash hasdg gdsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd , lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk, c jjkpackm kj ihgdiashhasdg hays v fg djjiaj djsjd , lmcgg nhycgygb cnjik d mjikhcg tujnn mckpkk, c jjkpackm kj ihgdiashhasdg gdsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd , lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk, c jjkpackm kj ihgdiashhasdg dsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd , lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk, c jjkp ackm kj ihgdiashhasdg hjy v fg djjiaj djsjd , lm cgg nhycgygb cnjik d mjikhcgtujnn mckpkk, c packm kj ihgdiashhasdg gdsihasd jddfg dgtrsd hjy v fg djjiaj djsjd , lmcgg nhycgygb Alice Bob sometimes (email)

But what if… Back in High school… DSN paper Alice Bob Jlduiy baios kjshh jcouhc lj ljxc jcjjzxkcj hyah dg dgf gh hnbbv hjy v fg djjiaj djsjd , lmcgg nhycgygb cnjik d ji khcgtu jnn mckpkk, c jjkpackm kj ihgdiash hasdg gdsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd , lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk, c jjkpackm kj ihgdiashhasdg hays v fg djjiaj djsjd , lmcgg nhycgygb cnjik d mjikhcg tujnn mckpkk, c jjkpackm kj ihgdiashhasdg gdsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd , lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk, c jjkpackm kj ihgdiashhasdg dsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd , lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk, c jjkp ackm kj ihgdiashhasdg hjy v fg djjiaj djsjd , lm cgg nhycgygb cnjik d mjikhcgtujnn mckpkk, c packm kj ihgdiashhasdg gdsihasd jddfg dgtrsd hjy v fg djjiaj djsjd , lmcgg nhycgygb Bob Alice

What would we like to achieve? • No bogus data – Still, inconsistencies are possible • Alice and Bob to detect inconsistencies – Rollback to consistent state • When server is correct (the common case) clients should have strong consistency and liveness for their operations – Linearizability and wait-freedom

Agenda • Motivation • Defining a Fail-Aware Untrusted Service • FAUST: Fail-Aware Untrusted STorage • Conclusions

Eventual Consistency • Client’s operations can complete even though they are only “weakly” consistent • Client is notified when its operations become “strongly” consistent – May invoke other operations without waiting for notifications • Eventual consistency is widely accepted and used – Starting with Bayou (SOSP 95) – Today in commercial systems, e. g. , Amazon’s Dynamo (SOSP 07)

Fail-Awareness [Fetzer & Cristian 96] • Provide notifications when the service does not guarantee its specified semantics – means that the service shouldn’t be used • Allows clients to take appropriate recovery actions

Fail-Aware Untrusted Service • When the server is correct – strong consistency and strong liveness • Execution is always “weakly” consistent – at least causal consistency • Stability notifications (next slide) • Accurate failure notifications • Detection completeness: All operations eventually become stable unless failure is detected

Example with Correct Server • Alice & Bob are from Europe, Carlos is from America • Its daytime in Europe, and Carlos is asleep • Alice’s “stability cut”: • Stability w. r. t all clients = perfect consistency

Agenda • Motivation • Defining a Fail-Aware Untrusted Service • FAUST: Fail-Aware Untrusted STorage • Conclusions

Fork consistency Server may present different views to clients [Mazières&Shasha, PODC 02] “Fork” their views of the history In such case, their views are forked ever after (never again see each others new updates) or else, server is exposed as faulty

Fork-Consistency – an Example C 1 is forked from C 3 C 2 is forked from C 1 C 1 C 2 write(X 1, u) read(X 3) C 3 is forked from C 2 → null read(X 3) → null read(X 1) → u C 3 write(X 3, w) start (X 1= X 2 = X 3 = null) C 2 write(X 1, u) read(X 1) → u C 3 write(X 3, w) read(X 1) → null read(X 3) → null C 1 read(X 3) → null

What about the common case? SUBMIT write operation read operation REPLY: I scheduled op 1, op 2, … before you REPLY: I scheduled of corresponding write value, signed context op 1, op 2, … before you COMMIT op (signed context) Client C 1 C 2 Server My context is: start (I am the first C 2 is forked from C 1 operation) write(X 1, u) read(X 1) → null A Join – not allowed with fork consistency read(X 1) → u Something is wrong! My context is: start (I am the first operation) C 2 must receive signed context of the write Must wait until the write commits What if the write crashes? Formal proof in [Cachin, Shelat, Shraer 07]

We need a weaker consistency notion Two weaker conditions have been defined recently: Fork-* [Li & Mazieres, NSDI 2007] Fork-sequential-consistency [Oprea & Reiter, DISC 2006] We show: both require blocking, in executions with 1 correct server C 2

Weak fork At-most one join: If server forks the views of two clients, they may see at most one more common operation Trust server about last operations by other clients in context Same as in Fork-* No checks needed Weaker than Fork-* Causally consistent views Stronger than Fork-* C 2 No blocking needed - allows wait-free protocols. In fact, suffices to achieve all properties of FAUST

USTOR: The untrusted storage protocol SUBMIT read operation REPLY: I scheduled op 1, op 2, … before you value, signed context of last committed op of writer COMMIT op (signed context) Client C 1 C 1 C 2 Server claims that C 1 committed no operations write(X 1, u) read(X 1) → null C 2 Server read(X 1) → u write(X 1, u) write(X 1, v) read(X 1) → null read(X 1) → u read(X 1) → v Server can hide context of second write, but must provide context of first write C 2 detects the error

Some properties of USTOR • Linearizable executions with correct server – A correct server schedules & processes the ops in order of receipt • Wait-free with correct server – A read does not need to wait till some write COMMITs – COMMIT info can arrive with SUBMIT of next operation • All executions are causally consistent – From properties of Weak-fork • Detects server failure, if server returns inconsistent context information • What’s missing for FAUST? – Accurate notifications to clients: operation stability, server failure – Completeness: for every op, eventually one of the following happens: • op indicated as stable with respect to other clients • server failure detected

From USTOR to FAUST Application stability notifications read / write server failure notifications FAUST Protocol read / write USTOR Protocol (client side) PROBE CONTEXT FAILURE SUBMIT COMMIT REPLY Client-Server Channel Client-to-Client Comm.

FAUST Layer at client Ci • Maintain the latest known context: contexti • Periodically (when no user ops) read data stored by others – Find out their latest known context, update contexti – Ci always makes operations – others will see that contexti changes • If no new info from Ck for a long time: – send a direct PROBE message (e. g. , email) to Ck – Ck responds with CONTEXT message containing contextk • If contextk is inconsistent with contexti – send a FAILURE message to all – report failure to application • Otherwise, if contextk includes j-th operation of Ci – j-th operation reported stable at Ci w. r. t. Ck

Conclusion • Need for semantics and client-side tools for services provided by “clouds” – Clients should be able to detect if semantics are violated – ACM SIGACT News Column 34, Distributed Computing in the Clouds; http: //www. ee. technion. ac. il/~idish/sigact. News • Our suggestion: Fail-Aware Untrusted Service – FAUST provides such semantics for simple storage • When a read/write operation completes causal consistency (at least) is guaranteed • Eventually, more information is obtained. Then: – Indicate to client that his operation preserves stronger semantics OR: report server failure • Common case: when server is correct, each operation completes independently of others + perfect consistency