Building Dependable Distributed Systems Chapter 7 Wenbing Zhao Department of Electrical and Computer Engineering Cleveland State University wenbing@ieee. org Building Dependable Distributed Systems, Copyright Wenbing Zhao 1
Outline n Byzantine general problem q n Practical Byzantine fault tolerance q q n By Leslie Lamport, Robert Shostak, & Marshall Pease By Miguel Castro and Barbara Liskov, OSDI’ 99 http: //www. pmg. csail. mit. edu/papers/osdi 99. pdf Zyzzyva 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 2 Wenbing Zhao
The Byzantine Generals Problem n n n Abstract model of a computer system that may have faulty components Faulty components may send conflicting information to different parts of the system Scenario where Byzantine Generals must reach agreement in the presence of traitors 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 3 Wenbing Zhao
The Byzantine Generals Scenario General Commanding General Lieutenant General Byzantine Army Division Enemy City General Lieutenant General Traitorous General Byzantine Army Division 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 4 Wenbing Zhao
Byzantine Generals Problem n A commanding general must send an order to his n-1 lieutenants such that q q n n n IC 1. All loyal lieutenants obey the same order IC 2. If the commanding general is loyal, then every loyal lieutenant obeys the order he sends IC 1 = Agreement clause IC 2 = Validity clause IC 1 and IC 2 are called interactive consistency Conditions 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 5 Wenbing Zhao
Byzantine Agreement Protocol n Assumption: q Every message that is sent is delivered correctly n q The receiver of a message knows who sent it n q Traitors cannot interfere with messages they do not sent Traitors cannot spoof messages The absence of a message can be detected n Traitors cannot prevent an agreement by not sending => Synchronous system + no spoofing 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 6 Wenbing Zhao
Byzantine Agreement Protocol (f=1) n n Round 1: the commander sends a value to each of the lieutenants Round 2: each of the lieutenants sends the value it received to its peers At the end of round 2, each lieutenant check to see if there is a majority opinion (attack or retreat). We have a solution if there is Question is: how many generals needed to tolerate f number of traitors? Note: f>=2 cases are much more complicated and expensive (Oral message algorithms) 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 7 Wenbing Zhao
Unsolvable Situations – N=3, f=1 Attack lieutenant Commander He said Retreat Attack lieutenant Commander Attack lieutenant 3/16/2018 Retreat He said Retreat Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing lieutenant 8 Wenbing Zhao
Byzantine Agreement Protocol (f=1) Commander Attack lieutenant He said Retreat lieutenant He said Attack He said Retreat He said Attack n Retreat lieutenant He said Attack If there are f traitors, then there must be at least 3 f + 1 total generals for IC 1 and IC 2 to hold 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 9 Wenbing Zhao
Byzantine Agreement Protocol (f=1) n n Under our assumption, if message digital signature is used and assuming the signature cannot be forged, we need only N=2 f+1 to tolerate f traitors q The commander still can send different information to different lieutenant, but a lieutenant cannot lie about what the commander has told him In asynchronous systems, N=2 f+1 is not sufficient q We have to stop after collecting f+1 input because the f faulty traitors could simply refrain from sending q Unfortunately there might be f inputs from traitors 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 10 Wenbing Zhao
Introduction to BFT Paper n n The growing reliance of industry and government on online information services Malicious attacks become more serious and successful More software errors due to increased size and complexity of software This paper presents “practical” algorithm for state machine replication that works in asynchronous systems like the Internet 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 11 Wenbing Zhao
Assumptions n n n Asynchronous distributed system The network may fail to deliver, delay, duplicate or deliver them out of order Faulty nodes may behave arbitrarily Independent node failures The adversary cannot delay correct nodes indefinitely All messages are cryptographically signed by their sender and these signatures cannot be subverted by the adversary 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 12 Wenbing Zhao
Service Properties n n n A (deterministic) service is replicated among ≥ 3 f+1 processors. Resilient to ≤ f failures Safety: All non-faulty replicas guaranteed to process the same requests in the same order Liveness: Clients eventually receive replies to their requests 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 13 Wenbing Zhao
Optimal Resiliency n n n Imagine non-faulty processors trying to agree upon a piece of data by telling each other what they believe the data to be A non-faulty processor must be sure about a piece of data before it can proceed f replicas may refuse to send messages, so each processor must be ready to proceed after having received (n-1)-f messages q Total of n-1 other replicas 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 14 Wenbing Zhao
Optimal Resiliency But what if f of the (n-1)-f messages come from faulty replicas? n To avoid confusion, the majority of messages must come from non-faulty nodes, i. e, (n-f-1)/2 ≥ f => Need a total of ≥ 3 f+1 replicas n 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 15 Wenbing Zhao
BFT Algorithm in a Nutshell Backup f + 1 Match (OK) Client Primary Backup 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 16 Wenbing Zhao
Replicas and Views Set of replicas (R): |R| ≥ 3 f + 1 R 0 R 1 R 2 ……… R|R-1| 0 View 1 For view v: primary p is assigned such that p= v mod |R| 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 17 Wenbing Zhao
Safeguards n n n If the client does not receive replies soon enough, it broadcasts the request to all replicas If the request has already been processed, the replicas simply re-send the reply (replicas remember the last reply message they sent to each client) If the primary does not multicast the request to the group, it will eventually be suspected to be faulty by enough replicas to cause a view change 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 18 Wenbing Zhao
Normal Case Operation Client {REQUEST, o, t, c} Primary o – Operation t – Timestamp c - Client Timestamps are totally ordered such that later requests have higher timestamps than earlier ones 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 19 Wenbing Zhao
Normal Case Operation n n Primary p receives a client request m , it starts a three-phase protocol Three phases are: pre-prepare, commit 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 20 Wenbing Zhao
Pre-Prepare Phase Backup Primary <<PRE-PREPARE, v, n, d> , m> v – view number n – sequence number d – digest of the message D(m) m – message 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing Backup 21 Wenbing Zhao
Prepare Phase n A backup accepts the PRE-PREPARE message only if: q q The signatures are valid and the digest matches m It is in view v It has not accepted a PRE-PREPARE for the same v and n Sequence number is within accepted bounds 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 22 Wenbing Zhao
Prepare Phase n If backup i accepts the pre-prepare message it enters prepare phase by multicasting <PREPARE, v, n, d, i> n n to all other replicas and adds both messages to its log Otherwise does nothing Replica (including primary) accepts prepare message and adds them to its log, provided that q Signatures are correct q View numbers match the current view q Sequence number is within accepted bounds 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 23 Wenbing Zhao
Prepare Phase n n At replica i, prepared (m, v, n, i) = true, iff 2 f PREPARE from different backups that match the pre-prepare When prepared = true, replica i multicasts <COMMIT, v, n, d , i> to other replicas 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 24 Wenbing Zhao
Agreement Achieved n n If primary is non-faulty then all 2 f+1 non-faulty replicas agree on the sequence number If primary is faulty q q n Either ≥f+1 non-faulty replicas (majority) agree on some other sequence and the rest realize that the primary is faulty Or, all non-faulty replicas will suspect the primary is faulty When a faulty primary is replaced, the minority of confused non-faulty replicas are brought up to date up by the majority 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 25 Wenbing Zhao
Commit Phase n n n Replicas accept commit messages and insert them in their log provided signatures are same Define committed and committed-local predicates as q Committed (m, v, n) = true, iff prepared (m, v, n, i) is true for all i in some set of f+1 non-faulty replicas q Committed-local (m, v, n, i) = true iff the replica has accepted 2 f+1 commit message from different replicas that match the pre-prepare for m If Committed-local (m, v, n, i) is true for some non-faulty replica i, then committed (m, v, n) is true 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 26 Wenbing Zhao
Commit Phase n n n Replica i executes the operation requested by m after committed-local (m, v, n, i) = true and i’s state reflects the sequential execution of all requests with lower sequence numbers The PRE-PREPARE and PREPARE phases of the protocol ensure agreement on the total order of requests within a view The PREPARE and COMMIT phases ensure total ordering across views 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 27 Wenbing Zhao
Normal Operation Reply n n All replicas sends the reply <REPLY, v, t, c, i, r>, directly to the client v = current view number t = timestamp of the corresponding request i = replica number r = result of executing the requested operation c = client id Client waits for f+1 replies with valid signatures from different replicas, and with same t and r, before accepting the result r 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 28 Wenbing Zhao
Normal Case Operation: Summery Request Pre-prepare Prepare Commit Reply C Primary: 0 1 2 Faulty: 3 3/16/2018 X Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 29 Wenbing Zhao
Garbage Collection n Used to discard messages from the log For the safety condition to hold, messages must be kept in a replica’s log until it knows that the requests have been executed by at least f+1 non-faulty replicas Achieved using a checkpoint, which occur when a request with sequence number (n) is divisible by some constant is executed 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 30 Wenbing Zhao
Garbage Collection n When a replica i produces a checkpoint it multicasts a message <CHECKPOINT, n, d, i> to other replicas Each replica collects checkpoint messages in its log until it has 2 f+1 of them for sequence number n with same digest d This creates a stable checkpoint and the replica discards all the pre-prepare, prepare and commit messages 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 31 Wenbing Zhao
View Changes n n Triggered by timeouts that prevent backups from waiting indefinitely for request to execute If the timer of backup expires in view v, the backup starts a view change to move to view v+1 by, q q Not accepting messages (other than checkpoint, viewchange, and new-view messages) Multicasting a VIEW-CHANGE message 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 32 Wenbing Zhao
View Changes n VIEW-CHANGE message is defined as <VIEW-CHANGE, v+1, n, C, P, i> where, C = 2 f + 1 checkpoint messages P = set of sets Pm Pm = a PRE-PREPARE msg + all PREPARE messages for all messages with committed = false 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 33 Wenbing Zhao
View Change - Primary n n Primary p of view v+1 receives 2 f valid VIEW-CHANGE messages Multicasts a <NEW-VIEW, v+ 1, V, O> message to all other replicas where q q n V = set of 2 f valid VIEW-CHANGE messages O = set of reissued PRE-PREPARE messages Moves to view v+1 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 34 Wenbing Zhao
View Changes - Backups n n Accepts NEW-VIEW by checking V and O Sends PREPARE messages for everything in O q n These PREPARE messages carry view v+1 Moves to view v+1 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 35 Wenbing Zhao
Events Before the View Change n n n Before the view change we have two groups of non-faulty replicas: the Confused minority and the Agreed majority A non-faulty replica becomes Confused when it is kept by the faulty's from agreeing on a sequence number for a request It can't process this request and so it will time out, causing the replica to vote for a new view 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 36 Wenbing Zhao
Events Before the View Change n n n The minority Confused replicas send a VIEW-CHANGE message and drop off the network The majority Agreed replicas continue working as long as the faulty's help with agreement The two groups can go out of synch but the majority keeps working until the faulty's cease helping with agreement 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 37 Wenbing Zhao
System State: Faulty Primary Is Erroneous View Change Possible? System State Confused Minority Agreed Majority ≤f non-faulty replicas ≥f+1 non-faulty replicas Agreed Majority ≥f+1 non-faulty replicas Confused Minority ≤f non-faulty replicas Adversary f non-faulty replicas f faulty replicas 3/16/2018 f non-faulty replicas f faulty replicas P P ≤ 2 f replicas: NOT enough to change views Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 38 Wenbing Zhao
Events Before the View Change n Given ≥f+1 non-faulty replicas that are trying to agree, the faulty replicas can either help that or hinder that ➲ ➲ n If they help, then agreement on request ordering is achieved and the clients get ≥f+1 matching replies for all requests with the faulty's help If they hinder, then the ≥f+1 non-faulty's will time out and demand for a new view When the new majority is in favor of a view change, we can proceed to the new view 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 39 Wenbing Zhao
System State: Faulty Primary Is it possible to continue processing requests? System State Confused Minority ≤f non-faulty replicas Agreed Majority ≥f+1 non-faulty replicas Adversary f non-faulty replicas f faulty replicas 3/16/2018 ≥f+1 non-faulty replicas Adversary P P f faulty replicas YES ≥ 2 f+1 replicas: enough for agreement Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 40 Wenbing Zhao
System State: Faulty Primary Majority now large enough to independently move to a new view Confused Minority ≤f non-faulty replicas Confused Majority 2 f+1 non-faulty replicas Enough to agree to change views Agreed Majority ≥f+1 non-faulty replicas Adversary P f non-faulty replicas f faulty replicas YES ≥ 2 f+1 replicas: enough for agreement Faulty replicas cease helping with agreement 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 41 Wenbing Zhao
Liveness n n Replicas must move to a new view if they are unable to execute a request To avoid starting a view change too soon, a replica that multicasts a view-change message for view v+1, waits for 2 f+1 view-change messages and then starts the timer T If the timer T expires before receiving new-view message it starts the view change for view v+2 The timer will wait 2 T before starting a view-change from v+2 to v+3 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 42 Wenbing Zhao
Liveness n n n If a replica receives f+1 valid view-change messages from other replicas for views greater than its current view, it sends a view-change message for the smallest view in the set, even if T has not expired Faulty replicas cannot cause a view-change by sending a view-change message since a viewchange will happen only if at least f+1 replicas send view-change message The above techniques guarantee liveness, unless message delays grow faster than the timeout period indefinitely 3/16/2018 Building Dependable Distributed Systems, EEC 688/788: Secure & Dependable Copyright Wenbing Zhao Computing 43 Wenbing Zhao
Zyzzyva: Speculative BFT n n A replica speculatively executes a request as soon as it receives a valid pre-prepare msg Commitment of a request is moved to the client q q n A request is said to have completed at hthe issuing client if the corresponding reply can be safely delivered If a request completes at a client, the request will eventually be committed at the server replicas Prepare and commit phases are reduced to a single phase q View change has one more additional phase Building Dependable Distributed Systems, Copyright Wenbing Zhao 44
Zyzzyva: Speculative BFT n n History hash: helps client determine if its request has been ordered appropriately Server replica maintains a history hash for each request ordered and appends the history hash hs = H(hs-1, ds) to the reply for the request with sequence number s q q ds: digest for the request Previous hash: hs-1, new hash with request s executed: hs H(): hash function Prefix concept: his is a prefix of hsj if sj > si, and there exist a set of requests with sequence numbers si+1, si+2, …, sj-1 with digests dsi+1, dsi+2, …, dsj-1 such that hsi+1=H(hsi, dsi+1), hsi+2 = H(hsi+1, dsi+2), …, hsj=H(hsj-1, dsj) Building Dependable Distributed Systems, Copyright Wenbing Zhao 45
Zyzzyva: Speculative BFT n n Safety: given any two requests that have completed, they must have been assigned two different sequence numbers. Furthermore, if the two sequence numbers are i and j an i < j, the history hash hi must be a prefix of hj Liveness: if a nonfaulty client issues a request, the request eventually completes Building Dependable Distributed Systems, Copyright Wenbing Zhao 46
Zyzzyva: Agreement Protocol n n A client maintains a complete timer for each request A request may complete at the client in one of two cases: q q Case 1: client receives 3 f+1 matching replies => all replicas have executed the request in the same total order Case 2: client receives at least 2 f+1 matching replies when timer expires. The client then initiates another round of msg exchange with the server replicas before the request is declared as complete Building Dependable Distributed Systems, Copyright Wenbing Zhao 47
Zyzzyva: Agreement Protocol, Case 1 Building Dependable Distributed Systems, Copyright Wenbing Zhao 48
Zyzzyva: Agreement Protocol, Case 2 Building Dependable Distributed Systems, Copyright Wenbing Zhao 49
Zyzzyva: Case 2 n Commit msg: contains a commit certificate: q q n Local-commit: a replica sends a local-commit when it receives a valid commit msg from client q n A list of 2 f+1 replica ids Signed component of spec-response from each of the 2 f+1 replicas Verify history hash When the client receives 2 f+1 or more local-commit, the request is completed and can be delivered Building Dependable Distributed Systems, Copyright Wenbing Zhao 50
Zyzzyva: View Change Protocol n When a view change is triggered q q n Sufficient number of backups timed out the current primary When the client receives spec-response msgs with different sequence numbers or different history hash values In the 2 nd case, client broadcasts a POM msg to all replicas q q Replica initiates a view change A replica also rebroadcasts POM upon receiving one => to speed up view change Building Dependable Distributed Systems, Copyright Wenbing Zhao 51
Zyzzyva: View Change Protocol n What is special in Zyzzyva q q n Impact on view change: q q n At most one round msg exchange during normal operation for agreement => equivalent to prepare phase (or commit phase), for case 2. Replicas would have commit certificate For case 1, replicas would not possess a commit certificate Need an additional round msg exchange – “I hate the primary” Need to change condition on including a request in new-view msg Cannot use PBFT view change protocol: may lose liveness q q q Faulty primary could force f nonfaulty replicas to suspect itself, but cooperate with other f+1 nonfaulty replicas f faulty replicas stops sending spec-response Client cannot complete any request: only receive f+1 matching replies Building Dependable Distributed Systems, Copyright Wenbing Zhao 52
Zyzzyva: View Change Protocol n PBFT view change q n A replica abandons the current view as soon as it suspects the primary: stops participating agreement for current view Zyzzyva view change q q Make sure a view change will actually take place before abandoning the current view Accomplished by using the “I hate primary” msg exchange: a replica abandons the current view only when it receives f+1 “I hate primary” msgs Building Dependable Distributed Systems, Copyright Wenbing Zhao 53
Zyzzyva: View Change Protocol n Zyzzyva view change: deal with case 1 q q A replica includes all order-req msgs received since latest stable checkpoint, or the most recent commit certificate New primary adopts request-seq# binding if f+1 or more matching order-req msgs The primary may see multiple sets for same seq# (for different requests): can select any of the bindings because none of them could have been completed A backup should accept the decision by the new primary for the ordering Building Dependable Distributed Systems, Copyright Wenbing Zhao 54
Скачать презентацию Building Dependable Distributed Systems Chapter 7 Wenbing Zhao
d81060a0a45127951dc703dc93a4d954.ppt