Reliable Distributed Systems Astrolabe The Internet Massive

Reliable Distributed Systems Astrolabe

The Internet Massive scale. Constant flux Source: Burch and Cheswick

Demand for more “autonomic”, intelligent behavior n Human beings constantly adapt as their environment changes n n n You bike up hill… start to breath hard and sweat. At the top, cool off and catch your breath It gets cold so you put on a sweater But computer systems tend to be rigid and easily disabled by minor events

Typical examples n Customers find that many Web Services systems are perceived as unreliable n n n End-user gets erratic response time Client could be directed to the wrong server site But the WS software isn’t at fault! n n Usually these problems arise from other systems to which WS is connected A snarl of spaghetti sits behind the front end

A tale of woe n Human operators lack tools to see state of system n n n Can’t easily figure out what may be going wrong In fact operators cause as much as 70% of all Web-related downtime! And they directly trigger 35% of crashes

Sample tale of woe n Fly. High. com Airlines maintains the “Fly. High. com Nerve Center” in Omaha n n n It has an old transactional mainframe for most operations Connected to this are a dozen newer relational database applications on clusters These talk to ~250, 000 client systems using a publish-subscribe system

Fly. High. com Architecture Omaha transactional system Enterprise clusters: flight plans, seat assignments, employee schedules, etc Seat request Anchorage check-in

Anchorage goes down n Problem starts in Omaha n n n System operator needs to move a key subsystem from computer A to B But the DNS is slow to propagate the change (maybe 48 to 72 hours) Anchorage still trying to talk to Omaha n n So a technician fixes the problem by typing in the IP address of B Gets hired by United with a hefty raise

Fly. High. com Architecture Omaha A B transactional system Enterprise clusters: flight plans, seat assignments, employee schedules, etc 126. 18. 17. 103 Seat request a. Fly. High. com Anchorage check-in

Six months later… n Time goes by and now we need to move that service again Anchorage crashes again… n n But this time nobody can figure out why! Hunting for references to the service or even to B won’t help Need to realize that the actual IP address of B is wired into the application now Nobody remembers what the technician did or why he did it!

What about big data centers? n n Discussed early in the course Let’s have another look

A glimpse inside e. Stuff. com “front-end applications” Pub-sub combined with point-to-point communication technologies like TCP LB LB LB service service Legacy applications

Tracking down a problem n Has the flavor of “data mining” n n n Perhaps, the “product popularity service” is timing out on the “availability” service But this might not be something we routinely instrument Need a way to “see” status and pose questions about things not normally instrumented

Monitoring a hierarchy of sets n n n A set of data centers, each having A set of services, each structured as A set of partitions, each consisting of A set of programs running in a clustered manner on A set of machines

Jim Gray: “A RAPS of RACS” n n RAPS: A reliable array of partitioned services RACS: A reliable array of clusterstructured server processes A set of RACS RAPS Ken Birman searching for “digital camera” x y z Pmap “B-C”: {x, y, z} (equivalent replicas) Here, y gets picked, perhaps based on load

Jim Gray: “A RAPS of RACS” Services are hosted at data centers but accessible system -wide Data center B Data center A Query source Update source pmap Logical partitioning of services l 2 P map Server pool Operators can control pmap, l 2 P map, other parameters. Large-scale multicast used to disseminate updates Logical services map to a physical resource pool, perhaps many to one

Homeland Defense and Military Data Mining n n Issue is even worse for a new generation of “smart” data mining applications Their role is to look for something on behalf of the good guys n Look for images of tall thin guys with long white beards on horseback

Two options n We could ship all the data to the analyst’s workstation n E. g. ship every image, in real-time If N machines gather I images per second, and we have B bytes per image, the load on the center system grows with N*I*B. With A analysts the load on the network grows as N*I*B*A Not very practical.

Two options n Or, we could ship the work to the remote sensor systems n n n They do all the work “out there”, so each just searches the images it is capturing Load is thus quite low Again, a kind of data mining and monitoring problem… not so different from the one seen in data centers!

Autonomic Distributed Systems n Can we build a new generation of middleware in support of these new adaptive systems? n n n Middleware that perceives the state of the network It can represent this knowledge in a form smart applications can exploit Although built from large numbers of rather dumb components the emergent behavior is adaptive. These applications are more robust, more secure, more responsive than any individual component When something unexpected occurs, they can diagnose the problem and trigger a coordinated distributed response They repair themselves after damage

Astrolabe n Intended as help for applications adrift in a sea of information n Structure emerges from a randomized peer-to-peer protocol n This approach is robust and scalable even under extreme stress that cripples more traditional approaches Developed at Cornell n By Robbert van Renesse, with many others helping… n Just an example of the kind of solutions we need n Astrolabe is a form of knowledge representation for selfmanaged networks Astrolabe

Astrolabe: Distributed Monitoring Name swift n 0 1 6. 2 1. 5 1. 1 0. 8 0. 9 1 0 4. 1 cardinal n 2. 0 1. 8 2. 1 1. 9 3. 1 falcon n Load Weblogic? SMTP? Word Version 4. 5 2. 7 3. 6 5. 3 1 0 … 6. 0 Row can have many columns Total size should be k-bytes, not megabytes Configuration certificate determines what data is pulled into the table (and can change) ACM TOCS 2003

Astrolabe in a single domain n n Each node owns a single tuple, like the management information base (MIB) Nodes discover one-another through a simple broadcast scheme (“anyone out there? ”) and gossip about membership n n Nodes also keep replicas of one-another’s rows Periodically (uniformly at random) merge your state with some else…

State Merge: Core of Astrolabe epidemic Name Time Load Weblogic? SMTP? Word Versio n swift 2011 2. 0 0 1 6. 2 falcon 1971 1. 5 1 0 4. 1 cardinal 2004 4. 5 1 0 6. 0 swift. cs. cornell. edu Name Load Weblogic ? SMTP? Word Version swift 2003 . 67 0 1 6. 2 falcon 1976 2. 7 1 0 4. 1 cardinal. cs. cornell. edu Time 2201 3. 5 1 1 6. 0

State Merge: Core of Astrolabe epidemic Name Time Load Weblogic? SMTP? Word Versio n swift 2011 2. 0 0 1 6. 2 falcon 1971 1. 5 1 0 4. 1 cardinal 2004 4. 5 1 0 6. 0 swift. cs. cornell. edu swift cardinal Name Load Weblogic ? SMTP? Word Version swift 2003 . 67 0 1 6. 2 falcon 1976 2. 7 1 0 4. 1 cardinal. cs. cornell. edu Time 2201 3. 5 1 1 6. 0 2011 2201 2. 0 3. 5

State Merge: Core of Astrolabe epidemic Name Time Load Weblogic? SMTP? Word Versio n swift 2011 2. 0 0 1 6. 2 falcon 1971 1. 5 1 0 4. 1 cardinal 2201 3. 5 1 0 6. 0 swift. cs. cornell. edu Name Load Weblogic ? SMTP? Word Version swift 2011 2. 0 0 1 6. 2 falcon 1976 2. 7 1 0 4. 1 cardinal. cs. cornell. edu Time 2201 3. 5 1 1 6. 0

Observations n Merge protocol has constant cost n n One message sent, received (on avg) per unit time. The data changes slowly, so no need to run it quickly – we usually run it every five seconds or so Information spreads in O(log N) time But this assumes bounded region size n In Astrolabe, we limit them to 50 -100 rows

Big system will have many regions n n Astrolabe usually configured by a manager who places each node in some region, but we are also playing with ways to discover structure automatically A big system could have many regions n n Looks like a pile of spreadsheets A node only replicates data from its neighbors within its own region

Scaling up… and up… n With a stack of domains, we don’t want every system to “see” every domain n n Cost would be huge So instead, we’ll see a summary Name Time Load Weblogic SMTP? Word ? Name Time Load Weblogic SMTP? Version Word ? Version Name 2011 Time 2. 0 Load Weblogic 1 SMTP? 6. 2 Word swift 0 ? Version swift Name 2011 Time 2. 0 Load 0 Weblogic 1 SMTP? 6. 2 Word falcon 1976 2. 7 1 4. 1 ? 0 Version SMTP? 6. 2 Word cardinal 2201 3. 5 1 swift Name 2011 Time 2. 0 1 Load 0 Weblogic 1 6. 0 falcon 1976 2. 7 1 4. 1 ? 0 Version cardinal 2201 3. 5 swift 2011 2. 0 1 0 1 1 6. 0 6. 2 falcon 1976 2. 7 1 0 4. 1 cardinal 2201 3. 5 1 1 6. 0 cardinal. cs. cornell. edu

Astrolabe builds a monitoring hierarchy Dynamically changing query output is visible system-wide Name Avg Load WL contact SF 2. 6 123. 45. 61. 3 123. 45. 61. 17 NJ 1. 8 127. 16. 77. 6 127. 16. 77. 11 Paris 3. 1 14. 66. 71. 8 SQL query “summarizes” data SMTP contact 14. 66. 71. 12 Name Load Weblogic? SMTP? Word Version swift 2. 0 0 1 falcon 1. 5 1 cardinal 4. 5 1 Name Load Weblogic? SMTP? Word Version 6. 2 gazelle 1. 7 0 0 4. 5 0 4. 1 zebra 3. 2 0 1 6. 2 0 6. 0 gnu . 5 1 0 6. 2 San Francisco … New Jersey …

Large scale: “fake” regions n These are n n n Computed by queries that summarize a whole region as a single row Gossiped in a read-only manner within a leaf region But who runs the gossip? n n Each region elects “k” members to run gossip at the next level up. Can play with selection criteria and “k”

Hierarchy is virtual… data is replicated Name Avg Load WL contact SMTP contact SF 2. 6 123. 45. 61. 3 123. 45. 61. 17 NJ 1. 8 127. 16. 77. 6 127. 16. 77. 11 Paris 3. 1 14. 66. 71. 8 14. 66. 71. 12 Name Load Weblogic? SMTP? Word Version swift 2. 0 0 1 falcon 1. 5 1 cardinal 4. 5 1 Name Load Weblogic? SMTP? Word Version 6. 2 gazelle 1. 7 0 0 4. 5 0 4. 1 zebra 3. 2 0 1 6. 2 0 6. 0 gnu . 5 1 0 6. 2 San Francisco … New Jersey …

What makes Astrolabe a good fit n Notice how hierarchical database abstraction “emerges” without ever being physically represented on any single machine n n Moreover, this abstraction is very robust It scales well… localized disruptions won’t disrupt the system state… consistent in eyes of varied beholders Yet individual participant runs a nearly trivial peerto-peer protocol Supports distributed data aggregation, data mining. Adaptive and self-repairing…

Worst case load? n A small number of nodes end up participating in O(logfanout. N) epidemics n n n Here the fanout is something like 50 In each epidemic, a message is sent and received roughly every 5 seconds We limit message size so even during periods of turbulence, no message can become huge. n n Instead, data would just propagate slowly Haven’t really looked hard at this case

Data Mining n In client-server systems we usually n n n Collect the data at the server Send queries to it With Astrolabe n n n Send query (and configuration certificate) to edge nodes They pull in desired data and query it User sees a sort of materialized result

Pros and Cons n Pros: n n n As you look “up” the hierarchy the answer you see can differ for different users (“where can I get some gas? ”) Parallelism makes search fast Cons: n n Need to have agreement on what to put into the aggregations Everyone sees the same hierarchy

Our platform in a datacenter

Dealing with legacy apps? n Many advocate use of a wrapper n Take the old crufty program… now it looks “data center enabled” (e. g. it might connect to Astrolabe or other tools)

We’ve seen several P 2 P tools n Scalable probabilistically reliable multicast based on P 2 P (peer-to-peer) epidemics n n Replication with strong properties n n n We could use this for the large multicast fanouts Could use it within the RACS DHTs or other content indexing systems Perhaps, gossip repair mechanism to detect and eliminate inconsistencies

Solutions that share properties n n Scalable Robust against localized disruption Have emergent behavior we can reason about, exploit in the application layer Think of the way a hive of insects organizes itself or reacts to stimuli. There are many similarities

Revisit our goals n Are these potential components for autonomic systems? ü ü ü n Middleware that perceives the state of the network It represent this knowledge in a form smart applications can exploit Although built from large numbers of rather dumb components the emergent behavior is adaptive. These applications are more robust, more secure, more responsive than any individual component When something unexpected occurs, they can diagnose the problem and trigger a coordinated distributed response They repair themselves after damage We seem to have the basis from which to work!

Brings us full circle n n Our goal should be a new form of very stable “autonomic middleware” Have we accomplished this goal? n n Probabilistically reliable, scalable primitives They solve many problems Gaining much attention now from industry, academic research community Fundamental issue is skepticism about peer-to -peer as a computing model

Conclusions? n n n We’re at the verge of a breakthrough – networks that behave like autonomic infrastructure on behalf of smart applications Could open the door to big advances But industry has yet to deploy these ideas and may think about them for a long time before doing so