PNUTS Yahoo s Hosted Data Serving Platform n n

PNUTS: Yahoo!’s Hosted Data Serving Platform n n Brian F. Cooper, Raghu Ramakrishnan, Utkarsh Srivastava, Adam Silberstein, Philip Bohannon, Hans-Arno Jacobsen, Nick Puz, Daniel Weaver and Ramana Yerneni Yahoo! Research With some additions by S. Sudarshan

How do I build a cool new web app? n Option 1: Code it up! Make it live! n n Scale it later It gets posted to slashdot Scale it now! Flickr, Twitter, My. Space, Facebook, … 2

How do I build a cool new web app? n Option 2: Make it industrial strength! n Evaluate scalable database backends Evaluate scalable indexing systems Evaluate scalable caching systems Architect data partitioning schemes Architect data replication schemes Architect monitoring and reporting infrastructure n Write application n n n n Go live Realize it doesn’t scale as well as you hoped Rearchitect around bottlenecks 1 year later – ready to go! 3

Example: social network updates Brian Sonja Jimi Brandon Kurt What are my friends up to? Sonja: Brandon: 4

Example: social network updates www. flickr. com 6 8 12 15 16 17 Jimi Mary Sonja Brandon Mike Bob

What do we need from our DBMS? n Web applications need: n Scalability n n And the ability to scale linearly Geographic scope High availability Web applications typically have: n Simplified query needs n n No joins, aggregations Relaxed consistency needs n Applications can tolerate stale or reordered data 6

What is PNUTS? 7

What is PNUTS? A B 42342 42521 E W C 66354 W D E 12352 75656 E C F 15677 E A B C D E F Parallel database 42342 42521 66354 12352 75656 15677 E W W E C E Indexes and views CREATE TABLE Parts ( ID VARCHAR, Stock. Number INT, Status VARCHAR A 42342 E … B 42521 W ) C 66354 W D E F 12352 75656 15677 Geographic replication Structured, flexible schema E C E Hosted, managed infrastructure 8

Query model n Per-record operations n n Multi-record operations n n Get Set Delete Multiget Scan Getrange Web service (RESTful) API 9

Detailed architecture Clients Data-path components REST API Routers Tablet controller Message Broker Storage units 10

Detailed architecture Local region Remote regions Clients REST API Routers YMB Tablet controller Storage units 11

Tablet splitting and balancing Each storage unit has many tablets (horizontal partitions of the table) Storage unit may become a hotspot Storage unit Tablet Overfull tablets split Tablets may grow over time Shed load by moving tablets to other servers 12

Query processing 13

Range queries Apple Avocado Grapefruit…Pear? Banana Blueberry Canteloupe Grape Kiwi Lemon MIN-Canteloupe SU 1 Canteloupe-Lime SU 3 Lime-Strawberry SU 2 Strawberry-MAX SU 1 Router Grapefruit…Lime? Lime…Pear? Lime Mango Orange Strawberry Tomato Watermelon Storage unit 1 Storage unit 2 Storage unit 3 16

Updates 1 8 Sequence # for key k Write key k Routers Message brokers 3 7 Sequence # for key k 2 Write key k 4 Write key k 5 SU SU SU 6 SUCCESS Write key k 17

Yahoo Message Bus n n Distributed publish-subscribe service Guarantees delivery once a message is published n n n Logging at site where message is published, and at other sites when received Guarantees messages published to a particular cluster will be delivered in same order at all other clusters Record updates are published to YMB by master copy n All replicas subscribe to the updates, and get them in same order for a particular record 18

Asynchronous replication and consistency 19

Asynchronous replication 20

Consistency model n Goal: make it easier for applications to reason about updates and cope with asynchrony n What happens to a record with primary key “Brian”? Record Update inserted v. 1 v. 2 Update v. 3 Update v. 4 Update Delete Update v. 5 v. 6 Generation 1 v. 7 v. 8 Time 21

Consistency model Read Stale version v. 1 v. 2 v. 3 v. 4 Stale version v. 5 v. 6 Generation 1 v. 7 Current version v. 8 Time 22

Consistency model Read up-to-date Stale version v. 1 v. 2 v. 3 v. 4 Stale version v. 5 v. 6 Generation 1 v. 7 Current version v. 8 Time 23

Consistency model Read-critical(required version): Stale version v. 1 v. 2 v. 3 v. 4 Read ≥ v. 6 Stale version v. 5 v. 6 Generation 1 v. 7 Current version v. 8 Time 24

Consistency model Write Stale version v. 1 v. 2 v. 3 v. 4 Stale version v. 5 v. 6 Generation 1 v. 7 Current version v. 8 Time 25

Consistency model Test-and-set-write(required version) Write if = v. 7 ERROR Stale version v. 1 v. 2 v. 3 v. 4 Stale version v. 5 v. 6 Generation 1 v. 7 Current version v. 8 Time 26

Consistency model Write if = v. 7 ERROR Stale version Current version Mechanism: per record mastership v. 1 v. 2 v. 3 v. 4 v. 5 v. 6 Generation 1 v. 7 v. 8 Time 27

Record and Tablet Mastership n n Data in PNUTS is replicated across sites Hidden field in each record stores which copy is the master copy n n n Record also contains origin of last few updates n n n updates can be submitted to any copy forwarded to master, applied in order received by master Mastership can be changed by current master, based on this information Mastership change is simply a record update Tablets mastership n n Required to ensure primary key consistency Can be different from record mastership 28

Other Features n n Per record transactions Copying a tablet (on failure, for e. g. ) n n n Request copy Publish checkpoint message Get copy of tablet as of when checkpoint is received Apply later updates Tablet split n Has to be coordinated across all copies 29

Query Processing n Range scan span tablets n n Only one tablet scanned at a time Client may not need all results at once n n Continuation object returned to client to indicate where range scan should continue Notification n n One pub-sub topic per tablet Client knows about tables, does not know about tablets n n Automatically subscribed to all tablets, even as tablets are added/removed. Usual problem with pub-sub: undelivered notifications, handled in usual way 30

Experiments 31

Experimental setup n Production PNUTS code n n Three PNUTS regions n n n Enhanced with ordered table type 2 west coast, 1 east coast 5 storage units, 2 message brokers, 1 router West: Dual 2. 8 GHz Xeon, 4 GB RAM, 6 disk RAID 5 array East: Quad 2. 13 GHz Xeon, 4 GB RAM, 1 SATA disk Workload n n n 1200 -3600 requests/second 0 -50% writes 80% locality 32

Inserts n n n required 75. 6 ms per insert in West 1 (tablet master) 131. 5 ms per insert into the non-master West 2, and 315. 5 ms per insert into the non-master East. 33

10% writes by default 34

Scalability 35

Request skew 36

Size of range scans 37

Related work n Distributed and parallel databases n n n Distributed filesystems n n Ceph, Boxwood, Sinfonia Distributed (P 2 P) hash tables n n Especially query processing and transactions Big. Table, Dynamo, S 3, Simple. DB, SQL Server Data Services, Cassandra Chord, Pastry, … Database replication n Master-slave, epidemic/gossip, synchronous… 38

Conclusions and ongoing work n PNUTS is an interesting research product n n n Research: consistency, performance, fault tolerance, rich functionality Product: make it work, keep it (relatively) simple, learn from experience and real applications Ongoing work n n n Indexes and materialized views Bundled updates Batch query processing 39

Thanks! n n cooperb@yahoo-inc. com research. yahoo. com 40