Supporting Disconnected Operations in Publish Subscribe Systems Vinod Muthusamy

Supporting Disconnected Operations in Publish/Subscribe Systems Vinod Muthusamy Joint work with Milenko Petrovic, Ioana Burcea, H. -Arno Jacobsen, Eyal de Lara University of Toronto 2004 IEEE International Conference on Mobile Data Management

Agenda • Part I – Publish/Subscribe – Centralized/distributed models – Benefits – Applications • Part II – Mobile Publish/Subscribe – Disconnected operation in distributed publish/subscribe systems • Motivation and Problem • Solutions – Evaluation • Factors • Results • Conclusions

The Publish/Subscribe Model TSX Stock markets NASDAQ NYSE Publisher AMG N=5 8 12 ORCL= HON=24 MSFT =27 Publisher Publications IN =84 IBM JNJ= 58 =19 TC Broker Subscriptions: IBM > 85 ORCL < 10 JNJ > 60 Notification Subscriptions Subscriber

Distributed Publish/Subscribe • Hierarchy of brokers [Siena, JEDI] • Subscriptions to root • Publications to root and multicast down to interested subscribers Broker Broker stocks/ibm/* Publisher stocks/* stocks/oracle

Publish/Subscribe Benefits • Simple interface • Decoupling of producers and consumers of data – Address • Content-based routing • Anonymity – – Platform Space Time Representation (semantic) • Efficient data dissemination (scalability) – Push model – Multicast

Publish/Subscribe Applications • • News dissemination Location based services Workflow management E-commerce Auctions Distributed gaming Software updates delivery Sensor networks • Existing systems – Research: To. PSS, Siena, JEDI, Gryphon, Hermes – Industry: IBM, Precache, TIBCO, Talarian

Part II • How does disconnected operation work in distributed publish/subscribe systems? • Why is disconnected operation a problem? • What are some solutions? • How do these solutions perform?

Disconnected Operation: Motivation Broker • User’s laptop connected at work • Disconnect at the end of the day • Reconnect at home • Events published while a subscriber is disconnected should be stored somewhere and replayed upon reconnection Broker Subscriber Broker

Disconnected Operation [JEDI] • • • Subscriber receives publications Subscriber disconnects Broker stores publications Subscriber connects Transfer state to new broker & replay old publications • New publications go directly to subscriber t 1 t 2 At Old Broker t 3 Disconnected Disconnect (moveout) Publisher Broker t 4 At New Broker Connect Receive (movein) new events Subscriber

Disconnected Operation [JEDI] • • • Subscriber receives publications Subscriber disconnects Broker stores publications Subscriber connects Transfer state to new broker & replay old publications • New publications go directly to subscriber t 1 t 2 t 3 Publisher Broker t 4 moveout At Old Broker Disconnected Disconnect (moveout) At New Broker Connect Receive (movein) new events Subscriber

Disconnected Operation [JEDI] • • • Subscriber receives publications Subscriber disconnects Broker stores publications Subscriber connects Transfer state to new broker & replay old publications • New publications go directly to subscriber t 1 t 2 At Old Broker t 3 Disconnected Disconnect (moveout) Publisher Broker t 4 At New Broker Connect Receive (movein) new events Subscriber

Disconnected Operation [JEDI] • • • Subscriber receives publications Subscriber disconnects Broker stores publications Subscriber connects Transfer state to new broker & replay old publications • New publications go directly to subscriber t 1 t 2 t 3 Publisher Broker t 4 movein At Old Broker Disconnected Disconnect (moveout) At New Broker Connect Receive (movein) new events

Disconnected Operation [JEDI] • • • Subscriber receives publications Subscriber disconnects Broker stores publications Subscriber connects Transfer state to new broker & replay old publications • New publications go directly to subscriber t 1 t 2 At Old Broker t 3 Disconnected Disconnect (moveout) t 4 At New Broker Connect Receive (movein) new events Publisher Broker

Problem: Unicast State Transfer Broker Ø Publish/subscribe is efficient [multicast] Broker Ø State transfer is inefficient [unicast] Ø What’s the bandwidth overhead of unicast state transfer?

State Transfer Optimizations: Prefetching • Eagerly migrate state while the user is disconnected • Exploits knowledge of future mobility patterns • Shortens perceived length of disconnection to the system • Less state to transfer • State transfer period hidden from user t 1 t 2 t 3 Broker t 4 moveout At Old Broker Disconnected At New Broker tp Disconnect (moveout) Connect Receive (movein) new events Broker

State Transfer Optimizations: Logging • Brokers cache recent publications • Exploits locality • Only partial state transfer needed if locality exists • Overhead could be worse if no locality exists • Requires cache space at brokers Broker

State Transfer Optimizations: Home-Broker • Mobile client logically reconnects to “home” broker • Eliminates state transfer overhead • Increases perceived disconnection period – Unicast transfer even when connected Broker movein

Evaluation: Factors Affecting Mobility Network Mobility Ø Bandwidth and latency of links Ø Broker placement Ø Broker topology Ø Number of brokers Ø Connection and disconnection periods Ø Predictive or repetitive mobility Ø Group mobility Application Ø No. of publishers and subscribers Ø Publication rate Ø Subscription specificity Ø Subscription (interest) locality Ø Message size

Evaluation: Setup • Simulation Environment – NS 2 network simulator – Implemented state transfer optimizations • Parameters – Topology • Metropolitan Area Network • 4 levels of degree 4 64 leaf brokers – Subscribers: 200, 400, 600, 800 – Publishers: 100 – Locality: random, 30%, 60%, 90% • • • 1 • Metric – Total message traffic – State transfer overhead (unicast/multicast) 64

Commute Scenario • Simulate evening commute home – Subscribers work downtown (20 inner brokers) – Start commute between 4: 00 pm and 6: 00 pm – 40% live in city center (30 inner brokers) • Take 15 min to 45 min to commute – 60% live in outskirts (34 outer brokers) • Take 45 min to 90 min to commute • Few total disconnections • Long disconnection periods Outskirts City center 1 Outskirts 64 Downtown

Commute Scenario: Average Overhead • • • Standard: almost 100% overhead Logging: Slight improvement Prefetching: negligible overhead Home broker: poor due to sustained unicast Peak overhead results – Up to 3 x for Standard – Up to 27 x for Home-broker

Commute Scenario: Total Message Cost • 800 subscribers • Same relative performance • Message cost tracks # of state transfers • Home-broker overhead even after reconnection • Max 10% concurrent state transfers

Random Scenario • More ad-hoc mobility – Subscriber starts at random broker – Disconnects for 10 min to 30 min – Walk (5 km/h), city driving (50 km/h), highway driving (100 km/h) – Reconnects to a random broker within range – Remains connected for 10 min to 30 min – (repeat) • More disconnections • Shorter disconnection periods

Random Scenario: Average Overhead • Standard: 100% overhead • Prefetching overhead now noticeable – Due to shorter disconnections • Logging diverges more from Standard – Due to hidden increase in locality with population – Due to more disconnections

Random Scenario: Effect of Locality • 800 subscribers • Adjust subscription locality by varying % of subscribers with identical subscriptions • Logging can approach “ideal” Prefetching • Others’ increasing overhead is due to decreasing multicast, and constant unicast

Random Scenario: Effect of Log Size • For Logging approach • Log size 0, 400, 800, 1200 events • Diminishing returns of increasing log size

Pervasive Scenario • Persistent connectivity – Similar mobility patterns as Random scenario – Disconnections are few seconds long • E. g. Cellular handoffs between cells • Many disconnections • Short disconnection periods

Pervasive Scenario: Average Overhead • Event migration is small portion of state transfer overhead • Similar overhead (except homebroker) – Due to very short disconnections

Conclusions • The publish/subscribe model lends itself well to mobile applications • Mobility can break a distributed pub/sub system – Network overhead can double with only 10% concurrent movein’s – Stored events typically dominate overhead, not subscriptions – With sufficient locality, Logging approaches Prefetching • Future Work – Other scenarios: realistic traces, mobile publishers – Investigate effects of state transfer traffic on latency – Develop more state transfer optimizations • Multicast state transfer • Logging at intermediate brokers • Smarter cache replacement policies

Thank You

Pub/Sub Optimizations Covering Subscriptions Advertisements Publications Broker stocks/* Advertisements Broker stocks/* Broker stocks/ibm/* Publisher Ø Reduce subscription traffic Ø Reduce publication traffic to the root

Covering and Advertisements • Our scenarios did not benefit significantly from covering and ads • Forwarding of events dominate overhead – 98% of overhead in Commute and Random scenarios – Covering only helps to quench subscriptions • There is little subscriber locality (in our scenarios) – Most events must propagate to the root broker – Ads only help to quench upstream publications

Toronto Publish/Subscribe System (To. PSS) Research Areas To. PSS A-To. PSS (matching algorithms) (approximate) M-To. PSS (mobile) S-To. PSS p 2 p-To. PSS (semantic) (peer-to-peer) L-To. PSS Rb-To. PSS (location-based correlation) X-To. PSS (rule-based) Information consumers subscribe to information of interest. Information producers publish information. To. PSS-broker(s) match and route relevant information to interested subscribers. (semi-structured data; XML) Ad hoc-To. PSS (ad hoc networking) persistent-To. PSS (Subject Spaces) Federated-To. PSS (federation of To. PSS brokers)