Monitoring Streams- A New Class of Data Management

Monitoring Streams- A New Class of Data Management Applications Presented by Qing Cao at CS@UVA

Table of contents n n n n Introduction Aurora System Model Aurora Optimization Real-Time Operation Details Critique Conclusion Discussion throughout the talk 2

Introduction Scenario User ID and Status Armed various sensors Brightness Sensor RPM, temperature, pressure, oil status, … RFID tagged Components Pressure Sensor 3

Auto Service Database 4 G Wireless Network GPS Service center Notify Instead of Query Repair center Home visit service 4

Scenario Summary n n Data Streams rather than Static Data Paradigm shift from HADP to DAHP Can traditional Database be used to handle this kind of scenarios? According to the authors, NO! 5

Comparison Monitoring Application Data Active Human Passive Traditional DBMS Data Passive Human Active Managing History of values required Very hard or inefficient Approximate query result required Not supported Real-time requirement required Not supported Typical model 6

So… n Quote: The primary goal of the Aurora project is to build a single infrastructure that can efficiently and seamlessly meet the requirements of such demanding applications. To this end, we are currently critically rethinking many existing data management and processing issues, as well as developing new proactive data processing concepts and techniques. 7

Implementation - trigger History of values Query management Data stream : Not in Real. Time : no scalable way to support latest location of the car : often update new triggers or queries requested by 3 rd party Trigger : they are not scalable CHALLENGE Query register Data CHALLENGE Stream ? ? ? CHALLENGE DBMS Update query : millions update in Data short time burst Submitter Output CHALLENGE Optimization : Is it helpful doing massive optimization during high load? Messaging Systems Qo. S : can not ensure service 8 for premium customers

Implementation middleware Data stream : sometimes lost or delivered lately Update query : millions update in short time burst query Query management : has to use new query language Query CHALLENGE register Resource usage : are we efficiently using the system? CHALLENGE Data. CHALLENGE Stream ? ? ? CHALLENGE DBMS History of values Data : no scalable way to Submitter find latest location of the car Output Query Optimization Processor : Can not benefit from query optimization Messaging Systems Qo. S : can not ensure service 9 for premium customers

Implementation - Aurora management Query Data stream : new stream : sometimes lost or processing delivered lately architecture Update queries Update query : new update : millionsstream in processing short time burst architecture query : has to use new query : intuitive stream algebra and GUI language Query CHALLENGE register Resource usage : : trainwe efficiently are scheduling & feed back from/to Qo. S using the system? CHALLENGE Data. CHALLENGE Stream CHALLENGE Output CHALLENGE DBMS History ofof values History the values Data : new stream processing : no scalable way to Submitter find latest location of architecture the car Query Optimization Processor : not benefit : Canrun-time optimization from query optimization Messaging Systems Qo. S : specified by. Qo. S application : can not ensure service administrator & 10 for premium load shedding customers

System model of Aurora User application Qo. S spec Query spec Aurora System External data source Historical Storage Operator boxes data flow Continuous & ad hoc queries 11 Application administrator

Implementation - Aurora inputs Storage Manager outputs Router Q 1 σ μ Q 2 Scheduler Data Stream Qm Buffer manager Box Processors Catalog Output Persistent Store Q 1 Q 2 Load Shedder Qo. S Monitor Qn 12

Aurora Query Semantics n Traditional ¡ ¡ n Structured Query Language Declarative query on static data Aurora ¡ Data flow model for data stream n ¡ Application manager will construct queries using GUI Stream Query Algebra n Queries are processed by SQu. Al operators on the data stream 13

Operators Discussion n n n n Slide Tumble Latch Resample Filter Drop Map Group. By MAP+GROUPBY = CASE 14

Query model Qo. S spec Storage b 1 b 2 b 3 app continuous query Connection point b 4 Qo. S spec b 5 view b 6 ad-hoc query b 7 b 8 b 9 app Qo. S spec 15

Optimization n Dynamic continuous query optimization ¡ ¡ ¡ n Inserting projections Combining boxes Reordering boxes Ad hoc query optimization ¡ ¡ 1 st stage : replace implementation (Filter/Join) 2 nd stage : same as continuous query 16

Run. Time Operation n n Qo. S Data Structure Storage Management Real-time Scheduling Load Shedding 17

Whole Structure Revisited inputs Storage Manager outputs Router Q 1 σ μ Q 2 Scheduler Data Stream Qm Buffer manager Box Processors Catalog Output Persistent Store Q 1 Q 2 Load Shedder Qo. S Monitor Qn 18

Aurora from Above Qo. S App. . . App . . . Qo. S 19

Runtime Operation Scheduling: Minimizing Per Tuple Processing Overhead … zyx A A (z) A (y) A (x) B B (A (z)) B (A (y)) B (A (x)) = Scheduler Action Train Scheduling: Box Trains: Tuple Trains: … zyx AB A B (A (z)) B (A (y)) B (A (x)) A (z, y, x) B B (A (z), A (y), A (x)) 20

Performance 21

Disucssion n Solution approach ¡ n Query model ¡ n Rethink about everything for the requirements Data flow style query specification and Qo. S Optimization ¡ ¡ ¡ Dynamic runtime optimization Train scheduling Qo. S specification based resource management 22

Discussion n Can it works in a distributive manner? ¡ n Aurora project What is the final result? ¡ After intensive searching of the tens of papers published on this subject, I finally finds what was implemented: 23

The final Result n n The Aurora stream-processing engine. Aurora is currently operational. It consists of some 100 K lines of C++ and Java and runs on both Unix- and Linux-based platforms. 24

Graphical Interface 25

GUI for an Example 26

Critique n n The overall approaches lacks in novelty, e. g. stream operators are ad-hoc. The overall result is not impressing. The project output is no more than a toy java program. Papers published lack in originality, depth, and overlap too much. 27

Conclusion n n Aurora is a large project that aims at stream query based engine design. Various new approaches are presented. No comparison results found in any paper. What do you think? 28

Extra on Aurora n n n n Aurora is the Latin word for "dawn". A polar light (caused by solar wind and seen near the poles). The collective noun for a group of polar bears. Several aircraft. Several vessels. Several Companies. In space: ¡ ¡ n In fiction: ¡ ¡ n n n An asteroid, discovered by J. C. Watson, in september 6, 1867. The Aurora Programme, a strategy of the European Space Agency. A superhero in the Marvel Universe. One of the Spacer worlds in Isaac Asimov's fiction One of at least four distinct music groups: a UK house group, also known as Aurora UK; a California-based ambient group; a contemporary Christian R&B group; a Mexican Latin music band. The name of the game engine that runs Neverwinter Nights, the toolset is called the Aurora toolset because of this. AND the aurora system as presented today. 29

THANK YOU! 30