Data Mining Algorithms for Large-Scale Distributed Systems Presenter: Ran Wolff Joint work with Assaf Schuster 2003
What is Data Mining? The automatic analysis of large database The discovery of previously unknown patterns The generation of a model of the data
Main Data Mining Problems Association rules n Description Classification n Fraud, Churn Clustering n Analysis He who does this and that will usually do some other thing too These attributes indicate a good behavior - those indicate bad behavior. There are three types of entities
Examples – Classification Customers purchase artifacts in a store Each transaction is described in terms of a vector of features The owner of the store tries to predict which transactions are fraudulent n n Example: young men who buy small electronics during rash-hours Solution: do not respect checks
Examples – Associations Amazon tracks user queries n Suggests to each user additional books he would usually be interested in Supermarket finds out “people who buy diapers also buy beer” n Place diapers and beer at opposite sides of the supermarket
Examples – Clustering Resource location n Find the best location for k distribution centers Feature selection n n Find 1000 concepts which summarize a whole dictionary Extract the meaning out of a document by replacing each work with the appropriate concept w Car for auto, etc.
Why Mine Data of LSD Systems? Data mining is good It is otherwise difficult to monitor an LSD system: lots of data, spread across the system, impossible to collect Many interesting phenomena are inherently distributed (e. g. , DDo. S), it is not enough to just monitor a few nodes
An Example Peers in the Kazza network reveal to the system which files they have on their disks in exchange to access to the files of their peers The result is a 2 M peers database of people recreational preferences Mining it, you could discover that Matrix fans are also keen of Radio-Head songs n n Promote RH performances in Matrix-Reloaded Ask RH to write the music for Matrix-IV
What is so special about this problem? Huge systems – Huge amounts of data Dynamic setting n n System – join / depart Data – constant update Ad-hoc solution Fast convergence
Our Work We developed an association rule mining algorithm that works well in LSD Systems n n n Local and therefore scalable Asynchronous and therefore fast Dynamic and therefore robust Accurate – not approximated Anytime – you get early results fast
In a Teaspoon A distributed data mining algorithm can be described as a series of distributed decisions Those decisions are reduced to a majority vote We developed a majority voting protocol which has all those good qualities The outcome is an LSD association rule mining (still to come: classification)
Problem Definition – Association Rule Mining (ARM)
Solution to Traditional ARM
Large-Scale Distributed ARM
Solution of LSD-ARM No termination Anytime solution Recall Precision
Majority Vote in LSD Systems Unknown number of nodes vote 0 or 1 n n n Nodes may dynamically change their vote Edges are dynamically added / removed An infra-structure w detects failure w ensures message integrity w maintains a communication forest Each node should decide if the global majority is of 0 or 1
Majority Vote in LSD Systems – cont. Because of the dynamic settings, the algorithm never terminates Instead we measure the percent of correct outputs In static periods that percent ought to converge to 100% In stationary periods we will show it converges to a different percentage n Assume the overall percentage of ones remains the same, but they are constantly switched
LSD-Majority Algorithm Nodes communicates by exchanging messages <s, c> Node u maintains: n n n su – its vote, cu – one (for now) <suv, cuv>– the last <s, c> it had sent to v <svu, cvu>– the last <s, c> it had received from v
LSD-Majority – cont. Node u calculates: n n Captures the current knowledge of u Captures the current agreement between u and v
LSD-Majority – Rational It is OK if the current knowledge of u is more extreme than what it had agreed with v The opposite is not OK n v might assume u supports its decision more strongly than u actually does Tie breaking prefers a negative decision
LSD-Majority – The Protocol
LSD-Majority – The Protocol The same decision is applied whenever n n n a message is received su changes an edge fails or recovers
LSD-Majority – Example
LSD-Majority Results
Proof of Correctness Will be given in class
Back from Majority to ARM To decide whether an itemset is frequent or not
Back from Majority to ARM To decide whether a rule is confident or not
Additionally Create candidates based on the ad-hoc solution Create rules on-the-fly rather than upon termination Our algorithm outputs the correct rules without specifying their global frequency and confidence
Eventual Results By the time the database is scanned once, in parallel, the average node has discovered 95% of the rules, and has less than 10% false rules.
Скачать презентацию Data Mining Algorithms for Large-Scale Distributed Systems Presenter
cc23bc1f6236fe9a4d53f65d95e291bc.ppt