Correlating Instrumentation data to system states: A building block for automated diagnosis and control Ira Cohen, Jeffrey S. Chase et al.
Introduction n Networked systems continue to grow in scale n Complex behavior stemming from interaction of n n n Workload Software structure Hardware Traffic conditions System goals Pervasive System needed to manage such a system n Examples? n n n HP’s Openview IBM’s Tivoli (Aggregates + displays graphically)
Introduction n Two approaches to build self managing systems n A priori models n n n Event-condition-action rules Not based on real systems (Disadvantages? ) n n Difficult and costly Unreliable, does not take account of all
Introduction n Statistical learning techniques n Assumes little to no domain knowledge n n Hence “general” Problem! n Still have to identify techniques that are powerful enough to induce effective models that are: n n n Efficient Accurate Robust
Goals n Automatically analyze instrumentation data from network services in order to n n Forecast Diagnose Repair failure conditions We use the Tree-Augmented Naïve Bayesian Networks (TANs) as the basis for n n Diagnosis Forcasting n n System-level instrumentations in a 3 -tier network service. Widely used in various fields, but TANs are not used in the context of computer systems.
Goals n Analyzed data from 124 metrics gathered from n 3 tiered e-commerce site under synthetic load n n Httperf Java Pet. Store as platform TAN model select combination of metrics and threshold values that complies with Service Level Objectives for average response time. Results later
What is a TAN? n n Bayesian network is an annotated directed acyclic graph encoding a joint probability distribution Naïve Bayesian Network n n n State var S is only parent of all other vertices Assumes all metrics are fully independent given S TANs consider relationships among metrics themselves, with constraint that each metric has only one other parent than S
Why Use a TAN? n n Based on premise that a relatively small subset of metrics and threshold values is sufficient to approximate the distribution accurately Outperforms generalized Bayesian networks and other alternatives in both n n Cost Accuracy
Why use a TAN? n n n Useful forecasting failures and violations Possible to induce models that predict SLO violations in near future, even when system is stable Automated controller can invoke directly n n Identify impending violation Respond n n n Loading Adding resources Cheap model to induce Possible to maintain multiple models Periodic refresh
Setup n System is 3 -tier webservice n n n Apache Middleware (BEA Web. Logic) Oracle db 3 Servers with HP Openview to collect statistics Load Generator is httperf SLO indicator processes the logs to determine compliance
Interpretability and Modifiability n TANs offer other advantages n n n Interpretability Modifiability Influence of each metric can be quantified in a probabilistic model n n n Analysis catalogs each type of violation according to the metrics and values that correlate with observed instances Strength is given from prob value occurring in different states Gives insight to causes of violations and how to repair
Workloads n Varies several characteristics n n Aggregate req rate Number of concurrent connections Fraction of data-intensive vs app-intensive requests This is to exercise the model-induction methodology by providing it with a wide range of M, P pairs n n Where M = sample of values for system metrics P = vector of app-level performance measurements
Workloads n n RAMP: Increasing concurrency STEP: Background + Step function n Background constant traffic Bursty, hour long bursts BUGGY: Increasing aggregate req. rate
Results n n Varied SLO thresholds to explore effect on induced models To eval accuracy of models under varying conditions Trained and evaled TAN classifier for each of 31 different SLO definitions Baseline: accuracy of 60 -pctile SLO classifier (MOD) and CPU as metric.
Results n n n Overall BA of TAN is 87 -94% 90+% for all experiments 6% False alarm for 2 experiments, 17% for BUGGY Single metric is not sufficient to capture pattern of SLO violations (CPU) Small number of metrics is sufficient to capture pattern (3 -8) Sensitive to workload and SLO definition (MOD always has high detection rate, but generate false alarms at increasing rate as SLO thresh increases)
Conclusion n TANs are attractive for self-managing systems n n n Build system models automatically No a priori knowledge required Generalizes to wide range of conditions Zeroes in on most relevant metrics Practical
Conclusion n Possible work to adapt this to changing conditions Close the loop for automated diagnosis and control Ultimately most successful model is a hybrid of n n Automatically induced models A priori models
Questions?
Скачать презентацию Correlating Instrumentation data to system states A building
78f0bf1e2111554f9fa7ddf8921d4eeb.ppt