Скачать презентацию Understanding Performance in Operating Systems Andy Wang COP Скачать презентацию Understanding Performance in Operating Systems Andy Wang COP

adc55cc0d59847a3f8583de7555dec2b.ppt

  • Количество слайдов: 71

Understanding Performance in Operating Systems Andy Wang COP 5611 Advanced Operating Systems Understanding Performance in Operating Systems Andy Wang COP 5611 Advanced Operating Systems

Outline n Importance of operating systems performance n Major issues in understanding operating systems Outline n Importance of operating systems performance n Major issues in understanding operating systems performance n Issues in experiment design

Importance of OS Performance n Performance is almost always a key issue in operating Importance of OS Performance n Performance is almost always a key issue in operating systems File system research n OS tools for multimedia n Practically any OS area n n Since everyone uses the OS (sometimes heavily), everyone is impacted by its performance n A solution that doesn’t perform well isn’t a solution at all

Importance of Understanding OS Performance n Great, so we work on improving OS performance Importance of Understanding OS Performance n Great, so we work on improving OS performance n How do we tell if we succeeded? n Successful research must prove its performance characteristics to a skeptical community

So What? n Proper performance evaluation is difficult n Knowing what to study is So What? n Proper performance evaluation is difficult n Knowing what to study is tricky n Performance evaluations take a lot of careful work n Understanding the results is hard n Presenting them effectively is challenging

For Example, n An idea - save power from a portable computer’s battery by For Example, n An idea - save power from a portable computer’s battery by using its wireless card to execute tasks remotely n Maybe that’s a good idea, maybe it isn’t n How do we tell? n Performance experiments to validate concept

But What Experiments? n What tasks should we check? n What should be the But What Experiments? n What tasks should we check? n What should be the conditions of the portable computer? n What should be the conditions of the network? n What should be the conditions of the server? n How do I tell if my result is statistically valid?

Issues in Understanding OS Performance n Techniques for understanding OS performance n Elements of Issues in Understanding OS Performance n Techniques for understanding OS performance n Elements of performance evaluation n Common mistakes in performance evaluation n Choosing proper performance metrics n Workload design/selection n Monitors n Software measurement tools

Techniques for Understanding OS Performance n Analytic modeling n Simulation n Measurement n Which Techniques for Understanding OS Performance n Analytic modeling n Simulation n Measurement n Which technique is right for a given situation?

Analytic Modeling + Sometimes relatively quick + Within limitations of model, testing alternatives usually Analytic Modeling + Sometimes relatively quick + Within limitations of model, testing alternatives usually easy – Mathematical tractability may require simplifications – Not everything models well – Question of validity of model

Simulation + Great flexibility + Can capture an arbitrary level of detail – Often Simulation + Great flexibility + Can capture an arbitrary level of detail – Often a tremendous amount of work to write and run – Testing a new alternative often requires repeating a lot of work – Question of validity of simulation

Experimentation + Lesser problems of validity + Sometimes easy to get started – Can Experimentation + Lesser problems of validity + Sometimes easy to get started – Can be very labor-intensive – Often hard to perform measurement – Sometimes hard to separate out effects you want to study – Sometimes impossible to generate cases you need to study

Elements of Performance Evaluation n Performance metrics n Workloads n Proper measurement technique n Elements of Performance Evaluation n Performance metrics n Workloads n Proper measurement technique n Proper statistical techniques n Minimization of effort n Proper data presentation techniques

Performance Metrics n The criteria used to evaluate the performance of a system n Performance Metrics n The criteria used to evaluate the performance of a system n E. g. , response time, cache hit ratio, bandwidth delivered, etc. n Choosing the proper metrics is key to a real understanding of system performance

Workloads n The requests users make on a system n If you don’t evaluate Workloads n The requests users make on a system n If you don’t evaluate with a proper workload, you aren’t measuring what real users will experience n Typical workloads Stream of file system requests n Set of jobs performed by users n List of URLs submitted to a Web server n

Proper Performance Measurement Techniques n You need at least two components to measure performance Proper Performance Measurement Techniques n You need at least two components to measure performance 1. A load generator To apply a workload to the system 2. A monitor To find out what happened

Proper Statistical Techniques n Computer performance measurements generally not purely deterministic n Most performance Proper Statistical Techniques n Computer performance measurements generally not purely deterministic n Most performance evaluations weigh the effects of different alternatives n How to separate meaningless variations from vital data in measurements? n Requires proper statistical techniques

Minimizing Your Work n Unless you design carefully, you’ll measure a lot more than Minimizing Your Work n Unless you design carefully, you’ll measure a lot more than you need to n A careful design can save you from doing lots of measurements n Should identify critical factors n And determine the smallest number of experiments that gives a sufficiently accurate answer

Proper Data Presentation Techniques n You’ve got pertinent, statistically accurate data that describes your Proper Data Presentation Techniques n You’ve got pertinent, statistically accurate data that describes your system n Now what? n How to present it Honestly n Clearly n Convincingly n

Why Is Performance Analysis Difficult? n Because it’s an art - it’s not mechanical Why Is Performance Analysis Difficult? n Because it’s an art - it’s not mechanical n You can’t just apply a handful of principles and expect good results n You’ve got to understand your system n You’ve got to select your measurement techniques and tools properly n You’ve got to be careful and honest

Some Common Mistakes in Performance Evaluation n No goals n Biased goals n Unsystematic Some Common Mistakes in Performance Evaluation n No goals n Biased goals n Unsystematic approach n Analysis without understanding n Incorrect performance metrics n Unrepresentative workload n Wrong evaluation technique

More Common Performance Evaluation Mistakes n Overlooking important parameters n Ignoring significant factors n More Common Performance Evaluation Mistakes n Overlooking important parameters n Ignoring significant factors n Inappropriate experiment design n No analysis n Erroneous analysis n No sensitivity analysis

Yet More Common Mistakes n Ignoring input errors n Improper treatment of outliers n Yet More Common Mistakes n Ignoring input errors n Improper treatment of outliers n Assuming static systems n Ignoring variability n Too complex analysis n Improper presentation of results n Ignoring social aspects n Omitting assumptions/limitations

Choosing Proper Performance Metrics n Three types of common metrics: n Time (responsiveness) n Choosing Proper Performance Metrics n Three types of common metrics: n Time (responsiveness) n Processing rate (productivity) n Resource consumption (utilization) n Can also measure various error parameters

Response Time n How quickly does system produce results? n Critical for applications such Response Time n How quickly does system produce results? n Critical for applications such as: n Time sharing/interactive systems n Real-time systems n Parallel computing

Processing Rate n How much work is done per unit time? n Important for: Processing Rate n How much work is done per unit time? n Important for: n Determining feasibility of hardware n Comparing different configurations n Multimedia

Resource Consumption n How much does the work cost? n Used in: n Capacity Resource Consumption n How much does the work cost? n Used in: n Capacity planning n Identifying bottlenecks n Also helps to identify the “next” bottleneck

Typical Error Metrics n Successful service (speed) n Incorrect service (reliability) n No service Typical Error Metrics n Successful service (speed) n Incorrect service (reliability) n No service (availability)

Characterizing Metrics n Usually necessary to summarize n Sometimes means are enough n Variability Characterizing Metrics n Usually necessary to summarize n Sometimes means are enough n Variability is usually critical

Essentials of Statistical Evaluation n Choose an appropriate summary n Mean, median, and/or mode Essentials of Statistical Evaluation n Choose an appropriate summary n Mean, median, and/or mode n Report measures of variation n Standard deviation, range, etc. n Provide confidence intervals (³ 95%) n Use confidence intervals to compare means

Choosing What to Measure n Pick metrics based on: n Completeness n (Non-)redundancy n Choosing What to Measure n Pick metrics based on: n Completeness n (Non-)redundancy n Variability

Designing Workloads n What is a workload? n Synthetic workloads n Real-World benchmarks n Designing Workloads n What is a workload? n Synthetic workloads n Real-World benchmarks n Application benchmarks n “Standard” benchmarks n Exercisers and drivers

What is a Workload? n A workload is anything a computer is asked to What is a Workload? n A workload is anything a computer is asked to do n Test workload: any workload used to analyze performance n Real workload: any workload observed during normal operations n Synthetic workload: any workload created for controlled testing

Real Workloads + They represent reality – Uncontrolled n Can’t be repeated n Can’t Real Workloads + They represent reality – Uncontrolled n Can’t be repeated n Can’t be described simply n Difficult to analyze n Nevertheless, often useful for “final analysis” papers

Synthetic Workloads + Controllable + Repeatable + Portable to other systems + Easily modified Synthetic Workloads + Controllable + Repeatable + Portable to other systems + Easily modified – Can never be sure real world will be the same

What Are Synthetic Workloads? n Complete programs designed specifically for measurement May do real What Are Synthetic Workloads? n Complete programs designed specifically for measurement May do real or “fake” work n May be adjustable (parameterized) n n Two major classes: n Benchmarks n Exercisers

Real-World Benchmarks n Pick a representative application and sample data n Run it on Real-World Benchmarks n Pick a representative application and sample data n Run it on system to be tested n Modified Andrew Benchmark, MAB, is a realworld benchmark + Easy to do, accurate for that sample application and data – Doesn’t consider other applications and data

Application Benchmarks n Variation on real-world benchmarks n Choose most important subset of functions Application Benchmarks n Variation on real-world benchmarks n Choose most important subset of functions n Write benchmark to test those functions + Tests what computer will be used for – Need to be sure it captures all important characteristics

“Standard” Benchmarks n Often need to compare general-purpose systems for general-purpose use Should I “Standard” Benchmarks n Often need to compare general-purpose systems for general-purpose use Should I buy a Compaq or a Dell PC? n Tougher: Mac or PC? n n Need an easy, comprehensive answer n People writing articles often need to compare tens of machines

“Standard” Benchmarks (cont’d) n Often need comparisons over time n How much faster is “Standard” Benchmarks (cont’d) n Often need comparisons over time n How much faster is this year’s Pentium Pro than last year’s Pentium? n Writing new benchmark undesirable n Could be buggy or not representative n Want to compare many people’s results

Exercisers and Drivers n For I/O, network, non-CPU measurements n Generate a workload, feed Exercisers and Drivers n For I/O, network, non-CPU measurements n Generate a workload, feed to internal or external measured system I/O on local OS n Network n n Sometimes uses dedicated system, interface hardware

Advantages and Disadvantages of Exercisers + Easy to develop, port + Incorporates measurement + Advantages and Disadvantages of Exercisers + Easy to develop, port + Incorporates measurement + Easy to parameterize, adjust – High cost if external – Often too small compared to real workloads

Workload Selection n Services exercised n Completeness n Level of detail n Representativeness n Workload Selection n Services exercised n Completeness n Level of detail n Representativeness n Timeliness n Other considerations

Services Exercised n What services does system actually use? n Speeding up response to Services Exercised n What services does system actually use? n Speeding up response to keystrokes won’t help a file server n What metrics measure these services?

Completeness n Computer systems are complex n Effect of interactions hard to predict n Completeness n Computer systems are complex n Effect of interactions hard to predict n So must be sure to test entire system n Important to understand balance between components

Level of Detail n Detail trades off accuracy vs. cost n Highest detail is Level of Detail n Detail trades off accuracy vs. cost n Highest detail is complete trace n Lowest is one request, usually most the common request n Intermediate approach: weight by frequency

Representativeness n Obviously, workload should represent desired application n Again, accuracy and cost trade Representativeness n Obviously, workload should represent desired application n Again, accuracy and cost trade off n Need to understand whether detail matters

Timeliness n Usage patterns change over time n File size grows to match disk Timeliness n Usage patterns change over time n File size grows to match disk size n If using “old” workloads, must be sure user behavior hasn’t changed n Even worse, behavior may change after test, as result of installing new system n “Latent demand” phenomenon

Other Considerations n Loading levels n Full capacity n Beyond capacity n Actual usage Other Considerations n Loading levels n Full capacity n Beyond capacity n Actual usage n Repeatability of workload

Monitors n A monitor is a tool used to observe system activity n Proper Monitors n A monitor is a tool used to observe system activity n Proper use of monitors is key to performance analysis n Also useful for other system observation purposes

Event-Driven Vs. Sampling Monitors n Event-driven monitors notice every time a particular type of Event-Driven Vs. Sampling Monitors n Event-driven monitors notice every time a particular type of event occurs Ideal for rare events n Require low per-invocation overheads n n Sampling monitors check the state of the system periodically Good for frequent events n Can afford higher overheads n

On-Line Vs. Batch Monitors n On-line monitors can display their information continuously n Or, On-Line Vs. Batch Monitors n On-line monitors can display their information continuously n Or, at least, frequently n Batch monitors save it for later n Usually using separate analysis procedures

Issues in Monitor Design n Activation mechanism n Buffer issues n Data compression/analysis n Issues in Monitor Design n Activation mechanism n Buffer issues n Data compression/analysis n Priority issues n Abnormal events monitoring n Distributed systems

Activation Mechanism n When do you collect the data? n Several possibilities: n When Activation Mechanism n When do you collect the data? n Several possibilities: n When an interesting event occurs, trap to data collection routine n Analyze every step taken by system n Go to data collection routine when timer expires

Buffer Issues n Buffer size should be big enough to avoid frequent disk writes Buffer Issues n Buffer size should be big enough to avoid frequent disk writes n But small enough to make disk writes cheap n Use at least two buffers, typically n One to fill up, one to record n Must think about buffer overflow

Data Compression or Analysis n Data can be literally compressed n Or can be Data Compression or Analysis n Data can be literally compressed n Or can be reduced to a summary form n Both methods save space But at the cost of extra overhead n Sometimes can use idle time for this n But idle time might be better spent dumping data to disk n

Priority of Monitor n How high a priority should the monitor’s operations have? n Priority of Monitor n How high a priority should the monitor’s operations have? n Again, trading off performance impact against timely and complete data gathering n Not always a simple question

Monitoring Abnormal Events n Often, knowing about failures and errors more important than knowing Monitoring Abnormal Events n Often, knowing about failures and errors more important than knowing about normal operation n Sometimes requires special attention n System may not be operating very well at the time of the failure

Monitoring Distributed Systems n Monitoring a distributed system is not dissimilar to designing a Monitoring Distributed Systems n Monitoring a distributed system is not dissimilar to designing a distributed system n Must deal with: Distributed state n Unsynchronized clocks n Partial failures n

Tools For Software Measurement n Code instrumentation n Tracing packages n System-provided metrics and Tools For Software Measurement n Code instrumentation n Tracing packages n System-provided metrics and utilities n Profiling

Code Instrumentation n Adding monitoring code to the system under + + + – Code Instrumentation n Adding monitoring code to the system under + + + – – – study Usually most direct way to gather data Complete flexibility Strong control over costs of monitoring Requires access to the source Requires strong knowledge of code Strong potential to affect performance

Typical Types of Instrumentation n Counters + Cheap and fast n But low level Typical Types of Instrumentation n Counters + Cheap and fast n But low level of detail n Logs + More detail n But more costly n Require occasional dumping or digesting n Timers

Tracing Packages n Allow dynamic monitoring of code that doesn’t have built-in monitors n Tracing Packages n Allow dynamic monitoring of code that doesn’t have built-in monitors n Akin to debuggers + Allows arbitrary insertion of code + No recompilation required + Tremendous flexibility + No overhead when you’re not using it – Somewhat higher overheads – Effective use requires access to source

System-Provided Metrics and Utilities n Many operating systems provide users access to some metrics System-Provided Metrics and Utilities n Many operating systems provide users access to some metrics n Most operating systems also keep some form of accounting logs n Lots of information can be gathered this way

Profiling n Many compilers provide easy facilities for + + – – profiling code Profiling n Many compilers provide easy facilities for + + – – profiling code Easy to use Low impact on system Requires recompilation Provides very limited information

Introduction To Experiment Design n You know your metrics n You know your factors Introduction To Experiment Design n You know your metrics n You know your factors n You’ve got your instrumentation and test loads n Now what?

Goals in Experiment Design n Obtain maximum information with minimum work n Typically meaning Goals in Experiment Design n Obtain maximum information with minimum work n Typically meaning minimum number of experiments n More experiments aren’t better if you have to perform them n Well-designed experiments are also easier to analyze

Experimental Replications n A run of the experiment with a particular set of levels Experimental Replications n A run of the experiment with a particular set of levels and other inputs is a replication n Often, you need to do multiple replications with a single set of levels and other inputs n For statistical validation

Interacting Factors n Some factors have effects completely independent of each other n Double Interacting Factors n Some factors have effects completely independent of each other n Double the factor’s level, halve the response, regardless of other factors n But the effects of some factors depends on the values of other factors n Interacting factors n Presence of interacting factors complicates experimental design

Basic Problem in Designing Experiments n Your chosen factors may or may not interact Basic Problem in Designing Experiments n Your chosen factors may or may not interact n How can you design an experiment that captures the full range of the levels? n With minimum amount of work

Common Mistakes in Experimentation n Ignoring experimental error n Uncontrolled parameters n Not isolating Common Mistakes in Experimentation n Ignoring experimental error n Uncontrolled parameters n Not isolating effects of different factors n One-factor-at-a-time experiment designs n Interactions ignored n Designs require too many experiments