Performance Engineering Laboratories Computer Engineering Department King Fahd

Performance Engineering Laboratories Computer Engineering Department King Fahd University of Petroleum & Minerals (KFUPM), Dhahran

Agenda n n Performance Engineering Laboratory (PEL) Independent performance evaluation Services Track record n n Web server performance comparisons Streaming media server performance evaluation Network traffic analysis Opportunities for working with us 6/3/2002 2 © PEL

Performance Engineering Laboratory (PEL) n n A facility established in computer engineering department at KFUPM Services n n n Independent evaluation of client products Product spec’s evaluation Performance comparison with competitor’s products n n Feedback to developers Technical marketing input Network traffic analysis Target products and services n IT products n n Network infrastructure products n n 6/3/2002 Proxy caches, LDAP serves, and layer-4 switches High-end systems n n Web servers, server accelerators, and streaming media servers Parallel, SMP, and DSM systems ISPs and carriers 3 © PEL

Why Evaluate Performance n Performance is central to computer systems n n n Competition demands efficient products n n n Marketing a product that is slower than its competitors is hard Highly efficient products can cut cost for customer Performance is central to any R&D effort n n New hardware is typically faster than the existing one New software is supposed to be “better” than the existing one Need to compare similar architectures, algorithms, systems, etc. Determine the efficacy of new designs Understand the differences between different systems Comparison (meaningful!) of a product with competitor’s products n n 6/3/2002 Sales team can’t provide it Sales team can effectively use it 4 © PEL

Market Opportunities n n Continued demand for client-server and IT products Large number of competing products n n n Performance is increasingly becoming the distinguishing factor among similar product Users are becoming increasingly aware of the performance issues and demand efficient products Time constraints to undertake detailed performance analysis n n Especially true with small to medium sized companies with small product development teams Performance evaluation often left for QA teams n n Goal of QA is to have a reliable, properly functioning product rather than an efficient product Third-party performance engineers can greatly help 6/3/2002 5 © PEL

Independent Evaluation n Concerns with independent performance evaluation n n QA can do it Sales team can do it; even compare performance with competitors Confidentiality may be compromised Our vision n n Anyone can generate measurements but few can “read” them to understand the story they tell Independent evaluation is significantly more credible for end user or IT manager who decides to buy the product n n n 6/3/2002 Sales team’s evaluation result is always predictable: “our” product is orders of magnitude better than “their” product Predictable no information!! Independent performance evaluation teams work closely with the developers in a professional manner 6 © PEL

Services n Primary services n Stand-alone product performance, Qo. S, and reliability analysis n n n n Product Specs evaluation Comparative analysis Network traffic and workload characterization Modeling based analysis Application parallelization Application profiling and/or system monitoring Other services n n Setup of in-house testing infrastructure Customized tools n n 6/3/2002 Automated performance regression testing tools Training 7 © PEL

Resources n We have two types of resources n n 6/3/2002 Skilled human resources A client-server based testing environment 8 © PEL

PEL Testbed PEL infrastructure being used for proxy performance evaluation 6/3/2002 9 © PEL

Track Record n Performance evaluation projects n n n Traffic analysis projects n n A campus web traffic analysis Parallel application performance evaluation n Comparison of Apache and Microsoft IIS web servers Comparison of Darwin Quicktime and Microsoft Windows Media Server Evaluation of automatically parallelized CFD code for high-end DSMs A trace-driven and measurement based memory performance evaluation for parallel applications on DSMs Design and evaluation of monitoring systems 6/3/2002 10 © PEL

Web Server Performance Comparisons Server: - Pentium 650 MHz - 256 MB RAM Clients: - Pentium 166 MHz - 64 - 128 MB RAM LAN: - 100 Mbps - Layer 2 switch Same server host runs Apache under Linux and IIS under MS Windows 2000 Server 6/3/2002 11 © PEL

Web Server Performance Comparisons n n Apache shows higher throughput with larger file sizes IIS shows higher throughput for average (~10 KB) file sizes 6/3/2002 12 © PEL

Web Server Performance Comparisons n n IIS offers lower latency at high load and small file sizes Apache shows lower latency with large file sizes only n 6/3/2002 Apache is network throughput limited here (~90 Mb/s max with 100 Mb/s switch) 13 © PEL

Web Server Performance Comparisons n So, which web server is better n n n Apache can show better throughput but our results are limited due to available network bandwidth (100 Mb/s per port) IIS shows high throughput and low latency for the average WWW document size (~10 KB) case with high transaction loads Other conclusion n Usually, it is an exaggeration to say that one product is better than the other n n This is usually true under specific workload conditions This information is useful for developers to tune their code Performance evaluation by sales departments won’t tell this Also, don’t under-estimate Microsoft products on Windows platforms… 6/3/2002 14 © PEL

Comparison of Streaming Media Servers Server: - Pentium 166 MHz - 128 MB RAM Clients: - Pentium 166 MHz - 48 - 64 MB RAM Switch: - 100 Mbps - Layer 2 switch n n Server machine runs Darwin Streaming Server under Linux Same server machine runs Windows Media Server under Win 2 K Server 6/3/2002 15 © PEL

Comparison of Streaming Media Servers 6/3/2002 16 © PEL

Comparison of Streaming Media Servers n Peak throughput n n n Memory performance n n n Indicated by 100% CPU usage Windows Media Server delivers significantly larger throughput at higher load that Darwin Streaming Server WMS shows high cache and page fault rates at high loads but still delivers better throughput Better exploitation of latency hiding opportunities offered by the processor through OS, compiler, and application Don’t expect more from a freely available media server!! n 6/3/2002 Darwin is available in public domain with source code from www. apple. com/quicktime 17 © PEL

Campus WWW Traffic Analysis 6/3/2002 18 © PEL

Campus WWW Traffic Analysis Top 10 Sites n n Web site popularity based on one month long logs from MS Proxy 2. 0 server Characteristic heavy-tail distribution of frequency of visits 6/3/2002 19 © PEL

Campus WWW Traffic Analysis n n Largest number of documents accessed are images followed by text documents Statistics reflect accesses over one month (Feb. 2002) 6/3/2002 20 © PEL

Campus WWW Traffic Analysis n n Analysis of arbitrarily selected 24 hours of proxy operation Low throughput with high latency 6/3/2002 21 © PEL

Campus WWW Traffic Analysis 6/3/2002 22 © PEL

Campus WWW Traffic Analysis n n Profile shows bandwidth saving by proxy However, most common case shows highest latency as well n 6/3/2002 Contrary to common perception, bandwidth is not the cause of long latencies experienced 23 © PEL

Opportunities to Work With Us n Short term contracts n n n Longer term contracts n n n Suitable for typically one particular product or service Turn around time of only a few weeks Suitable for multiple products and/or services Long-term relationship with one or more product development/deployment teams Points of contact n n 6/3/2002 Dr. Sadiq M. Sait (sadiq@ccse. kfupm. edu. sa) Dr. Abdul Waheed M. A. Sattar (awaheed@ccse. kfupm. edu. sa) 24 © PEL