Monitoring and Discovery in a Web Services Framework: Functionality and Performance of Globus Toolkit MDS 4 Jennifer M. Schopf Argonne National Laboratory UK National e. Science Centre (Ne. SC) Sept 11, 2006
What is a Grid l Resource sharing u u l Sharing always conditional: issues of trust, policy, negotiation, payment, … Coordinated problem solving u l Computers, storage, sensors, networks, … Beyond client server: distributed data analysis, computation, collaboration, … Dynamic, multi institutional virtual orgs u Community overlays on classic org structures u Large or small, static or dynamic 2
Why is this hard/different? l Lack of central control u u l Where things run When they run Shared resources u l Contention, variability Communication u Different sites implies different sys admins, users, institutional goals, and often “strong personalities” 3
So why do it? l Computations that need to be done with a time limit l Data that can’t fit on one site l Data owned by multiple sites l Applications that need to be run bigger, faster, more 4
What Is Grid Monitoring? l Sharing of community data between sites using a standard interface for querying and notification u u Data of interest to more than one person u l Data of interest to more than one site Summary data is possible to help scalability Must deal with failures u l Both of information sources and servers Data likely to be inaccurate u Generally needs to be acceptable for data to be dated 5
Common Use Cases l Decide what resource to submit a job to, or to transfer a file from l Keep track of services and be warned of failures l Run common actions to track performance behavior l Validate sites meet a (configuration) guideline 6
OUTLINE l Grid Monitoring and Use Cases l MDS 4 u u Higher level services u l Information Providers Web. MDS Deployments u Metascheduling data for Tera. Grid u Service failure warning for ESG l Performance Numbers l MDS For You! 7
What is MDS 4? l Grid level monitoring system used most often for resource selection and error notification u u l Uses standard interfaces to provide publishing of data, discovery, and data access, including subscription/notification u l Aid user/agent to identify host(s) on which to run an application Make sure that they are up and running correctly WS Resource. Properties, WS Base. Notification, WS Service. Group Functions as an hourglass to provide a common interface to lower level monitoring tools 8
Information Users : Schedulers, Portals, Warning Systems, etc. WS standard interfaces for subscription, registration, notification Standard Schemas (GLUE schema, eg) Cluster monitors (Ganglia, Hawkeye, Clumon, and Nagios) Services (GRAM, RFT, RLS) Queuing systems (PBS, LSF, Torque) 9
Web Service Resource Framework (WS RF) l Defines standard interfaces and behaviors for distributed system integration, especially (for us): u u Standard XML based service information model Standard interfaces for push and pull mode access to service data l Notification and subscription 10
MDS 4 Uses Web Service Standards l WS Resource. Properties u u u l WS Base. Notification u l Defines a mechanism by which Web Services can describe and publish resource properties, or sets of information about a resource Resource property types defined in service’s WSDL Resource properties can be retrieved using WS Resource. Properties query operations Defines a subscription/notification interface for accessing resource property information WS Service. Group u Defines a mechanism for grouping related resources and/or services together as service groups 11
MDS 4 Components l Information providers u u l Higher level services u u u l Index Service – a way to aggregate data Trigger Service – a way to be notified of changes Both built on common aggregator framework Clients u l Monitoring is a part of every WSRF service Non WS services are also be used Web. MDS All of the tool are schema agnostic, but interoperability needs a well understood common language 12
Information Providers l l Data sources for the higher level services Some are built into services Any WSRF compliant service publishes some data automatically u WS RF gives us standard Query/Subscribe/Notify interfaces u GT 4 services: Service. Meta. Data. Info element includes start time, version, and service type name u Most of them also publish additional useful information as resource properties u 13
Information Providers: GT 4 Services l Reliable File Transfer Service (RFT) u l Community Authorization Service (CAS) u l Service status data, number of active transfers, transfer status, information about the resource running the service Identifies the VO served by the service instance Replica Location Service (RLS) Note: not a WS u Location of replicas on physical storage systems (based on user registrations) for later queries u 14
Information Providers (2) l Other sources of data u Any executables u Other (non WS) services u Interface to another archive or data store u File scraping l Just need to produce a valid XML document 15
Information Providers: Cluster and Queue Data l Interfaces to Hawkeye, Ganglia, Clu. Mon, Nagios u u u l Basic host data (name, ID), processor information, memory size, OS name and version, file system data, processor load data Some condor/cluster specific data This can also be done for sub clusters, not just at the host level Interfaces to PBS, Torque, LSF u Queue information, number of CPUs available and free, job count information, some memory statistics and host info for head node of cluster 16
Other Information Providers l File Scraping u u l Mostly used for data you can’t find programmatically System downtime, contact info for sys admins, online help web pages, etc. Others as contributed by the community! 17
Higher Level Services l Index Service u l Trigger Service u l Caching registry Warn on error conditions All of these have common needs, and are built on a common framework 18
MDS 4 Index Service l Index Service is both registry and cache u u l l Subscribes to information providers In memory default approach u l l Datatype and data provider info, like a registry (UDDI) Last value of data, like a cache DB backing store currently being discussed to allow for very large indexes Can be set up for a site or set of sites, a specific set of project data, or for user specific data only Can be a multi rooted hierarchy u No *global* index 19
MDS 4 Trigger Service l Subscribe to a set of resource properties l Evaluate that data against a set of pre configured conditions (triggers) l When a condition matches, action occurs u Email is sent to pre defined address u Website updated 20
Common Aspects 1) Collect information from information providers u Java class that implements an interface to collect XML formatted data u “Query” uses WS Resource. Property mechanisms to poll a WSRF service u “Subscription” uses WS Notification subscription/notification u “Execution” executes an administrator supplied program to collect information 2) Common interfaces to external services u These should all have the standard WS RF service interfaces 21
Common Aspects (2) 3) Common configuration mechanism u Maintain information about which information providers to use and their associated parameters u Specify what data to get, and from where 4) Services are self cleaning u Each registration has a lifetime u If a registration expires without being refreshed, it and its associated data are removed from the server 5) Soft consistency model u Flexible update rates from different IPs u Published information is recent, but not guaranteed to be the absolute latest u Load caused by information updates is reduced at the expense of having slightly older information u Free disk space on a system 5 minutes ago rather than 2 seconds ago 22
Aggregator Framework is a General Service l This can be used for other higher level services that want to u u u l Archive Service u l Subscribe to data, put it in a database, query to retrieve, currently in discussion for development Prediction Service u l Subscribe to Information Provider Do some action Present standard interfaces Subscribe to data, run a predictor on it, publish results Compliance Service u Subscribe to data, verify a software stack match to definition, publish yes or no 24
Web. MDS User Interface l l l Web based interface to WSRF resource property information User friendly front end to Index Service Uses standard resource property requests to query resource property data XSLT transforms to format and display them Customized pages are simply done by using HTML form options and creating your own XSLT transforms Sample page: u http: //mds. globus. org: 8080/webmds/webm ds? info=indexinfo&xsl=servicegroupxsl 25
Web. MDS Service 26
27
28
29
Web. MDS E Site 1 Trigger action A Rsc 1. a Site 1 Index Rsc 1. b I GRAM (PBS) Ganglia/PBS Trigger Service F Site 2 Index GRAM (LSF) I RFT C West Coast Index Site 3 Index App B Index B Rsc 2. b Ganglia/LSF Rsc 1. d Rsc 3. a D VO Index Rsc 1. c I Site 3 Rsc 2. a I Rsc 3. b GRAM I Hawkeye RLS 31
Web. MDS E Site 1 Trigger action A Rsc 1. a Site 3 Rsc 2. a Rsc 3. a D VO Index Site 1 Index Trigger Service F Site 2 Index C West Coast Index Site 3 Index App B Index B Rsc 2. b Rsc 1. b I Rsc 3. b GRAM I Hawkeye I GRAM (PBS) RLS Container Ganglia/PBS Rsc 1. c I GRAM (LSF) Ganglia/LSF Rsc 1. d I ABC RFT I Service Index Registration RFT 32
Web. MDS E Site 1 Trigger action A Rsc 1. a Site 1 Index Rsc 1. b I GRAM (PBS) Ganglia/PBS Trigger Service F Site 2 Index GRAM (LSF) I RFT C West Coast Index Site 3 Index App B Index B Rsc 2. b Ganglia/LSF Rsc 1. d Rsc 3. a D VO Index Rsc 1. c I Site 3 Rsc 2. a I Rsc 3. b GRAM I Hawkeye RLS 33
Web. MDS E Site 1 Trigger action A Rsc 1. a Site 1 Index Rsc 1. b I GRAM (PBS) Ganglia/PBS Trigger Service F Site 2 Index GRAM (LSF) I RFT C West Coast Index Site 3 Index App B Index B Rsc 2. b Ganglia/LSF Rsc 1. d Rsc 3. a D VO Index Rsc 1. c I Site 3 Rsc 2. a I Rsc 3. b GRAM I Hawkeye RLS 34
Web. MDS E Site 1 Trigger action A Rsc 1. a Site 1 Index Rsc 1. b I GRAM (PBS) Ganglia/PBS Trigger Service F Site 2 Index GRAM (LSF) I RFT C West Coast Index Site 3 Index App B Index B Rsc 2. b Ganglia/LSF Rsc 1. d Rsc 3. a D VO Index Rsc 1. c I Site 3 Rsc 2. a I Rsc 3. b GRAM I Hawkeye RLS 35
Any questions before I walk through two current deployments? l Grid Monitoring and Use Cases l MDS 4 u u Higher level services u l Information Providers Web. MDS Deployments u Metascheduling Data for Tera. Grid u Service Failure warning for ESG l Performance Numbers l MDS for You! 36
Working with Tera. Grid l Large US project across 9 different sites u l Starting to explore Meta. Scheduling approaches u l Different hardware, queuing systems and lower level monitoring packages Currently evaluating almost 20 approaches Need a common source of data with a standard interface for basic scheduling info 37
Cluster Data l Provide data at the subcluster level u Sys admin defines a subcluster, we query one node of it to dynamically retrieve relevant data l Can also list per host details l Interfaces to Ganglia, Hawkeye, Clu. Mon, and Nagios available now u Other cluster monitoring systems can write into a. html file that we then scrape 38
Cluster Info l Unique. ID l l Benchmark/Clock speed Number of nodes in a cluster/subcluster l Storage. Device l Processor l Main. Memory l Operating. System l Architecture u l Disk names, mount point, space available TG specific Node properties 39
Data to collect: Queue info l Interface to PBS (Pro, Open, Torque), LSF l LRMSType l Running. Jobs l LRMSVersion l Waiting. Jobs l Default. GRAMVersion and port and host l Free. CPUs l Max. Wall. Clock. Time l Total. CPUs l Max. CPUTime l Status (up/down) l Max. Total. Jobs l Total. Jobs (in the queue) l Max. Running. Jobs 40
How will the data be accessed? l Java and command line APIs to a common TG wide Index server u l One common web page for TG u l Alternatively each site can be queried directly http: //mds. teragrid. org Query page is next! 41
42
Status l Demo system running since Autumn ‘ 05 u u l Queuing data from SDSC and NCSA Cluster data using Clu. Mon interface All sites in process of deployment u Queue data from 7 sites reporting in u Cluster data still coming online 43
Earth Systems Grid Deployment l Supports the next generation of climate modeling research l Provides the infrastructure and services that allow climate scientists to publish and access key data sets generated from climate simulation models l Datasets including simulations generated using the Community Climate System Model (CCSM) and the Parallel Climate Model (PCM l Accessed by scientists throughout the world. 44
Who uses ESG? l In 2005 u l ESG web portal issued 37, 285 requests to download 10. 25 terabytes of data By the fourth quarter of 2005 u Approximately two terabytes of data downloaded per month l 1881 registered users in 2005 l Currently adding users at a rate of more than 150 per month 45
What are the ESG resources? l Resources at seven sites u u u u l Argonne National Laboratory (ANL) Lawrence Berkeley National Laboratory (LBNL) Lawrence Livermore National Laboratory (LLNL) Los Alamos National Laboratory (LANL) National Center for Atmospheric Research (NCAR) Oak Ridge National Laboratory (ORNL) USC Information Sciences Institute (ISI) Resources include u u u u Web portal HTTP data servers Hierarchical mass storage systems OPe. NDAP system Storage Resource Manager (SRM) Grid. FTP data transfer service [ Metadata and replica management catalogs 46
47
The Problem: l Users are 24/7 l Administrative support was not! l Any failure of ESG components or services can severely disrupt the work of many scientists The Solution l Detect failures quickly and minimize infrastructure downtime by deploying MDS 4 for error notification 48
ESG Services Being Monitored Service Being Monitored Grid. FTP server OPe. NDAP server Web Portal HTTP Dataserver Replica Location Service (RLS) servers Storage Resource Managers Hierarchical Mass Storage Systems ESG Location NCAR LANL, NCAR LANL , LBNL, LLNL, NCAR, ORNL LANL, LBNL, NCAR, ORNL 49
Index Service l Site wide index service is queried by the ESG web portal u Generate an overall picture of the state of ESG resources displayed on the Web 50
51
Trigger Service l Site wide trigger service collects data and sends email upon errors Information providers are polled at pre defined services u Value must be matched for set number of intervals for trigger to occur to avoid false positives u Trigger has a delay associated for vacillating values u Used for offline debugging as well u 52
1 Month of Error Messages Total error messages for May 2006 Messages related to certificate and configuration problems at LANL Failure messages due to brief interruption in network service at ORNL on 5/13 HTTP data server failure at NCAR 5/17 RLS failure at LLNL 5/22 Simultaneous error messages for SRM services at NCAR, ORNL, LBNL on 5/23 RLS failure at ORNL 5/24 RLS failure at LBNL 5/31 47 38 2 1 1 3 1 1 53
1 Month of Error Messages Total error messages for May 2006 Messages related to certificate and configuration problems at LANL Failure messages due to brief interruption in network service at ORNL on 5/13 HTTP data server failure at NCAR 5/17 RLS failure at LLNL 5/22 Simultaneous error messages for SRM services at NCAR, ORNL, LBNL on 5/23 RLS failure at ORNL 5/24 RLS failure at LBNL 5/31 47 38 2 1 1 3 1 1 54
Benefits l Overview of current system state for users and system administrators u u l Failure notification u u l System admins can identify and quickly address failed components and services Before this deployment, services would fail and might not be detected until a user tried to access an ESG dataset Validation of new deployments u l At a glance info on resources and services availability Uniform interface to monitoring data Verify the correctness of the service configurations and deployment with the common trigger tests Failure deduction u u A failure examined in isolation may not accurately reflect the state of the system or the actual cause of a failure System wide monitoring data can show a pattern of failure messages that occur close together in time can be used to deduce a problem at a different level of the system Eg. 3 SRM failures EG. Use of MDS 4 to evaluate file descriptor leak 56
OUTLINE l Grid Monitoring and Use Cases l MDS 4 u u Trigger Service u l Index Service Information Providers Deployments u Metascheduling Data for Tera. Grid u Service Failure warning for ESG l Performance Numbers l MDS for You! 57
Scalability Experiments l MDS index u u l Clients u u l l 20 nodes also dual 2. 6 GHz Xeon, 3. 5 GB RAM 1, 2, 3, 4, 5, 6, 7, 8, 16, 32, 64, 128, 256, 384, 512, 640, 768, 800 Nodes connected via 1 Gb/s network Each data point is average of 8 minutes u u l Dual 2. 4 GHz Xeon processors, 3. 5 GB RAM Sizes: 1, 10, 25, 50, 100 Ran for 10 mins but first 2 spent getting clients up and running Error bars are SD over 8 mins Experiments by Ioan Raicu, U of Chicago, using Di. Perf 58
Size Comparison l In our current Tera. Grid demo u u Host data 3 attributes for approx 900 nodes u 12 attributes of sub cluster data for 7 subclusters u l 17 attributes from 10 queues at SDSC and NCSA ~3, 000 attributes, ~1900 XML elements, ~192 KB. Tests here 50 sample entries u element count of 1113 u ~94 KB in size 59
60
61
MDS 4 Stability Vers. Index Size Time up (Days) Queries Processed Query Per Sec. Roundtrip Time (ms) 4. 0. 1 25 66+ 81, 701, 925 14 69 4. 0. 1 50 66+ 49, 306, 104 8 115 4. 0. 1 100 33 14, 686, 638 5 194 4. 0. 0 1 14 93, 890, 248 76 13 4. 0. 0 1 96 623, 395, 877 74 13 62
Index Maximum Size Heap Size (MB) 64 128 256 512 1024 1536 Approx. Max. Index Entries 600 1275 2650 5400 10800 16200 Index Size (MB) 1. 0 2. 2 4. 5 9. 1 17. 7 26. 18 63
Performance l Is this enough? u u l We don’t know! Currently gathering up usage statistics to find out what people need Bottleneck examination u u In the process of doing in depth performance analysis of what happens during a query MDS code, implementation of WS N, WS RP, etc 64
MDS For You l Grid Monitoring and Use Cases l MDS 4 u u Higher level services u l Information Providers Web. MDS Deployments u Metascheduling Data for Tera. Grid u Service Failure warning for ESG l Performance Numbers l MDS for You! 65
How Should You Deploy MDS 4? l Ask: Do you need a Grid monitoring system? l Sharing of community data between sites using a standard interface for querying and notification u Data of interest to more than one site u Data of interest to more than one person u Summary data is possible to help scalability 66
What does your project mean by monitoring? l Display site data to make resource selection decisions l Job tracking l Error notification l Site validation l Utilization statistics l Accounting data 67
What does your project mean by monitoring? l Display site data to make resource selection decisions l Job tracking l Error notification l Site validation l Utilization statistics l Accounting data MDS 4 a Good Choice! 68
What does your project mean by monitoring? l Display site data to make resource selection decisions l Job tracking – generally application specific l Error notification l Site validation l Utilization statistics – use local info l Accounting data use local info and reliable messaging – AMIE from TG is one option Think about other tools 69
What data do you need l There is no generally agreed upon list of data every site should collect l Two possible examples u What TG is deploying l u http: //mds. teragrid. org/docs/mds 4 TG overview. pdf What GIN Info is collecting l http: //forge. gridforum. org/sf/wiki/do/view. Page/projects. gin/w iki/GINInfo. Wiki l Make sure the data you want is actually theoretically possible to collect! l Worry about the schema later 70
Building your own info providers l l l See the developer session! Some pointers… List of new providers u l http: //www. globus. org/toolkit/docs/development/4. 2 drafts/info/providers/index. html How to write info providers: u u u http: //www. globus. org/toolkit/docs/4. 0/info/usefulrp /rpprovider overview. html http: //www unix. mcs. anl. gov/~neillm/mds/rp provider documentation. html http: //globus. org/toolkit/docs/4. 0/info/index/WS_M DS_Index_HOWTO_Execution_Aggregator. html 71
How many Index Servers? l Generally one at each site, one for full project l Can be cross referenced and duplicated l Can also set them up for an application group or any subset 72
What Triggers? l What are your critical services? 73
What Interfaces? l Command line, Java, C, and Python come for free l Web. MDS give you the simepl one out of the box l Can stylize like TG and ESG did – very straight forward 74
What will you be able to do? l Decide what resource to submit a job to, or to transfer a file from l Keep track of services and be warned of failures l Run common actions to track performance behavior l Validate sites meet a (configuration) guideline 75
Summary l MDS 4 is a WS based Grid monitoring system that uses current standards for interfaces and mechanisms l Available as part of the GT 4 release u l Currently in use for resource selection and fault notification Initial performance results aren’t awful – we need to do more work to determine bottlenecks 76
Where do we go next? l Extend MDS 4 information providers u More data from GT 4 services u Interface to other data sources l Inca, GRASP, Pin. GER Archive, Net. Logger l Additional deployments l Additional scalability testing and development u u Database backend to Index service to allow for very large indexes Performance improvements to queries – partial result return 77
Other Possible Higher Level Services l Archiving service u u The next high level service we’ll build Currently a design document internally, should be made external shortly l Site Validation Service (ala Inca) l Prediction service (ala NWS) l What else do you think we need? Contribute to the roadmap! u http: //bugzilla. globus. org 78
Other Ways To Contribute l Join the mailing lists and offer your thoughts! u mds dev@globus. org u mds user@globus. org u mds announce@globus. org l Offer to contribute your information providers, higher level service, or visualization system l If you’ve got a complementary monitoring system – think about being an Incubator project (contact incubator commiters@globus. org, or come to the talk on Thursday) 79
Thanks l l l MDS 4 Core Team: Mike D’Arcy (ISI), Laura Pearlman (ISI), Neill Miller (UC), Jennifer Schopf (ANL) MDS 4 Additional Development help: Eric Blau, John Bresnahan, Mike Link, Ioan Raicu, Xuehai Zhang This work was supported in part by the Mathematical, Information, and Computational Sciences Division subprogram of the Office of Advanced Scientific Computing Research, U. S. Department of Energy, under contract W 31 109 Eng 38, and NSF NMI Award SCI 0438372. ESG work was supported by U. S. Depart ment of Energy under the Scientific Discovery Through Advanced Computation (Sci. DAC) Program Grant DE FC 02 01 ER 25453. This work also supported by DOESG Sci. DAC Grant, i. VDGL from NSF, and others. 80
81
For More Information l Jennifer Schopf jms@mcs. anl. gov u http: //www. mcs. anl. gov/~jms u l Globus Toolkit MDS 4 u l http: //www. globus. org/toolkit/mds MDS related events at Grid. World u MDS for Developers l u Monday 4: 00 5: 30, 149 A/B MDS “Meet the Developers” session l Tuesday 12: 30 1: 30, Globus Booth 82
Скачать презентацию Monitoring and Discovery in a Web Services Framework
dc1674c54660fb8f629cfbabfe9bf04b.ppt