An Agent Based, Dynamic Service System to Monitor, Control and Optimize Distributed Systems ACAT - November 2008 ERICE 1 Iosif Legrand, Harvey Newman, Ramiro Voicu , Costin Grigoras, Catalin Cirstoiu, Ciprian Iosif Legrand November Dobre 2008 April 2007 Iosif Legr
The Mon. ALISA Framework Ø Mon. ALISA is a Dynamic, Distributed Service System capable to collect any type of information from different systems, to analyze it in near real time and to provide support for automated control decisions and global optimization of workflows in complex grid systems. Ø The Mon. ALISA system is designed as an ensemble of autonomous multithreaded, self-describing agent-based subsystems which are registered as dynamic services, and are able to collaborate and cooperate in performing a wide range of monitoring tasks. These agents can analyze and process the information, in a distributed way, and to provide optimization decisions in large scale distributed applications. Iosif Legrand November 2008 April 2007 Iosif Legr
Distributed Object Systems CORBA , DCOM “Traditional” Distributed Object Models (CORBA, DCOM) Server The Stub is linked to the Client. The Client must know about the service from the beginning and needs the right stub for it Lookup Stub Service Lookup Skeleton Service CLIENT “IDL” Compiler The Server and the client code must be created together !! Iosif Legrand November 2008
Distributed Object Systems Web Services WSDL/SOAP Server CLIENT Lookup WSDL Service Lookup Interface Service The client can dynamically generate the data structures and the interfaces for using remote objects based on WSDL Platform independent Iosif Legrand November 2008
Mobile Code and Distributed Services Any well suited protocol for the application Lookup Proxy Service CLIENT Dynamic Code Loading Services can be used dynamically ØRemote Services Proxy == RMI Stub ØMobile Agents Proxy == Entire Service Ø“Smart Proxies” Proxy adjusts to the client Act as a true dynamic service and provide the necessary functionally to be used by any other services that require such information (Jini, interface to WSDL / SOAP) Ø Ø Ø mechanism to dynamically discover all the “Service Units" remote event notification for changes in the any system lease mechanism for each registered unit Iosif Legrand November 2008 April 2007 Iosif Legr
Mon. ALISA Service & Data Handling Postgres n Data Store Web Service WS Clients and service WSDL SOAP Data Cache Service & DB io rat t is eg R Lookup Service D is co ve ry Data (via ML Proxy) Predicates & Agents Applications Configuration Control (SSL) AGENTS FILTERS / TRIGGERS Collects any type of information Monitoring Modules Clients or Higher Level Services Dynamic Loading Push and Pull Iosif Legrand November 2008 April 2007 Iosif Legr
The Mon. ALISA Architecture HL services Proxies Agents Mon. ALISA services Network of JINI-Lookup Services Secure & Public Regional or Global High Level Services, Repositories & Clients Secure and reliable communication Dynamic load balancing Scalability & Replication AAA for Clients Distributed System for gathering and analyzing information based on mobile agents: Customized aggregation, Triggers, Actions Distributed Dynamic Registration and Discoverybased on a lease mechanism and remote events Fully Distributed System with no Single Point of Failure Iosif Legrand November 2008 April 2007 Iosif Legr
Registration / Discovery Admin Access and AAA for Clients Application Mon. ALISA Service Registration (signed certificate) Discovery Client (other service) Lookup Service Trust keystore Services Proxy Multiplexer Applications Mon. ALISA Services Proxy Multiplexer Admin SSL connection Lookup Service Mon. ALISA Service Trust keystore Data Filters & Agents Client authentication Client (other service) AAA services Iosif Legrand November 2008 April 2007 Iosif Legr
Monitoring Grid sites, Running Jobs, Network Traffic, and Connectivity Running Jobs JOBS TOPOLOGY ACCOUNTING Iosif Legrand November 2008 April 2007 Iosif Legr
Monitoring architecture in ALICE Ap. Mon run tim e eg at ed n ope files ed eu nts Qu Age b Jo di us sk ed My. SQL ets Servers sock mi g mb rate yte d s API Services at a Ap. Mon Castor. Grid Scripts Ap. Mon Mona. Lisa Repository y rox My. P tus sta Ap. Mon D lo Ap. Mon. ALISA LCG Site Ali. En Job Agent 10 gr nr. o f files Ap. Mon job sta tus cpu Ali. En Job Agent ksi 2 k Ap. Mon. ALISA @CERN Ali. En SE Ap. Mon ad e tiv ac ions ss se Cluster Monitor Ali. En CE Ap. Mon Ag Ap. Mon Ali. En Job Agent j st obs at us Mon. ALISA @Site rss Ali. En Job Agent Ap. Mon Ali. En Brokers Ap. Mon ses z Ali. En SE Ap. Mon ces vs Ali. En TQ Ali. En Optimizers pro Ali. En Job Agent Ap. Mon job slots u cp e tim Ap. Mon f sp ree ac e Ali. En Job Agent Ali. En IS Cluster Monitor net In/o ut Ali. En CE LCG Tools Long History DB Iosif Legrand November 2008 Alerts Actions
ALICE : Global Views, Status & Jobs http: //pcalimonitor. cern. ch Iosif Legrand November 2008 April 2007 Iosif Legr
ALICE: Job status – history plots Iosif Legrand November 2008 April 2007 Iosif Legr
ALICE: Resource Usage monitoring Ø Cumulative parameters q CPU Time & CPU KSI 2 K q Wall time & Wall KSI 2 K q Read & written files q Input & output traffic (xrootd) Ø Running parameters q Resident memory q Virtual memory Ø Open files q Workdir size q Disk usage q CPU usage Ø Aggregated per site Iosif Legrand November 2008 April 2007 Iosif Legr
ALICE: Job agents monitoring Ø From Job Agent itself q Requesting job q Installing packages q Running job q Done q Error statuses Ø From Computing Element q Available job slots q Queued Job Agents q Running Job Agents Iosif Legrand November 2008 April 2007 Iosif Legr
Monitoring the Execution of Jobs and the Time Evolution SPLIT JOBS LIFELINES for JOBS Summit a Job Job DAG Job 1 Job 2 Job 31 Job 32 Iosif Legrand November 2008 April 2007 Iosif Legr
Local and Global Decision Framework Two levels of decisions: local (autonomous), • Traffic • Jobs • Hosts • Apps global (correlations). Actions triggered by: values above/below given thresholds, absence/presence of values, ML Service Actions based on global information Actions based on local information Global ML Services correlations between any values. • Temperature • Humidity alerts (emails/instant msg/atom feeds), • A/C Power • … Action types: ML Service running an external command, automatic charts annotations in the repository, Sensors Local decisions Global decisions running custom code, like securely ordering a ML service to (re)start a site service. Iosif Legrand November 2008 April 2007 Iosif Legr
ALICE: Automatic job submission Restarting Services My. SQL daemon is automatically restarted when it runs out of memory Trigger: threshold on VSZ memory usage ALICE Production jobs queue is kept full by the automatic submission Trigger: threshold on the number of aliprod waiting jobs Administrators are kept up-to-date on the services’ status Trigger: presence/absence of monitored information Iosif Legrand November 2008 April 2007 Iosif Legr
Automatic actions in ALICE is using the monitoring information to automatically: resubmit error jobs until a target completion percentage is reached, submit new jobs when necessary (watching the task queue size for each service account) production jobs, RAW data reconstruction jobs, for each pass, restart site services, whenever tests of Vo. Box services fail but the central services are OK, send email notifications / add chart annotations when a problem was not solved by a restart, dynamically modify the DNS aliases of central services for an efficient load-balancing. Most of the actions are defined by few lines configuration files. Iosif Legrand November 2008 April 2007 Iosif Legr
Monitoring USLHCnet r Operations & management assisted by agent-based software r Used on the new CIENA equipment used for network managment Iosif Legrand November 2008 April 2007 Iosif Legr
USLHCnet: Precise measurements for the Operational Status on the WAN Link r Operations & management assisted by agent-based software r Used on the new CIENA equipment used for network managment Iosif Legrand November 2008 April 2007 Iosif Legr
USLHCnet: Traffic on different segments Iosif Legrand November 2008 April 2007 Iosif Legr
USLHCnet: Accounting for Integrated Traffic Iosif Legrand November 2008 April 2007 Iosif Legr
The Ultra. Light Network BNL ESnet IN /OUT 23 Iosif Legrand November 2008
Available Bandwidth Measurements Embedded Pathload module. Iosif Legrand November 2008 April 2007 Iosif Legr
Monitoring Network Topology, Latency, Routers NETWORKS ROUTERS AS Iosif Legrand November Real Time Topology Discovery & Display 2008 April 2007 Iosif Legr
EVO : Real-Time monitoring for Reflectors and the quality of all possible connections Iosif Legrand November 2008 April 2007 Iosif Legr
ating a Dynamic, Global, Minimum Spanning Tree to optimize the con A weighted connected graph G = (V, E) with n vertices and m edges. The quality of connectivity between any two reflectors is measured every second. Building in near real time a minimumspanning tree with addition constrains 27 Iosif Legrand November 2008
Dynamic MST to optimize the Connectivity for Reflectors Frequent measurements of RTT, jitter, traffic and lost packages The MST is recreated in ~ 1 S case on communication problems. Iosif Legrand November 2008 April 2007 Iosif Legr
EVO: Optimize how clients connect to the system for best performance and load balancing Iosif Legrand November 2008 April 2007 Iosif Legr
FDT – Fast Data Transfer Ø FDT is an application for efficient data transfers. Ø Easy to use. Written in java and runs on all major platforms. Ø It is based on an asynchronous, multithreaded system which is using the NIO library and is able to: Ø stream continuously a list of files Ø use independent threads to read and write on each physical device Ø transfer data in parallel on multiple TCP streams, when necessary Ø use appropriate size of buffers for disk IO and networking Ø resume a file transfer session 30 Iosif Legrand November 2008
FDT – Fast Data Transfer Control connection / authorization Pool of buffers Kernel Space Data Transfer Sockets / Channels Restore the files from buffers Independent threads per device 31 Iosif Legrand November 2008
FDT features The FDT architecture allows to "plug-in" external security APIs and to use them for client authentication and authorization. Supports several security schemes : • IP filtering • SSH • GSI-SSH • Globus-GSI • SSL Ø User defined loadable modules for Pre and Post Processing to provide support for dedicated MS system, compression … Ø FDT can be monitored and controlled dynamically by Mon. ALISA services Ø 32 Iosif Legrand April 2007 November 2008 Iosif Legrand
FDT – Memory to Memory Tests in WAN ~9. 4 Gb/s ~9. 0 Gb/s CPUs Dual Core Intel Xenon @ 3. 00 GHz, 4 GB RAM, 4 x 320 GB SATA Disks Connected with 10 Gb/s Myricom 33 Iosif Legrand October 2006 November 2008 Iosif Legrand
Disk -to- Disk transfers in WAN 1 U Nodes with 4 Disks MB/s NEW YORK GENEVA 4 U Disk Servers with 24 Disks CERN CALTECH Reads and writes on 4 SATA disks in parallel on each server Reads and writes on two 12 -port RAID Controllers in parallel on each server Mean traffic ~ 210 MB/s ~ 0. 75 TB per hour Mean traffic ~ 545 MB/s ~ 2 TB per hour u Lustre read/ write ~ 320 MB/s between Florida and Caltech u Works with xrootd u Interface to d. Cache using the dcap protocol Iosif Legrand October 2007 November 2008 April 2007 Iosif Legrand Iosif Legr
Monitoring Optical Switches Dynamic restoration of lightpath if a segment has problems Iosif Legrand November 2008 April 2007 Iosif Legr
Monitoring the Topology and Optical Power on Fibers for Optical Circuits Controlling Port power monitoring Iosif Legrand Glimmerglass Switch Example November 2008 April 2007 Iosif Legr
“On-Demand”, End to End Optical Path Allocation >FDT A/file. X B/path/ CREATES AN END TO END PATH < 1 s OS path available Configuring interfaces Starting Data Transfer Real time monitoring Internet DATA Re g APPLICATION ul ar IP p at h Mon. ALISA Distributed Service System LISA sets up - Network Interfaces - TCP stack - Kernel parameters - Routes LISA APPLICATION “use eth 1. 2, …” 37 OS Agent B LISA Agent TL 1 LISA AGENT Monitor Control A Mon. ALISA Service Optical Switch Active light p ath Iosif Legrand Detects errors and automatically recreate the path in less than the TCP timeout November 2008
Controlling Optical Planes Automatic Path Recovery 200+ MBytes/sec From a 1 U Node CERN Geneva USLHCnet Internet 2 Starlight FDT Transfer CALTECH Pasadena Manlan “Fiber cut” simulations The traffic moves from one transatlantic line to the other one FDT transfer (CERN – CALTECH) continues uninterrupted TCP fully recovers in ~ 20 s 38 Iosif Legrand 2 1 4 3 4 fiber cut emulations 4 Fiber cuts simulations November 2008
End to End Path Provisioning on different layers Default IP route Site A Layer 3 Site B VCAT and VLAN channels Layer 2 Optical path Layer 1 Monitor layout / Setup circuit Monitor interfaces traffic Monitor host & end-to-end paths / Setup end-host parameters Control transfers and bandwidth reservations Iosif Legrand November 2008 April 2007 Iosif Legr
“On-Demand”, L 2 Dynamic Channel and Path Allocation APPLICATION >FDT A/file. X B/path/ path or channel allocation Configuring interfaces Starting Data Transfer Reg ula r IP pa Reg th ular Local VLANs MAP Local VLANs to WAN channels or light paths IP p ath Recommended to use two NICs -one for management /one for data -- bonding two NICs to the same IP Iosif Legrand November 2008 April 2007 Iosif Legr
The Need for Planning and Scheduling for Large Data Transfers In Parallel Sequential 2. 5 X Faster to perform the two reading tasks sequentially Iosif Legrand November 2008 April 2007 Iosif Legr
Dynamic Path Provisioning Queueing and Scheduling Channel allocation based on VO/Priority, [ + Wait time, etc. ] Create on demand a End-to-end path or Channel & configure end-hosts Automatic recovery (rerouting) in case of errors Dynamic reallocation of throughputs per channel: to manage priorities, control time to completion, where needed u Reallocate resources requested but not used u u Request User Realtime Feedback Scheduling Control Monitoring End Host Agents 42 Iosif Legrand November 2008
Dynamic priority for FDT Transfers on common segments Priority 4 Priority 2 Priority 8 Iosif Legrand November 2008 April 2007 Iosif Legr
Bandwidth Challenge at SC 2005 151 Gbs ~ 500 TB Total in 4 h Iosif Legrand November 2008 April 2007 Iosif Legr
FDT & Mon. LISA Used at SC 2006 17. 7 Gb/s Disk to Disk on 10 Gb/s link used in Both directions from Florida to Caltech 45 Iosif Legrand April 2007 November 2008 Iosif Legrand
SC 2006 Official BWC 46 Hyper BWC Iosif Legrand April 2007 November 2008 Iosif Legrand
SC 2007 +80 Gb/s disk to disk Iosif Legrand November 2008 April 2007 Iosif Legr
Communities using Mon. ALISA Major Communities q q q q ALICE CMS ATLAS EVO LGC RUSSIA UNAM Grid (Mx) ITU q q q USLHCNET ULTRALIGHT GLORIAD ABILENE Ro. Edu. NET Enlightened Mon. ALISA Today USLHCne Running 24 X 7 VRVS ALICE at ~340 Sites t Ø Collecting ~ 1 000 - parameters in near real-time Ø Update rate of 20, 000 parameter updates per second http: //monalisa. caltech. edu OSG Ø Monitoring Ø 12, 000 computers Ø > 100 WAN Links Ø Thousands of Grid jobs running EVO concurrently Iosif Legrand November 2008 April 2007 Iosif Legr
Скачать презентацию An Agent Based Dynamic Service System to Monitor
ece92d7b3c4e4d3e7ec676068adf15be.ppt