Enabling Grids for E-scienc. E Additional tools for EGEE users Three days were not enough for everything… Gergely Sipos MTA SZTAKI Budapest www. eu-egee. org www. glite. org EGEE-II INFSO-RI-031688 2 nd Rio Grid School 3 -5. July, 2007 Rio de Janeiro, Brazil
Overview Enabling Grids for E-scienc. E • Programming APIs for g. Lite services – WMProxy (~WMS API) – SEE-GRID File Management API (~LFC&LCG API) – GFAL API you already know… • Additional services in g. Lite – R-GMA • RESPECT – Initiative to collect useful tools that work: – Grid. Way – GANGA – P-GRADE you already know. . . EGEE-II INFSO-RI-031688 2
Enabling Grids for E-scienc. E WMProxy www. eu-egee. org www. glite. org EGEE-II INFSO-RI-031688
The WMProxy Enabling Grids for E-scienc. E • The WMProxy is the service responsible to provide access to the WMS functionality through a Web Service Interface • The g. Lite WMProxy Server can be either accessed directly through the published WSDL, the C++ command line interface, or the API • In these two links you can find a guide about how to interact with the main services exposed by the WMProxy through the provided API Java and C++ API https: //grid. ct. infn. it/twiki/bin/view/GILDA/Api. Java. WMProx y https: //grid. ct. infn. it/twiki/bin/view/GILDA/WMProxy. CPPAP I 4 EGEE-II INFSO-RI-031688
WMS Architecture overview Enabling Grids for E-scienc. E Logging and bookkeeping service LB Proxy Client WMProxy LCG Job Controller Condor. G • Command line client • WS client • C++, Java, Python API Workload Manager g. Lite CE CE g. Lite WMS EGEE-II INFSO-RI-031688 5
References Enabling Grids for E-scienc. E • API Documentation http: //trinity. datamat. it/projects/EGEE/wiki/apidoc/3. 1/ htmljava/index. html • Datamat – WMProxy quick start http: //trinity. datamat. it/projects/EGEE/wiki. php? n =WMProxy. Client. Quick. Start • JDL Attributes guide for WMProxy https: //edms. cern. ch/document/590869/1 • WMProxy user guide https: //edms. cern. ch/document/674643/1 EGEE-II INFSO-RI-031688 9
Enabling Grids for E-scienc. E SEE-GRID File management API www. eu-egee. org www. glite. org EGEE-II INFSO-RI-031688
Overview Enabling Grids for E-scienc. E • SEE-GRID File Management Java API supports most of the data management operations offered by LFC and LCG_UTILS. • These Java API are compatible with LCG 2. 7. x and g. Lite grid middleware. EGEE-II INFSO-RI-031688 11
Catalog management Enabling Grids for E-scienc. E EGEE-II INFSO-RI-031688 12
Enabling Grids for E-scienc. E EGEE-II INFSO-RI-031688 Directory management: Directory. Item class 13
Enabling Grids for E-scienc. E EGEE-II INFSO-RI-031688 Directory management: Directory. Item class 14
Enabling Grids for E-scienc. E EGEE-II INFSO-RI-031688 File management: File. Item class 15
Enabling Grids for E-scienc. E File management: File. Item class EGEE-II INFSO-RI-031688 16
References Enabling Grids for E-scienc. E • SEE-GRID File Management Java API Documentation http: //grid 02. rcub. bg. ac. yu/LFCJava. API/fles/docs/java doc/version 1. 2/index. html • Source code (version 1. 2) http: //grid 02. rcub. bg. ac. yu/LFCJava. API/files/downloa ds/SEE-GRIDFile. Management. APIv 1. 2. zip • Source code (version 1. 1) http: //grid 02. rcub. bg. ac. yu/LFCJava. API/files/downloa ds/SEE-GRIDFile. Management. APIv 1. 1. zip EGEE-II INFSO-RI-031688 17
Enabling Grids for E-scienc. E R-GMA www. eu-egee. org www. glite. org EGEE-II INFSO-RI-031688
What is R-GMA ? Enabling Grids for E-scienc. E • Uniform method to access and publish both information and monitoring data. • From a user's perspective, an R-GMA installation currently appears similar to a single relational database. • GMA (Grid Monitoring Architecture) was developed by the Global Grid Forum (Predecessor of Open Grid Forum – www. ogf. org) • R-GMA (Relational GMA) was created: – To simplify use of GMA – To give a relational view EGEE-II INFSO-RI-031688 19
R-GMA Enabling Grids forrunning Processes E-scienc. E on Computing Elements P 1 R-GMA cmd line tools OR programming API VIRTUAL DATABASE TABLE 1, Producer P 1 details P 3 P 2 TABLE 1, Colum defs TABLE 2, Producer P 1 details TABLE 2, Colum defs TABLE 2, Producer P 2 details TABLE 3, Colum defs TABLE 4, Colum defs SQL “CREATE TABLE” SQL “INSERT” SCHEMA TABLE 2, Producer P 3 details TABLE 3, Producer P 2 details TABLE 3, Producer P 1 details TABLE 3, Producer P 3 details MEDIATOR R-GMA cmd line tools OR programming API SQL “SELECT” C 1 C 2 REGISTRY There is no central repository!!! There is only a “Virtual Database”. Client applications running on the UI (or embedded in a portal) Schema is a list of table definitions: additional tables/schema can be defined by applications Registry is a list of data producers with all its details. Producers publish data. Consumers read data published. EGEE-II INFSO-RI-031688 20
More information Enabling Grids for E-scienc. E • R-GMA overview page. – http: //www. r-gma. org/ • R-GMA in EGEE – http: //hepunx. rl. ac. uk/egee/jra 1 -uk/ • R-GMA command line tool – http: //hepunx. rl. ac. uk/egee/jra 1 -uk/glite-r 1/command-line. pdf • R-GMA Browser Home Page – https: //rgmasrv. ct. infn. it: 8443/R-GMA/ EGEE-II INFSO-RI-031688 21
Enabling Grids for E-scienc. E EGEE NA 4 RESPECT initiative Recommended External Software Packages for Egee Communi. Ties www. eu-egee. org www. glite. org EGEE-II INFSO-RI-031688
Motivations Enabling Grids for E-scienc. E • EGEE g. Lite middleware: implementations of base grid services that serve as a platform for high-level services • Not aim to provide a comprehensive solution for any particular grid application • Software from external providers must be used in conjunction with g. Lite to satisfy fully the needs of the user community • The array of available grid software is vast! • RESPECT (Recommended External Software Packages for EGEE Communities) program aims identifying useful, well-supported software for EGEE users EGEE-II INFSO-RI-031688 23
Benefits to EGEE users Enabling Grids for E-scienc. E • Having a set of external software packages that enhances the functionality of the g. Lite middleware • Rduces the amount of application development, and generally accelerates the adoption of grid technologies • Reduce the pressure on the EGEE middleware activity to provide solutions for services outside of the core functionality • Integration and testing activities in EGEE can concentrate on core g. Lite issues • Increasing the number of users via a more attractive platform and having more varied services EGEE-II INFSO-RI-031688 24
RESPECT tools Enabling Grids for E-scienc. E • RESPECT provides list of software that focuses on those packages that are genuinely useful for an EGEE application and that are generic enough to be useful to other applications • The RESPECT program is not a general repository of grid software • Current RESPECT tools: – GANGA – Grid. Way – P-GRADE Portal • Further information: http: //egeena 4. lal. in 2 p 3. fr/ “Grid software” menu EGEE-II INFSO-RI-031688 25
Grid. Way one of the tools recognised by EGEE’s RESPECT program Alternative to WMS Higher level command line UI Examples of use: Alternative broker – no need for close CE-SE Many similar jobs Resources outside EGEE also to be used User-site-specific policies are required (priorities of users’ jobs) …
DRMAA Application-Infrastructure decoupling CLI. C, . java Grid. Way Globus, g. Lite, … PBS SGE Grid Middleware $> Grid Meta. Scheduler Results Applications What is Grid. Way? Infrastructure • standard API (OGF DRMAA) • Command Line Interface • open source • job execution management • resource brokering • Basic Grid services • Standard interfaces • end-to-end (e. g. TCP/IP) • highly dynamic & heterogeneous • high fault rate
Who is using Grid. Way in EGEE? Massive Ray Tracing CD-HIT workflow Fusion Biomed Users Grid. Way Users EGEE RB • DRMAA interface • VO Schedulers Grid. Way • Services: BDII, GRAM, Grid. FTP • EGEE Resource Broker g. Lite SGE Cluster PBS Cluster
Where can I get Grid. Way? Download the software • From the Gridway webpage: www. gridway. org • From the ETICS repository • From the Globus CVS repository (cvs. globus. org) Install the software • Install it on your desktop computer OR • Ask your institute to make a central installation OR • Ask your VO to make a central installation More Information • Gridway webpage: www. gridway. org • Application porting with Grid. Way http: //www. gridway. org/successstories/applicationporting. php • Infrastructures using Grid. Way http: //www. gridway. org/successstories/projectsinfrastructures. php
Enabling Grids for E-scienc. E GANGA www. eu-egee. org www. glite. org EGEE-II INFSO-RI-031688
Introduction Enabling Grids for E-scienc. E Goals: • provide a simple and consistent way of preparing, organising and executing jobs on different computing infrastructures • provide a clean interface which can be used: – interactively (CLI / python interpreter) – as a Python API for scripting – through a GUI • • • Make it easy and integrated with application environment Allow quick transition between local PC, cluster, Grid. . . Organize work, keep history of jobs, . . . EGEE-II INFSO-RI-031688 37
Motivation Enabling Grids for E-scienc. E • In practice users deal with multiple computing backends FULL RUN PANDA PBS SGE Local PC EGEE-II INFSO-RI-031688 TEST LSF DEBUG 38
Motivation Enabling Grids for E-scienc. E • FAQ: running applications on multiple computing backends I must learn many interfaces PBS LSF SGE Local. PC PANDA How to configure my applications? Do I get a consistent view on all my jobs? EGEE-II INFSO-RI-031688 39
Ganga Enabling Grids for E-scienc. E • Ganga: Job Management Interface – a utility which you download to your computer § or it is already installed in your institute in a shared area • for example: /nfs/sw/ganga/install/4. 3. 2 § it is an add-on to installed software – comes with a set of plugins for some applications § open - other applications and backend may be easily added • even by users Ganga. Framework Ganga Application Software LSF Client LCG UI EGEE-II INFSO-RI-031688 Application Plugins Backend Plugins. . 40
Ganga Job Enabling Grids for E-scienc. E Where the Ganga journey starts … Mandatory Executable EGEE Optional EGEE-II INFSO-RI-031688 41
Example Application: the ATLAS data analysis application Enabling Grids for E-scienc. E $ ganga athena --in. DS my. Input. Dataset. txt --outputdata my. Output. root --split 3 --maxevt 100 --lsf job. Options. py j = Job() j. application=Athena() j. application. prepare() j. application. option_file='job. Options. py‘ Scripting mode quick mix CLIP mode application j. inputdata=DQ 2 Dataset() j. inputdata. type='DQ 2_LOCAL' j. inputdataset=“my. Input. Dataset. txt” inputdata j. outputdata=DQ 2 Output. Dataset() j. outputdata=[‘my. Output. root'] outputdata j. splitter = Athena. Splitter. Job(numsubjobs=3) j. merger = Athena. Output. Merger() Splitter & Merger flexible j. backend = LSF() j. submit() j 2 = j. copy() j 2. backend=LCG( CE=’ce 102. cern. ch: 2119/jobmanager-lcglsf-grid_2 nh_atlas’ ) j 2. submit() EGEE-II INFSO-RI-031688 42
More information Enabling Grids for E-scienc. E • • • Ganga Home: http: //cern. ch/ganga Official Ganga User’s Guide: http: //cern. ch/ganga/user/html/Ganga. Introduction/ Ganga. Tutorial GPI Reference Manual : http: //ganga. web. cern. ch/ganga/release/4. 3. 2/reports/html/Man uals/Ganga. Tutorial. Manual. html • Looking for help: project-ganga-developers@cern. ch EGEE-II INFSO-RI-031688 43
Summary Enabling Grids for E-scienc. E • g. Lite services can be accessed through programming APIs too – GFAL API (yesterday), WMProxy, SEE-GRID • R-GMA – Relational database that mediates between your job and you • RESPECT program: EGEE NA 4 initiative to identify useful tools that work and has user support – List of software is at http: //egeena 4. lal. in 2 p 3. fr/ – Grid. Way: Broker and higher level command line client § Alternative to WMS; parametric jobs – GANGA § Object oriented cmd line interface for WMS; parametric jobs, splitter, merger components – P-GRADE: § Web portal, workflow and parameter study support EGEE-II INFSO-RI-031688 44
Enabling Grids for E-scienc. E Questions? www. eu-egee. org www. glite. org EGEE-II INFSO-RI-031688
Скачать презентацию Enabling Grids for E-scienc E Additional tools for
c4121482fd05a54218ae95c9c17f63a3.ppt