P-GRADE Portal and GEMLCA A workflow-oriented portal and

P-GRADE Portal and GEMLCA: A workflow-oriented portal and application hosting environment Miklos Kozlovszky m. kozlovszky@sztaki. hu MTA SZTAKI (Hungarian Academy of Sciences) www. portal. p-grade. hu www. cpc. wmin. ac. uk/gemlca 1

Contents • • Motivation of creating the tools P-GRADE Portal and GEMLCA in a nutshell Lifecycle of GEMLCA / P-GRADE applications Services provided for application developers • Introduction to the hands-on exercises • Hands-on 2

Context Application toolkits, standards Higher-level grid services (brokering, …) Basic Grid services: AA, job submission, info, … Graphical interface Middleware independent services and interfaces of P-GRADE/GEMLCA Middleware specific clients Grid middleware services 3

Current situation and trends in Grid computing • Fast evolution of Grid systems and middleware: – GT 2, OGSA, GT 3 (OGSI), GT 4 (WSRF), LCG-2, g. Lite, … • Many production Grid systems are built with them – EGEE (LCG-2 g. Lite), UK NGS (GT 2), Open Science Grid (GT 2 GT 4), Nordu. Grid (~GT 2) • Although the same set of core services are available everywhere, they are implemented in different ways – – Data services (file management) Computation services (job submission) Security services (proxy based single sign-on) Brokers (not in every middleware, but e. g. in g. Lite - WMS) 4

P-GRADE Portal in a nutshell • General purpose, workflow-oriented computational Grid portal. Supports the development and execution of workflow based Grid applications – a Grid orchestration environment • Based on Grid. Sphere web portal framework – Functionalities are accessed through portlets – Easy to expand with new portlets (e. g. application-specific portlets) – Easy to tailor to end-user or community needs Developed by SZTAKI (1. 0 in 2003, now 2. 5) Grid services supported by P-GRADE Portal 2. 5: TODAY’S FOCUS • • Service Job execution File storage Certificate management Information system Brokering EGEE grids (LCG/g. Lite) Globus 2 grids Computing Element GRAM Storage Element, File catalog Grid. FTP server My. Proxy server, VOMS server BDII MDS-2, MDS-4 Workload Management System Job monitoring Mercury Workflow & job visualization PROVE Solves Grid interoperability problem at the workflow level 6

GEMLCA extension of the P-GRADE Portal • P-GRADE Portal extended with GEMLCA Grid service back-end – To share jobs and legacy codes as application components with others – A step towards collaborative e-Science • Developed by the University of Westminster (London) • Support for Globus 4 grids (besides GT 2 and EGEE) • Available on the NGS and OGF GIN LCG / g. Lite VOs Globus 2 VOs P-GRADE Portal job job GEMLCA Globus 4 VOs 7

Related projects The development, operation and training of P-GRADE Portal and GEMLCA is supported by the following projects: – SEE-GRID www. see-grid. eu Development, application support – Coregrid www. coregrid. net Research, development – EGEE www. eu-egee. org g. Lite training, application development – ICEAGE www. iceage-eu. org Grid training and education 8

A Grid application in the GEMLCA / P-GRADE Portal • A directed acyclic graph where – Nodes represent jobs or services (a batch program executed on a computing resource) – Ports represent input/output files the components expect/produce – Arcs represent file transfer operations • Semantics of the workflow: – A job can be executed if all of its input files are available – Responsibility of the built-in workflow manager 9

Three levels of parallelism within a P-GRADE Portal application • The workflow concept of the GEMLCA/ P-GRADE Portal enables the efficient parallelization of complex problems • Semantics of the workflow enables two levels of parallelism: Multiple instances of the same workflow process different data files – Parallel execution inside a workflow node – Parallel execution among workflow nodes – Parametric sweep execution of the workflow (SIMD) Multiple nodes can run parallel The job/service can be a parallel code 10

Workflow-level Grid interoperability: The GIN Resource Testing portal Grid Interoperability Now VO Portal: OGF effort to demonstrate workflow level grid interoperability between major production Grids and to monitor these resources 12

The typical user scenario Part 1 - development phase My. Proxy servers SAVE WF / PS Portal server REUSE WORKFLOW COMPONENTS Grid services START EDITOR OPEN & EDIT or DEVELOP WORKFLOW or PS WF 13

The typical user scenario Part 2 - execution phase My. Proxy servers TRANSFER FILES, SUBMIT JOBS DOWNLOAD PROXY CERTIFICATES VISUALIZE JOBS and WORKFLOW PROGRESS Portal server MONITOR JOBS Grid services DOWNLOAD (SMALL) RESULTS Keep large files on Grid storage resources 14

The typical user scenario Part 3 - collaborative phase My. Proxy servers Share workflow components with other users of the same portal Portal server Grid services Export and share workflows with users of the same, or another portal 15

Inside the portal server Client • • P-GRADE Portal server Technology specific gateways Tomcat File transfer Proxy management P-GRADE Portal portlets (JSR-168, Gridsphere 2): Load monitoring Workflow, Certificates, Information System, Settings, GEMLCA DAGMan workflow manager shell scripts Grid middleware clients Grid Java Webstart Optional workflow editor plug-in: Web browser GEMLCA service (WSRF) Grid middleware services (WMS, LFC, SE, …) Mercury API Information system clients Co. G API & scripts Mercury monitor service Information systems My. Proxy server & VOMS 16

Workflow Editor Defining the graph Define a Directed Acyclic Graph (DAG) of jobs and services (GEMLCA jobs): 1. Drag & drop components: nodes and ports 2. Define their properties 3. Connect ports by channels (no cycles, no loops, no conditions…) 17

Workflow Editor Properties of a job component Properties of a job: • Type of executable • Client side location of the binary • Number of required processors • Command line parameters • The resource to be used for the execution: • Grid (VO) • Resource / broker 18

Workflow Editor Defining broker jobs Select a Grid with broker! (*_BROKER) Ignore the resource field! If default JDL is not sufficient use the built-in JDL editor! 19

Workflow Editor Built-in JDL editor for brokered jobs Rank & Requirement JDL look at the g. Lite Users’ manual! 20

Workflow Editor Properties of a service component (GEMLCA job) Properties of a service: • The location of the service: • Grid (VO) • Resource / broker • An application (binary) associated with that resource • Input parameter values for the service 21

Workflow Editor Defining job / service input-output data File properties Type: input: the component reads output: the component writes File type: local: originates from my desktop remote: originates from a grid storage element File: location of the file File storage type (for outputs only): Permanent: final result Volatile: used only for inter-component data transfer 22

How to refer to an I/O file? Input file Output file Local file • Client side location: result. dat c: experiments11 -04. dat • LFC logical file name (LFC file catalog is required – EGEE VOs) lfn: /grid/gilda/sipos/11 -04_-_result. dat • Grid. FTP address (in Globus Grids): gsiftp: //somengshost. ac. uk/mydir/11 -04. dat gsiftp: //somengshost. ac. uk/mydir/result. dat Remote file 23

Workflow level file transfer by the workflow manager Grid infrastructure LOCAL INPUT FILES & BINARIES LOCAL OUTPUT FILES Portal server LOCAL INPUT FILES & BINARIES User level storage GEMLCA repository LOCAL OUTPUT FILES Binaries of GEMLCA jobs Storage elements REMOTE INPUT FILES REMOTE OUTPUT FILES Computing elements 24

Job / service level file transfer by the workflow manager 0 1 Grid infrastructure 2 3 Generated by the portal Computing Element LOCAL INPUT FILE Portal server Pre script 0 binary LOCAL OUTPUT FILE Post script 2 Generated by the portal Storage Elements REMOTE INPUT 1 FILE Custom file transfer REMOTE OUTPUT FILE 3 25

Reminder: grid files in JDL • Example JDL file name file lfn: logical Executable = Catalog to find file RB uses File “grid. Test”; The file itself is NOT transferred by location the middleware! Std. Error = “stderr. log”; Your binary must transfer Std. Output = “stdout. log”; input/output grid files! Input. Sandbox = {“/home/joda/test/grid. Test”}; Output. Sandbox = {“stderr. log”, “stdout. log”}; Input. Data = “lfn: /grid/VOname/mydir/testbed 0 -00019”; Requirements = other. Architecture==“INTEL” && can tranfer the file Higher level tools other. Op. Sys==“LINUX” && other. Free. Cpus >=4; for you. E. g. P-GRADE Portal Rank = “other. Glue. Host. Benchmark. SF 00”; Your code does not have to “speak” storage protocols if it is developed in P-GRADE Portal! 26

Information system portlet to browse computing elements Graphical interface for BDII servers 27

Workflow execution Main steps 1. 2. 3. 4. 5. Download proxies Submit workflow Observe workflow progress If some error occurs correct the graph Download result 28

Certificate Manager Certificates portlet • To start your session on the Grid you must create a proxy certificate on the portal server • “Certificates” portlet: • to upload a proxy into My. Proxy servers • to download a proxy from My. Proxy into the portal server 29

Certificate Manager Multi-grid portal Multi-proxy environment Multiple proxies can be available on the portal server at the same time! Certificate from EGEE CA: SEE-GRID CEs and SEs Certificate from Hungarian CA: HUNGRID CEs and SEs 31

Certificates, proxies with g. Lite VOs: Download My. Proxy server VOMS server Proxy 1 Proxy 2 VOMS ext. Proxy 2 I have to do this every time when I want to execute workflows Portal server VOMS ext. Proxy 2 Grid services 33

Workflow Management (workflow portlet) • The portlet presents the status, size and output of the available workflow in the “Workflow” list • It has a Quota manager to control the users’ storage space on the server • The portlet also contains the “Abort”, “Attach”, “Details”, “Delete” and “Delete all” buttons to handle execution of workflows • The “Attach” button opens the workflow in the Workflow Editor • The “Details” button gives an overview about the jobs of the workflow 34

Workflow Execution (observation by the workflow portlet) White/Red/Green color means the job is initial/running/finished state 35

Workflow Execution (observation by the workflow portlet) White/Red/Green color means the job is initial/running/finished state 36

Workflow Execution (observation by the workflow portlet) White/Red/Green color means the job is initial/running/finished state 37

Workflow Execution (observation by the workflow portlet) White/Red/Green color means the job is initial/running/finished state 38

Workflow Execution (observation by the workflow portlet) White/Red/Green color means the job is initialised/running/finished 39

On-Line Monitoring both at the workflow and job levels (workflow portlet) - The portal monitors and visualizes workflow progress - The portal monitors and visualizes parallel jobs (if they are prepared for Mercury monitor) 40

Rescuing a failed workflow 1. A job failed during workflow execution Read the error log to know why 41

Rescuing a failed workflow 2. Don’t touch the finished jobs! Map the failed job onto a different resource or download a new proxy for it The execution can continue from the point of failure 42

Downloading the results… 43

Sharing a successfully finished job with other users: GEMLCA repository 44

Collaborative grid applications Combine services and your code in the same workflow! Service invocation Job submission 45

File Management through LFC and LCG • File / Directory management through LFC and LCG – – – listing LFC hosts for the selected VO browsing a LFC directory creating a new directory removing a directory/file displaying details (owner-group info, last modification, access rights) of a directory/file renaming a directory/file changing access rights of a directory/file uploading a (local) file to a storage element downloading a file from a storage element listing replicas of a file In the frame of the Portal Developer Alliance replicating a file Birsen Omay from the Middle East Technical University has created this new P-GRADE deleting a replica of a file Portal extension. 46

File Management through LFC and LCG LFC Host Portal server Storage Element LFC file and directory management 47

My. Proxy Credential Manager Getting information about a My. Proxy credential • My. Proxy Credential Management – getting info about a previously stored My. Proxy credential – changing passphrase of a My. Proxy credential – removing a credential In the frame of the Portal Developer Alliance Birsen Omay from the Middle East Technical University has created this new P-GRADE Portal extension 48

My. Proxy Credential Manager Getting information about a My. Proxy credential My. Proxy server access details: Hostname Port number User name (from upload) Password (from upload) Display information about My. Proxy credential 49

Certificates, proxies: Getting information My. Proxy server Get information (owner, start date, end date) about the credential stored for “username” from My. Proxy Server Portal server Request information about the credential Grid services 50

My. Proxy Credential Manager Getting information about a My. Proxy credential Information about the credential for username “birsen” 51

Certificates, proxies: Changing Passphrase My. Proxy server Modify password of the credential stored for “username” Portal server Change password for “username” Grid services 52

Certificates, proxies: Removing a Credential My. Proxy server Remove the credential for “username” from My. Proxy server Portal server Grid services Destroy the credential for “username” 53

How to get access? • P-GRADE Portal service is available: – SEE-GRID infrastructure – Central European VO of EGEE – GILDA: Training VO of EGEE – Many national Grids (UK National Grid Service, Hun. Grid, Turkish Grid, etc. ) – US Open Science Grid, Tera. Grid – Economy-Grid, Swiss Bio. Grid, Bio and Biomed EGEE VOs, Baltic. Grid – OGF Grid Interoperability Now (GIN) VO portal. p-grade. hu/index. php? m=5&s=0 54

Summary and conclusion • P-GRADE Portal hides the complexity of Grid systems – Globus 2, Globus 4, LCG, g. Lite • Various components can be integrated into workflows • • • Sequential codes MPI codes Legacy code services (with the GEMLCA-specific version) • Workflows can be executed as parameter studies – Storage management – Generators – Collectors • Your code does not have to contain grid specific calls • Graphical interfaces for – grid application development – certificate management – application execution and monitoring • Support for collaborative work – Share workflow components – Share workflows • Built by standard portlet API customizable to specific needs 55

Learn once, use everywhere Develop once, execute anywhere Questions? www. portal. p-grade. hu pgportal@lpds. sztaki. hu 56