Grid Application Support by the P-GRADE Portal Gergely

Grid Application Support by the P-GRADE Portal Gergely Sipos sipos@sztaki. hu MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences www. lpds. sztaki. hu/pgportal@lpds. sztaki. hu CEOS WGISS Meeting, May 8 -12, 2006, Budapest 1

Outline • • • Motivation of our work P-GRADE Portal in a nutshell Grid application development Grid application execution and monitoring Support for collaborative work Conclusions CEOS WGISS Meeting, May 8 -12, 2006, Budapest 2

Layered view of Grid systems E-Scientists Application toolkits, portals Higher-level grid services: brokering, data replica, … Basic Grid services: security, job submission, info, … Graphical grid tools: P-GRADE Portal Command line grid tools Grid middleware: e. g. Globus, EGEE, … Resources to be shared CEOS WGISS Meeting, May 8 -12, 2006, Budapest 3

Motivation to the P-GRADE Portal • Fast evolution of Grid middleware technologies and tools: – GT 2, OGSA, GT 3 (OGSI), GT 4 (WSRF), LCG-2, g. Lite, … • Many production Grids are now freely available for e. Scientists – EGEE (LCG-2 g. Lite), UK NGS (GT 2), US Open Science Grid (GT 2 GT 4), Nordu. Grid (ARC), … • The same set of services are available everywhere, but implemented in different ways – Computation services, data services, security services, (brokers) Let’s provide a technology-neutral, intuitive graphical interface for the most common Grid middleware services! CEOS WGISS Meeting, May 8 -12, 2006, Budapest 4

P-GRADE Portal in a nutshell • General purpose, workflow-oriented computational Grid portal. Supports the development and execution of workflow-based Grid applications. • Based on standard portlet framework (Gridsphere) – Easy to expand with new portlets (e. g. application-specific portlets) – Easy to tailor to community needs • Grid services supported by the portal: Service Job execution File storage EGEE grids Globus grids Computing Element GRAM Storage Element Grid. FTP server Certificate management Information system Brokering My. Proxy BDII MDS-2 Workload Management System --- Job monitoring Mercury Workflow & job visualization PROVE The P-GRADE Portal hides middleware technologies and solves Grid interoperability problem at the workflow level CEOS WGISS Meeting, May 8 -12, 2006, Budapest 5

What is a P-GRADE Portal workflow? • a directed acyclic graph where – Nodes represent jobs (batch programs to be executed on a computing resource) – Ports represent input/output files the jobs expect/produce – Arcs represent file transfer operations • semantics of the workflow: – A job can be executed if all of its input files are available CEOS WGISS Meeting, May 8 -12, 2006, Budapest 6

Two levels of parallelism • The workflow concept of the P-GRADE Portal enables the efficient parallelization of complex problems • Semantics of the workflow enables two levels of parallelism: – Parallel execution inside a workflow node – Parallel execution among workflow nodes Multiple jobs run parallel CEOS WGISS Meeting, May 8 -12, 2006, Budapest Each job can be a parallel program 7

Ultra-short range weather forecast (Hungarian Meteorology Service) Forecasting dangerous weather situations (storms, fog, etc. ), crucial task in the protection of life and property 25 x 10 x 25 x 5 x Processed information: surface level measurements, highaltitude measurements, radar, satellite, lightning, results of previous computed models Requirements: • Execution time < 10 min • High resolution (1 km) CEOS WGISS Meeting, May 8 -12, 2006, Budapest 8

Multi-Grid portal: Demonstration @ EGEE User Forum Portal Server in London P-GRADE Portal server Portal Client in Geneva Exe Manchester Executable Leeds Service Invocation. West. Focus GT 4 Executable Budapest e bl cuta EGEE (CE) Poznan NGS GT 2 (UK) Grid (UK) Executable Job Repository (GEMLCA) CEOS WGISS Meeting, May 8 -12, 2006, Budapest Uo. W Brunel 9

The typical user scenario Part 1 - development phase Certificate servers SAVE WORKFLOW Portal server Grid services START EDITOR OPEN & EDIT or DEVELOP WORKFLOW CEOS WGISS Meeting, May 8 -12, 2006, Budapest 10

The typical user scenario Part 2 - execution phase Certificate servers TRANSFER FILES, SUBMIT JOBS DOWNLOAD PROXY CERTIFICATES VISUALIZE JOBS and WORKFLOW PROGRESS Portal server MONITOR JOBS Grid services SUBMIT WORKFLOW DOWNLOAD RESULTS CEOS WGISS Meeting, May 8 -12, 2006, Budapest 11

Opening the workflow editor The editor is a Java Webstart application download and installation is only a click! CEOS WGISS Meeting, May 8 -12, 2006, Budapest 12

Workflow Editor Define DAG of batch jobs: 1. Drag & drop components: jobs and ports 2. Connect ports by channels (no cycles, no loops) 3. Define job and port properties CEOS WGISS Meeting, May 8 -12, 2006, Budapest 13

Job property window Properties of a job: • Binary executable • (Type of executable) • (Number of processors) • Command line parameters • The resource to be used for the execution: • Grid • Broker / resource CEOS WGISS Meeting, May 8 -12, 2006, Budapest 14

Support for manual resource selection: information system browser The information system portlet queries EGEE and Globus information systems CEOS WGISS Meeting, May 8 -12, 2006, Budapest 15

Support for broker-based resource selection 1. Select a broker Grid/VO for the job 2. (Specify extra ranks & requirements for the job in Job Description Language) 3. The broker will find the best resource for your job! CEOS WGISS Meeting, May 8 -12, 2006, Budapest 16

Defining input/output data File type: input: required by the job output: produced by the job File location: local: my desktop remote: grid storage site File name: unique name of the file File storage type: Permanent: final result of the WF Volatile: only used for inter-job data transfer CEOS WGISS Meeting, May 8 -12, 2006, Budapest 17

The typical user scenario Execution phase – step 1: Certificate servers DOWNLOAD PROXY CERTIFICATES Portal server Grid services CEOS WGISS Meeting, May 8 -12, 2006, Budapest 18

Prerequisite of workflow execution: Download user certificates 1. Certificate server details: • Hostname • Port number • User name • Password 2. Certificate parameters: • Lifetime CEOS WGISS Meeting, May 8 -12, 2006, Budapest 19

Result of certificate download Multiple certificates for multiple Grids/VOs can be available on the portal server at the same time! SEE-GRID resources HUNGRID resources CEOS WGISS Meeting, May 8 -12, 2006, Budapest 20

Workflow Execution (observation by the workflow portlet) White/Red/Green color means the Meeting, May 8 -12, 2006, Budapest job is initial/running/finished state CEOS WGISS 21

Workflow Execution (observation by the workflow portlet) White/Red/Green color means the Meeting, May 8 -12, 2006, Budapest job is initial/running/finished state CEOS WGISS 22

Workflow Execution (observation by the workflow portlet) White/Red/Green color means the Meeting, May 8 -12, 2006, Budapest job is initial/running/finished state CEOS WGISS 23

Workflow Execution (observation by the workflow portlet) White/Red/Green color means the Meeting, May 8 -12, 2006, Budapest job is initial/running/finished state CEOS WGISS 24

Workflow Execution (observation by the workflow portlet) White/Red/Green color means the Meeting, May 8 -12, 2006, Budapest job is initialised/running/finished CEOS WGISS 25

On-Line application monitoring - The portal monitors and visualizes parallel jobs - The portal monitors and visualizes workflows: -Input data transfer -Job execution CEOS WGISS Meeting, May 8 -12, 2006, Budapest -Output data transfer 26

Downloading the results… - Download small files in a single ZIP file - Keep large files in the Grid for future analysis! CEOS WGISS Meeting, May 8 -12, 2006, Budapest 27

Collaborative work by the P-GRADE Portal • Export and import workflows and execution results from/into the portal • Share legacy applications as services with other portal users • Share successfully executed jobs as services with other portal users • Integrate other users’ legacy applications and jobs into your workflow CEOS WGISS Meeting, May 8 -12, 2006, Budapest 28

References • P-GRADE Portal service is available for – SEE-GRID infrastructure – HUNGRID VO of EGEE – Central European VO of EGEE – GILDA: Training VO of EGEE – UK National Grid Service • Under preparation for – US Open Science Grid, Economy-Grid, Swiss Bio. Grid, Bio. Info. Grid, Turkish Grid • P-GRADE portal can be installed for any public/private EGEE or Globus Grid/VO Please contact us! CEOS WGISS Meeting, May 8 -12, 2006, Budapest 29

Main conclusion: Easy-to-use, technology-neutral Grid portal for e-Scientists • The P-GRADE Portal hides differences of Grids • Globus X – LCG 2 – g. Lite Grid interoperability • Transparent switching between Grid technologies • Graphical tools for application development, execution and monitoring • Seq & parallel components can be integrated into large Grid applications • Your code does not have to contain grid specific calls • Support for collaborative research • Share workflows • Share legacy applications and jobs as application services • Built by standard portlet API • customizable to specific application areas and user groups (such as earth scientists…) CEOS WGISS Meeting, May 8 -12, 2006, Budapest 30

Further information… – Visit the www. lpds. sztaki. hu/pgportal site (manuals, slide shows, installation procedure, etc. ) – Visit or request a training event! (see list of events on P-GRADE Portal homepage) • Lectures, demos, hands-on tutorials, application development support – Get an account for a portal service: • • VOCE portal - SZTAKI SEEGRID portal – SZTAKI HUNGrid portal – SZTAKI NGS portal – University of Westminster – Install a portal for your community: If you are the administrator of a Grid/VO SZTAKI is pleased to help you install a portal for your community! CEOS WGISS Meeting, May 8 -12, 2006, Budapest 31

• P-GRADE Portal tutorial in Paris (free event!) 22 -23, June – lectures, hands-on, application support See details on P-GRADE Portal homepage! • Joint Regional CE EGEE and SEE-GRID Summer School on Grid Application Support 3 -8 July, Budapest, SZTAKI www. egee. hu/grid 06 Thank you! Questions? CEOS WGISS Meeting, May 8 -12, 2006, Budapest 32