EGI-In SPIRE Getting started with the European Grid

EGI-In. SPIRE Getting started with the European Grid Infrastructure Gergely Sipos Senior User Community Support Officer EGI. eu gergely. sipos@egi. eu 3/19/2018 EGI-In. SPIRE RI-261323 1 www. egi. eu

Outline • • • Introduction to EGI, main services, getting access Demo (David Groep): Finding your CA, requesting a certificate Lifecycle of a grid job, low level access: command line tools Demo (Antonio Calanducci): Submitting a job from command line Complex jobs, development of complex applications Demo: Developing and managing complex application with P-GRADE portal Providing grid applications for end users Demo (Antonio Calanducci): g. Library application Services for users, Q&A EGI-In. SPIRE RI-261323 2 www. egi. eu

What is a Grid? A grid consists of distributed resources controlled by separate organisations that be systematically used securely by users external to that organisation • Resources can include: – – Commodity or HPC clusters Disk or tape storage Instruments Data Archives or Digital Libraries EGI-In. SPIRE RI-261323 3 www. egi. eu

European Grid Infrastructure • A sustainable, production grid infrastructure for e-Science applications, communities • European Data Grid (EDG) – Explore concepts in a testbed • Enabling Grid for E-scienc. E (EGEE) – Moving from prototype to production • European Grid Infrastructure (EGI) – Routine usage of a sustainable e-infrastructure EGI-In. SPIRE RI-261323 4 www. egi. eu

EGI Collaboration Research Community N G I E I R O Research Community N G I Research Community E I R O N G I EGI. eu EGI-In. SPIRE RI-261323 NGI: National Grid Infrastructure/Initiative EIRO: European International Research Organisation 5 www. egi. eu

The EGI-In. SPIRE Project Integrated Sustainable Pan-European Infrastructure for Researchers in Europe • A 4 year project – Project cost € 72 M – Effort: 9261 PMs – 51 partners Funded Un-Funded • EGI. eu, 40 NGIs, 2 EIROs 8 institutes from Asia Pacific • • NA 1: Project & Consortium Management NA 2: External Relations NA 3: User Community Coordination SA 1: Reliable Operation of the infrastructure SA 2: Provisioning the Software Infrastructure SA 3: Support for Heavy User Communities JRA 1: Support for Operational Tools EGI-In. SPIRE RI-261323 6 www. egi. eu

The European Grid Infrastructure What is happening now? ~200, 000 CPUs (cores) ~30 Pb disk ~40 Pb tape ~24 million jobs/month 336 sites 59 countries ~70 active VOs Grid Real Time Monitor: http: //gridportal. hep. ph. ic. ac. uk/rtm EGI-In. SPIRE RI-261323 www. egi. eu 8

Infrastructure, middleware, Virtual Organisations Infrastructure elements (sites) are provided by National Grid Initiatives (NGIs) NGI INTERNET NGI NGI EGI-In. SPIRE RI-261323 9 www. egi. eu

Infrastructure, middleware, Virtual Organisations Infrastructure elements (sites) are provided by National Grid Initiatives (NGIs) Middleware runs on each EGI site to provide – Data services – Computation services – Security service – High level services NGI INTERNET NGI NGI EGI-In. SPIRE RI-261323 10 www. egi. eu

Infrastructure, middleware, Virtual Organisations Infrastructure elements (sites) are provided by National Grid Initiatives (NGIs) Middleware runs on each EGI site to provide – Data services – Computation services – Security service – High level services EGI resources and users form Virtual Organisations (VO): basis for collaboration EGI-In. SPIRE RI-261323 NGI INTERNET NGI NGI 11 www. egi. eu

Infrastructure, middleware, Virtual Organisations Infrastructure elements (sites) are provided by National Grid Initiatives (NGIs) Middleware runs on each EGI site to provide – Data services – Computation services – Security service – High level services EGI resources and users form Virtual Organisations (VO): basis for collaboration EGI-In. SPIRE RI-261323 NGI INTERNET NGI NGI 12 www. egi. eu

Example: Biomed Virtual Organization https: //twiki. cern. ch/twiki/bin/view/EGEE/Life. Sciences Sites: • 36, mostly from Europe and Asia • • 65, 584 CPUs ~741 TB storage Portal services are also available Members: • Anybody working with biomedical applications • No geographical restriction • Currently ~250 members EGI-In. SPIRE RI-261323 13 www. egi. eu

EGI middleware • Provided by an external project: – European Middleware Initiative (EMI) • Three different stacks - with similar (not the same!) functionality and with some common services: – ARC: www. nordugrid. org/middleware – g. Lite: www. glite. org – UNICORE: www. unicore. eu • VOs can decide which middleware to install – Linux is used on all the grid servers – EGI clients can run Windows too! EGI-In. SPIRE RI-261323 14 www. egi. eu

Getting access to an EGI VO • Users (and machines) are identified by grid certificates. Certificates issued by Certification Authorities (CA) • You CA Join VO: Once VO Membership Service (VOMS) keeps list of people who are registered to a VO • Obtain certificate: Once Renew certificate: Annually Users’ steplist – Obtain certificate from your national CA: http: //www. igtf. net – Register at the VO • List of VOs: http: //cic. gridops. org/ – VO manager authorizes You • VOMS DB updated • Your identity is replicated onto VO resource within 24 hours – Use the services of the grid EGI-In. SPIRE RI-261323 VO manager Register VO Management Service Replicating DB: once a day VO database Grid sites VIRTUAL ORGANISATION 15 www. egi. eu

Find your national CA! http: //igtf. net CAs in Asia. Pacific region EGI-In. SPIRE RI-261323 CAs in Europe and Africa CAs in America 16 www. egi. eu

Obtaining a grid certificate User generates public/private key pair in browser or in files. Private Key encrypted on local disk: passphrase EGI-In. SPIRE RI-261323 17 www. egi. eu

Obtaining a grid certificate User generates public/private key pair in browser or in files. Cert Request Public Key User sends public key to CA and shows proof of identity. ID Private Key encrypted on local disk: passphrase EGI-In. SPIRE RI-261323 18 www. egi. eu

Obtaining a grid certificate User generates public/private key pair in browser or in files. Cert Request CA signature links identity and public key in certificate. CA informs user. Public Key User sends public key to CA and shows proof of identity. Cert ID Private Key encrypted on local disk: passphrase EGI-In. SPIRE RI-261323 19 www. egi. eu

EGI-In. SPIRE Demo Obtaining a grid user certificate EGI-In. SPIRE RI-261323 20 www. egi. eu

Joining a VO: Example: VOCE – VO of Central Europe Certificate must be stored in the browser to register to VOCE EGI-In. SPIRE RI-261323 21 www. egi. eu

EGI User Community Organisation Virtual Research Community Grid authorization VO Research communities VO Virtual Research Community VO User User EGI-In. SPIRE RI-261323 User • • • High Energy Physics (HEP) Life Sciences (LS) Earth Sciences (ES) Grid Observatory (GO) Computational Chemistry (CC) • Astronomy & Astrophysics (AA) • Fusion (F) Members User Heavy User Communities (~Largest VRCs): 13, 381 users in >140 Registered VOs 22 www. egi. eu

EGI-In. SPIRE Using EGI Command line tools EGI-In. SPIRE RI-261323 24 www. egi. eu

Typical use case User environment Create job definition Submit job (batch executable + small inputs)Broker Information System service query create Proxy (~login) Retrieve status & (small) output files Retrieve output Job status publish state Submit job Logging Site X of YOUR VO Computing service VO Management Service (VOMS) EGI-In. SPIRE RI-261323 Job status Logging and bookkeeping service process Storage Service Read/write files 25 www. egi. eu

Typical use case User environment Create job definition Submit job (batch executable + small inputs)Broker Share applications create Proxy (~login) Information System service query Share services Retrieve status & (small) output files Retrieve output Job status publish state Submit job Logging Site X of YOUR VO Storage Service Share resources Computing service VO Management Service (VOMS) EGI-In. SPIRE RI-261323 Job status Logging and bookkeeping service process Read/write files Share data 26 www. egi. eu

A job definition in g. Lite (Job Description Language - JDL) $ cat hostname. jdl Type = "Job"; Job. Type = "Normal"; Executable = "/bin/hostname"; Std. Output = "hostname. out"; Std. Error = "hostname. err"; Output. Sandbox = {"hostname. err", "hostname. out"}; Arguments = "-f"; • Generated by a high level environment OR • Written by hand EGI-In. SPIRE RI-261323 28 www. egi. eu

Job management commands in g. Lite WMS version LCG-2 WMS glite-wms-job-delegate-proxy -d deleg. ID Delegate proxy Submit Status Logging Output Cancel Compatible resources g. Lite WMS via WMProxy g. Lite 3. 1+ edg-job-submit glite-wms-job-submit [-o joblist]jdlfile [-d deleg. ID] [-a] [-o joblist] jdlfile edg-job-status glite-wms-job-status [-v verbosity] [-i joblist] job. IDs edg-job-get-logging-info glite-wms-job-logging-info [-v verbosity] [-i joblist] job. IDs edg-job-get-output glite-wms-job-output [-dir outdir] [-i joblist] job. IDs edg-job-cancel glite-wms-job-cancel [-i joblist] job. ID edg-job-list-match glite-wms-job-list-match jdlfile [-d deleg. ID] [-a] jdlfile g. Lite User Guide: http: //glite. web. cern. ch/glite/documentation EGI-In. SPIRE RI-261323 29 www. egi. eu 29

EGI-In. SPIRE Demo Job management with g. Lite command line tools EGI-In. SPIRE RI-261323 30 www. egi. eu

EGI-In. SPIRE Complex jobs EGI-In. SPIRE RI-261323 31 www. egi. eu

Realistic use cases • Generating many jobs with similar parameters (hundreds, thousands, . . . ) • Managing many jobs • Dependencies among jobs (often data dependency) • Working with large number of files (identifying, locating, replicating files) EGI-In. SPIRE RI-261323 32 www. egi. eu

Support for complex jobs in g. Lite • Complex job types in g. Lite: – Collection: many independent jobs – Directed Acyclic Graph (DAG): jobs with data dependencies – Parametric jobs: similar jobs PARAMETRIC. JDL [ Type = “Parametric"; Parameter. Start = 0; Parameter. Step =2; Parameters= 6; _PARAM_ runs from 0 to 10 Arguments = “inputfigure_PARAM_. jpg"; Std. Output = “transformed_PARAM_. jpg"; Output. Sandbox = {" transformed_PARAM_. jpg ", …}; …] EGI-In. SPIRE RI-261323 33 www. egi. eu

High level tools for EGI users Applications Toolkits, frameworks, portals Command line & APIs Higher-level g. Lite services (broker, info system, . . . ) Basic middleware services: computing, storage, security EGI-In. SPIRE RI-261323 • Most VOs also have toolkits, frameworks, portals on top of the middleware • Some tools are domain specific, some are not • RESPECT Programme: – Recommended External Software Packages for EGI Communi. Ties – Identify third-party software that works well with EGI middleware • Expand the functionality of the grid infrastructure, • Reduce duplicated development • Speed the porting of new applications to the grid 34 www. egi. eu

RESPECT • Identify third-party software that works well with EGI. – http: //appdb. egi. eu/ Applications & tools • Simplified Access – P-GRADE, Ganga, Migrating Desktop, g-Eclipse, i 2 glogin, Virtual Control Room, Parallel ROOT • Workload Management (~brokering) – Grid. Way Metascheduler, DIANE • Monitoring – Dashboard • Middleware extensions – GRel. C, Instrument Element, Sto. RM, Distributed Grid Storage, ESIP EGI-In. SPIRE RI-261323 35 www. egi. eu

EGI-In. SPIRE Demo Developing grid applications with P-GRADE Portal EGI-In. SPIRE RI-261323 www. egi. eu

Typical user scenario Job compilation phase UPLOAD JOB SOURCE(S) Portal server Client COMPILE – EDIT Grid services DOWNLOAD BINARI(ES) 37

Typical user scenario Workflow development phase SAVE WORKFLOW Portal server Grid services Client IMPORT WORKFLOW START EDITOR OPEN & EDIT WORKFLOW ADD BINARIES DSpace WF repository 38

Typical user scenarios Workflow execution phase My. Proxy Certificate servers TRANSFER FILES, SUBMIT JOBS DOWNLOAD PROXY CERTIFICATES VISUALIZE JOBS and WORKFLOW PROGRESS Portal server MONITOR JOBS Grid services Client DOWNLOAD (SMALL) RESULTS 39

EGI-In. SPIRE Applications for end users EGI-In. SPIRE RI-261323 41 www. egi. eu

User community support User support is key to achieve sustainability • Sharing applications with the community – EGI Application Database: http: //appdb. egi. eu/ • Transparent integration of end user applications with the grid – Make EGI usable for non-IT scientist – Customise interfaces to community/scientific needs • Approaches – Customise existing portals (science gateways) to new grid applications (e. g. g. USE) – Develop grid-aware end user environments (e. g. g. Library) EGI-In. SPIRE RI-261323 www. egi. eu

EGI Application Database http: //appdb. egi. eu/ EGI-In. SPIRE RI-261323 www. egi. eu

EGI-In. SPIRE Demo g. Library application: Preserving Federico De Roberto’s Cultural Heritage EGI-In. SPIRE RI-261323 www. egi. eu

Federico De Roberto cultural heritage • De Roberto, an Italian writer of the XIX/XX century, born in Naples, but spending his life in Catania • has left to the humanistic community numerous works • Those are made up of valuable and hard-to-manage pieces: manuscripts, typescripts, drafts with handwritten corrections, magazines, cuts, sketches, photos, etc. . 45 EGI-In. SPIRE RI-261323 www. egi. eu

Acquisition stage • Digitalization of manuscripts, typescripts, printed works – TIFF Files, one per page, 600 dpi, about 100 MB for A 3 • High resolution scans for in-depth examination – PDF, one per work, 300 dpi, varying file sizes 40 -400 MB • Overall examination of works – 8000 sheets/scans, 3 Terabyte of disk space – Different physical formats, A 3/A 4/custom size • Embedded Metadata – TIFF with embedded metadata to provide scan physical features and information about the content • Image. Width, Image. Height, XResolution, File. Size, Creation. Date, Modify. Date • Description, Keywords, Caption. Writer, Title, Author, Copyright Status, Copyright Notice – Added with Photoshop after the digitalization phase (Adobe XMP format) 46 EGI-In. SPIRE RI-261323 46 www. egi. eu

g. Library features • g. Library is the INFN/COMETA platform that provides a simple yet powerful system to store, organize, search and retrieve “digital assets” in repositories built on grid infrastructures – hide the underlying technical details to the users • What we mean by “digital asset”: – data + metadata 47 EGI-In. SPIRE RI-261323 www. egi. eu

De Roberto DR on Grid with g. Library • Goals: – store the 8000 scans of De Roberto Heritage ----> Data Grid Storage Elements – enable an ubiquitous and 24/24 h access to scientists ---> Web Application – document organization for a quick search ---> Metadata Services – long-term digital preservation of data ---> redundancy through Replicas of files on several Storage Elements – simple and easy-to-use system for searches, organization, upload and download of digitalized documents on the Grid -----> 48 EGI-In. SPIRE RI-261323 www. egi. eu

g. Library architecture 49 EGI-In. SPIRE RI-261323 www. egi. eu

In collaboration with Dr. Pistagna (Uni. CT) • Mobile interface to access Grid Digital Libraries: – i. Phone, i. Pod Touch, i. Pad • Advantages: – provides an extremely intuitive and touch-based user interface to grid storages and metadata even to not-expert users – (to do) automatic selection of the closest replica, according to the user physical location retrieved by the integrated GPS – offline access to the assets already downloaded 50 EGI-In. SPIRE RI-261323 www. egi. eu

Reference • Contacts: – antonio. calanducci@ct. infn. it, glibrary@ct. infn. it • De Roberto repository prototype: – https: //glibrary. ct. infn. it/deroberto/ • Videos: – http: //www. youtube. com/watch? v=VNN 3 Onpm. UUU – http: //www. youtube. com/watch? v=Ih. FFj. HD 8 Is. I • Scientific publications: – A. Calanducci, R. Barbera, J. Sevilla, A. De Filippo, M. Saso, S. Iannizzotto, F. De Mattia, F. Vicinanza. “Data Grids for Conservation of Cultural Inheritance”, 1 st International Workshop on Data Grids for e-Science (Da. Gre. S 09) at ACM International Conference on Computing Frontiers, May 18 -20, 2009 (http: //www. computingfrontiers. org/2009/) • https: //glibrary. ct. infn. it/m/Da. GRes-editor. pdf – A. Calanducci, C. Cherubino, L. N. Ciuffo, D. Scardaci, “A Digital Library Management System for the Grid”, Fourth International Workshop on Emerging Technologies for Next-generation GRID (ETNGRID 2007) at 16 th IEEE International Workshops on Enabling Technologies: Infrastructures for Collaborative Enterprises (WETICE-2007), GET/INT Paris, France, June 18 -20, 2007 (http: //etngrid. diit. unict. it/2007/index. html). • https: //glibrary. ct. infn. it/glibrary/downloads/g. Library_paper_v 2. pdf 51 EGI-In. SPIRE RI-261323 51 www. egi. eu

EGI-In. SPIRE Services for users EGI-In. SPIRE RI-261323 52 www. egi. eu

User support in EGI International User Communities EGI. eu User Community Support Team Local user communities NGI User Support Team Local user communities National user communities NGI User Support Team Local user communities NGI User Support Team . . . NGI User Support Team Technical services for EGI User Support: application database, training database, etc. EGI-In. SPIRE RI-261323 User Support services: • Identification of potential users • Consultancy • Training • Porting scientific applications • Providing access to scientific applications • VO setup and support • Development of new software services (portals, toolkits, middleware extensions, etc. ) • Collecting feedback • Documentation • Helpdesk • Community integration 53 www. egi. eu

Technical services for EGI users and NGI User Support Teams • Training – List of training events: http: //training. egi. eu/index. cfm – Repository of talks, exercises: http: //library. nesc. ed. ac. uk: 8080/ • Application Support – Application database: http: //appdb. egi. eu • Virtual Organization Support – Operations portal: http: //cic. gridops. org/ • Direct User Support – Global Grid User Support: www. ggus. org • Central entry point: www. egi. eu • National entry points operated by NGIs – NGI User Support Team presentations: Friday, 9: 00 -10: 30, Room Yakult EGI-In. SPIRE RI-261323 54 www. egi. eu

EGI-In. SPIRE Thank you! EGI-In. SPIRE RI-261323 www. egi. eu