Update on EU Data Grid progress and plans

Update on EU Data. Grid progress and plans for EGEE Fabrizio Gagliardi EU Data. Grid Project Leader Fabrizio. Gagliardi@cern. ch www. edg. org

Overview u Project Outline u Atlas task force u CMS stress test u Tutorials u Relationships u Future with other grid projects Directions (EGEE) u Summary LCG-PEB 21 January 2003 EDG/EGEE status 2

The Project u 9. 8 M Euros EU funding over 3 years u 90% for middleware and applications (HEP, Earth Obs. and Bio Med. ) u Three u Total n year phased developments & demos (2001 -2003) of 21 partners Research and Academic institutes as well as industrial companies u Related projects and activities: n Data. TAG (2002 -2003) n Cross. Grid (2002 -2004) n GRIDSTART (2002 -2004) n Grace (2002 -2004) LCG-PEB 21 January 2003 EDG/EGEE status 3

Data. GRID project priorities After initial middleware development and testbed deployment effort has been refocused on quality and stability u Quality Policy Statement published http: //eu-datagrid. web. cern. ch/eu-datagrid/WP 12/default. htm u List of priorities defined at the project retreat http: //documents. cern. ch/age? a 021130 u Followed-up at project conference http: //www. tomiexpress. hu/datagrid/ u Show-stoppers found by users on the application testbed are the highest priority u Incremental improvements to current release driven by the needs of the applications (HEPCAL) LCG-PEB 21 January 2003 EDG/EGEE status 4

ATLAS-EDG Task Force u ATLAS is eager to use Grid tools for the Data Challenges n n u ATLAS Data Challenges are already on the Grid (Nordu. Grid, USA) The DC 1/phase 2 (to start in October) is expected to be done using the Grid tools to a bigger extent ATLAS-EDG Task Force was put together in August with the aims: n u (by Oxana Smirnova) To assess the usability of the EDG testbed for the immediate production tasks To introduce the Grid awareness to the ATLAS collaboration The Task Force has representatives both from ATLAS and EDG: 40 members (!) on the mailing list, ca 10 of them working nearly full-time The initial task: to process 5 input partitions of the Dataset 2000 at the EDG Testbed + one non-EDG site (Karlsruhe); if this works, continue with other datasets LCG-PEB 21 January 2003 EDG/EGEE status 5

Achievements (by Oxana Smirnova): u A team of hard-working people across the Europe u ATLAS software (release 3. 2. 1) is packaged into relocatable RPMs, distributed and validated elsewhere u DC 1 production script is “gridified”, submission script is produced u User-friendly testbed status monitor deployed u 5 Dataset 2000 input files are replicated to 5 sites (2 @ each) u u u After fixing the “long jobs” problem, 50% of the planned challenge is performed (5 researchers × 10 jobs) – unfortunately, only CERN testbed was fully available With the rest of the testbed being fixed, jobs are getting scheduled and executed elsewhere Second test: 4 input files (ca 400 MB each) replicated to 4 sites; 250 jobs submitted, adjusted to run ca 4 hours each. The jobs were distributed across all the testbed by the Resource Broker LCG-PEB 21 January 2003 EDG/EGEE status 6

Summary u Advantages of the Grid: n n n u ( by Oxana Smirnova ) Possibility to execute tasks and move files over a distributed computing infrastructure by using one single personal certificate (no need to memorize dozens of passwords) Possibility do distribute the workload adequately and automatically, without logging in explicitly to each remote system Possibility to do worldwide production in a perfectly coordinated way, using identical software (RPMs), scripts and databases Where we are now: n Several Grid toolkits are on the market n EDG – probably the most elaborated, but still in development n n u This development goes way faster with the help of the users running real applications Common efforts of the ATLAS-EDG Task Force proved that it is possible to execute real tasks on the EDG Testbed already now Thanks all the members for the efforts so far, but there’s more to be done! LCG-PEB 21 January 2003 EDG/EGEE status 7

CMS/EDG stress test status Andrea Sciabà on behalf of CMS & EDG collaboration CCS general meeting December 3, 2002

Sites and resources Site CE No. of CPU SE Disk space (GB) CERN lxshare 0227 122 lxshare 0393 100 lxshare 0384 1000(=100 10) CNAF testbed 008 40 grid 007 g 1000 RAL gppce 05 12 gppse 05 360 (NIKHEF) tbn 09 20 tbn 03 35 Lyon ccgridli 03 120 (shared) ccgridli 07 200 cmsgrid 002 513(+513) 400 ccgridli 08 Legnaro cmsgrid 001 Total LCG-PEB 21 January 2003 48 762 3721 EDG/EGEE status 9

CMSIM events vs. time LCG-PEB 21 January 2003 EDG/EGEE status 10

Current issues u The biggest problems related to the Information System: n n n u Symptom: no resources are found Cause: instability of the MDS when it is overloaded Solution: submitting jobs at a lower rate improves the chances of success Symptom: the RB gets stuck (no job ever starts) Cause: investigating. . . Symptom: grid elements disappear from the II Cause: services on some machines stopped working Solution: restart the services Symptom: timeouts when copying the input sandbox Symptom: log file lost (“Stdout does not contain useful data”) Cause: several (no free files/inodes, broken connect. between CE & RB, …) Problems related to the replica manager: n Symptom: file registration in the RC fails from time to time LCG-PEB 21 January 2003 EDG/EGEE status 11

Current issues u u None of these problems is a show-stopper and they happen just in a fraction of the jobs! Fixes are already there for some of them (but not yet deployed) LCG-PEB 21 January 2003 EDG/EGEE status 12

Conclusions u u u 50000 events (FZ files) produced in ~ 2 days! The CMS Task Force has made impressive progress and the first results are promising. A few issues have been identified and solutions are being worked out/applied The entire task force shows a fruitful cooperation between CMS and EDG! LCG-PEB 21 January 2003 EDG/EGEE status 13

Tutorials The tutorials are aimed at users wishing to "gridify" their applications using EDG software and are organized over 2 full consecutive days. Approx. 100 people have followed the tutorial since August. http: //hep-proj-grid-tutorials. web. cern. ch/hep-proj-grid-tutorials/ October: DAY 2 DAY 1 3 & 4 – CERN u Data Management u Introduction to Grid computing 31 & Nov 1 - CERN and overview of the Data. Grid u LCFG, fabric mgmt & sw project December distribution & installation u Security 2 & 3 – Edinburgh u Applications and Use cases 5 & 6 - Italy u Testbed overview u Future Directions 9 & 10 – NIKHEF lunch u Job Submission 12 - Cracow u hands-on exercises: data mgmt lunch More sessions will be organised in u hands-on exercises: job submission the future LCG-PEB 21 January 2003 EDG/EGEE status 14

Related Grid projects Through links with sister projects, there is the potential for a truely global scientific applications grid LCG-PEB 21 January 2003 EDG/EGEE status 15

Interaction with sister projects u Cross. Grid u Nordu. Grid n Using the same security certs. n Testbed sites install EDG software n Involved in EDG architecture work s n n Extending it for needs of intensive interactive applications s Participating in the EDG testing activities s Representatives in each projects architecture & management groups s s Good ideas for gatekeeper and MDS configuration Helped develop GDMP and GSI extensions for Replica Catalog Involved in Glue schema work Security policy n u Data. TAG n n n (EDT) EDT is deploying EDG sw to investigate inter-operability with US projects (i. VDGL, Gri. Phy. N, PPDG) Results feedback into EDG software releases s (e. g. GLUE compatible information providers/consumers) Mware testing Working in WP 8 (HEP applications) u i. VDGL/Gri. Phy. N/PPDG n n US members in EDG architecture group Looking for common packaging and toolkit usage solutions No strict boundaries with a large cross-fertilization of ideas, software and people Data. GRID is learning from the experiences in these projects LCG-PEB 21 January 2003 EDG/EGEE status 16

Plans for the future u Further development in 2003 n Further iterative improvements to middleware driven by LCG and users needs n More extensive testbeds providing more computing resources n Prepare EDG software for future migration to Open Grid Services Architecture u Interaction with LCG n LCG intends to make use of the Data. GRID middleware n LCG is contributing to Data. GRID s Testbed support and infrastructure Get access to more computing resources in HEP computing centres Testing and verification Reinforce the testing group and maintain a certification testbed Fabric management and middleware development n s u New n EU project (EGEE) Make plans to preserve current major asset of the project: probably the largest Grid development team in the world s Eo. I for FP 6 ( www. cern. ch/egee-ei ) LCG-PEB 21 January 2003 EDG/EGEE status 17

EGEE vision u Goal n u Enabling Grids for E-science in Europe create a general European Grid production quality infrastructure on top of present and future EU RN infrastructure Build on n EU and EU member states major investment in Grid Technology n u Several pioneering prototype results n u Applications Largest Grid development team in the world Goal can be achieved for about € 100 m/4 years on top of the national and regional initiatives Approach n n Leverage current and planned national and regional Grid programmes (e. g. LCG) EGEE Work closely with relevant industrial Grid developers, NRNs and US Geant network LCG-PEB 21 January 2003 EDG/EGEE status 18

Work done so far u Eo. I for FP 6: www. cern. ch/egee-ei submitted on June 7 th u Several follow up meetings u u An editorial board an Interim Task Force established to prepare a position paper and a presentation for a EU Grid workshop in Brussels on October 3 -4 Both bodies extended to follow-up with the EU (IST 02, ER 02, individual contacts) LCG-PEB 21 January 2003 EDG/EGEE status 19

GÉANT and GRIDs: The model GRIDs use GÉANT infrastructure Application areas GRIDs platforms International dimension LCG-PEB 21 January 2003 R&D on GRIDs GÉANT profits from technological innovation GRIDs empowered GÉANT network EDG/EGEE status 20

Instruments Research Infrastructures IST Programme 665 M Euro GÉANT, GRIDs, other ICT-RI 100 + 200 M Euro Structuring the ERA Programme 3. 825 M Euro • Integrated Projects • Integrated Infrastructure Initiatives • Networks of Excellence • Coordinated actions • Specific Targeted Projects • Support actions • Coordinated actions • More info on: http: //www. cordis. lu/ist/fp 6/activities. htm • Support actions Separate calls for proposals! LCG-PEB 21 January 2003 EDG/EGEE status 21 K. Baxevanidis EU 2. 655 M Euro

Communication Network Development Call u u 45 -47 Million Euros available in the first EU call (Dec 17 th, 2002) Hard to get the whole budget, we will need to share with one, two, more projects and a lot of competition to be expected (1200 Eo. Is received in this area!) u Focus on support and integration of already established Grid infrastructures u Build a Grid production layer on top of the EU RN infrastructure u No major funds for H/W, CS research or application development (in a first approximation) LCG-PEB 21 January 2003 EDG/EGEE status 22

Integrated Infrastructure Initiative (I 3) u Three lines of funding supported (with possible budget breakdown): n Networking activities (nothing to do with networks…): s n Specific service activities: s n This is the overhead: management, coordination, dissemination and outreach (7 -10% of the total funding) Provision and procurement of Grid services (60% of total funding) Joint research activity s s LCG-PEB 21 January 2003 Engineering development to improve the services provided by the Grid infrastructure (20% of total funding) Application support and focused R&D (10% of total funding) EDG/EGEE status 23

Networking activities u Coordination and management of the participating Grid infrastructures n u Management structure to be defined Dissemination, training and outreach n n u Leverage EDG and other project tutorials Proposal from Terena received User clubs, industry forum etc. LCG-PEB 21 January 2003 EDG/EGEE status 24

Specific service activities u u Integration of major national and international Grid infrastructures Two tier structure: n n n 1 st Tier: Major Grid centres (6 -8). Must satisfy minimum level of Grid resources and staffing 2 nd Tier: POPs in all other Geant supported countries EU resources for doubling the 1 st tier centres Grid support staff, a central operation centre and a distributed call and support centre Interface to Geant follow-on project Mostly staff and overhead (computer fabrics and storage provided by the partners) LCG-PEB 21 January 2003 EDG/EGEE status 25

Joint research activity u u u Focus on hardening and re-engineering of Middleware Leverage current EU Grid projects and international Grid technology developers (large and established M/W development community) 8 -10 WPs with critical mass in a single geographical center, dedicated WP managers hired by the project and reporting to the project technical management (possible international and industrial participation) u Quality assurance group, integration, certification and distribution group with industrial quality u International senior advisory group for project review, long term technology development and direction LCG-PEB 21 January 2003 EDG/EGEE status 26

Additional activities u Application support: n n u high level interface and portals user requirements (a la HEPCAL) CS focused activity: n Long term CS issues for production quality Grids LCG-PEB 21 January 2003 EDG/EGEE status 27

Distribution of responsibilities Motivation: provide transparent, effective process for proposal preparation EGEE Executive Committee: n n n Responsible for defining Work Packages and setting up Task Forces to deliver technical content for proposal. Max ~10 persons for effective process Should represent stakeholders with major, proven computer and human resources to contribute to EGEE US has observer status (Ian Foster) EGEE technical advisory board: n n Advise the Executive Committee on the overall architecture and specific technical issues US participation confirmed LCG-PEB 21 January 2003 EDG/EGEE status 28

Distribution of responsibilities EGEE Editorial board: n Responsible for gathering input from taskforces, overall editing of proposal, filling out administrative forms and maintaining timeline EGEE National Partners board: n n Responsible for coordination and communicating with interested parties on national/regional level. Ideally one person per country/region Consulted by Executive Committee during preparation of proposal, to ensure adequate transparency – must be seen as impartial EGEE interest group: n All institutes, companies, organisations interested to remain informed about progress of EGEE proposal. Includes potential subcontractors for different workpackages LCG-PEB 21 January 2003 EDG/EGEE status 29

EGEE proposal timeline Tentative Schedule (continued) • EU call out on Dec 17 th • Draft 1: overall project structure end of February 2003 • Draft 2: with detailed workpackages end of March 2003 • Final proposal including admin and management end of April 2003 • Submission by May 6 th 2003 • First feedback from EU in June-July • Contract negotiation late summer, fall ’ 03 • Contract signature by the end of ’ 03 • Start of project Q 1 -Q 2 ‘ 04 LCG-PEB 21 January 2003 EDG/EGEE status 30

Summary u ATLAS/Data. GRID EDG task force has been a successful experience for u CMS stress test still on-going is a major advance on production quality performance in view of next EU EDG review on February 4 -5 u Deployment of a very large production Grid testbed being explored with the EU (EGEE) u This needs to be done in close collaboration with LCG and the US Grid developers for the maximum benefit of the LCG experiments and potential application to other international scientific communities (also good for long term future of HEP…) LCG-PEB 21 January 2003 EDG/EGEE status 31