Enabling Grids for E-scienc E EGEE Status

Enabling Grids for E-scienc. E EGEE – Status and Future of the World’s Largest Grid Infrastructure Dieter Kranzlmüller Deputy Project Director EGEE CERN, Switzerland www. eu-egee. org INFSO-RI-508833

Cracow Grid Workshop Series Enabling Grids for E-scienc. E INFSO-RI-508833

Presentation overview Enabling Grids for E-scienc. E • EGEE operates a Production Infrastructure: – Operations – Middleware – Applications • EGEE as Partner • The Next Phase: EGEE-II INFSO-RI-508833

The EGEE project Enabling Grids for E-scienc. E • Objectives – consistent, robust and secure service grid infrastructure for many applications – improving and maintaining the middleware – attracting new resources and users from industry as well as science • Structure – 71 leading institutions in 27 countries, federated in regional Grids – leveraging national and regional grid activities worldwide – funded by the EU with ~32 M Euros for first 2 years starting 1 st April 2004 INFSO-RI-508833

Achievements Enabling Grids for E-scienc. E • Infrastructure >170 sites >15 000 CPUs >5 PB storage >10 000 concurrent jobs >60 Virtual Organisations INFSO-RI-508833 ng ceedi ex !! ready tions! e al ta We’r expec t ymen deplo

EGEE services Enabling Grids for E-scienc. E • Production service – Based on the LCG-2 service – With new resource centres and new applications encouraged to participate – Stable, well-supported infrastructure, running only well-tested and reliable middleware • Pre-production service – Run in parallel with the production service (restricted nr of sites) – First deployment of new versions of the middleware – Applications test-bed • GILDA testbed – https: //gilda. ct. infn. it/testbed. html – Complete suite of Grid elements and applications § Testbed, CA, VO, monitoring – Everyone can register and use GILDA for training and testing INFSO-RI-508833 15 sites on 3 continents (all of them GÉANT sites)

Production Service Enabling Grids for E-scienc. E Scale of the production service – ~16 K CPUs/170 sites – Other national & regional grids: ~60 sites, ~6000 processors greatly exceeds no of sites planned for the end of EGEE • • 100 000 No of jobs run • 10 000 100 Interoperability demonstrated with OSG (ongoing work with ARC) > 2. 2 million jobs Jan-Oct 2005 – Daily averages, sustained over a month 2200 – 10 100 ~6 M k. SI 2 K. cpu. hours INFSO-RI-508833 ~700 cpu years Month

Achievements Enabling Grids for E-scienc. E • Infrastructure >170 sites >15 000 CPUs >5 PB storage >10 000 concurrent jobs >60 Virtual Organisations • Middleware – Now at g. Lite release 1. 4 § Focus on basic services, easy installation and management § Industry friendly open source license INFSO-RI-508833

EGEE Middleware Enabling Grids for E-scienc. E www. g. Lite. org • First release of g. Lite end of March 2005 – Now at g. Lite v 1. 4 • Guiding principles – Lightweight services – Interoperability & Co-existence with deployed infrastructures – Robust: Performance & Fault Tolerance – Portable – Service oriented architecture – Site autonomy – Open source license • Development / Integration / Testing – Workload Management – Information Systems – Security – Data Management LCG-1 LCG-2 g. Lite-1 g. Lite-2 Globus 2 based Web services based g. Lite offers a complete data management solution in a distributed environment building on existing technology INFSO-RI-508833

Architecture & Design Enabling Grids for E-scienc. E • Design team includes – Representatives from middleware providers (Ali. En, Condor, EDG, Globus, …) – Colleagues from the Operations activity – Partners from related projects (e. g. OSG) • g. Lite development takes into account input and experiences from applications, operations, related projects – Effective exchange of ideas, requirements, solutions and technologies – Coordinated development of new capabilities – Open communication channels – Joint deployment and testing of middleware – Early detection of differences and disagreements g. Lite is not “just” a software stack, it is a “new” framework for international collaborative middleware development INFSO-RI-508833

Software stack and origin of services in release 1 (simplified) Enabling Grids for E-scienc. E • Computing Element – Gatekeeper (Globus) – Condor-C (Condor) – CE Monitor (EGEE) – Local batch system (PBS, LSF, Condor) • Workload Management – WMS (EDG) – Logging and bookkeeping (EDG) – Condor-C (Condor) • Information and Monitoring – R-GMA (EDG) • Storage Element – g. Lite-I/O (Ali. En) – Reliable File Transfer (EGEE) – Grid. FTP (Globus) – SRM: Castor (CERN), d. Cache (FNAL, DESY), other SRMs • Catalog – File/Replica & Metadata Catalogs (EGEE) • Security – GSI (Globus) – VOMS (Data. TAG/EDG) – Authentication for C and Java based (web) services (EDG) Now doing rigorous scalability and performance tests on pre-production service INFSO-RI-508833

Code & Usage Enabling Grids for E-scienc. E • g. Lite core (supported by EGEE) – ~200 packages – 23 logical deployment modules flexible deployment – 700. 000 LOCs § mostly in Java, C, and C++ • g. Lite releases have been produced – Tested, Documented, with Installation and Release notes – Subsystems used on § Service Challenges § Pre-Production Services § Production Service – And by other communities (e. g. DILIGENT) INFSO-RI-508833

The Full Picture Enabling Grids for E-scienc. E Applications High Level Grid Middleware etc glogin for interactivity Basic Grid Middleware Grid sites connected by Research Networks INFSO-RI-508833

Achievements Enabling Grids for E-scienc. E • Infrastructure >170 sites >15 000 CPUs >5 PB storage >10 000 concurrent jobs >60 Virtual Organisations Add itio nal 4 d 8 ap om ain s ar plicat ion e in s fr eva • Middleware om lua tion – Now at g. Lite release 1. 4 sta ge § Focus on basic services, easy installation and management § Industry friendly open source license • Applications: >20 applications from 7 domains – High Energy Physics – Astronomy – Geo-Physics – Biomedicine – Financial Simulation – Earth Sciences – Computational Chemistry INFSO-RI-508833

Applications Example: WISDOM Enabling Grids for E-scienc. E • Grid-enabled drug discovery process for neglected diseases – In silico docking § compute probability that potential drugs dock with target protein – To speed up and reduce cost required to develop new drugs • WISDOM (World-wide In Silico Docking On Malaria) – First biomedical data challenge – 46 million ligands docked in 6 weeks § Target proteins from malaria parasite § Molecular docking applications: Autodock and Flex. X § ~1 million virtual ligands selected – 1 TB of data produced – 1000 computers in 15 countries § Equivalent to 80 CPU years Never done for a neglected disease Never done on a large scale production infrastructure INFSO-RI-508833

Applications Example: MAGIC Enabling Grids for E-scienc. E • Major Atmospheric Gamma Imaging Cherenkov telescope (MAGIC) – Origin of VHE -rays (30 Ge. V – Te. V) § § Active Galactic Nuclei (AGN) Supernova Remnants Unidentified EGRET sources Gamma Ray Bursts – Huge hadronic background MC simulations § to simulate the background of one night, 70 CPUs (P 4 2 GHz) need to run 19200 days – Observation data are big too! • MAGIC Grid – Use three national Grid centres as backbone – All are members of EGEE • MAGIC telescope II will start 2007 (scalability) Towards a virtual observatory for VHE -rays INFSO-RI-508833

EGEE User Forum Enabling Grids for E-scienc. E • 3 -day workshop – Presentations by users from different thematic areas – EGEE presentations • Goals – Understand expectations, status and evolution across EGEE and related projects – Promote cross-application fertilisation • 1 -3 March 2006 • CERN, Switzerland • http: //egee-intranet. web. cern. ch/egee-intranet/User-Forum/index. html INFSO-RI-508833

User information & support Enabling Grids for E-scienc. E • More than 150 training events across many countries – >1200 people trained induction; application developer; advanced; retreats – Material archive online with ~250 presentations • Public and technical websites as well as dissemination material constantly evolving to expand information and keep it up to date • 4 conferences organized ~ 300 @ Cork ~ 400 @ Den Haag ~ 450 @ Athens ~ 460 @ Pisa INFSO-RI-508833

EGEE and Industry Enabling Grids for E-scienc. E • Industry as – partner – to increase know-how on Grid technologies – user – for R&D applications – provider – of established Grid services, such as call centres, support centres and computing resource provider centres • Industry Forum – Raise awareness of the project among industries – Encourage businesses to participate ability to “experience” EGEE Grid in early stages INFSO-RI-508833

Why work with EGEE? Enabling Grids for E-scienc. E • A global Grid infrastructure helps to provide easier access for – Small research groups – Scientists from many different fields – Remote and still developing countries • …to new technologies – Produce and store massive amounts of data – Transparent access to millions of files across different administrative domains – Low cost access to large computing resources § Mobilise quickly large amounts of CPU on short notice – High-end facilities • …and helps to find new ways to collaborate – Develop applications using distributed complex workflows – Eases distributed collaborations – New modes of community building – Easier access to higher education INFSO-RI-508833

EGEE as partner Enabling Grids for E-scienc. E • Ongoing collaborations – with non EU partners in EGEE: US, Israel, Russia, Korea, Taiwan… – with other European infrastructure projects: § GÉANT § DEISA § SEE-GRID – with other European grid projects § DILIGENT § GRIDCC – with non-European projects: § OSG: Open. Science. Grid (USA) § NAREGI (Japan) • EGEE as incubator – >10 related projects have been created INFSO-RI-508833

Related projects Enabling Grids for E-scienc. E INFSO-RI-508833

Toward EGEE-II Enabling Grids for E-scienc. E • EGEE-II proposal submitted to the EU – On 8 September 2005 – Proposed start 1 April 2006 • Natural continuation of EGEE – Emphasis on providing an infrastructure for e-Science increased support for applications increased multidisciplinary Grid infrastructure more involvement from Industry – Expanded consortium § > 90 partners in 32 countries (Non-European partners in USA, Korea and Taiwan) § related projects world-wide Grid infrastructure increased international collaboration INFSO-RI-508833

Project Activities Enabling Grids for E-scienc. E • Similar structure to EGEE with increased focus on: • Services – New countries joining – New activity for sw integration and testing • Networking – Increase support for applications – Reinforce outreach, dissemination and training § Closer links to industry (CERN openlab project) § Extend coverage to all regions • Coordination: – Technical Coordination Group (TCG) – User Information Group (UIG) INFSO-RI-508833

EGEE => EGEE-II Enabling Grids for E-scienc. E EGEE Project • 01. 04 -31. 03. 06 • Number of Partners: 71 • Countries: 27 • Total Budget: 32 M€ EGEE-II Proposal • 01. 04. 06 -31. 03. 08 • Number of Partners: 92 • Countries: 32 • Total Budget: 37 M€ (+ 15%) • CE Federation Requested Contribution: 1. 7 M€ • CE Federation Requested Contribution: 2. 7 M€ (+ 60%) INFSO-RI-508833

The ACEs Model Austrian Grid Czech Republic Hungary Poland Central European Grid Consortium EGEE – Enabling Grids for E-science Slovenia Slovakia

CE Federation Contribution Enabling Grids for E-scienc. E • Cross. Grid Flood Application ported to g. Lite/EGEE – Best Demo at 3 rd EGEE Conference, Athens – Selected for focused EGEE Review in December 2005 • VOCE – Discussed as model for regional contribution/coordination • • glogin g. Lite P-GRADE as EGEE Portal Porting g. Lite to IA-64 … INFSO-RI-508833

Summary Enabling Grids for E-scienc. E • Grids represent powerful new tools for science – The potential return to international scientific communities is enormous and opens the path for commercial and industrial applications – Many applications are already benefiting from Grid technologies Window of opportunity to move grids from research to production, as networks did a few years ago • EGEE offers: – A mechanism for linking together the people, resources and data of many scientific communities – A set of middleware for “gridfying” applications with documentation, training and support – Regular forums for linking with grid experts, other communities and industry • EGEE-II will further extend support for user communities and applications Success will lead to the adoption of grids as the main computing infrastructure for science If we succeed then the potential return to international scientific communities will be enormous and open the path for commercial and industrial applications INFSO-RI-508833

Contacts Enabling Grids for E-scienc. E • EGEE Website http: //www. eu-egee. org • How to join http: //public. eu-egee. org/join/ • EGEE Project Office projectoffice@eu-egee. org INFSO-RI-508833