International Computing e-Infrastructures Past Present and Future

International Computing e-Infrastructures: Past, Present and Future. . . Fabrizio Gagliardi EMEA Director Technical Computing Microsoft Corporation

Outline of the talk Some personal introductory remarks Definition of e-Infrastructures Need for e-Infrastructures Recent past history Current situation, accomplishments and challenges Outlook for the future CGW'05 2

Definition of e-Infrastructures to support wide geographically distributed communities which share problems and resources to work towards common goals Leveraging international network interconnectivity Based on safe AAA architecture Need persistent software middleware (S/W is integral part of the infrastructure) CGW'05 3

‘Grids’: A Catch-All Marketing Term ‘Grids’ mean many different things to many different people/companies: P 2 P desktop cycle-stealing Linked Supercomputer Centers Managed virtual distributed clusters Internet access to giant, distributed repositories Virtualization of data center IT resources Out-sourcing to “utility compute centers” Sharing resources distributed among different administrative domains (Ian Foster) Ø For Microsoft, Grids are about Data Management as much as Compute Cycles CGW'05 4

Need for e-Infrastructures Science, industry and commerce are more and more digital, process vast amounts of data and need massive computing power We live in a “flat” world: Science is more and more an international collaboration and often requires a multidisciplinary approach Need to use technology for the good cause Fight Digital/Divide Industrial uptake has become essential CGW'05 5

Recent past history Meta-computing and distributed computing early examples in the 80’ and 90’ (CASA, I-Way, Unicore, Condor etc. ) EU-US workshop in Annapolis in 1999 on large scientific data bases: http: //www. cacr. caltech. edu/euus/ EU FP 5 and US Trillium and national Grids EU FP 6, US OSG, NAREGI/Japan… CGW'05 6

Chronology IGO-----ETICS/EGO---- EDG start 2001 CHEP 2000 EGEE-II start 2006 EGEE start 2004 EGEE-XX 2008? We are here CGW'05 7

ARCADE 2002 Barcelona, 14 th February 2002 “Unlimited” bandwidth, breaking the frontiers of computing: the path in Europe from FP 5 to FP 6. Antonella Karlson Research Networks Unit, DG INFSO, EC antonella. karlson@cec. eu. int CGW'05 "The views expressed in this presentation are those of the author and do not necessarily reflect the views of the 8

GRIDs - IST projects (~36 m Euro) An integrated approach Applications EGSO CROSSGRID Middleware GRIP EUROGRID & Tools DAMIEN GRIDSTART GRIA GRIDLAB DATAGRID DATATAG Underlying Infrastructures Industry / business CGW'05 Science 9

GRIDs: Examples of large testbeds Ü DATAGRID, CROSSGRID T NCTURE A EASTRU G FR IN • 17 European countries • Collaboration of more than 2000 scientists Application requirements: • Computing > 20 TFlops/s • Downloads > 0. 5 PBytes • Network speeds at 10 CGW'05 10 Gbps

GRIDs: Examples of large testbeds Ü DATATAG (cross-Atlantic testbed) (2 Gbps) Links with US projects (Gri. Phy. N, PPDG, i. VDGL, …) CGW'05 11

Current situation: accomplishments and challenges Many Grids around the world, very few maintained as a persistent infrastructure (best example is the “secret” Google Grid) Need for public and open Grids (OSG, EGEE and related projects, NAREGI, and TERAGRID, DEISA good prototypes) Persistence, support, sustainability, long term funding, easy access are the major challenges CGW'05 12

Projects in Europe (I) Enabling Grids for E-scienc. E • Access to IT-resources (connectivity, computing, data, instrumentation…) for scientists: – Providing e-Infrastructure § § Géant 2 EGEE DEISA SEE-GRID – Benefiting from e-Infrastructure § § § DILIGENT SIMDAT GRIDCC Core. GRID Grid. Lab – Concertation: GRIDSTART, Grid. Coord – Grid mobility: Akogrimo INFSO-RI-508833 CGW'05 13

Projects in Europe (II) Enabling Grids for E-scienc. E • Sample of National Grid projects: – Austrian Grid Initiative – Dutch. Grid – France: § e-Toile § ACI Grid – Germany § D-Grid § Unicore D-GRID – Grid Ireland – Italy § INFNGrid § GRID. IT – Nordu. Grid – UK e-Science § National Grid Service § OMII § Grid. PP project INFSO-RI-508833 CGW'05 14

Policy Forums Enabling Grids for E-scienc. E • The e-Infrastructures Reflection Group (e. IRG) – Mission: study and promote policies for easy and cost-effective shared use of electronic resources in Europe – 25 countries (government-appointed representatives), EU: 2 members – White Papers • European Strategy Forum on Research Infrastructures (ESFRI) – Role: to support a coherent approach to policy-making on research infrastructures in Europe, and to act as an incubator for international negotiations about concrete initiatives – representatives of the 25 EU Member States, appointed by Research Ministers and a representative of the European Commission • ESFRI + e. IRG: European roadmap for new research infrastructures of pan-European interest (10 -20 years) INFSO-RI-508833 CGW'05 15

Enabling Grids for E-scienc. E Géant 2 • GÉANT 2 is the 7 th generation of the pan-European research and education network, successor to the multi-gigabit research network GÉANT. – Official start: 1 September 2004, Duration: 4 years – Funding: EC, national research, education networks – Managed by DANTE • Goal: – To connect 34 countries through 30 national research and education networks (NRENs) – using multiple 10 Gbps wavelengths • Status: – Equipment and services currently in operations (officially inaugurated by Commissioner Reding last June) – Transition from GÉANT network to GÉANT 2 gradually completing, started in the first quarter of 2005 INFSO-RI-508833 CGW'05 16

EU Grid technology & infrastructure (I) Enabling Grids for E-scienc. E New Grid Research Projects in FP 6 EU Funding: 52 MILLION - Start: SUMMER 2004 GRIDCOORD Building the ERA in Grid research K-WF Grid inteli. GRID Knowledge based workflow & collaboration Semantic Grid based virtual organisations Grid-based generic enabling application technologies to facilitate solution of industrial problems Onto. Grid SIMDAT Uni. Grid. S Extended OGSA Implementation based on UNICORE EU-driven Grid services architecture for busines. S and industry Next. GRID HPC 4 U Fault tolerance, dependability for Grid Mobile Grid architecture and services for dynamic virtual organisations Akogrimo European-wide virtual laboratory for longer term Grid research-creating the foundation for next generation Grids Core. GRID Specific support action Integrated project Network of excellence Knowledge Services for the semantic Grid Datamining tools & services Provenance Trust and provenance for Grids Specific targeted research project From a talk by Ulf Dahlsten, Den Haag, Nov 2004 INFSO-RI-508833 CGW'05 17

EU Grid technology & infrastructure (II) Enabling Grids for E-scienc. E Building the European e. Infrastructure for research 2000 2001 TEN 155 network 2002 2003 2004 GÉANT network 2005 2006 2007 2008 GÉANT network (FP 6) IPv 6 testbeds IPv 6 actions Grid testbeds Grid enabled Infrastructures (EGEE, DEISA, SEE-GRID, …) (other) testbeds FP 5 FP 6 FP 7 Complementary to National infrastructures From a talk by Ulf Dahlsten, Den Haag, Nov 2004 INFSO-RI-508833 CGW'05 18

EU Grid technology & infrastructure (III) Enabling Grids for E-scienc. E MUPPET ogy va lida tion EUQo. S EE SA ID cture EG EI GR astru EUROLABS D EE- Infr S e IPv 6 Task Force support Courtesy of Specific Support Actions K. Baxevanidis, EU INFSO-RI-508833 nol er Und tion otia neg Flexible Quality of Service Assurance GRIDCC DILIGENT IPv 6 TF SC ech Optical solutions for Grid infrastruct. New user communities using Grids – Digital Libraries in ser U …t Real time Grid for remote control of instruments e. Infrastructure – ent… m Testbeds olve v LOBSTER Experimental testbeds Traffic monitoring CGW'05 19

EU Grid technology & infrastructure (V) Enabling Grids for E-scienc. E • e. Infrastructure – achievements • Connectivity service • • Computing, storage service • • GÉANT network: 10 Gbit/s, IPv 6 enabled, 3900 Research Centres connected EGEE: production quality, >10000 CPUs, >5 PB storage, training, coverage of 27 countries DEISA: Supercomputer network, reaching 40 Tflop/s • Testbeds • Rich set of technologies tested/ verified (IPv 6, Grids, Optical, End-to-End Qo. S, Security, Mobility…) and communities involved (scientific, industry) • International links • USA, Russia, Mediterranean, Asia, Latin America. . . From a talk by Ulf Dahlsten, Den Haag, Nov 2004 INFSO-RI-508833 CGW'05 20

Outlook for the future CGW'05 21

Supercomputing Goes Personal 1991 1998 2005 System Cray Y-MP C 916 Sun HPC 10000 Shuttle @ New. Egg. com Architecture 16 x Vector 4 GB, Bus 24 x 333 MHz Ultra. SPARCII, 24 GB, SBus 4 x 2. 2 GHz x 64 4 GB, Gig. E OS UNICOS Solaris 2. 5. 1 Windows Server 2003 SP 1 GFlops ~10 ~10 Top 500 # 1 500 N/A Price $40, 000 $1, 000 (40 x drop) < $4, 000 (250 x drop) Customers Government Labs Large Enterprises Every Engineer & Scientist Applications Classified, Climate, Physics Research Manufacturing, Energy, Finance, Telecom Bioinformatics, Materials Sciences, Digital Media CGW'05 22

The Continuing Trend Towards Decentralized, Networked Resources Grids of personal & departmental clusters Personal workstations & departmental servers Minicomputers Mainframes CGW'05 23

Leverage IT Industry’s Existing R&D Parallel applications development High-productivity IDEs Integrated debugging/profiling/tracing/analy sis Code designer wizards Concurrent programming frameworks Platform optimizations Dynamic, profile-guided optimization New programming abstractions Digital experimentation Collaboration-enhanced Office productivity tools Structure experiment data and derived results in a manner appropriate for human reading/reasoning (as opposed to optimizing for query processing and/or storage efficiency) Enable collaboration among colleagues (Scientific) workflow environments Distributed systems issues Web Services & HPC grids Automated orchestration Visual scripting Provenance Security Interoperability Scalability Dynamic Systems Management Self (re)configuration & tuning Reliability & availability RDMS + data mining Ease-of-use Advanced indexing & query processing Advanced data mining algorithms CGW'05 24

Scientific Information Worker: Past and Future Past Buy lab equipment Keep lab notebook Run experiments by hand Assemble & analyze data (using stat pkg) Collaborate by phone/email; write up results with Latex Metaphor: Physical experimentation “Do it yourself” Lots of disparate systems/pieces Future Buy hardware & software Automatic provenance Workflow with 3 rd party domain packages Excel & Access/SQL-Server Office tool suite with collaboration support Metaphor: Digital experimentation Turn-key desktop supercomputer Single integrated system CGW'05 25

Where Grids will be in 5 years? Like in the past ES, AI, networking, OS they will disappear from the hot research (and hype) space and become mainstream technology Major Grids already work in production (EGEE: 18’ 000 computers, Google: 100’ 000 computers? . . . ) Major IT vendors will integrate Grid middleware in their standard products (industrial uptake) Computing and data resources will become commodities on the Internet ISPs will offer a wide range of services Grid based, a full mature market will develop for these services The result will be a tremendous computing and data processing power which will enable a new set of scientific applications and generate large revenues for business applications A potential leveler for a worldwide science and economy => digital Divide could be moderated CGW'05 26

… And time will tell how wrong we are in our predictions now See you back here next year! CGW'05 27