The Grid Ian Foster Argonne National Laboratory University

The Grid Ian Foster Argonne National Laboratory University of Chicago Globus Alliance www. mcs. anl. gov/~foster

Grid Computing is in the News … foster@mcs. anl. gov 2 ARGONNE öCHICAGO

Why Should You Care? 1) Grid is a disruptive technology [Vision] u It ushers in a virtualized, collaborative, distributed world 2) Grid addresses pain points now [Reality] u Grids are built not bought, but are delivering real benefits in commercial settings 3) An open Grid is to your advantage [Future] u Standards are being defined now that will determine the future of this technology foster@mcs. anl. gov 3 ARGONNE öCHICAGO

Why Should You Care? 1) Grid is a disruptive technology [Vision] u It ushers in a virtualized, collaborative, distributed world 2) Grid addresses pain points now [Reality] u Grids are built not bought, but are delivering real benefits in commercial settings 3) An open Grid is to your advantage [Future] u Standards are being defined now that will determine the future of this technology foster@mcs. anl. gov 4 ARGONNE öCHICAGO

Quality, economies of scale The Power Grid: On-Demand Access to Electricity Decouple production & consumption, enabling l On-demand access l Economies of scale l Consumer flexibility l New devices Time foster@mcs. anl. gov 5 ARGONNE öCHICAGO

But Computing Isn’t Really Like Electricity! l How about “access computing resources like we access Web content”? u Þ We have no idea where a website is, or on what computer or operating system it runs Two interrelated opportunities 1) Enhance economy, flexibility, access by virtualizing computing resources 2) Deliver entirely new capabilities by integrating distributed resources foster@mcs. anl. gov 6 ARGONNE öCHICAGO

Virtualization • Automatically connect applications to services • Dynamic & intelligent provisioning Applications: Delivery Application Virtualization Application Services: Distribution Infrastructure Virtualization Servers: Execution • Dynamic & intelligent provisioning • Automatic failover Source: The Grid: Blueprint for a New Computing Infrastructure (2 nd Edition), 2004 foster@mcs. anl. gov ARGONNE öCHICAGO 7

Distributed System Integration NY Financial Institution – Insurance Group UK Financial Institution NY Financial Institution – Capital Markets Group foster@mcs. anl. gov 8 ARGONNE öCHICAGO

And Ultimately … foster@mcs. anl. gov 9 ARGONNE öCHICAGO

The Grid “Resource sharing & coordinated problem solving in dynamic … virtual organizations” 1. Enable integration of distributed service & resources 2. Using general-purpose protocols & infrastructure 3. To achieve useful qualities of service “The Anatomy of the Grid”, Foster, Kesselman, ARGONNE öCHICAGO Tuecke, 2001 foster@mcs. anl. gov 10

Terminology (1) Cluster Grid foster@mcs. anl. gov Enterprise Grid 11 Global Grid ARGONNE öCHICAGO

Terminology (2) Grid Utility Computing Autonomic Computing foster@mcs. anl. gov Service. Oriented Architecture 12 ARGONNE öCHICAGO

Why Should You Care? 1) Grid is a disruptive technology [Vision] u It ushers in a virtualized, collaborative, distributed world 2) Grid addresses pain points now [Reality] u Grids are built not bought, but are delivering real benefits in commercial settings 3) An open Grid is to your advantage [Future] u Standards are being defined now that will determine the future of this technology foster@mcs. anl. gov 13 ARGONNE öCHICAGO

Grid Addresses Pain Points Now l Low utilization of enterprise resources l High cost of provisioning for peak demand l Inadequate resources prevent use of advanced applications l Lack of information integration foster@mcs. anl. gov 14 ARGONNE öCHICAGO

Mission Criticality Grid Deployment Trends Corporate Scientific Department foster@mcs. anl. gov Enterprise 15 Collaboration Internet ARGONNE öCHICAGO

Early Commercial Applications Grid Services Market Opportunity 2005 Leading adopters (Oct 2003) * • Financial services: 31% • Life sciences: 26% • Manufacturing: 18% Manufacturing Financial Services Energy Derivatives Analysis Seismic Analysis Statistical Analysis Reservoir Analysis Portfolio Risk Analysis Mechanical/ Electronic Design LS / Bioinformatics Other Entertainment Process Simulation Cancer Research Finite Element Analysis Drug Discovery Digital Rendering Protein Folding Massive Multi-Player Games Failure Analysis Protein Sequencing Streaming Media Web Applications Weather Analysis Code Breaking/ Simulation Academic “Gridified” Infrastructure Sources: IDC, 2000 and Bear Stearns- Internet 3. 0 - 5/01 Analysis by SAI foster@mcs. anl. gov Rocket Science 16 Business Service, The 451 Group ARGONNE öCHICAGO *Grids 2004: From To

Grid Deployment Strategies l A range of excellent commercial & open source products for resource federation u u Federate enterprise information resources u l Federate enterprise computing resources Globus Toolkit®: inter-enterprise sharing But, “Grids are built, not bought” u l Integration with other enterprise systems is needed to deliver complete solution Start small & with well-defined ROI case u Grow based on experience foster@mcs. anl. gov 17 ARGONNE öCHICAGO

Novartis PC Grid links 3, 700 desktop systems u u Research & development applications Potentially mainstream business computing > 5 teraflop/s computing power Estimate savings of $200 M over 3 years foster@mcs. anl. gov 18 “We have projects we calculate would take 6 years on a single supercomputer. Today, the run time is 12 hours. ” Peter Sany, Novartis CIO ARGONNE öCHICAGO

University of Texas foster@mcs. anl. gov 19 ARGONNE öCHICAGO

Royal Dutch/Shell Improve accuracy and speed of summarization & scientific modeling applications u u More robust, scalable IT infrastructure that adjusts as volumes fluctuate Open standards ease integration of software Cut processing time of seismic data, while improving the quality of the output Focus employees on key scientific, not IT problems foster@mcs. anl. gov 20 “Grid computing is important to Shell because it offers the potential to create a truly unlimited resource, with a uniform interface to a variety of services. This is a significant opportunity for Shell to engage its independent companies in closer cooperation. ” J. N. Buur, Principal Research Physicist, Shell International Exploration and Production B. V. ARGONNE öCHICAGO

Kansai Electric Power Co. l l Japan’s second largest electric utility company has various information in a heterogeneous, distributed database environment Integrate information across departments and affiliated companies to enable information sharing foster@mcs. anl. gov 21 Application DB Client CRM Accounting Workflow Federated Database Oracle Server DB 2 Server ADABAS Server NOTES Server ARGONNE öCHICAGO

NEES (Network for Earthquake Engineering Simulation) Collaboratory U. Nevada Reno www. neesgrid. org

NEESgrid at Work UIUC Experimental Model U. Colorado Experimental Model NCSA Computational Model m 1 f 2 f 1 f 2 m 1 foster@mcs. anl. gov 23 ARGONNE öCHICAGO

CMS High Energy Physics l Generate data for a physics experiment u Simulate 1. 5 million events for physics studies @ ~500 sec per event u 2 months continuous running across 5 sites u Managed by a single person foster@mcs. anl. gov 24 ARGONNE öCHICAGO

Why Should You Care? 1) Grid is a disruptive technology [Vision] u It ushers in a virtualized, collaborative, distributed world 2) Grid addresses pain points now [Reality] u Grids are built not bought, but are delivering real benefits in commercial settings 3) An open Grid is to your advantage [Future] u Standards are being defined now that will determine the future of this technology foster@mcs. anl. gov 25 ARGONNE öCHICAGO

Grid Vision, Marketing, and Reality l Vision u l Marketing u l Computing resources can be shared like content on the Wb Have we got a [Data, compute, knowledge, information, desktop, PC, enterprise, cluster, …] Grid for you! Reality u u Commercial products mostly noninteroperable Open source tools offer de facto standards, but are also far from a complete solution foster@mcs. anl. gov 26 ARGONNE öCHICAGO

Standards Matter! l Open, standard protocols u u Avoid product/vendor lock-in u Enable innovation/competition on end points u l Enable interoperability Enable ubiquity In Grid space, must address how we u Describe, discover, & access resources u Monitor, manage, & coordinate, resources u Account & charge for resources For many different types of resource foster@mcs. anl. gov 27 ARGONNE öCHICAGO

Increased functionality, standardization Developing Grid Standards Managed shared virtual systems Research Open Grid Services Arch Web services, etc. Internet standards Custom solutions 1990 foster@mcs. anl. gov Real standards Multiple implementations Globus Toolkit Defacto standard Single implementation 1995 2000 28 2005 2010 ARGONNE öCHICAGO

Open Grid Services Architecture Adopt service-oriented architecture u Key to virtualization, discovery, composition, local-remote transparency + Standard service description & access u Leverage industry standard Web services + Distributed service management protocols u A “component model for Web services” = A framework for creating, managing, & delivering interoperable services “The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration”, Foster, Kesselman, ARGONNE öCHICAGO Nick, Tuecke, 2002 foster@mcs. anl. gov 29

Open Grid Services Architecture (GGF, OASIS, W 3 C) Users in Problem Domain X Application & Integration Technology for Problem Domain X Generic Virtual Service Access and Integration Layer Job Submission Brokering Registry Workflow Banking Authorisation OGSA Structured Data Integration Data Transport Resource Usage Transformation Structured Data Access OGSI: Interface to Grid Infrastructure Web Services: Basic Functionality Compute, Data & Storage Resources Distributed Structured Data Relational XML Semi-structured - foster@mcs. anl. gov Virtual Integration Architecture ARGONNE öCHICAGO 30

Open Grid Services Infrastructure Client Introspection: • What port types? • What policy? • What state? Lifetime management • Explicit destruction • Soft-state lifetime Grid. Service (required) Grid Service Handle handle resolution Grid Service Reference Data access Service data element Other standard interfaces: factory, notification, collections Service data element Implementation Hosting environment/runtime (“C”, J 2 EE, . NET, …) foster@mcs. anl. gov 31 ARGONNE öCHICAGO

OGSA Standards: Status l WSDL 1. 2 in progress in W 3 C l OGSI 1. 0 completed July 2003 in GGF l WS-Agreement draft in GGF l WS-Management drafts in OASIS l OGSA Data Access & Integration in GGF l WS-Security specifications in OASIS l SAML & XACML in OASIS l WS-Addressing status unclear l Other work in progress foster@mcs. anl. gov 32 ARGONNE öCHICAGO

Standards: Summary l Grid and Web Services are merging u l Grid is an aggressive use case of Web Services standards landscape is in flux u u OGSI/A will need to evolve with it Uncertain status of security & policy standards continues to be a big source of concern l Grid services standards landscape heating up l W 3 C, OASIS, GGF are key standards orgs l Open source software important for adoption foster@mcs. anl. gov 33 ARGONNE öCHICAGO

OGSA Status: Implementations l Globus Toolkit v 3: Linux for the Grid u u l Open source middleware, commercial support A range of computation & data management, registry, and security functions Some nice announced OGSI-based products u u IBM, Avaki, Platform, Sun, NEC, HP, UD, Entropia, Data. Synapse, Insors, Oracle, etc. Read the fine print: “Intent is to use OGSIbased products, ” “OGSA-compliant software, ” “Embraces fundamental OGSA concepts” foster@mcs. anl. gov 34 ARGONNE öCHICAGO

Globus Toolkit History: An Unreliable Memoir Only Globus. Org; not downloads from: NMI UK e. Science EU Data. Grid IBM Platform etc. The Grid: Blueprint for a New Computing Infrastructure published DARPA, MPICH-G NSF begin released funding Grid work Globus Project wins Global Information Infrastructure Award GT 2. 0 Released Physiology of the Grid Paper Released GT 2. 0 beta Released NSF GRIDS Center Initiated Anatomy of the Grid Paper Released Significant GT 1. 1. 4 and Commercial MPICH-G 2 Released Interest in Grids NSF & European Commission Initiate Many New Grid Projects First GT 1. 1. 3 Released Early Application Successes Reported GT 1. 0. 0 Released GT 2. 2 Released GT 1. 1. 2 Released Euro. Globus Conference Held in Lecce GT 1. 1. 1 Released NASA initiates Information Power Grid foster@mcs. anl. gov 35 ARGONNE öCHICAGO

HP Grid Product Offerings l Globus Toolkit for HP-UX, Tru 64 Unix, Linux Globus Toolkit 2. 4 u Tested and optimized for HP platforms u Commercial support! u l HP Utility Data Center (UDC) Data Center operation & construction services u Service-centric design u Intent is to construct UDC products using OGSI-compliant services u l Enterprise Grid foster@mcs. anl. gov consulting 36 ARGONNE öCHICAGO

HP and Planetary Computing virtual data center computing utility or GRID value programmable data center switch compute fabric storage grid-enabled systems clusters UDC Tru 64, HP-UX, Linux Open VMS clusters, Tru. Cluster, MC Service. Guard foster@mcs. anl. gov today 37 shared, traded resources ARGONNE öCHICAGO

Pop Quiz: The Grid Is … a) A collaboration & resource sharing infrastructure for scientific applications b) A distributed service integration and management technology c) A disruptive technology that enables a virtualized, collaborative, distributed world d) An open source technology & community e) A marketing slogan f) All of the above foster@mcs. anl. gov 38 ARGONNE öCHICAGO

Summary: Why You Should Care 1) Grid is a disruptive technology [Vision] u It ushers in a virtualized, collaborative, distributed world 2) Grid addresses pain points now [Reality] u Grids are built not bought, but are delivering real benefits in commercial settings 3) An open Grid is to your advantage [Future] u Standards are being defined now that will determine the future of this technology foster@mcs. anl. gov 39 ARGONNE öCHICAGO

To Learn More Deep tech & strategic info Jan 20 -23, 2004, San Fran www. globusworld. org 2 nd Edition www. mkp. com/grid 2 Working, research Become a groups Three meetings a year Globus corporate affiliate www. ggf. org Weekly email & web newsletter www. gridtoday. com foster@mcs. anl. gov Commercial 40 conf & expo Background information www. mcs. anl. gov/ ~foster ARGONNE öCHICAGO