Computational Grids and the Globus Project Ian Foster

Computational Grids and the Globus Project Ian Foster Argonne National Laboratory The University of Chicago Globus Co-PI: Carl Kesselman, USC/ISI

Credits l Thanks to the Globus teams at ANL, USC/ISI, NCSA, and elsewhere – Steve Tuecke, Gregor von Laszewski, Steve Fitzgerald, Randy Butler, Brian Toonen, and many others l Research supported by DARPA, DOE, NSF, NASA; equipment from Cisco Systems Ian Foster ARGONNE CHICAGO

Overview l Computational grids – Why, what, where l Globus project and its research activities – Focus on resource management l U. S. Grid deployment activities – NASA, DOE, DISCOM l Application overviews – Focus on on-line instrumentation l Summary Ian Foster ARGONNE CHICAGO

Context “Imagine a computing & information infrastructure designed to power research & education by providing pervasive access to all significant information resources” nationally. . . or internationally. . . Ian Foster ARGONNE CHICAGO

Grid Application Examples Online analysis of instrument data Teleimmersion/distance collaboration Trans. Atlantic remote visualization/steering Record-setting distributed supercomputing Ian Foster Parameter studies with deadline scheduling ARGONNE CHICAGO

“The Grid” l l l Pervasive, reliable, secure, highperformance access to leading-edge computational, data, instrument, display & human resources New problem-solving methodologies based on this coupling of geographically distributed resources Advanced network services that enable of the above (“middleware”) Ian Foster ARGONNE CHICAGO

What are “Grids”? l Grid infrastructures support applications that – Involve coordinated use of end-system resources, often in different admin domains – Often require high performance – Require enhanced services beyond those provided by today’s Internet l l Major production infrastructures & testbeds being deployed at this moment Grid R&D sponsored by DARPA Quorum, NSF PACIs, NASA IPG, NGI (DOE in particular) Ian Foster ARGONNE CHICAGO

Creating a Usable Grid : Grid Services (“Middleware”) l Standard grid services that – Provide uniform, high-level access to a wide range of resources (including networks) – Address interdomain issues of security, policy, etc. – Permit application-level management and monitoring of end-to-end performance l Middleware-level and higher-level APIs and tools targeted at application programmers – Map between application and Grid Ian Foster ARGONNE CHICAGO

Unifying Concept: An Integrated Grid Architecture Applns … a rich variety of applications. . . Appln Toolkits Remote data toolkit Grid Services Protocols, authentication, policy, resource management, instrumentation, discovery, etc. Grid Fabric Ian Foster Remote comp. toolkit Remote viz toolkit Async. collab. toolkit . . . Remote sensors toolkit Archives, networks, computers, display devices, etc. ; associated local services ARGONNE CHICAGO

The Globus Project: Argonne, USC/ISI, NCSA, Aerospace, NASA Ames, LBNL, others l Basic research in grid-related technologies – Resource management, security, adaptation, etc. l Development of Globus toolkit – Core services for grid-enabled tools & applns l Construction of large grid testbed: GUSTO – Largest grid testbed in terms of sites & apps l Application experiments – Tele-immersion, distributed computing, etc. Ian Foster ARGONNE CHICAGO

The Globus Project and the IGA Examples: SF-Express, X-ray source data, astrophysics, … rich variety Applns crystallography, collaborative of design, remote quantum chem. Ongoing: HEP & climate data, combustion corridor, remote applications. . . Allegra, distributed AMR, Corridor. One, Hot. Page, . . . Collaborative efforts: e. g. , data Supported partners: e. g. , tele. Remote Async. Remote Appln remote viz, commodity Remote grid, immersion, parameter studies, data collab. . tools, PSEs, Toolkits Message Passing Interface, viz comp. grid, debuggers, desktop sensors toolkit toolkit etc. , etc. couplers, etc. Research Globus Toolkit Protocols, authentication, policy, resource Deployment Grid Security, instrumentation, discovery, Subsets of Globus etc. , etc. Services management, Authentication, resource mgmt, management, information, toolkit technology instrumentation, communication, fault deployed as detection, data access, production Gridadaptivity Archives, networks, computers, display devices, etc. ; other services as and infrastructure Fabric policy, associated local services etc. , etc. when available Ian Foster ARGONNE CHICAGO

GUSTO Testbed Map Ian Foster ARGONNE CHICAGO

Globus Grid Services l The Globus toolkit provides a range of basic Grid services – Security, information, fault detection, communication, resource management, . . . l These services are simple and orthogonal – Can be used independently, mix and match – Programming model independent l For each there are well-defined APIs l Standards are used extensively – E. g. , LDAP, GSS-API, X. 509, . . . Ian Foster ARGONNE CHICAGO

Grid Security Infrastructure l Single-sign on, run anywhere [if authorized] l PKI, X. 509 certificates l l Identity/credential mapping at each resource Allows programs to act as user for limited period: delegation of rights Ian Foster ARGONNE CHICAGO

Single sign-on via “globus-id” CREDENTIAL User Assignment of User Proxy credentials to “user proxies” Globus Technical Details: Grid Security Services Provide Single Sign-On Credential Site 2 Site 1 GRAM Process GSI Process Ticket Kerberos Ian Foster Authenticated interprocess communication Mutual user-resource authentication Process GRAM Process Mapping to local ids GSI GSS-API: multiple Public Key low-level mechanisms Certificate Process ARGONNE CHICAGO

An Example of Globus Services at Work: Resource Management Architecture “ 10 GFlops, EOS data, 20 Mb/sec -- for 20 mins” Info service: location + selection Resource Broker “ 20 Mb/sec” GRAM Globus Resource Allocation Managers GRAM Ian Foster Fork LSF EASYLL Condor etc. Metacomputing “What computers? ” Directory “What speed? ” Service “When available? ” “ 50 processors + storage from 10: 20 to 10: 40 pm” GRAM ARGONNE CHICAGO

Globus and the National Grid l Selected “Grid Services” are being migrated into the infrastructure – Grid information service – Grid security infrastructure – Grid resource management services l Simultaneously these and other Globus services are being applied to develop – Grid-enabled tools – Grid-enabled applications l An ongoing iterative refinement process Ian Foster ARGONNE CHICAGO

Other Potential University Participants ANL ESnet Abilene NWU Diff. Serv Policy Tools AADS Switch supporting MREN PVCs UIUC Diff. Serv Enabled Router The Alliance National Technology Grid Diff. Serv Measurement Servers Wisc. UChicago UIC EMERGE MREN/ESnet Testbed Globus Project Grid Testbed Partnerships Include. . . NASA’s Information Power Grid Ian Foster ARGONNE CHICAGO

Case Study: Online Instrumentation Advanced Photon Source wide-area dissemination real-time collection archival storage desktop & VR clients with shared controls tomographic reconstruction Ian Foster ARGONNE CHICAGO DOE X-ray source grand challenge: ANL, USC/ISI, NIST, U. Chicago

Current Technology Focus Areas l Advanced resource management techniques – GARA: Globus Arch. for Resv. & Allocation l High-end data-intensive applications – “Data Grid” l Interfaces to commodity technologies – Co. G-Kit: Commodity Grid Toolkits l Distance visualization – NOVA: Network Optimized Visualization Arch. With supporting work on info/instr. , policy, accounting, authentication/authorization, etc. Ian Foster ARGONNE CHICAGO

Ian Foster ARGONNE CHICAGO

GARA and Differentiated Services Server Client GARA API Diffserv Resource Manager Ian Foster Diffserv Manager ARGONNE CHICAGO

Scheduling Bulk Transfer and High-Priority Transfers Ian Foster ARGONNE CHICAGO

The Grid Forum http: //www. gridforum. org l l l IETF-like community forum for discussion & definition of Grid infrastructure First two meetings (June 16 -18, Oct 18 -20) attracted 150 people 9 working groups established in security, information infrastructure, resource management, accounting, etc. l Next mtg: San Diego Feb/Mar 2000 l See also European Grid Forum – www. egrid. org Ian Foster ARGONNE CHICAGO

Summary: Globus Contributions l l l Architecture that addresses inter-domain issues Toolkit model for building grid-enabled tools and applications Information-rich approach to adaptivity and high performance New technologies for security, resource management, etc. Construction of production environments and testbeds allowing realistic evaluation Ian Foster ARGONNE CHICAGO

Summary l l Grids require Grid services that make resources accessible and usable and Grid toolkits for application development The Globus project is building essential services and partnering with tool developers Widespread adoption by groups building production infrastructure Significant application success stories in a range of problem classes Ian Foster ARGONNE CHICAGO

For More Information. . . l Globus: www. globus. org l Grid Forum: www. gridforum. org l Grid book “The Grid: Blueprint for a Future Computing Infrastructure, ” I. Foster & C. Kesselman (Eds), Morgan. Kaufmann, 1999 http: //www. mkp. com/grids Ian Foster ARGONNE CHICAGO