GRIDS Center Grid Research Integration Development Support

GRIDS Center Grid Research Integration Development & Support http: //www. grids-center. org Chicago - NCSA – SDSC - USC/ISI - Wisconsin

GRIDS, part of the NSF Middleware Initiative (NMI) 1. The Information Sciences Institute (ISI) at the University of Southern California (Carl Kesselman) 2. The University of Chicago (Ian Foster) 3. The National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana. Champaign (Randy Butler) 4. The University of California at San Diego (Phil Papadoupolus) 5. The University of Wisconsin at Madison (Miron Livny) Part of the NSF Middleware Initiative (NMI) www. grids-center. org

GRIDS Enabling Seamless Collaboration GRIDS will help distributed communities pursue common goals Scientific research u Engineering design u Education u Artistic creation u Focus is on the enabling mechanisms required for collaboration u Resource sharing as a fundamental concept Part of the NSF Middleware Initiative (NMI) www. grids-center. org

GRIDS Grid Computing Rationale The need for flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resource See “The Anatomy of the Grid: Enabling Scalable Virtual Organizations” by Foster, Kesselman, Tuecke at http: //www. globus. org (in the “Publications” section) The need for communities (“virtual organizations”) to share geographically distributed resources as they pursue common goals while assuming the absence of: u u central location central control omniscience existing trust relationships Part of the NSF Middleware Initiative (NMI) www. grids-center. org

GRIDS Elements of Grid Computing Resource sharing u u Computers, storage, sensors, networks Sharing is always conditional, based on issues of trust, policy, negotiation, payment, etc. Coordinated problem solving u Beyond client-server: distributed data analysis, computation, collaboration, etc. Dynamic, multi-institutional virtual organizations u Community overlays on classic org structures u Large or small, static or dynamic Part of the NSF Middleware Initiative (NMI) www. grids-center. org

GRIDS Resource-Sharing Mechanisms 1. Should address security and policy concerns of resource owners and users 2. Should be flexible and interoperable enough to deal with many resource types and sharing modes 3. Should scale to large numbers of resources, participants, and/or program components 4. Should operate efficiently when dealing with large amounts of data & computational power Part of the NSF Middleware Initiative (NMI) www. grids-center. org

GRIDS Grid Applications Science portals u u u Help scientists overcome steep learning curves of installing and using new software Solve advanced problems by invoking sophisticated packages remotely from Web browsers or "thin clients” Portals are currently being developed in biology, fusion, computational chemistry, and other disciplines Distributed computing u u High-speed workstations and networks can yoke together an organization's PCs to form a substantial computational resource E. g. , U. S. and Italian mathematicians pooled resources for one week, aggregating 42, 000 CPUdays to solve "Nug 30" Part of the NSF Middleware Initiative (NMI) www. grids-center. org

Grid Portals Part of the NSF Middleware Initiative (NMI)

Distributed Computing to Evaluate AIDS Drugs Community = u u u 1000 s of home computer users Philanthropic computing vendor (Entropia) Research group (Scripps) Common goal= advance AIDS research Part of the NSF Middleware Initiative (NMI)

GRIDS More Grid Applications Large-scale data analysis u u Science increasingly relies on large datasets that benefit from distributed computing and storage E. g. , the Large Hadron Collider at CERN will generate many petabytes of data from high-energy physics experiments, with single-site storage impractical for technical and political reasons Computer-in-the-loop instrumentation u u u Data from telescopes, synchrotrons, and electron microscopes are traditionally archived for batch processing Grids are permitting quasi-real-time analysis that enhances the instruments’ capabilities E. g. , with sophisticated “on-demand” software, astronomers may be able to use automated detection techniques to zoom in on solar flares as they occur Part of the NSF Middleware Initiative (NMI) www. grids-center. org

Large-scale Data Analysis Part of the NSF Middleware Initiative (NMI)

GRIDS Still More Grid Applications Collaborative work u u Researchers often want to aggregate not only data and computing power, but also human expertise Grids enable collaborative problem formulation and data analysis u u E. g. , an astrophysicist who has performed a large, multi-terabyte simulation could let colleagues around the world simultaneously visualize the results, permitting real-time group discussion E. g. , civil engineers collaborate to design, execute, & analyze shake table experiments Part of the NSF Middleware Initiative (NMI) www. grids-center. org

Collaboration via Online Access to Scientific Instruments Part of the NSF Middleware Initiative (NMI)

GRIDS Grids and Industry Grid computing has much in common with major industrial thrusts u u Business-to-business, Peer-to-peer, Application Service Providers, Storage Service Providers, Distributed Computing, Internet Computing, etc. Outsourcing increases decentralization of resources Sharing issues are not adequately addressed by existing technologies u Complicated requirements: “run program X at site Y subject to community policy P, providing access to data at Z according to policy Q” Companies like IBM, Platform Computing and Microsoft are getting substantively involved with the open-source Grid community (e. g. , web services and Grid services) Part of the NSF Middleware Initiative (NMI) www. grids-center. org

GRIDS e. Business Grids 1. Engineers at a multinational company collaborate on the design of a new product 2. A multidisciplinary analysis in aerospace couples code and data in four companies 3. An insurance company mines data from partner hospitals for fraud detection 4. An application service provider offloads excess load to a compute cycle provider 5. An enterprise configures internal & external resources to support e. Business workload Part of the NSF Middleware Initiative (NMI) www. grids-center. org

GRIDS and the NSF Middleware Initiative GRIDS is one of two NMI teams; the other is EDIT NMI seeks standard components and mechanisms u Authentication, authorization, policy u Resource discovery and directory u Remote access of computers, data, instruments Also seeks: u Integration with end-user tools (conferencing, data analysis, data sharing, distributed computing, etc. ) u Integration with campus infrastructures u Integration with commercial technologies Part of the NSF Middleware Initiative (NMI) www. grids-center. org

GRIDS Deliverables for NMI Release 1. 0 On May 7, NMI Release 1. 0 was issued (see www. nsf-middleware. org), including deliverables from the GRIDS and EDIT teams GRIDS software in NMI-R 1 will include new versions of: u Globus Toolkit™ u Condor-G u Network Weather Service u package also includes KX. 509 Part of the NSF Middleware Initiative (NMI) www. grids-center. org

GRIDS The Globus Toolkit™ The de facto standard for Grid computing u u u A modular “bag of technologies” addressing key technical problems facing Grid tools, services and applications Made available under liberal open source license Simplifies collaboration across virtual organizations l Authentication u Grid Security Infrastructure (GSI) Scheduling u Globus Resource Allocation Manager (GRAM) u Dynamically Updated Request Online Coallocator (DUROC) File transfer l Global Access to Secondary Storage (GASS) u Grid. FTP Resource description l l u u Monitoring and Discovery Service (MDS) Part of the NSF Middleware Initiative (NMI) www. grids-center. org

GRIDS u High performance computing (HPC) is often measured in operations per second; with high throughput computing (HTC), Condor permits increased processing capacity over longer periods of time l l u u Condor-G CPU cycles/day (week, month, year? ) under non-ideal circumstances “How many times can I run simulation X in a month using all available machines? ” The Condor Project develops, deploys, and evaluates mechanisms and policies for HTC on large collections of distributed systems NMI-R 1 will include Condor-G, an enhanced version of the core Condor software optimized to work with Globus Toolkit™ for managing Grid jobs Part of the NSF Middleware Initiative (NMI) www. grids-center. org

GRIDS Network Weather Service u u u From UC Santa Barbara, NWS monitors and dynamically forecasts performance of network and computational resources Uses a distributed set of performance sensors (network monitors, CPU monitors, etc. ) for instantaneous readings Numerical models’ ability to predict conditions is analogous to weather forecasting – hence the name l For use with the Globus Toolkit and Condor, allowing dynamic schedulers to provide statistical Quality-of-Service readings l NWS forecasts end-to-end TCP/IP performance (bandwidth and latency), available CPU percentage and available nonpaged memory l NWS automatically identifies the best forecasting technique for any given resource Part of the NSF Middleware Initiative (NMI) www. grids-center. org

GRIDS KX. 509 for Converting Kerberos Certificates to PKI Stand-alone client program from the University of Michigan u u u For a Kerberos-authenticated user, KX. 509 acquires a shortterm X. 509 certificate that can be used by PKI applications Stores the certificate in the local user's Kerberos ticket file Systems that already have a mechanism for removing unused kerberos credentials may also automatically remove the X. 509 credentials Web browser may then load a library (PKCS 11) to use these credentials for https The client reads X. 509 credentials from the user’s Kerberos cache and converts them to PEM, the format used by the Globus Toolkit Part of the NSF Middleware Initiative (NMI) www. grids-center. org

GRIDS Integration Issues Ten NMI testbed sites will be early adopters, seeking integration of enterprise and Grid computing u Eight sites to be announced soon by SURA u Two further sites: Cal. Tech and USC Via NMI partnerships, GRIDS will help identify points of intersection and divergence between Grid and enterprise computing u Directory services u Authorization, authentication and security u Emphasis is on open standards and architectures as the route to successful collaboration Part of the NSF Middleware Initiative (NMI) www. grids-center. org