Tera Grid Communication and Computation Tal Lavian tlavian nortelnetworks

Tera. Grid Communication and Computation Tal Lavian tlavian@nortelnetworks. com Many slides and most of the graphics are taken from other slides Tera. Grid Comm & Comp 1

Agenda l Introduction l Some applications l Tera. Grid Architecture l Globus toolkit l Future comm direction l Summary Tera. Grid Comm & Comp 2

The Grid Problem Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations Some relation to Sahara Service composition: computation, servers, storage, disk, network… Sharing, cooperating, peering, brokering… Tera. Grid Comm & Comp 3

Tera. Grid Wide Area Network NCSA, ANL, SDSC, Caltech Star. Light International Optical Peering Point (see www. startap. net) Abilene 4 x ane ( ackpl DTF B ps) 40 Gb : Chicago Indianapolis Urbana Los Angeles San Diego I-WIRE OC-48 (2. 5 Gb/s, Abilene) Multiple 10 Gb. E (Qwest) Multiple 10 Gb. E (I-WIRE Dark Fiber) ANL • Solid lines in place and/or available by October 2001 • Dashed I-WIRE lines planned for summer 2002 Source: Charlie Catlett, Argonne Tera. Grid Comm & Comp UIC Starlight / NW Univ Multiple Carrier Hubs Ill Inst of Tech Univ of Chicago Indianapolis (Abilene NOC) NCSA/UIUC 4

The 13. 6 TF Tera. Grid: Computing at 40 Gb/s Site Resources 26 24 8 4 HPSS External Networks Caltech External Networks Site Resources HPSS SDSC 4. 1 TF 225 TB Site Resources HPSS External Networks 5 Argonne NCSA/PACI 8 TF 240 TB Tera. Grid Comm Tera. Grid/DTF: NCSA, SDSC, Caltech, & Comp Argonne External Networks Site Resources Uni. Tree 5 www. teragrid. org

4 Tera. Grid Sites Have Focal Points l SDSC – The Data Place u Large-scale and high-performance data analysis/handling u Every Cluster Node is Directly Attached to SAN l NCSA – The Compute Place u Large-scale, Large Flops computation l Argonne – The Viz place u Scalable Viz walls l Caltech – The Applications place u Data and flops for applications – Especially some of the Gri. Phy. N Apps l Specific machine configurations reflect this Tera. Grid Comm & Comp 6

Tera. Grid building blocks l Distributed, multisite facility u single site and “Grid enabled” capabilities Ø uniform compute node selection and interconnect networks at 4 sites Ø central “Grid Operations Center” u at least one 5+ teraflop site and newer generation processors Ø SDSC at 4+ TF, NCSA at 6. 1 -8 TF with Mc. Kinley processors u at least one additional site coupled with the first Ø four core sites: SDSC, NCSA, ANL, and Caltech l Ultra high-speed networks (Static configured) u multiple gigabits/second Ø modular 40 Gb/s backbone (4 x 10 Gb. E) l Remote visualization u data from one site visualized at another Ø high-performance commodity rendering and visualization system Ø Argonne hardware visualization support Ø data serving facilities and visualization displays l NSF - $53 M award in August 2001 Tera. Grid Comm & Comp 7

Agenda l Introduction l Some applications l Tera. Grid Architecture l Globus toolkit l Future comm direction l Summary Tera. Grid Comm & Comp 10

What applications are being targeted for Grid-enabled computing? Traditional l Quantum Chromodynamics l Biomolecular Dynamics l Weather Forecasting l Cosmological Dark Matter l Biomolecular Electrostatics l Electric and Magnetic Molecular Properties Tera. Grid Comm & Comp 11

Multi-disciplinary Simulations: Aviation Safety Wing Models • Lift Capabilities • Drag Capabilities • Responsiveness Airframe Models Stabilizer Models • Deflection capabilities • Responsiveness Crew Capabilities - accuracy - perception - stamina - re-action times - SOP’s Engine Models Human Models Source NASA • Braking performance • Steering capabilities • Traction • Dampening capabilities Landing Gear Models • Thrust performance • Reverse Thrust performance • Responsiveness • Fuel Consumption Whole system simulations are produced by coupling all of the sub-system simulations Tera. Grid Comm & Comp 13

New Results Possible on Tera. Grid l Biomedical Informatics Research Network (National Inst. Of Health): u Evolving reference set of brains provides essential data for developing therapies for neurological disorders (Multiple Sclerosis, Alzheimer’s, etc. ). l Pre-Tera. Grid: u One lab u Small patient base u 4 TB collection l Post-Tera. Grid: u Tens of collaborating labs u Larger population sample u 400 TB data collection: more brains, higher resolution u Multiple scale data integration and analysis Tera. Grid Comm & Comp 14

Grid Communities & Applications: Data Grids for High Energy Physics ~PBytes/sec Online System ~100 MBytes/sec ~20 TIPS There are 100 “triggers” per second Each triggered event is ~1 MByte in size ~622 Mbits/sec or Air Freight (deprecated) France Regional Centre Spec. Int 95 equivalents Offline Processor Farm There is a “bunch crossing” every 25 nsecs. Tier 1 1 TIPS is approximately 25, 000 Tier 0 Germany Regional Centre ~100 MBytes/sec CERN Computer Centre Fermi. Lab ~4 TIPS Italy Regional Centre ~622 Mbits/sec Tier 2 ~622 Mbits/sec Institute ~0. 25 TIPS Physics data cache Institute ~1 MBytes/sec Tier 4 Caltech ~1 TIPS Tier 2 Centre Tier 2 Centre ~1 TIPS Physicists work on analysis “channels”. Each institute will have ~10 physicists working on one or more channels; data for these channels should be cached by the institute server Physicist workstations Tera. Grid Source Harvey Newman, Caltech. Comm & Comp 15

Agenda l Introduction l Some applications l Tera. Grid Architecture l Globus toolkit l Future comm direction l Summary Tera. Grid Comm & Comp 16

Grid Computing Concept l New applications enabled by the coordinated use of geographically distributed resources u E. g. , distributed collaboration, data access and analysis, distributed computing l Persistent infrastructure for Grid computing u E. g. , certificate authorities and policies, protocols for resource discovery/access l Original motivation, and support, from high-end science and engineering; but has wide-ranging applicability Tera. Grid Comm & Comp 17

Globus Hourglass l Focus on architecture issues u Propose set of core services as basic infrastructure Applications Diverse global services u Use to construct high-level, domainspecific solutions l Design principles u Keep participation cost low Core Globus services u Enable local control u Support for adaptation u “IP hourglass” model Local OS

Elements of the Problem l Resource sharing u Computers, storage, sensors, networks, … u Sharing always conditional: issues of trust, policy, negotiation, payment, … l Coordinated problem solving u Beyond client-server: distributed data analysis, computation, collaboration, … l Dynamic, multi-institutional virtual orgs u Community overlays on classic org structures u Large or small, static or dynamic Tera. Grid Comm & Comp 19

Gilder vs. Moore – Impact on the Future of Computing 10 x every 5 years 2 x/9 mo Log Growth 1 M h 10, 000 AN /M AN W cessor nce Pro ma Perfor 100 1995 t id dw an B 1997 1999 2001 2003 Tera. Grid Comm & Comp 10 x 2 x/18 mo 2005 2007 20

Improvements in Large-Area Networks l Network vs. computer performance u Computer speed doubles every 18 months u Network speed doubles every 9 months u Difference = order of magnitude per 5 years l 1986 to 2000 u Computers: x 500 u Networks: x 340, 000 l 2001 to 2010 u Computers: x 60 u Networks: x 4000 Moore’s Law vs. storage improvements vs. optical improvements. Graph from Scientific American (Jan 2001) by Cleo Vilett, source Vined Khoslan, Kleiner, Caufield and Perkins. Tera. Grid Comm & Comp 21

Evolving Role of Optical Layer 10 Capacity (Mb/s) 10 10 6 10 32 8 4 2 OC-192 c 10 Gb/s transport line rate 1. 7 Gb/s 10 OC-192 WDM 5 4 Service interface rates equal transport line rates 160 3 565 Mb/s 135 Mb/s 2 OC-48 TDM 10 Gb Ethernet OC-48 c Gb Ethernet OC-12 c Fast Ethernet OC-3 c Transport system capacity 10 1 Ethernet T 3 Data: LAN standards Data: Internet backbone T 1 85 90 95 2000 Year Source: IBM WDM research Tera. Grid Comm & Comp 22

Scientific Software Infrastructure One of the Major Software Challenges Peak Performance is skyrocketing (more than Moore’s Law) u but. . . u Efficiency has declined from 40 -50% on the vector supercomputers of 1990 s to as little as 5 -10% on parallel supercomputers of today and may decrease further on future machines Research challenge is software u Scientific codes to model and simulate physical processes and systems u Computing and mathematics software to enable use of advanced computers for scientific applications u Continuing challenge as computer architectures undergo fundamental changes: Algorithms that scale to thousandsmillions processors Tera. Grid Comm & Comp 23

Agenda l Introduction l Some applications l Tera. Grid Architecture l Globus toolkit l Future comm direction l Summary Tera. Grid Comm & Comp 24

Globus Approach l A toolkit and collection of services addressing key technical problems u Modular “bag of services” model u Not a vertically integrated solution u General infrastructure tools (aka middleware) that can be applied to many application domains l Inter-domain issues, rather than clustering u Integration of intra-domain solutions l Distinguish between local and global services Tera. Grid Comm & Comp 25

Globus Technical Focus & Approach l Enable incremental development of grid-enabled tools and applications u Model neutral: Support many programming models, languages, tools, and applications u Evolve in response to user requirements l Deploy toolkit on international-scale production grids and testbeds u Large-scale application development & testing l Information-rich environment u Basis for configuration and adaptation Tera. Grid Comm & Comp 26

Layered Grid Architecture (By Analogy to Internet Architecture) “Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services “Sharing single resources”: negotiating access, controlling use Collective Application Resource “Talking to things”: communication (Internet protocols) & security Connectivity Transport Internet “Controlling things locally”: Access to, & control of, resources Fabric Link Tera. Grid Comm & Comp For more info: www. globus. org/research/papers/anatomy. pdf Internet Protocol Architecture Application 27

Globus Architecture? l No “official” standards exist l But: u Globus Toolkit has emerged as the de facto standard for several important Connectivity, Resource, and Collective protocols u Technical specifications are being developed for architecture elements: e. g. , security, data, resource management, information Tera. Grid Comm & Comp 28

Agenda l Introduction l Some applications l Tera. Grid Architecture l Globus toolkit l Future comm direction l Summary Tera. Grid Comm & Comp 29

Static lightpath setting NCSA, ANL, SDSC, Caltech Abilene 4 x ane ( ackpl DTF B ps) 40 Gb : Chicago Indianapolis Urbana Los Angeles San Diego OC-48 (2. 5 Gb/s, Abilene) Multiple 10 Gb. E (Qwest) Multiple 10 Gb. E (I-WIRE Dark Fiber) • Solid lines in place and/or available by October 2001 • Dashed I-WIRE lines planned for summer 2002 Source: Charlie Catlett, Argonne Tera. Grid Comm & Comp 30

Lightpath for OVPN l Lightpath setup u u One or two-way Rates: OC 48, OC 192 and OC 768 Qo. S constraints On demand ASON l Aggregation of BW u OVPN u Video u HDTV Mirror Server ASON Optical Ring ASON OVPN video HDTV Optical fiber and channels Tera. Grid Comm & Comp 31

Dynamic Lightpath setting l Resource optimization (route 2) u Alternative lightpath l Route to mirror sites (route 3) ASON u Lightpath setup failed u Load balancing u Long response time Ø Congestion Ø Fault Route 3 Mirror Server Route 2 ASON Optical Ring ASON Route 1 main Server Tera. Grid Comm & Comp 32

Multiple Architectural Considerations Apps Clusters C O N T R O L Dynamically Allocated Lightpaths P L A N E Switch Fabrics Physical Monitoring Tera. Grid Comm & Comp 33

Agenda l Introduction l Some applications l Tera. Grid Architecture l Globus toolkit l Future comm direction l Summary Tera. Grid Comm & Comp 34

Summary l The Grid problem: Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations l Grid architecture: Emphasize protocol and service definition to enable interoperability and resource sharing l Globus Toolkit a source of protocol and API definitions, reference implementations l Current static communication. Next wave dynamic optical VPN l Some relation to Sahara u Service composition: computation, servers, storage, disk, network… u Sharing, cooperating, peering, brokering… Tera. Grid Comm & Comp 35

References l globus. org l griphyn. org l gridforum. org l grids-center. org l nsf-middleware. org Tera. Grid Comm & Comp 36

Backup Tera. Grid Comm & Comp 37

Wavelengths and the Future l Wavelength services are causing a network revolution: u Core long distance SONET Rings will be replaced by meshed networks using wavelength cross-connects u Re-invention of pre-SONET network architecture l Improved transport infrastructure will exist for IP/packet services l Electrical/Optical grooming switches will emerge at edges l Automated Restoration (algorithm/GMPLS driven) becomes technically feasible. l Operational implementation will take some time Tera. Grid Comm & Comp 38

Optical components Router/Switch SONET/SDH Gb. E OXC DWDM Fiber Tera. Grid Comm & Comp 39

Internet Reality Data Center SONET DWD M SONET Access Metro Long Haul Tera. Grid Comm & Comp Metro Access 40

OVPN on Optical Network Light Path Tera. Grid Comm & Comp 41

Three networks in The Internet ASON OPE ASON Access ASON Backbone ASON PX Metro Core Local Network (LAN) Long-Haul Core (WAN) Metro Network (MAN) Tera. Grid Comm & Comp 42

Data Transport Connectivity Packet Switch Circuit Switch l data-optimized l Voice-oriented u Ethernet u TCP/IP u SONET u ATM l Network uses u LAN u Metro and Core l Advantages u Efficient u Simple u Low cost u Reliable l Disadvantages u Complicate u High cost u Unreliable Efficiency ? Reliability Tera. Grid Comm & Comp 43

Global Lambda Grid Photonic Switched Network l 1 l 2 Tera. Grid Comm & Comp 44

The Metro Bottleneck Other Sites Access End User Metro Access Metro Ethernet LAN DS 1 DS 3 OC-12 OC-48 OC-192 DWDM n x l IP/DATA 1 Gig. E LL/FR/ATM 1 -40 Meg 10 G 40 G+ Tera. Grid Comm & Comp Core 45