Enabling e. Science Grid Technologies Today & Tomorrow Ian Foster Argonne National Laboratory University of Chicago Univa Corporation e. Science [n]: Large-scale science carried out through distributed collaborations—often leveraging access to large-scale data & computing
2 Overview l The need for Grid technology l Grid technology today l Future challenges & directions l Summary
3 Science Today is a Team Sport
NEES: Network for Earthquake Engineering Simulation Links instruments, data, computers, people
5 The Application-Infrastructure Gap Dynamic and/or Distributed Applications Shared Distributed Infrastructure B A 1 1 9 9
6 Bridging the Gap: Grid Technology Users l u u l Composition Service-oriented applications Wrap applications as services Compose applications into workflows Service-oriented infrastructure u Provision physical resources to support application workloads Workflows Invocation Appln Service Provisioning
7 Overview l The need for Grid technology l Grid technology today l Future challenges & directions l Summary
Grid Technology: Service-Oriented Infrastructure User Application Tool Uniform interfaces, security mechanisms, Web service transport, monitoring GRAM Computers Reliable File Transfer MDSIndex User Svc Host Env Specialized resource 8 User Application Tool User Svc Host Env My. Proxy Grid. FTP Storage DAIS Database
Web Services: Standards, Tools, Interoperability Custom Web Services Custom GT 4 WSRF Web Services WS-Addressing, WSRF, WS-Notification WSDL, SOAP, WS-Security Registry Administration GT 4 Container User Applications 9
10 Authentication & Authorization SSL/WS-Security with Proxy Services (running Certificates Authz Callout on user’s behalf) Access Compute Center Rights CAS or VOMS issuing SAML or X. 509 ACs Users Rights Local Policy on VO identity or attribute authority My. Proxy VO Rights’ KCA
11 The Many Dimensions of Scaling Index Disk-to-disk on Tera. Grid >2 Gbyte/s Local Update Bloom DB send filter (secs) (bits) 10 K 2 1 M 1 M Grid. FTP: High-performance, reliable data movement <1 2 24 10 M 5 M 7 175 50 M Replica location: LIGO = 40 M files, 10 sites
Scaling: Grid 2003 Workflows Sloan digital sky survey 12 Genome sequence analysis Physics data analysis
13 Application Examples Fusion Grid: 1000 s of jobs STAR: 5 TB transfer (SRM, Grid. FTP) Earth System Grid: O(100 TB) online data NASA/NVO: Mosaics from multiple sources
Global Community
68 Belgium 20 Finland 7 Ecuador 3 Mongolia 1 Mozambique 66 Venezuela 20 New Zealand 7 Macedonia 3 Sri Lanka 1 Tokelau 66 Romania 17 Nigeria 6 Bolivia 3 Wallis & Futuna Is. 1 Greenland 64 Indonesia 17 South Africa 6 Comoros 3 Belarus 1 Nepal 62 Mexico 16 Jordan 6 Zaire 3 Bangladesh 1 Swaziland 61 Turkey 16 Slovenia 6 Lebanon 2 Falkland Islands 1 Iraq 488 Germany 60 Malaysia 16 Afghanistan 5 Puerto Rico 2 Kuwait 1 Serbia 391 Brazil 58 Portugal 15 Denmark 5 Namibia 2 Sierra Leone 1 Barbados 57 Austria 15 Philippines 5 Togo 2 Trinidad & Tobago 1 Cambodia 54 Ireland 14 Vanuatu 5 Tajikistan 2 Guyana 1 Qatar 44 Hong Kong 14 Luxembourg 5 Paraguay 2 American Samoa 1 Saint Vincent 40 Hungary 14 Tunisia 5 Albania 2 Andorra 1 Laos 211 Viet Nam 38 Egypt 12 Virgin Is. (U. K. ) 5 Sudan 2 Georgia 1 San Marino 211 Spain 38 Argentina 12 Peru 4 Estonia 2 Cook Islands 1 Libya 34 Uruguay 12 Yemen 4 Camaroon 2 Turkmenistan 1 Benin 31 Ukraine 11 Norway 4 Ghana 2 Gabon 1 Angola 159 Australia 29 Slovakia 11 Algeria 4 Tuvalu 2 The Gambia 1 Chad 142 Singapore 25 Israel 11 Armenia 4 Costa Rica 2 Kazakhstan 1 Gibraltar 131 Greece 23 Yugoslavia 10 Iceland 4 Cuba 2 Macau 1 Haiti 119 Colombia 23 Iran 9 Zambia 4 UAE 2 Malta 1 Guatemala 111 Poland 22 Bulgaria 9 Virgin Is. (U. S. ) 4 Tonga 2 Jamaica 1 Malawi 109 Netherlands 22 Uzbekistan 9 Uganda 4 W. Samoa 2 Latvia 1 Equatorial Guinea 107 Thailand 22 Czech Rep. 9 Bosnia & Herz. 4 Tanzania 2 Turks & Caicos 1 Palau 94 Switzerland 22 N. Korea 8 Kenya 3 Syria 1 Bhutan 1 Bermuda 77 Chile 21 Lithuania 7 Zimbabwe 3 Bahamas 1 Ascension Island 1 Botswana 21 Croatia 7 Saudi Arabia 3 Ethiopia 1 Cyprus 1 Suriname 2753 China 1318 Japan 1017 India 750 U. K. 495 Italy 328 S. Korea 306 Taiwan 268 France 241 Canada 202 Russia 187 Pakistan 74 Sweden 38669 downloads in 2004 from globus. org 26776 U. S.
2753 China 1318 Japan 1017 India 750 U. K. 495 Italy 68 Belgium 20 Finland 7 Ecuador 3 Mongolia 1 Mozambique 66 Venezuela 20 New Zealand 7 Macedonia 3 Sri Lanka 1 Tokelau 66 Romania 17 Nigeria 6 Bolivia 3 Wallis & Futuna Is. 1 Greenland 64 Indonesia 17 South Africa 6 Comoros 3 Belarus 1 Nepal 62 Mexico 16 Jordan 6 Zaire 3 Bangladesh 1 Swaziland 61 Turkey 16 Slovenia 6 Lebanon 2 Falkland Islands 1 Iraq 38669 downloads in 2004 from globus. org 26776 U. S. Afghanistan Top 1610 488 Germany 60 Malaysia 391 Brazil 58 Portugal 15 Denmark 5 Puerto Rico 2 Kuwait 1 Serbia 5 Namibia 1 Barbados 15 Philippines 26776 U. S. 5 Togo 306 Taiwan 54 Ireland 14 Vanuatu 5 Tajikistan 268 France 44 Hong Kong 14 Luxembourg 2753 China 5 Paraguay 241 Canada 40 Hungary 14 Tunisia 1318 Japan 5 Albania 211 Viet Nam 38 Egypt 12 Virgin Is. (U. K. ) 5 Sudan 211 Spain 38 Argentina 12 Peru 1017 India 4 Estonia 202 Russia 34 Uruguay 12 Yemen 4 Camaroon 187 Pakistan 31 Ukraine 750 U. K. 11 Norway 4 Ghana 159 Australia 29 Slovakia 11 Algeria 495 Italy 4 Tuvalu. Rica 142 Singapore 25 Israel 11 Armenia 4 Costa 131 Greece 23 Yugoslavia 10 Iceland 4 Cuba 488 Germany 119 Colombia 23 Iran 9 Zambia 4 UAE 111 Poland 22 Bulgaria 391 Brazil 4 Tonga 9 Virgin Is. (U. S. ) 109 Netherlands 22 Uzbekistan 9 Uganda 4 W. Samoa 328 S. Korea 107 Thailand 22 Czech Rep. 9 Bosnia & Herz. 4 Tanzania 94 Switzerland 22 N. Korea 306 Taiwan 8 Kenya 3 Syria 328 S. Korea 77 Chile 74 Sweden 57 Austria 2 Sierra Leone 2 Trinidad & Tobago 1 Cambodia 2 Guyana 1 Qatar 2 American Samoa 1 Saint Vincent 2 Andorra 1 Laos 2 Georgia 1 San Marino 2 Cook Islands 1 Libya 2 Turkmenistan 1 Benin 2 Gabon 1 Angola 2 The Gambia 1 Chad 2 Kazakhstan 1 Gibraltar 2 Macau 1 Haiti 2 Malta 1 Guatemala 2 Jamaica 1 Malawi 2 Latvia 1 Equatorial Guinea 2 Turks & Caicos 1 Palau 1 Bhutan 1 Bermuda 21 Lithuania 7 Zimbabwe 3 Bahamas 1 Ascension Island 1 Botswana 21 Croatia 7 Saudi Arabia 3 Ethiopia 1 Cyprus 1 Suriname
17 Overview l The need for Grid technology l Grid technology today l Future challenges & directions l Summary
There’s Much More to Do: For Example, U. S. Dept of Energy Pacific Northwest National Lab • Environmental Molecular Sciences Lab Univ. of IL • Electron Microscopy Center for Materials Research • Center for Microanalysis of Materials Fermi National Accelerator Lab • Tevatron Stanford Linear Accelerator Center • B-Factory • Stanford Synchrotron Radiation Laboratory MIT • Bates Accelerator Argonne National Lab Center • Intense Pulsed Neutron Source • Plasma Science & • Advanced Photon Source Fusion Center • Argonne Tandem Linac Accelerator System Brookhaven National Lab • Relativistic Heavy Ion Collider • National Synchrotron Light Source Lawrence Berkeley National Lab Princeton Plasma Physics Lab • Advanced Light Source • National Center for Electron Microscopy • National Energy Research Scientific Computing Facility Thomas Jefferson National Accelerator Facility • Continuous Electron Beam Accelerator Facility Oak Ridge National Lab Sandia Combustion Research Facility 18 General Atomics - DIII-D Tokamak Los Alamos Neutron Science Center SC Laboratories SC User Facilities Institutions that Use SC Facilities James R. Mac. Donald Laboratory • High-Flux Isotope Reactor Surface Modification & Characterization Center • Spallation Neutron Source (under construction) Physics Accelerators Synchrotron Light Sources Neutron Sources Special Purpose Facilities Large Fusion Experiments
There’s Much More to Do: E. g. , U. S. Dept of Energy l 19 Goal: Any DOE scientist can access any DOE computer, software, data, instrument u u ~1000 instruments** (vs. maybe 10 online? ) u ~1000 scientific applns** (vs. 2 Fusion services) u ~10 PB of interesting data** (vs. 100 TB on ESG) u l ~25, 000 scientists* (vs. ~1000 DOE certs) ~100, 000 computers* (vs. ~3000 on Grid 3) Not to mention many external partners I. e. , we need to scale by 2 -3 orders of magnitude to have DOE-wide impact! * Rough estimate; ** WAG
20 Scaling Up: Service-Oriented Science Domain-dependent Content Expt design Expt output Telepresence monitor Services Domain-independent Simulation code Electronic notebook Portal server Simulation code Simulation server Metadata catalog Certificate authority Data archive Resources Experimental apparatus Servers, storage, networks
Establishing Policy Across Virtual Organizations VO AZA ATA: Attribute Authority AZA: Authorization Authority Subject ATA Subject AZA WS-Subject VO ATA Resource AZA Resource ATA WS Resource GT 4 -enabled services such as CAS, VOMS, Grid. Shib are initial steps in this direction 21
22 Overview l The need for Grid technology l Grid technology today l Future challenges & directions l Summary
23 Enabling e. Science l e. Science methods no longer optional but now vital to scientific competitiveness l We’ve demonstrated feasibility, but now need to scale up l Meanwhile, others are investing heavily u EU: ~$70 M/yr for “Grid” infrastructure, tech u UK: ~$60 M/yr for e. Science apps and tech u Germany, Italy, Netherlands, Japan, China, Korea, Singapore, Australia all have programs How does the U. S. stay competitive?
Acknowledgments l Carl Kesselman and Steve Tuecke, my long-time Globus co-conspirators l Gregor von Laszewski, Kate Keahey, Jennifer Schopf, Mike Wilde, other Argonne colleagues l Globus Alliance members at Argonne, U. Chicago, USC/ISI, Edinburgh, NCSA, PDC l Miron Livny, U. Wisconsin Condor project, Rick Stevens, Argonne & U. Chicago l Other partners in Grid technology, application, & infrastructure projects l DOE Sci. DAC program, NSF Middleware Initiative, NASA, IBM for generous support
25 For More Information l Globus Alliance u l Globus Consortium u l www. ggf. org Open Science Grid u l www. globusconsortium. com Global Grid Forum u l www. globus. org www. opensciencegrid. org Background information u www. mcs. anl. gov/~foster 2 nd Edition www. mkp. com/grid 2
Скачать презентацию Enabling e Science Grid Technologies Today Tomorrow
569bd69a234bbdf7dbc681aea11f215c.ppt