Some Grid Experiences Laura Pearlman USC Information Sciences

Some Grid Experiences Laura Pearlman USC Information Sciences Institute ICTP Advanced Training Workshop on Scientific Instruments on the Grid *Most of these slides are from Lee Liming’s Globus. World 2006 presentation “A Globus® Primer: What is the Grid and How Do I Use It? ”

To be Covered I. Grid computing problems II. Some notable U. S. Grids III. How grids are built and used in real life Globus. WORLD 2006 Globus Primer 2

Grid Computing Problems l Scientific problems that are big enough that they require people in several organizations to collaborate and share computing resources, data, and instruments. u u u u u Interactive simulation (climate modeling) Very large-scale simulation and analysis (galaxy formation, gravity waves, battlefield simulation) Engineering (parameter studies, linked component models) Experimental data analysis (high-energy physics) Image and sensor analysis (astronomy, climate study, ecology) Online instrumentation (microscopes, x-ray devices, etc. ) Remote visualization (climate studies, biology) Engineering (large-scale structural testing, chemical engineering) Biomedical applications Globus. WORLD 2006 Globus Primer 3

Some Core Problems - Heterogeneity l l l l Different authentication mechanisms across institutions Different mechanisms for monitoring system and application status across institutions Different ways to submit jobs Different ways to store & access files and data Different ways to keep track of data Different preferences in programming languages and environments Difficulty in tracking the causes of failures Conflicting requirements among groups that need to interoperate Globus. WORLD 2006 Globus Primer 4

Some Core Problems - Trust l Rigid use policies (authorization, Qo. S) vs rigid application assumptions. l Authorization needs to happen at many levels (communities, organizations, resource owners, etc. ). l Complicated social structures exceed the abilities of simple authorization systems. Globus. WORLD 2006 Globus Primer 5

Some Real-World Grids Globus. WORLD 2006 Globus Primer 6

Earth System Grid Goal: Give climate scientists easier access to the distributed data and resources that they require to perform their research. Developed new technologies for (1) creating and operating "filtering servers" capable of performing sophisticated analyses, and (2) delivering results to users. Globus. WORLD 2006 Globus Primer 7

Collaborative Engineering: NEES U. Nevada Reno www. neesgrid. org Globus. WORLD 2006 Globus Primer 8

NSF’s Tera. Grid* l. Tera. Grid DEEP: Integrating NSF’s most powerful computers (60+ TF) u 2+ UC/ANL PU NCSA PSC IU ORNL UCSD UT l. Tera. Grid WIDE Science Gateways: Engaging Scientific Communities u 90+ A National Science Foundation Investment in Cyberinfrastructure $100 M 3 -year construction (2001 -2004) $150 M 5 -year operation & enhancement (2005 -2009) * Slide courtesy of Ray Bair, Argonne National Laboratory Globus. WORLD 2006 PB Online Data Storage u. National data visualization facilities u. World’s most powerful network (national footprint) Globus Primer Community Data Collections u. Growing set of community partnerships spanning the science community. u. Leveraging NSF ITR, NIH, DOE and other science community projects. u. Engaging peer Grid projects such as Open Science Grid in the U. S. as peer Grids in Europe and Asia-Pacific. l. Base Tera. Grid Cyberinfrastructure: Persistent, Reliable, National u. Coordinated distributed computing and information environment u. Coherent User Outreach, Training, 9

Open Science Grid l $30 M over five years for effort to u sustain and evolve the distributed facility, u bring on board new communities & capabilities, u educate & train. l OSG hardware resources, applications and many other contributions come from OSG consortium members. l OSG technical work is performed together with collaborators & external projects l OSG has partners in Africa, Asia, Europe, North and South America. Text for this slide courtesy of Ruth Pordes Globus. WORLD 2006 Globus Primer 10

OSG Partners l l l l Autralian Partnerships for Advanced Computing (APAC) Data Intensive Science University Network (DISUN) Enabling Grids for E-Scienc. E (EGEE) Grid Laboratory of Wisconsin (GLOW) Grid Operations Center at Indiana University Grid Research and Education Group at Iowa (GROW) Nordic Data Grid Facility (Nordu. Grid) Northwest Indiana Computational Grid (NWICG) New York State Grid (NYSGrid) (in progress). Tera. Grid Texas Internet Grid for Research and Education (TIGRE) TWGrid (from Academica Sinica Grid Computing) Worldwide LHC Computing Grid Collaboration (WLCG) Slide courtesy of Ruth Pordes, OSG All Hands Meeting 2007 Globus. WORLD 2006 Globus Primer 11

OSG Snapshot 100 Resources across production & integration infrastructures Using production & research networks !ncrease in ~15 since Seattle Sustaining through OSG submissions: Measuring ~180 K CPUhours/day. ~Factor of 50% more (being measured) than in Seattle ~20, 000 cores (from 30 to 4000 cores per cluster) ~6 PB accessible Tapes ~4 PB Shared Disk 27 Virtual Organizations (+ 3 operations VOs) 25% non-physics. Slide courtesy of Ruth Pordes, OSG All Hands Meeting 2007 Globus. WORLD 2006 Globus Primer 12

MEDICUS Picture courtesy of Stephan Erberich Globus. WORLD 2006 Globus Primer 13

II. How Grids are Built and Used

Methodology l Building a Grid system or application is currently an exercise in software integration. u u u u l Define user requirements Derive system requirements or features Survey existing components Identify useful components Develop components to fit into the gaps Integrate the system Deploy and test the system Maintain the system during its operation This should be done iteratively, with many loops and eddies in the flow. Globus. WORLD 2006 Globus Primer 15

What End Users Need Secure, reliable, ondemand access to data, software, people, and other resources (ideally all via a Web Browser!) Globus. WORLD 2006 Globus Primer 16

How it Happens Web Browser Compute Server Simulation Tool Web Portal Registration Service Data Viewer Tool Chat Tool Credential Repository Telepresence Monitor Compute Server Camera Database service Data Catalog Certificate authority Users work Application services Collective services with client organize VOs & enable aggregate &/or applications access to other services virtualize resources Globus. WORLD 2006 Globus Primer Database service Resources implement standard access & management interfaces 17

How it Happens l Implementations are provided by a mix of Application-specific code u “Off the shelf” tools and services u Tools and services from the Grid community (Globus + others using the same standards) u l Glued together by… Application development u System integration u Globus. WORLD 2006 Globus Primer 18

The Importance of Community l All Grid technology is evolving rapidly. u u l Community is important! u u l Web services standards Grid interfaces Grid implementations Grid resource providers (ASP, SSP, etc. ) Best practices (OGF, OASIS, etc. ) Open source (Linux, Axis, Globus, etc. ) Application of community standards is vital. u u u Increases leverage Mitigates (a bit) effects of rapid evolution Paves the way for future integration/partnership Globus. WORLD 2006 Globus Primer 19