CEOS GRID PROTOTYPING PROJECT Presentation to Internet 2

CEOS GRID PROTOTYPING PROJECT Presentation to Internet 2 Member Meeting Spring 2003 Marriott Crystal Gateway April 9, 2003 Richard des. Jardins (rdesjardins@arc. nasa. gov) NASA Research and Education Network Kenneth Mc. Donald (ken. mcdonald@gsfc. nasa. gov) NASA Goddard Space Flight Center Yonsook Enloe (yonsook@harp. gsfc. nasa. gov) SGT, Inc. Allan Doyle (adoyle@intl-interfaces. com) International Interfaces April 2003 1

Summary of CEOS Grid Presentation & Points of Contact (POCs) • CEOS, WGISS Background – POC: Ken. Mc. Donald@gsfc. nasa. gov • CEOS Grid Prototyping Project – POC: Yonsook Enloe (yonsook@harp. gsfc. nasa. gov) • Network Team – POC: Jeff. Smith@gsfc. nasa. gov • Tech Team – POC: Allan Doyle (adoyle@intl-interfaces. com) • Main Issues – POC: Yonsook Enloe (yonsook@harp. gsfc. nasa. gov) • Applications – – – April 2003 USGS Data Delivery – POC: Stu Doescher (doescher@usgs. gov) NOAA NOMADS – POC: Glenn. Rutledge@noaa. gov ESA Data Integration – POC: Pedro. Goncalves@esa. int GSFC Advanced Data Grid – POC: Debbie. Ladwig@gsfc. nasa. gov GSFC/GMU EOSDIS Data Pools – POC: Liping. Di@gsfc. nasa. gov UAH Scientific Data Mining – POC: Sara Graves (sgraves@itsc. uah. edu) 2

Committee on Earth Observation Satellites (CEOS) • Committee on Earth Observation Satellites (CEOS) consists of world's government agencies responsible for civil Earth observation (EO) satellite programs, along with agencies that receive and process data acquired remotely from space. • Main goal of CEOS is to ensure that critical scientific questions related to Earth observation and global change are covered, and that satellite missions do not unnecessarily overlap each other. • CEOS specific objectives: – Cooperate in mission planning and in development of compatible data products, services, applications, and policies. – Provide focal point for international coordination of space-related EO activities. – Exchange policy and technical information to encourage complementarity and compatibility of observation and data exchange systems. • URL: www. ceos. org • POC: Ken. Mcdonald@gsfc. nasa. gov April 2003 3

CEOS Working Group on Information Systems and Services (WGISS) • • WGISS Goal: To stimulate and coordinate the development of member agency systems and services which manage and supply the data and information from participating agency Earth observation (EO) missions. WGISS specific objectives: – Assist EO data provider agencies to maximize the usefulness, easy access and application of EO data they gather and provide to diverse users worldwide. – Address needs of EO data and information users by aiding development of services that provide simpler and wider access to EO resources. – Emphasis is on using demonstration projects involving user groups to identify and solve critical interoperability issues associated with achievement of global services. • • • April 2003 WGISS has developed tools, standards and services to assist access to and use of EO data resources available online worldwide. URL: wgiss. ceos. org POC: Ken. Mcdonald@gsfc. nasa. gov 4

WGISS - New Structure Current Tasks: WGISS Technology and Services Subgroup Current Tasks: April 2003 International Directory Network CEOS Interoperable Catalog System Data Services Networks Archive EOGEO Workshop GRID Developing Countries CD-ROM CEOS Information Infrastructure WGISS Test Environment Projects and Applications Subgroup Current Tasks: Global Datasets Global Mapping Book WTF CEOP WTF Core Sites (WGCV) 5

CEOS Grid Prototyping Project • Oct 2002 -March 2003: Phase 1 Establish CEOS Grid Technology Core Testbed – Objectives: • Establish an immediate Grid capability base within participating CEOS agencies: – – Grid software Access to existing Grids Pilot applications Knowledgeable people • April 2003 -Sept 2003: Phase 2 Demonstrate CEOS Grid-enabled Applications – Objectives: • Demonstrate Grid-enabled applications, each involving at least two CEOS agency sites. • Show proof of concept. • Evaluate benefits. • Obtain lessons learned from infusion of Grid technologies from the Technology Core into real CEOS agency information systems and applications. April 2003 6

CEOS Grid Prototyping Project, Cont’d. • Fall 2003: Presentation to WGISS for approval to continue project into second year – Objectives: • Report to WGISS on accomplishments and "So what? " from first year. • Present specific proposed project plan and get approval to continue to second year. • Oct 2003 -Sept 2004: Phase 3 Create persistent CEOS Grid within WTFs – Objectives: • Infuse applicable Grid technologies into selected CEOS agency information systems and WTFs, to create a persistent CEOS Grid that would be available to support future CEOS agency initiatives. April 2003 7

CEOS Grid Prototyping Project, Cont’d. April 2003 8

CEOS Grid Prototyping Project, Cont’d. NOAA NCDC USGS EDC UAH April 2003 NASA GSFC ESA ESRIN GSFC/GMU 9

CEOS Grid Prototyping Project Organization Chart Technology and Services SG Wyn Cudlip Existing Grids Network Task Team (TT) Jeff Smith CEOS Grid Prototyping TT Yonsook Enloe negotiated relationships network support Engineering and Technical Allan Doyle USGS Data Delivery NOMADS Stu Doescher ESA Data Integration Glenn Rutledge GSFC/GMU EOSDIS Data Pools Liping Di April 2003 • Information Power Grid • Earth Systems Grid • EU Data Grid and Data TAG Ivan Petiteville GSFC Advanced Data Grid Debbie Ladwig UAH Scientific Data Mining Sara Graves 10

CEOS Grid Prototyping Project Network Team • Led by Goddard Space Flight Center (GSFC), supported by Ames Research Center (ARC) NASA Research and Education Network (NREN). • "Virtual" CEOS Grid Prototyping Network is actually made up of connectivity from several High Performance Research and Education Networks (HPRENs), e. g. , NASA Research and Education Network (NREN), Energy Science Network (ESnet), Internet 2 Abilene, European HPRENs. • Network team works to ensure adequate connectivity between testbed nodes: – Identify connectivity requirements (testbed network map). – Perform network performance testing. – Work to solve specific network connectivity problems as needed. • Developed CEOS Grid Firewall Best Common Practices (BCP) Document: – Working with USGS to implement, test and refine document. • POCs: Jeff. Smith@gsfc. nasa. gov, Dave Hartzell (dhartzell@arc. nasa. gov) April 2003 11

CEOS Grid Prototyping Project Tech Team • Focus is on: – Establishing CEOS Grid Technology Core Testbed, including defining, establishing, extending and documenting a base level of functionality at each participating testbed node and organization. • Summary Technical Work Plan: – Carry out regular technical coordination via weekly telecons, information exchange Web site, and "ceos-grid-tech@harp. gsfc. nasa. gov" email list. – Learn from existing Grid contacts. – Download free Grid software and install in testbed nodes. – Define "template" for specifications and activities needed to configure CEOS Grid core sites and partner sites. – Connect testbed nodes into Initial CEOS Grid Virtual Organization (VO). April 2003 12

CEOS Grid Prototyping Project Tech Team • Summary Technical Work Plan, Continued: – Define and execute core technology interoperability test suite (automated for regular testing and measurement). – Provide CEOS Grid Virtual Organization certificates to participants (certificates will be supplied by NASA Information Power Grid), and help applications negotiate access agreements with existing Grid VOs. – Assist application team leads to interconnect and interoperate their application sites with existing Grid VOs and CEOS partner sites. – Identify representatives to attend: • Global Grid Forum (GGF) Applications and Testbeds Research Group • OGC EO WG and Architecture SIG (which is beginning to focus on Grid) • APAN (Grid WG and Earth Monitoring WG) April 2003 13

CEOS Grid Prototyping Project Main Issues • POC: Yonsook Enloe (yonsook@harp. gsfc. nasa. gov) • CEOS Grid issues: – Six application projects with widely differing application areas. – Many issues are common to all six projects. – Project team is working together to gain insight into these common problems. • Issue 1: Lack of Grid software how-to install and use documents: – Team is producing how-to documentation: Grid Cookbook pages. – 1 st cookbook page: How to install and configure Globus 2. 2. – 2 nd cookbook page: How to install and configure Grid. FTP with multiple hosts and multiple clients. – 3 rd cookbook page (in progress): How to put simple applications on the Grid. • Issue 2: Lack of Grid expertise by participants: – Grid Experts (IPG, . . . ) are acting as consultants on various specialty topics. – Formed Tech Team to help each other and help later participants. April 2003 14

Main Issues, Cont’d. • Issue 3: Most agencies have firewalls. How to deal with these and how to configure to allow access? – Network Team is gathering requirements for firewalls and is drafting a "CEOS Grid Firewall Best Practices" document. – Technical POCs interested in this issue will review document with their firewall administrators and will iterate on the document. • Issue 4: Grid Monitoring: – Network bandwidth performance testing and checkout of network routing is being performed between testbed nodes. – Several Grid Monitoring tools (Map Center, Ganglia, NWS) are being studied and tried out. – Map Center monitoring tool can monitor host machine ports and perform process level monitoring. – CEOS Grid application sites are linking to this tool to try it out (work in progress). April 2003 15

Main Issues, Cont’d. • Issue 5: Certificate Authority (CA): How should host and user certificates be implemented in an international multi-agency consortium? – Small tiger team formed to study issue with Grid expert. – Procedure for CA has been drafted and is being reviewed. – Planning to use certificates from multiple sources (work in progress). • Issue 6: Catalog Issues: Because EO data have huge volumes from many sources, need a product catalog that is searchable and scalable. What kinds of catalog components are available on the Grid and do these components have the necessary capabilities for CEOS catalogs? – Grid experts on SDSC SRB/MCAT and Globus MCS invited to give presentations to entire team. – Catalog Tiger team formed (small team to study and analyze catalog issues and report back to the main team) (work in progress). • Issue 7: Putting EO applications on the Grid: – Small tiger team formed to study this issue and prototype at least one approach. – Prototyping OGC Web Map Server (WMS) and Web Coverage Server (WCS) on the Grid. – Will generate Cookbook page on putting WMS and WCS on the Grid. April 2003 16

Application: USGS Data Delivery • Led by USGS EROS Data Center (EDC) in Sioux Falls SD. • Goal: Explore use of GRID technologies (primarily Grid. FTP and Certificate Authority) for the delivery and reception of earth science data. • Application focus: – Delivery of earth science data from EDC to scientific user community. – Receiving data into the archive from producer/reception sites. • Explore how Grid technologies would replace current technologies being used: – Physical media (tape cartridges, CD/ROMs), primitive network protocols (semi-anonymous FTP and limited FTP push). – Security is major concern. • URL: Web site under development. • POC: Stu Doescher (doescher@usgs. gov) April 2003 17

Application: NOAA NOMADS • NOAA Operational Model Archive and Distribution System (NOMADS) goals: – Develop distributed Grid framework, promoting standards across multiple institutions. – Provide access to climate and numerical weather prediction (NWP) models for analysis and intercomparison. – Foster research within geoscience communities to study complex earth systems using multiple collections of distributed data. • Led by National Climatic Data Center (NCDC), with support from National Center for Environmental Prediction (NCEP), Geophysical Fluid Dynamics Laboratory (GFDL), and over a dozen other major collaborators. • Grid technologies: Grid. FTP, Grid Information Service (GIS), Certificates. • URL: www. ncdc. noaa. gov/oa/climate/nomads. html • POC: Glenn. Rutledge@noaa. gov April 2003 18

Application: ESA Data Integration • Led by European Space Agency (ESA) European Space Research Institute (ESRIN). • Developing Grid Portal for Earth Science Applications Browser: – Interfacing to EU Data. Grid, DOE Earth System Grid, other data warehouses, Open. GIS Consortium (OGC) Web Services (OWS). • Interfaces CEOS interoperability technologies with Grid environments to support on-demand user-driven data integration: – Catalogue Interoperability Protocol (CIP), Web Map Server (WMS), Archive Data Management, Selection and transfer of data, On-demand data product generation, Data product visualization. • HTML user interface implemented using client application with generic functions developed in Java. Script. • URL: giserver. esrin. esa. int • POC: Pedro. Goncalves@esa. int April 2003 19

Application: NASA GSFC Advanced Data Grid (ADG) • Led by NASA Goddard Space Flight Center (GSFC): • – Systems engineering, architecture and implementation support from Aerospace Corporation and GST Inc. – Grid support (Certificate Authority services and Grid resources and services) from NASA Ames Research Center (ARC) Information Power Grid. – Relationship with EOSDIS Data Pools Project. Primary Goals: – Assess scalability of Grid architecture/implementation for Earth Science Data Segment data life cycle management and workflow (primary focus on Data Grid issues, not Compute Grid issues). – Demonstrate realistic science application of relevance to NPP mission (www. jointmission. gsfc. nasa. gov) in fully Grid-enabled environment. • Technologies: – Globus Toolkit – Storage Resource Management: SDSC SRB/MCAT, LBNL SRM, Globus MCS, related tools. – Grid monitoring tools as required (e. g. , ganglia). • Data and Metadata: – Primarily EOS Data (MODIS) from Terra and Aqua Satellites, ECS Metadata Schema. • • April 2003 URL: Web site under development. POC: Debbie. Ladwig@gsfc. nasa. gov 20

Application: NASA GSFC/GMU EOSDIS Data Pools • • Led by NASA Earth Observing System (EOS) Data and Information System (DIS) Project at NASA Goddard Space Flight Center (GSFC), with technology development and testbed at George Mason University (GMU). Goal is to demonstrate integration of Grid and Open. GIS Consortium (OGC) Web Services (OWS): – – – • • April 2003 Provide interoperable, personalized, on-demand data access and services. Initial focus is on the NASA/EOSDIS Data Pools environment at four EOS Distributed Active Archive Centers (DAACs): Goddard Space Flight Center (GSFC), Langley Research Center (La. RC), National Snow and Ice Data Center (NSIDC) at University of Colorado at Boulder, EROS Data Center (EDC). Technology development site is at GMU Laboratory for Advanced Information Technology and Standards (LAITS). Integrate NASA HDF-EOS (EOSDIS standard data format) Web GIS Software Suite (NWGISS), which provides OGC web map, coverage and registries services, with Grid technologies which provide security, resource access and management, Grid information/monitoring, data access/transfer. Work with Grid teams at Argonne National Laboratory (ANL) and NASA Ames Research Center to make Globus geospatial enabled and OGC interface compatible. URL: laits. gmu. edu POC: Liping. Di@gsfc. nasa. gov 21

Application: UAH/NSSTC Scientific Data Mining • • Led by University of Alabama in Huntsville (UAH) Information Technology and Systems Center (ITSC) data mining tools using Earth Science data from the National Space Science and Technology Center (NSSTC) and other data centers Explore use of Grid software tools and resources for compute-intensive data mining and machine learning applications in the earth sciences: – – • • • April 2003 Investigate Grid-enabled data mining issues, e. g. , Grid resource monitoring and intelligent scheduling, to manage distributed data and compute resources in support of scientific data mining. Science focus is on developing supervised classifier of storm characteristics to identify dangerous storms with potential for heavy lightning. Leverage substantial UAH data mining expertise and software. Leverage ITSC testbed for NSF Middleware Initiative (NMI), to provide visibility into NMI for CEOS Grid developers, and to provide earth science and spatial data requirements and feedback to NMI middleware development and support team. Grid technologies: Globus Toolkit (Globus Packaging Technology (GPT), Grid Resource Information Service (GRIS), Grid. Resource Allocation Manager (GRAM), Grid. FTP, Monitoring and Discovery System (MDS), Grid Security Infrastructure (GSI)), Network Weather Service (NWS), Condor-G. URL: www. itsc. uah. edu/about. html POC: Sara Graves (sgraves@itsc. uah. edu) 22