Скачать презентацию DAAC Technology Snapshot Tom Kalvelage Land Processes DAAC Скачать презентацию DAAC Technology Snapshot Tom Kalvelage Land Processes DAAC

347b1d5bd15881ee4536757a0c614a1c.ppt

  • Количество слайдов: 24

DAAC Technology Snapshot Tom Kalvelage Land Processes DAAC Manager For The DAACs ESISS, February DAAC Technology Snapshot Tom Kalvelage Land Processes DAAC Manager For The DAACs ESISS, February 17, 2004 1

Objective of the presentation: • Present lessons learned from the collective experience of the Objective of the presentation: • Present lessons learned from the collective experience of the DAACs that may be relevant with respect to the Data and Information Management Plan. • Present experience with technology and its impacts on the DAACs. 2

ESE Data Centers NSIDC Cryosphere Polar Processes ASF SAR Products Sea Ice Polar Processes ESE Data Centers NSIDC Cryosphere Polar Processes ASF SAR Products Sea Ice Polar Processes SEDAC LP DAAC-EDC Land Processes & Features Human Interactions in Global Change GES DAAC-GSFC Upper Atmosphere Atmospheric Dynamics, Ocean Color, Hydrology Global Biosphere, Radiance Data ASDC-La. RC Radiation Budget, Clouds Aerosols, Tropospheric Chemistry PODAAC-JPL Ocean Circulation Air-Sea Interactions GHRC Hydrologic Cycle & Severe Weather ORNL Biogeochemical Dynamics EOS Land Validation 3

Background The DAACs are located at host institutions that care about the user communities Background The DAACs are located at host institutions that care about the user communities their DAAC serves. Each DAAC is unique, but together we are fully operational on a large scale serving all users. The DAACs use a variety of systems to help users. • Cross-DAAC systems like the EOSDIS Core System (ECS) and EOS Data Gateway. • Local DAAC systems like La. TIS, TRMM Support System, “Version 0” systems, ASTER Browse Tool, Mercury, etc. • Many ‘add-ons’ to existing systems (ECS and non-ECS). • Tools and scripts both online and delivered to users. In addition to systems; we also work with the science and applications communities to ensure their needs are met: • Users via User Working Groups, Science Investigator-led Processing Systems (SIPS), REASo. N CAN members, field campaigns, universities, applications community, education, national and international research projects and so on. 4

Lessons Learned While the DAACs are very diverse and each has its own particular Lessons Learned While the DAACs are very diverse and each has its own particular work, experience, and viewpoint, collectively there are some lessons learned across the DAACs that we believe may be relevant. 5

Developer Interaction with Operations Having DAAC Operations work closely with developers has contributed to Developer Interaction with Operations Having DAAC Operations work closely with developers has contributed to data system success. • User Services made sure new capability would satisfy users. • Operators provided realistic ops concepts and requirements. • Systems were more likely to be useable and/or efficiently operable as delivered. • Sustaining engineers provided installation and maintenance advice and requirements. • Integration and test has taken less time and requires less rework (i. e. “measure twice, cut once”) to get from integration, through test, to operations. • Ensured that the architecture and interfaces are flexible enough to be maintained and upgraded in the future. On-site developers from off-site development contractor can help, but care must be taken to avoid impacting operations or development. 6

DAAC Engineering/Development Capability A DAAC on-site engineering and development capability has significantly contributed to DAAC Engineering/Development Capability A DAAC on-site engineering and development capability has significantly contributed to mission success. It has: • enabled the DAACs to evaluate and/or implement new information technology, helping keep data systems current and improving services for users. • provided needed capability via scripts, tools, and subsystems when delivered systems (COTS or GFE) fall short. • This includes tools for users (e. g. , subsetting, reformatting) and for operations (e. g. , data management). • helped successfully integrate multiple delivered and developed systems (e. g. , EOSDIS Core System, EOS Data Gateway, Product Distribution System, etc). • provided improved capability to deploy advanced information technology into an operational or near-operational environment for more rapid feedback on operations readiness. 7

User Services (including order service and technical support) has significantly contributed to mission success. User Services (including order service and technical support) has significantly contributed to mission success. • User Services has worked best when it is an integral part of the DAAC, able to: • influence the engineering and operations of the DAAC on behalf of the users, and • educate the users about the DAAC’s engineering and operations. • User Services has been essential to explain to users the user search and order interfaces, data formats, data content, algorithms, applicability of the data, and potential ways of using the data. • Informal outreach performed by User Services has been instrumental in making the wider user community aware that the data are even available. 8

Big Systems vs. Little Systems Capability can be implemented in a wide spectrum of Big Systems vs. Little Systems Capability can be implemented in a wide spectrum of ways, from doing everything in one large system (one-size-fits-all) to doing each small thing in its own small system (one-size-fits-one). • Experience shows that doing 100% in one system doesn’t work due to significant inertia and cost. • Anything less than 100% (one-size-fits-most) will work, but collective experience does not tell us the best solution or equilibrium point – its different for each situation. Examples: • After evaluation, half of the DAACs did not use the big ECS system; that turned out to be a good choice. • A user interface providing some functionality to all data sets allowed unique work to be done by unique user interfaces. • Moving unique processing out of ECS to smaller systems, reduced the effort required to address unique requirements. • Supporting multiple missions with a ‘one-size-fits-most’ big system (e. g. , ECS) allowed us to have more mature systems and processes and less risk when new missions started up. 9

Systems In General Having worked with a number of systems, in general we have Systems In General Having worked with a number of systems, in general we have some lessons learned about them. • Systems designed with the maximum amount of COTS software (e. g. , ECS) do have a lot of functionality. However, complex COTS and hardware interdependencies combined with planned obsolescence cause significant integration work. • Systems that are not designed to be automated are very difficult to automate later. • Systems are more operable, and less costly to operate, if operated at or below specified performance levels (margin). • Regarding standards and formats, in general: • It’s best to use existing standards and formats. – Data in existing standards and formats were used, and did not require tool development. – Data in new standards and formats were not used, until tools were developed. • Translators are generally preferable to new tools. 10

Data Distribution on Hard Media Users continue to demand data distributed on hard media. Data Distribution on Hard Media Users continue to demand data distributed on hard media. • While hard media distribution is expected to decline, at least partially due to improved networks, several types of users have resisted this trend: • Users with poor or expensive network connections (such as international users). • Users with limited local storage capacity. • Users who just want the data on media. 11

Communications Maintaining good communication and working closely together amongst the DAACs has been productive. Communications Maintaining good communication and working closely together amongst the DAACs has been productive. • The DAACs began meeting together in the early 1990 s. • The DAACs were advised to form an alliance by a 1998 National Research Council Review, and did so. • The DAACs have balanced working independently and together: • The DAACs have very different user communities, measurements, products, environments, and cultures. • However, user service, data stewardship, and information technology are common to all DAACs. • Agreement is by consensus; difficult sometimes, but rewarding and worthwhile. 12

Information Technology Observations Hardware • The price/performance trend for utilizing commodity CPUs shows favorable Information Technology Observations Hardware • The price/performance trend for utilizing commodity CPUs shows favorable results. A real transitional jump will be when idle desktop PCs and disk can be used for science processing. • With disk and tape price/performance improving, full archive on-line disk storage with necessary backup will become more feasible. • WAN performance increases will further enable distributed EOSDIS data system architecture. Software • Database usage continues to be limited by throughput performance • Grid computing is certainly worth pursuing to determine its real applicability. • Affordable mass storage will facilitate a proliferation of ‘personally’ customized datasets. • Automated agents, already being utilized, will find additional areas for implementation (e. g. , ordering data and performing data services). Translators versus standards • Standard formats are necessary but not always enforceable. • Translators between standards need to be implemented. 13

Information Technology Observations Database Management • Methods of managing large quantities of both metadata Information Technology Observations Database Management • Methods of managing large quantities of both metadata and data need to be examined with long term archiving in mind. Data Migration • Data need to be migrated regularly to be preserved. • Plans for migration need to be an integral part of all data management schemes, not an afterthought. Longer term observations • We will need to better understand the relationship between data ownership, intellectual property rights, and really longterm preservation. • The latter is likely to require much more geographically dispersed replication of data than we have now - and will require automated migration from technology to technology in order to deal with technological obsolescence 14

Some Things That Keep Us Up At Night Examples: • Technology Fads - reaching Some Things That Keep Us Up At Night Examples: • Technology Fads - reaching the right and affordable balance between innovation and experience. • Additional infrastructure requirements (e. g. , security, statistics) that strain fixed resources. • Maintaining a highly competent IT staff. • Timely receipt of data and documentation from field campaigns. • Achieving metadata management and ‘harmony’ across datasets and user communities, including conversion between the many metadata standards used across the sciences. • Long term data stewardship and orphaned data sets. 15

Some Things That Get Us Excited Examples: • The reuse/adaptability of new technologies including Some Things That Get Us Excited Examples: • The reuse/adaptability of new technologies including new webbased visualization tools that enhance the use and usefulness of data. • Completing an implementation that the user community gets excited about (service, data access tools) because they find it useful. • Developing and implementing concepts and technologies including open standards that enable users to extract information from data, and knowledge out of information. • Collaborations to develop new products and services which employ maturing data management and advancing information technology. 16

Backup Slides 17 Backup Slides 17

Staff Functions Definitions • Operations: (data system operators, some on shift) • Sustaining Engineering: Staff Functions Definitions • Operations: (data system operators, some on shift) • Sustaining Engineering: (installs, tests software and in some cases hardware, maintains and tunes data system; e. g. , data base administration, systems administration of hardware and systems software resources) • Development Engineering: (designs, plans, and develops replacement systems and subsystems) • User Services: (interface to users, support in filling orders, answering user questions, DAAC outreach to users) • Mission (Science Data) Support: (interface to science data providers, DAAC-internal data set coordination, maintenance of data set documentation) • Management: overall coordination and resource planning 18

Functional Mix: FY 2000 and FY 2004 FY 2000 FY 2004 MANAGEMENT 8% MANAGEMENT Functional Mix: FY 2000 and FY 2004 FY 2000 FY 2004 MANAGEMENT 8% MANAGEMENT 7% OPERATIONS 30% MISSION (SCIENCE DATA) SUPPORT 25% USER SERVICES 12% DEV ENGR 7% SUS ENGR 19% OPERATIONS 24% MISSION (SCIENCE DATA) SUPPORT 23% USER SERVICES 13% SUS ENGR 24% DEV ENGR 8% 19

Functional Mix: Overall 2000 vs 2004 Each DAAC provided percentages for their FTE by Functional Mix: Overall 2000 vs 2004 Each DAAC provided percentages for their FTE by functional group. 20

Networks Observations • DAACs have at least 100 mb connectivity to their users with Networks Observations • DAACs have at least 100 mb connectivity to their users with most in the gigabit range • ASF currently has 10 mb connectivity • Internal DAAC networks are 100 mb or faster, with most at gigabit speeds 21

COTS in FY 2004 Examples of COTS packages used at DAACs. This list is COTS in FY 2004 Examples of COTS packages used at DAACs. This list is not comprehensive. 22

Acronyms ASDC ASF ASTER Atmospheric Sciences Data Center Alaska SAR Facility Advanced Spaceborne Thermal Acronyms ASDC ASF ASTER Atmospheric Sciences Data Center Alaska SAR Facility Advanced Spaceborne Thermal Emission and Reflection Radiometer CAN Cooperative Agreement Notice COTS Commercial Off the Shelf CPU Computer Processing Unit DAAC Distributed Active Archive Center DBMS Data Base Management System ECS EOSDIS Core System EDC EROS Data Center EOS Earth Observing System EOSDIS EOS Data and Information System ESE Earth Sciences Enterprise FDDI Fiber Distributed Data Interconnect FTE Full Time Equivalent FY Fiscal Year GES DAAC GSFC Earth Sciences DAAC GFE Government Furnished Equipment 23

Acronyms, continued GHRC GIS GSFC HIPPI IT La. RC La. TIS LP DAAC NSIDC Acronyms, continued GHRC GIS GSFC HIPPI IT La. RC La. TIS LP DAAC NSIDC ORNL PC PO DAAC REASo. N SEDAC SIPS SW TRMM WAN Global Hydrology Resource Center Geographic Infromation System Goddard Space Flight Center High-Performance Parallel Interface Information Technology Langley Research Center Langley TRMM Information System Land Processes DAAC National Snow and Ice Data Center DAAC Oak Ridge National Laboratory DAAC Personal Computer Physical Oceanography DAAC Research, Education, and Applications Solutions Network Socioeconomic Data and Applications Center Science Investigator-led Processing System Software Tropical Rain Measurement Mission 24 Wide Area Network