NMFS Enterprise Data Management DAARWG Meeting December 8, 2010 Jim Sargent, NOAA Fisheries Information Architect 1 NMFS Enterprise Data Management
The End in Mind: Take Aways • • Need to make quality data available to the world It will take a cultural shift - Which is HARD! NMFS EDM: One LO’s approach • Understand the comprehensive enterprise-wide process we have been through, and • Possibly leverage the fruits of our labors and learn from challenges DAARWG recommendations: – Expand scope beyond Access and Archive Requirements to full Data Management Life Cycle – Promote and support EDMC and LO data management efforts – Leverage similarities while respecting diversity of data – Review and support EDMC’s Procedural Directives – Consider: • Adopting a new vision of data management • Doing a comprehensive inventory NMFS Enterprise Data Management
Outline • A Case for a New Vision for Data and Data Management • One LO’s approach: NMFS EDM • Recommendations NMFS Enterprise Data Management
NMFS Enterprise Data Management
Deep Water Horizon Data Management Challenges Immediate Response • Need to quickly identify applicable / available data Pre and Post Disaster • • • Volume and diversity of data Collected for one-use; potential for multi-use Infrastructure needed to support multiple access mechanisms Releasability of data (organizational vs. technical) Coordination of collection and distribution Comparison and utilization of modeling outputs and observations Overall Need for: • Clear and consistent data management policy • Better data documentation (metadata) • An overarching response plan Prepared by Environmental Data Management Committee for NOAA Leadership NMFS Enterprise Data Management 5
It’s the Next Disaster, Do We Even Know What Data We Have? NMFS Enterprise Data Management
Flashback “NOAA is like a library without any card catalog …or even bookshelves” ……. . Dr. John A. Knauss, Former NOAA Administrator, 1986 NMFS Enterprise Data Management
Fast Forward to 2010: We Need to Share Our Data • President's Directive on Open Government – Transparency, participation, and collaboration – Timely publication of quality information • It’s the right thing to do! • Greater than archive and preservation issues – Not all access is done from archives NMFS Enterprise Data Management
Everyone is Wresting With This • Other federal agencies – IWGDD, NSF, ICES, ISO • DM maturing as a discipline – DAMA DM Book of Knowledge (DMBOK) – Certified Data Management Professionals • H. R. 5037 April 15, 2010 – Requires Federal agencies to develop public access policies • NMFS Science Center Data Management Reviews NMFS Enterprise Data Management
It’s About a Cultural Shift • Need to move from the academic paradigm – Publish or Parish Share or Parish – Science Information – Science Quality good enuf • Changing Culture is Hard – Tipping Point, Blink …. Malcolm Gladwell – Switch, Made to Stick … Phil and Dan Heath NMFS Enterprise Data Management
A New Vision? ? NOAA data assets are recognized and managed as a core agency resource, on par with financial and human resources. 11 NMFS Enterprise Data Management
One NOAA Line Office’s Approach…. NMFS Enterprise Data Management (EDM) Program NMFS Enterprise Data Management
Leadership Science Board NMFS Staff • Analysis & Research • Recommendations: • Policies • Procedural directives • Guidelines • Best practices • Implementation • Marketing NMFS Staff Approvals and support • Policies; • Procedural directives • Guidelines • Best practices FIMAC 13 NMFS Enterprise Data Management
FIMAC Workshop • Teams’ Plans developed • Required Resources Identified • Recommendations presented to and accepted by LC • FIMAC Created Data Documentation Inventory Procedure Directive Initiated drafted and vetted Policy drafted and vetted Jan 15, 2009 IA Position Filled • Teams Created • IMCs Created FIMC researched NMFS DM and developed EDM Recommendations Research, Analysis, and Recommendations 2008 Data Stewardship Teams Established Planning and Development of Teams 2009 NMFS Data Catalog established Policy Enacted A Brief History of EDM Time Implementation 2010 14 NMFS Enterprise Data Management
NMFS EDM MISSION: “To effect a cultural change in which all NMFS data are recognized and managed as a core agency resource, on par with financial and human resources. ” 15 NMFS Enterprise Data Management
NMFS EDM Vision NMFS customers can confidently find, access, and use our data 16 NMFS Enterprise Data Management
NMFS EDM Vision Internal and external constituents NMFS customers can confidently find, access, and use our data 17 NMFS Enterprise Data Management
NMFS EDM Vision Confidence in finding and trust in using our data NMFS customers can confidently find, access, and use our data 18 NMFS Enterprise Data Management
Using various NMFS portals, data. gov, geospatial. gov, etc) • Browse through an ordered hierarchies or taxonomonies • Search EDMusing: specific key words (controlled vocabularies) Vision • Discipline • • User tags (folksonomies) • Metadata include users’ comments • Minimize the number mouse clicks NMFS customers can confidently find, access, and use our data 19 NMFS Enterprise Data Management
NMFS EDM Vision Through confidentiality and security filters while using standard tools and formats NMFS customers can confidently find, access, and use our data 20 NMFS Enterprise Data Management
NMFS EDM Vision with Download selected data sufficient documentation, including quality indicators and warnings to effectively and properly use and understand the data NMFS customers can confidently find, access, and use our data 21 NMFS Enterprise Data Management
NMFS EDM Vision All NMFS enterprise data we choose to share NMFS customers can confidently find, access, and use our data 22 NMFS Enterprise Data Management
NMFS EDM Vision NMFS customers can confidently find, access, and use our data 23 NMFS Enterprise Data Management
NMFS Enterprise Data And Information Management Policy Overview A high level NMFS Policy Directive implemented by Operational Procedure Directives • Enacted by Eric Schwaab: June 2010 – • Directed RAs, SDs, and ODs to send 2 -3 people to NMFS Data Stewardship Workshop General Policy: All data shall: – – – be visible, accessible and understandable to authorized users; modeled, named and defined consistently across and within all NMFS programs; have a standard set of metadata; and be managed, controlled and shared by data stewards throughout the data management lifecycle. be publicly available generally within one-year of its collection 24 NMFS Enterprise Data Management
Procedural Directives Principles/Concepts • Developing Procedural Directives an iterative process • Data Documentation Procedural Directive: FMCs shall: – Develop their own plans for documenting and sharing data assets – Inventory and document data assets and tools in NMFs metadata repository, In. Port – Measure metadata quality with a rubric FY 11 will be a year of learning and practicing how to document and share our data • Develop, use, and refine metadata standards • Populate In. Port with discovery level metadata • Develop practice and refine procedures for sharing 25 NMFS Enterprise Data Management
Data Stewardship Teams A Scientist, A Data Librarian A Data Manager 26 NMFS Enterprise Data Management
NMFS Data Stewardship Workshop To develop a shared understanding of data stewardship, empower the data stewardship teams to lead the way for implementing the Data Documentation Procedure Directive, to develop best practice based on the experience during the initial documentation and inventory tasks. • Discuss New Data Management Policies and procedures • Develop best practices – How to document our data • e. g. , quality assessment – Spirals in data documentation – Rubrics and metrics • Q&A and Sidebar Communities 27 NMFS Enterprise Data Management
Next Steps • • Data Doc. PD approved – Dec 2010 Data Inventory Complete – Dec 2010 Data Doc. Implementation Plans Due- Feb 2011 Data Stewardship Workshop – Apr 2011 First cut Best Practices – Apr 2011 Data Doc. Implementation Plans Finalized – May 2011 Address Preservation and Terminology Socialize, Support, Promote, and Market NMFS Enterprise Data Management
Critical Success Factors • Management Commitment • Dedicated, Personally Committed Team – – – FIMAC Coordination Team EDM Partners Steady Hand at the Helm Executive Sponsorship • Drive Up and Then Drive Down – Harness the mavens • Socialize, Support, Promote, and Market 29 NMFS Enterprise Data Management
Challenges • Tipping Point almost reached but not there yet • Budget Shortfall – Perception that field not fully bought into it • Overworked Teams a high risk NMFS Enterprise Data Management
DAARWG Recommendations • Expand scope beyond Access and Archive Requirements to full Data Management life Cycle • Support cultural change needed to meet demand for quality data • Promote and Support EDMC and LO DM Efforts • Review and Support Procedural Directives • Leverage similarities while understanding diversity – Establish good data management practices for the whole data management lifecycle • Consider Recommending – A new Vision of Data Management – Conducting a comprehensive inventory with defined metadata NMFS Enterprise Data Management
32 NMFS Enterprise Data Management
33 NMFS Enterprise Data Management
Backup Slides 34 NMFS Enterprise Data Management
35 NMFS Enterprise Data Management
Data Sharing “Data should be made as widely and freely available as possible while safeguarding the privacy of participants, and protecting confidential and proprietary data” • Data shall be shared in data. gov, as appropriate, as a oneclick data asset, a one-click product, or by using a software tool (e. g. , FOSS) • Sufficient documentation to understand the data being shared must be published in In. Port and referred to or provided with the data • Data Stewards decide what data is appropriate for sharing • Sharing confidential data must conform to Agency policy 36 NMFS Enterprise Data Management
Timelines for Sharing Types of Data Does not include provisional, predecisional documents and preliminary analyses leading to final management actions 37 NMFS Enterprise Data Management
Principles for Effective Environmental Data Management 1. Data should be archived and accessible 2. Adequate resources for end-to-end management 3. Management activities should involve users 4. Interagency and international partnerships 5. Metadata are essential 6. Expert stewards required for management 7. Process to decide what data to archive 8. Archive must support discovery, access, and integration 9. Effective management requires a formal, ongoing planning process Prepared by Environmental Data Management Committee for NOAA Leadership National Research Council Committee on Archiving and Accessing Environmental and Geospatial Data at NOAA, 38 2007 NMFS Enterprise Data Management
Issues Identification Identifying the Problem The FIMC identified 12 key issues and interviewed their management to determine their priorities • Top priority Issues – – – Critical gaps No authoritative data inventory Insufficient metadata Data quality and consistency challenges Ability to integrate data Administrative systems • Other issues – – Data being lost Data not being archived for perpetuity Historical data that need rescuing Communications re: applications and IT return the Interestingly, ing lowest rank at issue was thot NMFS did n have buy-in r fo across FMCs. IM addressing 39 NMFS Enterprise Data Management
Issues Identification Identifying the Problem The FIMC identified 12 key issues and interviewed their management to determine their priorities • Top priority Issues – – – Critical gaps No authoritative data inventory Insufficient metadata Data quality and consistency challenges Ability to integrate data Administrative systems • Other issues – Data being lost – Data not being archived for perpetuity – Historical data that need rescuing – Communications re: applications and IT return the Interestingly, ing lowest rank at issue was thot NMFS did n have buy-in r fo across FMCs. IM addressing 40 NMFS Enterprise Data Management
NOAA’s Conceptual Framework 41 NMFS Enterprise Data Management
DAMA-DMBOK Functional Framework 42 NMFS Enterprise Data Management
Скачать презентацию NMFS Enterprise Data Management DAARWG Meeting December 8
d4a645795cc568f404bd171ecb768975.ppt