Introduction Context for the Week William Michener LTER

Introduction: Context for the Week William Michener LTER Network Office Department of Biology University of New Mexico 1

Ecological Informatics A broad interdisciplinary science that incorporates both conceptual and practical tools for the understanding, generation, processing, and propagation of ecological data and information. 2

Eco. Informatics 1. Basic infrastructure • • • Hardware/software Security/archival solutions/backup solutions Wireless communication 2. Concepts 3. Activities • • Data documentation (metadata) Database design and creation • Ecological and environmental databases • Taxonomic and biodiversity databases QA/QC • Analysis and visualization Data & information propagation and discovery (web) 3

Eco. Informatics 1. Basic infrastructure • • • Hardware/software – Murillo (Monday) Security/archival solutions/backup solutions – Murillo (Monday) Wireless communication 2. Concepts – Michener (Monday) 3. Activities • • Data documentation (metadata) – Michener & Romanello (Monday) Database design and creation • Ecological and environmental databases • Taxonomic and biodiversity databases QA/QC • Analysis and visualization Data & information propagation and discovery (web) 4

Eco. Informatics 1. Basic infrastructure • • • Hardware/software – Murillo (Monday) Security/archival solutions/backup solutions – Murillo (Monday) Wireless communication 2. Concepts – Michener (Monday) 3. Activities • • Data documentation (metadata) – Michener & Romanello (Monday) Database design and creation – Porter (Tuesday) • Ecological and environmental databases – Porter (Tuesday) • Taxonomic and biodiversity databases QA/QC – Vanderbilt (Tuesday) • Analysis and visualization Data & information propagation and discovery (web) 5

Eco. Informatics 1. Basic infrastructure • • • Hardware/software – Murillo (Monday) Security/archival solutions/backup solutions – Murillo (Monday) Wireless communication – Porter & Vande Castle (Wednesday) 2. Concepts – Michener (Monday) 3. Activities • • Data documentation (metadata) – Michener & Romanello (Monday) Database design and creation – Porter (Tuesday) • Ecological and environmental databases – Porter (Tuesday) • Taxonomic and biodiversity databases – Garneau (Wednesday) QA/QC – Vanderbilt (Tuesday) • Analysis and visualization – Vanderbilt (Wednesday) Data & information propagation and discovery (web) – White (Wed. ) 6

Science Environment for Ecological Knowledge (SEEK) • Large, 5 -yr, multi-investigator ITR project • Fundamental improvements for researchers – Global access to ecologically relevant data – Rapidly locate and utilize distributed computation – Capture, reproduce, extend analysis process • Informatics Training 7

SEEK Overview 8

Semantic Mediation • • Label data with semantic types Label inputs and outputs of analytical components with semantic types Data • Ontology Workflow Components Use reasoning engines to generate transformation steps – Beware analytical constraints • Use reasoning engine to discover relevant components 9

Ecological ontologies • • What was measured (e. g. , biomass) Type of measurement (e. g. , Energy) Context of measurement (e. g. , Psychotria limonensis) How it was measured (e. g. , dry weight) 10

Scientific workflows EML provides semi-automated data binding 11 Scientific workflows represent knowledge about the process; AMS captures this knowledge

Eco. Informatics 1. Basic infrastructure • • • Hardware/software – Murillo (Monday) Security/archival solutions/backup solutions – Murillo (Monday) Wireless communication – Porter & Vande Castle (Wednesday) 2. Concepts – Michener (Monday) 3. Activities • • Data documentation (metadata) – Michener & Romanello (Monday) Database design and creation – Porter (Tuesday) • Ecological and environmental databases – Porter (Tuesday) • Taxonomic and biodiversity databases – Garneau (Wednesday) QA/QC – Vanderbilt (Tuesday) • Analysis and visualization – Vanderbilt (Wednesday) Data & information propagation and discovery (web) – White (Wed. ) 4. SEEK • • • Overview Scientific workflows and analytical pipelines Ontologies and semantic mediation 5. Education – incorporating ecoinformatics into courses 12

Eco. Informatics 1. Basic infrastructure • • • Hardware/software – Murillo (Monday) Security/archival solutions/backup solutions – Murillo (Monday) Wireless communication – Porter & Vande Castle (Wednesday) 2. Concepts – Michener (Monday) 3. Activities • • Data documentation (metadata) – Michener & Romanello (Monday) Database design and creation – Porter (Tuesday) • Ecological and environmental databases – Porter (Tuesday) • Taxonomic and biodiversity databases – Garneau (Wednesday) QA/QC – Vanderbilt (Tuesday) • Analysis and visualization – Vanderbilt (Wednesday) Data & information propagation and discovery (web) – White (Wed. ) 4. SEEK • • • Overview – Michener (Thursday) Scientific workflows and analytical pipelines – Pennington (Thursday) Ontologies and semantic mediation 5. Education – incorporating ecoinformatics into courses 13

Eco. Informatics 1. Basic infrastructure • • • Hardware/software – Murillo (Monday) Security/archival solutions/backup solutions – Murillo (Monday) Wireless communication – Porter & Vande Castle (Wednesday) 2. Concepts – Michener (Monday) 3. Activities • • Data documentation (metadata) – Michener & Romanello (Monday) Database design and creation – Porter (Tuesday) • Ecological and environmental databases – Porter (Tuesday) • Taxonomic and biodiversity databases – Garneau (Wednesday) QA/QC – Vanderbilt (Tuesday) • Analysis and visualization – Vanderbilt (Wednesday) Data & information propagation and discovery (web) – White (Wed. ) 4. SEEK • • • Overview – Michener (Thursday) Scientific workflows and analytical pipelines – Pennington (Thursday) Ontologies and semantic mediation – Bower (Friday) 14 5. Education – incorporating ecoinformatics into courses – (Friday)

Questions ? ? ? 15

Cyber-Infrastructure Challenges & Ecoinformatics: An Ecologist’s Perspective William Michener LTER Network Office Department of Biology University of New Mexico 16

Today’s Road Map • Science • Cyber-Infrastructure Challenges • Ecoinformatics 17

Today’s Road Map • Science • Cyber-Infrastructure Challenges • Ecoinformatics 18

Most studies use a single scale of observation -- Commonly 1 m 2 The literature is biased toward single and small scale results 19

Variable Change Time (yrs) transition from one state or condition to another 20

Thinking Outside the “Box” Time LTER Biocomplexity Parameters ce pa S ? ? – NEON, CUAHSI, CLEANER, …. Increase in breadth and depth of understanding. . . 21

24 NSF LTER Sites in the U. S. and the Antarctic: > 1500 Scientists; 6, 000+ Data Sets— different themes, methods, units, structure, …. LTER 2 4 1 20 15 17 14 24 9 8 18 6 16 5 19 10 11 3 12 21 7 23 22 13 22

23

Today’s Road Map • Science • Cyber-Infrastructure Challenges • Ecoinformatics 24

Knowledge Information Data Phenomena 25

Knowledge Information Data Phenomena 26

Abstraction of phenomena 27

Knowledge Information Data Phenomena 28

Hunter-gatherers on Y Hither 29

A Paradigm Shift Harvesters Taxon 1 Taxon 2 Taxon 3 Taxon 4 Abiotic factors Integrated, Interdisciplinary Databases 30

Data Entropy Time of publication Information Content Specific details General details Retirement or career change Accident Death Time (Michener et al. 31 1997)

Semantics B A • • Schema transform Coding transform Taxon Lookup Semantic transform Imagine scaling!! C 32

Semantics—Linking Taxonomic Semantics to Ecological Data §Taxon concepts change over time (and space) §Multiple competing concepts coexist §Names are re-used for multiple concepts Elliot 1816 R. plumosa R. Plumosa v. intermedia Gray 1834 R. plumosa v. plumosa Chapman 1860 R. plumosa Rhynchospora plumosa s. l. Kral 1998 R. Plumosa v. interrupta R. intermedia R. pineticola R. plumosa v. pinetcola R. plumosa v. plumosa A B R. sp. 1 C from R. Peet 2002? 33

Knowledge Information Data Phenomena 34

Characteristics of Ecological Data High Data Volume (per dataset) Low Satellite Images Weather Stations Business Data Most Software Gene Sequences GIS Most Ecological Data Primary Productivity Biodiversity Surveys Population Data Soil Cores Complexity/Metadata Requirements High 35

What Users Really Want… 36

Data Collection Analysis Publish For Other Scientists Translation 37 Use By Non-Scientists

Today’s Road Map • Science • Cyber-Infrastructure Challenges • Ecoinformatics 38

Ecological Informatics A broad interdisciplinary science that incorporates both conceptual and practical tools for the understanding, generation, processing, and propagation of ecological data and information. 39

Project Initiation Data Design and Metadata Data Acquisition and Quality Control Data Manipulation and Quality Assurance Analysis and Interpretation Access and Archiving Publication 40

Research Program Investigators Studies Field Computer Entry Electronically Interfaced Field Equipment Electronically Interfaced Lab Equipment Experimental Design Methods Data Design Data Forms Quality Control Raw Data File Data Entry Quality Assurance Checks no Data verified? Data Contamination yes Summary Analyses Data Validated Investigators Archive Data File Archival Mass Storage Magnetic Tape / Optical Disk / Printouts Access Interface Publication Synthesis Metadata Off-site Storage Secondary Users 41

“Ecological Informatics” Activities • • • Project / experimental design Data acquisition QA/QC Data documentation (metadata) Data archival 42

“Ecological Informatics” Activities • • • Project / experimental design Data acquisition QA/QC Data documentation (metadata) Data archival 43

Project / Experimental Design Analyses Data / Database Design 44

Project / Experimental Design • Some Classic References – Green, R. H. 1979. Sampling Design and Statistical Methods for Environmental Biologists. John Wiley & Sons, Inc. , New York. – Resetarits, Jr. , W. J. and J. Bernardo (eds. ). 1998. Experimental Ecology. Oxford University Press, New York. – Scheiner, S. M. and J. Gurevitch (eds. ). 1993. Design and Analysis of Ecological Experiments. Chapman & Hall, New York. – Sokal, R. R. and F. J. Rohlf. 1995. Biometry. W. H. Freeman & Company, New York. – Underwood, A. J. 1997. Experiments in Ecology: Their Logical Design and Interpretation Using Analysis of Variance. Cambridge University Press, Cambridge, UK. 45

“Ecological Informatics” Activities • • • Project / experimental design Data acquisition QA/QC Data documentation (metadata) Data archival 46

Data Design • Conceptualize and implement a logical structure within and among data sets that will facilitate data acquisition, entry, storage, retrieval and manipulation. 47

Data Set Design: Best Practices • • Assign descriptive file names Use consistent and stable file formats Define the parameters Use consistent data organization Perform basic quality assurance Assign descriptive data set titles Provide documentation (metadata) from Cook et al. 2000 48

1. Assign descriptive file names • File names should be unique and reflect the file contents • Bad file names – Mydata – 2001_data • A better file name – Sevilleta_LTER_NM_2001_NPP. asc • • • Sevilleta_LTER is the project name NM is the state abbreviation 2001 is the calendar year NPP represents Net Primary Productivity data asc stands for the file type--ASCII 49

2. Use consistent and stable file formats • Use ASCII file formats – avoid proprietary formats • Be consistent in formatting – don’t change or re-arrange columns – include header rows (first row should contain file name, data set title, author, date, and companion file names) – column headings should describe content of each column, including one row for parameter names and one for parameter units – within the ASCII file, delimit fields using commas, pipes (|), tabs, or semicolons (in order of preference) 50

3. Define the parameters • Use commonly accepted parameter names that describe the contents (e. g. , precip for precipitation) • Use consistent capitalization (e. g. , not temp, Temp, and TEMP in same file) • Explicitly state units of reported parameters in the data file and the metadata (SI units are recommended) • Choose a format for each parameter, explain the format in the metadata, and use that format throughout the file – e. g. , use yyyymmdd; January 2, 1999 is 19990102 51

4. Use consistent data organization (one good approach) Station Date Temp Precip Units YYYYMMDD C mm HOGI 19961001 12 0 HOGI 19961002 14 3 HOGI 19961003 19 -9999 Note: -9999 is a missing value code for the data set 52

4. Use consistent data organization (a second good approach) Station Date Parameter Value Unit HOGI 19961001 Temp 12 C HOGI 19961002 Temp 14 C HOGI 19961001 Precip 0 mm HOGI 19961002 Precip 3 mm 53

5. Perform basic quality assurance • Assure that data are delimited and line up in proper columns • Check that there no missing values for key parameters • Scan for impossible and anomalous values • Perform and review statistical summaries • Map location data (lat/long) and assess errors • Verify automated data transfers • For manual data transfers, consider double keying data and comparing 2 data sets 54

6. Assign descriptive data set titles • Data set titles should ideally describe the type of data, time period, location, and instruments used (e. g. , Landsat 7). • Titles should be restricted to 80 characters. • Data set title should be similar to names of data files – Good: “Shrub Net Primary Productivity at the Sevilleta LTER, New Mexico, 2000 -2001” – Bad: “Productivity Data” 55

7. Provide documentation (metadata) • To be discussed in detail 56

Database Types • • • File-system based Hierarchical Relational Object-oriented Hybrid (e. g. , combination of relational and object-oriented schema) 57 Porter 2000

File-system-based Database Directory Files 58 Porter 2000

Hierarchical Database Project Data sets Variables Codes Investigators Locations Methods 59 Porter 2000

Relational Database Projects Data_id Location_id Data sets Location_id 60 Porter 2000

“Ecological Informatics” Activities • • • Project / experimental design Data acquisition QA/QC Data documentation (metadata) Data archival 61

High-quality data depend on: • Proficiency of the data collector(s) • Instrument precision and accuracy • Consistency (e. g. , standard methods and approaches) – Design and ease of data entry • Sound QA/QC • Comprehensive metadata (e. g. , documentation of anomalies, etc. ) 62

How are data to be acquired? • • Automatic Collection ? Tape Recorder Data Sheet Field entry into hand-held computer 63

What’s wrong with this data sheet? Plant ______________ ______________ Life Stage _______________ _______________ 64

Important questions • How well does the data sheet reflect the data set design? • How well does the data entry screen (if available) reflect the data sheet? 65

PHENOLOGY DATA SHEET Rio Salado - Transect 1 Collectors: _________________ Date: __________ Time: _____ Notes: _______________________________________________ Plant ardi arpu atca bamu zigr P/G = perennating or germinating V = vegetating B = budding FL = flowering FR = fruiting P/G P/G V V V V Life Stage B B B B FL FL FR FR M M M M S S S S M = dispersing S = senescing D = dead NP = not present D D D D NP NP 66

PHENOLOGY DATA SHEET Collectors Date: Notes: ardi Troy Maddux 16 May 1991 Time: V B Y N Y N P/G V B Y N deob P/G Y N asbr 13: 12 Cloudy day, 3 gopher burrows on transect Y N arpu Rio Salado - Transect 1 Y N P/G V B Y N Y N FL Y N FR M S D NP Y N Y N Y N Y N Y N 67

“Ecological Informatics” Activities • • • Project / experimental design Data acquisition QA/QC Data documentation (metadata) Data archival 68

Generic Data Processing Research Program Investigators Studies Field Computer Entry Electronically Interfaced Field Equipment Electronically Interfaced Lab Equipment Experimental Design Methods Data Design Data Forms Quality Control Raw Data File Data Entry Quality Assurance Checks no Data verified? Data Contamination yes Summary Analyses Data Validated Investigators Archive Data File Archival Mass Storage Magnetic Tape / Optical Disk / Printouts Access Interface Publication Synthesis Metadata Off-site Storage Secondary Users 69 Brunt 2000

“Ecological Informatics” Activities • • • Project / experimental design Data acquisition QA/QC Data documentation (metadata) Data archival 70

“Ecological Informatics” Activities • • • Project / experimental design Data acquisition QA/QC Data documentation (metadata) Data archival 71

Traditional Fates of Data Post-Publication • Paper to filing cabinets • Data to floppy disks or tape entropy • Data and information lost over time 72

Data Archive • A collection of data sets, usually electronic, stored in such a way that a variety of users can locate, acquire, understand use the data. • Examples: – ESA’s Ecological Archive – NASA’s DAACs (Distributed Active Archive Centers) 73

Cycles of Research “A Conventional View” on ti blica Pu Data s Analysis and modeling Problem Collection Planning 74

Cycles of Research on ti blica Pu “A New View” Archive of Data s Collection Analysis and modeling Secondary Observations Original Observations Planning Selection and extraction Problem Definition (Research Objectives) Planning 75

Keys to Success • Start small and keep it simple – building on simple successes is much easier than failing on large inclusive attempts. §Involve scientists - ecological data management is a scientific endeavor that touches every aspect of the research program. Scientists should be involved in the planning and operation of a data management system. §Support science – data management must be driven by the research and not the other way around, a data management system must produce the products and services that are needed by the community. 76

üIncreasing value of data over time Serendipitous Discovery Data Value Inter-site Synthesis Gradual Increase In Data Equity Methodological Flaws, Instrumentation Obsolescence Non-scientific Monitoring Time 77

LTER Data Access Policy 1) There are two types of data: Type I (data that is freely available within 2 -3 years) with minimum restrictions and, Type II (Exceptional data sets that are available only with written permission from the PI/investigator(s)). Implied in this timetable, is the assumption that some data sets require more effort to get on-line and that no "blanket policy" is going to cover all data sets at all sites. However, each site would pursue getting all of their data on-line in the most expedient fashion possible. 2) The number of data sets that are assigned TYPE II status should be rare in occurrence and that the justification for exceptions must be well documented and approved by the lead PI and site data manager. Some examples of Type II data may include: locations of rare or endangered species, data that are covered by copyright laws (e. g. TM and/or SPOT satellite data) or some types of census data involving human subjects. 78

Reasons to Not Share Data: • • Fear of getting scooped Number of publications will decrease People will find errors Someone will misinterpret my data 79

Benefits of Data Sharing • Publicity, accolades, media attention • • Renewed or increased funding Teaching: • long-term data sets adapted for teaching & texts • • Archival: back-up copy of critical data sets Research: • new synthetic studies • peer-reviewed publications • • Document global and regional change Conservation and resource management: • species and natural areas protection • new environmental laws 80

Brunt (2000) Ch. 2 in Michener and Brunt (2000) Porter (2000) Ch. 3 in Michener and Brunt (2000) Edwards (2000) Ch. 4 in Michener and Brunt (2000) Michener (2000) Ch. 7 in Michener and Brunt (2000) Cook, R. B. , R. J. Olson, P. Kanciruk, and L. A. Hook. 2000. Best practices for preparing ecological and ground-based data sets to share and archive. (online at http: //www. daac. ornl. gov/cgibin/MDE/S 2 K/bestprac. html) 81

Thanks !!! 82