QA 4 EO A Quality Assurance Framework for

QA 4 EO A Quality Assurance Framework for Earth Observation Greg Stensaas Chair of CEOS Working Group on Calibration and Validation USGS Remote Sensing Technologies Project Manager USGS Earth Resources Observation and Science (EROS) Center, Sioux Falls, SD Presentation to NASA Earth Science Data Systems (ESDS) Standards Process Group (SPG) Meeting 12 July 2011

Group on Earth Observation (GEO) l The Group on Earth Observations is coordinating efforts to build a Global Earth Observation System of Systems, or GEOSS. l http: //www. earthobservations. org/ 2

What is GEO? l Voluntary partnership of governments and international organizations l Provides a framework within which these partners can develop new projects and coordinate their strategies and investments l As of March 2011, GEO’s Members include 86 Governments and the European Commission and 61 related organizations l Constructing GEOSS on the basis of a 10 -Year Implementation Plan through 2015 l Use EO information for nine “Societal Benefit Areas” of disasters, health, energy, climate, water, weather, ecosystems, agriculture and biodiversity 3

GEO Societal Benefits GEOSS will yield a broad range of societal benefits, notably: l l l l l Reducing loss of life and property from natural and human-induced disasters Understanding environmental factors affecting human health and well-being Improving the management of energy resources Understanding, assessing, predicting, mitigating, and adapting to climate variability and change Improving water resource management through better understanding of the water cycle Improving weather information, forecasting and warning Improving the management and protection of terrestrial, coastal and marine ecosystems Supporting sustainable agriculture and combating desertification Understanding, monitoring and conserving biodiversity 4

Data Quality Information needed l l l The Global Earth Observation System of Systems (GEOSS) must deliver “timely, quality, long-term, global information ” to meet the needs of its nine “societal benefit areas” This will be achieved through the synergistic use of data derived from a variety of sources (satellite, airborne and surface-based) and the coordination of resources and efforts of the members Accomplishing this vision, starting from a system of disparate systems that were built for a multitude of applications, requires the establishment of an internationally coordinated framework to facilitate interoperability and harmonisation The success of this framework is dependent upon the successful implementation of a single key principle: u all EO data and derived products shall have associated with it a documented and fully traceable quality indicator (QI) Success also necessitates the means to efficiently communicate quality attributes to all stakeholders 5

GEOSS Quality Assurance Strategy l l The Committee on Earth Observation Satellites (CEOS) Working Group on Calibration and Validation (WGCV) lead a group to develop a GEOSS Quality Assurance Strategy as a direct response to the GEO Task DA-06 -02/DA-09 -01 a: GEOSS Quality Assurance Strategy u l Develop a GEO data quality assurance strategy and implementation process, beginning with space-based observations and expanding to in-situ observations, taking account of existing associated GEOSS quality assurance work, and including the quality issues of derived Earth observation information products. A Quality Assurance framework for Earth Observation (QA 4 EO) has been established and is now being implemented within GEO and CEOS 6

Committee on Earth Observation Satellites - http: //www. ceos. org/ Working Group on Calibration and Validation (WGCV) Subgroups • Atmospheric Composition (ESA) • Infrared Visible & Optical Sensors (NPL) • Land Product Validation (NASA) • Microwave (CSSAR) • Synthetic Aperture Radar (DLR) • Terrain Mapping (UCL) 7

QA 4 EO Workshops l CEOS Working Group on Calibration and Validation (WGCV) worked with many agencies/organizations to develop the QA 4 EO principles and operational implementation details during meetings and workshops Geneva, WMO Oct. 2007 Guiding principles Gaithersburg, NIST May 2008 Establishing an operational framework Antalya, TÜBİTAK Sept. 2009 Facilitating Implementation 8

QA 4 EO Essential Principle l l Measurement/processes are only significant if their “quality” is specified In order to achieve the vision of GEOSS, Quality Indicators (QIs) should be ascribed to data and products, at each stage of the data processing chain - from collection and processing to delivery. u A QI should provide sufficient information to allow all users to readily evaluate a product’s suitability for their particular application, i. e. its “fitness for purpose”. u To ensure that this process is internationally harmonised and consistent, the QI needs to be based on a documented and quantifiable assessment of evidence demonstrating the level of traceability to internationally agreed (where possible SI) reference standards. 9

QA 4 EO Essential Principle Data and derived products shall have associated with them an QA 4 EO indicator of their quality to enable users to assess its suitability for their application. Essential Principle “fitness for purpose” Quality Indicators (QIs) should be ascribed to data and Products. AQuality provide sufficient QI should Indicators information to allow all users to readily evaluate its “fitness for purpose”. QI needs to be based on a documented and quantifiable assessment of evidence demonstrating the level of Traceability to internationally agreed (where possible SI) reference standards. 10

What QA 4 EO is… it’s a general framework based on 1 essential principle and composed of 7 key guidelines These are “living documents” (i. e. v. 4. 0) and they offer a flexible approach to allow the effort for the tailoring of the guidelines to be commensurate with the final objectives. It is a user (costumer) driven process. 11

…and what is not …not a certification body …not a set of standards for QC/QA activities and processes that would limit competitiveness or innovation and evolution of technology and methodologies …not a framework developed with a top-down approach …the QA 4 EO process and its implementation should not be judgemental and bureaucratic 12

QA 4 EO Documents l The framework and key guideline documents were peerreviewed by representatives from different Earth Observation communities u Approved by WGCV (28 th Plenary Meeting, Oct. 2008) u Endorsed by CEOS (22 nd CEOS Plenary, Nov. 2008) u Reviewed by World Meteorological Organization (WMO) and Global Space-based Inter-Calibration System (GSICS) (early 2009) l A guide was issued in order to provide a new user with an overview and guidance on getting started with QA 4 EO l QA 4 EO documents including the framework, key guidelines, and the guide can be found on the QA 4 EO web site: http: //qa 4 eo. org/documentation. html 13

QA 4 EO Definitions l Quality Indicator u u l A mean of providing “a user” of data or derived product, (i. e. which is the results of a process) sufficient information to assess its suitability for a particular application. This “information” should be based on a quantitative assessment of its traceability to an agreed reference measurement standard (ideally SI) but can be presented as numeric or text descriptor providing the quantitative linkage is defined Traceability u Property of a measurement result whereby the result can be related to a reference through a documented unbroken chain of calibrations each contributing to the measurement uncertainty. 14

Further Definitions l Reference (measurement) Standard: u u l Realization of the definition of a given quantity, ideally with stated quantity value and associated measurement uncertainty, used as a reference. Can be individual or community defined. Uncertainty: u u Non-negative parameter characterising the dispersion of the quantity values that are being attributed to a measurand (quantity), based on the information used. Where possible, this should be derived from an experimental evaluation but can also be an estimate based on other information, e. g. experience. 15

QA 4 EO Guides l The QA 4 EO principles provide the framework and introduce the key guidelines: u u u u QA 4 EO-QAEO-GEN-DQK-001 A guide to establish a Quality Indicator on a satellite sensor derived data product QA 4 EO-QAEO-GEN-DQK-002 A guide to content of a documentary procedure to meet the Quality Assurance requirements of GEO QA 4 EO-QAEO-GEN-DQK-003 A guide to “reference standards” in support of Quality Assurance requirements of QA 4 EO-QAEO-GEN-DQK-004 A guide to comparisons – organization, operation and analysis to establish measurement equivalence to underpin the QA requirements of QA 4 EO-QAEO-GEN-DQK-005 A guide to establishing validated models, algorithms and software to underpin the Quality Assurance requirements of QA 4 EO-QAEO-GEN-DQK-006 A guide to expression of uncertainty of measurements QA 4 EO-QAEO-GEN-DQK-007 A guide to establishing quantitative evidence of traceability to underpin the Quality Assurance requirements of QA 4 EO 16

Why QA 4 EO? • Underpins unambiguous bias evaluation • Enables robust validation through international harmonisation of ground truth instrumentation and methods • Facilitates interoperability: - Sensor to sensor - Across technical domains (e. g. optical / SAR) - Observing platforms (Satellite, Aircraft, in-situ) - Understand compare different models 17

Why QA 4 EO? • Allows new information services; e. g. , • How much carbon is in an acre of forest? • And how sure are we? • Allows innovation in our efforts to monitor in the longterm for climate; i. e. , • SI traceability • NOT high-risk “normalization” • Allows integration of defined data for understood model output 18

We need to understand the error l All systems and models have some degree of error u l l Data record error is essential for use in climate and change assessment Different groups can produce defensible, but statistically inconsistent estimates of trends Bias between sensors need to be defined and there is a strong need for better error characterization The types of errors recently defined by Climate Observation System (GCOS) u u l l Calibration compensates for errors and improves measurement accuracy possible Accuracy: The rms difference between the single or averaged values of a variable and the truth. Stability: The extent to which accuracy of a time average remains constant over a longer time period (e. g. , annual average relative to decadal average). Errors should be specified on a FOV basis. Aggregated error estimates are not sufficient Single sensor products are simpler than merged products Error correlations are also important to document The importance of specifying each error requirement depends on the application 19

Implementation at various level of organization l l l Datasets provided to GEO will require information with Quality Indicators in order to support decision and policy makers QA 4 EO builds upon ‘best practice’ from all sectors, not just Earth Observation Each entity is responsible for implementing QA 4 EO in its program u At Agency Level l Each Agency has to organise the QA 4 EO implementation at its own level u For Example: European Space Agency (ESA) has incorporated QA 4 EO in Calibration and Validation efforts QA 4 EO is referenced in ESA statements of work and standards l Referenced in ISO standards At CEOS level l WGCV is in charge of coordinating QA 4 EO at CEOS Level At GEO and SBA levels l Workshop to identify the best setup to manage QA 4 EO at the GEO and SBA levels l u u 20

Future QA 4 EO Activities l Harmonization and implementation is aided by QA 4 EO guidelines and methods are continually evolving l QA 4 EO is cross-cutting and is evolving to meet the needs of all SBAs by establishing case studies and training, currently, international partners are working this process via GEO/CEOS l Continuing to coordinate and define international joint efforts and activities u l E. g. ; Continuing to perform GEO/CEOS intercomparison and intercalibration campaigns GEO/CEOS is developing showcases highlighting the need for QA 4 EO u Showcases on Forest Carbon Tracking (FCT), Atmospheric Composition (AC), and Global Digital Elevation Models (DEM) l A questionnaire is under-development to help providers/users assess their QA 4 EO compliance l Future QA 4 EO Workshop on Providing Quality Information in Harmonized Earth Observation Data 21

Future QA 4 EO Workshop • Workshop on Providing Quality Information in Harmonized Earth Observation Data by 2015 • Hosted by UKSA at Rutherford Appleton Laboratory (RAL) near Oxford, UK on Oct 18 -20, 2011 • Key SBA implementation examples showing the benefit of and the need for enhanced quality in GEO and CEOS data and processes • Establish a roadmap of key objectives that will cover technical, coordination, and governance aspects of QA 4 EO • Your active participation is critical to moving forward with the GEO’s vision for a GEOSS • Workshop registration: http: //qa 4 eo. org/index. html 22

QA 4 EO Executive Summary l It is critical that data and derived products are easily accessible in an open manner and have associated with them an indicator of their quality traceable to reference standards (preferably SI) to enable users to assess its suitability for their application i. e. its “fitness for purpose”. l This Quality Indicator needs to be unequivocal in its interpretation and derivation, yet sufficiently flexible, to be implemented across the full range of EO activities which are coordinated through GEO. l The Quality Assurance Framework for Earth Observation (QA 4 EO) was established to show the need for quality and achieve this task. 23

QA 4 EO Executive Summary l QA 4 EO implementation is supported by a framework document and a set of key guidelines to assist in its interpretation and implementation l Organizations that fund and oversee the development and execution of Earth Observation programs are responsible for implementing the QA 4 EO key guidelines to their delivered data products l A GEO Oversight Panel, composed of diverse members throughout the EO community, are responsible for managing QA 4 EO documents, clearly communicating QA 4 EO concepts to data providers and users alike, and mediating QA 4 EO disputes between data providers and users 24

Questions

CEOS Structure 2010 -2011 Troika Past Chair INPE/Brazil Troika Systems Engineering Office (SEO) Chair ASI/Italy Future Chair ISRO/India Permanent Secretariat CEOS Executive Officer (CEO) Chair Agency ESA, EUMETSAT, JAXA, NASA, NOAA USGS (CEO) & NOAA (DCEO) Strategic Implementation Team (SIT) Chair: JAXA Vice Chair: NASA (2 -year terms) NASA WGCV WGISS WGEdu WGClimate Chair USGS Chair GISTDA Chair NOAA & INPE Chair EC-JRC Vice Chair CSA Vice Chair JAXA Vice Chair N/A Vice Chair NOAA WGCV=Working Group on Calibration and Validation WGISS=Working Group on Information Systems and Services WGEdu=Working Group on Education, Training and Capacity Building WGClimate = Working Group on Climate (1 -year terms) CEOS SBA Coordinators - Agriculture - Biodiversity - Climate - Disasters - Ecosystems - Energy - Health - Water - Weather Virtual Constellations for GEO Atmospheric Composition - NASA - ESA Precipitation - JAXA - NASA Land Imaging - USGS - ISRO - INPE Ocean Surface Topography - NOAA - EUMETSAT Ocean Color Radiometry - EC-JRC - JAXA - NASA Ocean Surface Vector Wind - NOAA - ISRO - EUMETSAT 26