WTF-CEOP Status and Plan Data Mining and Data

WTF-CEOP Status and Plan “Data Mining” and “Data Integration” September 17, 2003 Osamu Ochiai/NASDA Ben Burford/RESTEC Toshio Koike/University of Tokyo CEOS/WGISS, Chiang Mai – September 17, 2003 1

Contents 1. 2. 3. 4. 5. 6. CEOP Data WTF-CEOP (“Data Mining”) The science CEOP Data Integration CEOP Interoperability Way Forward CEOS/WGISS, Chiang Mai – September 17, 2003 2

- CEOP Consolidated Enhanced Observation Period. • In-situ data (36 Reference Sites globally) • Satellite Data (200 Tbytes, subset and full scene) • Model Output Data (NWP from 10 Agencies) EOP-1: 1 July 2001 thru 30 September 2001 EOP-2: 1 October 2001 thru 30 September 2002 EOP-3: 1 October 2002 thru 30 September 2003 EOP-4: 1 October 2003 thru 31 December 2004 CEOS/WGISS, Chiang Mai – September 17, 2003 3

CEOP In-situ Data CEOS/WGISS, Chiang Mai – September 17, 2003 4

Satellite Data • Subset scenes over Reference Sites • Full scenes over research areas • 200 TBytes • TRMM • MODIS • CERES CEOS/WGISS, Chiang Mai – September 17, 2003 • ASTER • MISR • ENVISAT 5 • AQUA • ADEOS-II • ALOS

Model Output Sources Major National and Multi-National Centers Bo. M, CPTEC, ECMWF, ECPC, JMA, DAO, GLDAS, NCEP, NCMRWF, UKMO NWP – Numerical Weather Prediction data Max Planck Institute for Meteorology (MPIM) at Hamburg, Germany CEOS/WGISS, Chiang Mai – September 17, 2003 6

Resolution of the data (JMA) (example : Lindenburg) 288*145 (1. 25 x 1. 25 degree) 3 D - 23 pressure levels (fundamental variables) CEOS/WGISS, Chiang Mai – September 17, 2003 640*320 (Native Gaussian, T 213) 2 D (TOA, Surface processes ) 7 320*160 (4 grids average ) 3 D - 40 levels (heating rates)

NWP Assimilation and Forecast Day N 00 Day N+1 03 A 06 09 A F F 12 15 A F 18 21 A F F F 00 03 06 A F F 09 A F F 12 15 A F F 18 21 00 A F Assimilation A F F Forecast F Repeat 12 -36 hour forecast NCEP Global Model Forecast (12 -36 hour) 03 421. 6 MBytes/day 153. 9 GBytes/year CEOS/WGISS, Chiang Mai – September 17, 2003 06 09 12 15 F F 8 18 21 24 27 30 33 36 F F F Forecast

Model Output Data (1/2) CEOP has requested 82 variables from the providers of Model data shortwave downward flux shortwave upward flux longwave upward flux temperature pressure humidity zonal wind meridional wind pressure velocity horizontal kinetic energy (KE) geopotential (g. Z) cloud water convective latent heating rate stable latent heating rate convective moistening rate stable moistening rate turbulent heating rate CEOS/WGISS, Chiang Mai – September 17, 2003 short-wave heating rate long-wave heating rate water vapor zonal flux water vapor meridional flux water vapor vertical flux water vapor flux divergence energy (Cp. T+g. Z+KE) zonal flux energy (Cp. T+g. Z+KE) meridional flux energy (Cp. T+g. Z+KE) vertical flux energy flux divergence local time tendency of temperature local time tendency of KE local time tendency of moisture Local time tendency of cloudwater surface pressure skin temperature 2 -meter specific humidity 9 u-component at 10 m v_component at 10 m potential temperature at 10 m specific humidity at 10 m snow water equivalent Snow depth vegetation water planetary boundary layer height shortwave downward flux shortwave upward flux longwave downward flux longwave upward flux sensible heating latent heating Snow and frozen ground conversion to soil water meridional wind stress zonal wind stress

Model Output Data (2/2) 10 m turbulent kinetic energy in a layer precipitation (total) precipitation (snow) surface runoff baseflow local surface pressure tendency local skin temperature tendency local snow water equivalent tendency soil moisture soil temperature infiltration rate local soil moisture tendency local temperature tendency subsurface temperature ground Heat Flux base flow runoff Precipitation type 1 rain or 2 snow elevation Surface albedo station land/sea/ice mask 0(land)or 1(sea)or(2)ice CEOP is mapping these 82 variables to: • the standard GRIB list of variables • the variable names of model data • the variable names of in-situ data • the variable names of MOLTS • the variable names of satellite data CEOS/WGISS, Chiang Mai – September 17, 2003 Cloud cover aerosol concentration surface exchange coefficient roughness length Vegetation cover Water table (wells) streamflow Stream discharge Reservoir storage This mapping among variable names will be a key component of “Data Integration” for CEOP data! 10

Data Integration “Data Integration” • • • Users can access all types of data in all locations Models can be run on data from all areas. Model output can be compared. Common formats Common data types Consistent data variable names What else can we do to expand “Data Integration”? CEOS/WGISS, Chiang Mai – September 17, 2003 11

CEOP “Data Mining” (WTF-CEOP) Validate Soil Moisture Algorithm • Input data - AMSR data (6. 9 GHz, 18. 7 GHz, 36. 5 GHz) • Validation data - Soil Moisture in-situ data First • Get the scenes of AMSR data over a CEOP Reference Site (Mongolia) • Data for July, August and September, 2001 CEOS/WGISS, Chiang Mai – September 17, 2003 12

Mongolia Reference Site AMSR Data (6. 9 GHz, 18. 7 GHz, 36. 5 GHz) Soil Moisture In-situ Data AWS - Automated Weather Station (6) ASSH – Automatic Station for Soil Hydrology (12) CEOS/WGISS, Chiang Mai – September 17, 2003 13

Two years of validation Mongolia Reference Site/AMSR Data: 2 scenes/day x 365 = 730 scenes of AMSR data (times 3 frequencies = 2190 scenes) 2 scenes/day x 365 x 2 yr = 1, 460 scenes of AMSR data (extract 17, 520 pixels from 114 GBytes of data) 36 Reference Sites: 52, 560 scenes (4. 1 TBytes of data) CEOS/WGISS, Chiang Mai – September 17, 2003 14

WTF-CEOP User Interface CEOP Data Integration Center “Data Analysis System” User Interface Select Location Reference Site Tibet Mongolia Himalayas NSCSSJ etc. Select Time Range From: 01 07 2001 From: 30 09 2001 In-situ Data Ps (mb) Ts (deg. C) RH (%) Td (deg. C) Mv (%) Ws (m/s) etc. Select Data In-situ Satellite MOLTS Model Output CEOS/WGISS, Chiang Mai – September 17, 2003 15 Satellite Data TRMM/PR TRMM/TMI ADEOS II /AMSR ADEOS-II/GLI etc. AMSR Data 6. 9 GHz 10. 7 GHz 18. 7 GHz 23. 8 GHz 36. 5 GHz 89. 0 GHz

Data Mining Request Prepare a “Distributed Data Mining Request” • Time Range – July 1 to Sep. 30, 2001. • AMSR data (6. 9 GHz, 18. 7 GHz, 36. 5 GHz) • Soil Moisture in-situ data • Locations (12 sets of latitude/longitude values) Format for Data Mining Request (Use OGC Web Coverage Server (WCS) format? ) CEOS/WGISS, Chiang Mai – September 17, 2003 16

NASDA User Interface Module 1. User I/F (menus). 2. Send Data Mining Request. NASA Data Mining Module 3. Do data mining of Satellite Data. NASDA Data Mining Module 3. Do data mining of Satellite Data. NASDA Final Module 4. Do data mining of CEOP in-situ data. 5. Send satellite and in-situ data results set to user. CEOS/WGISS, Chiang Mai – September 17, 2003 17 ESA Data Mining Module 3. Do data mining of Satellite Data.

NASDA, NASA, ESA Satellite “Data Mining Module” 1. Catalog Search for all AMSR scenes over Mongolia (bounding box search) for July 1 to Sep. 30, 2001 • Use agency catalog search client (e. g. EDG) • Prepare a file with scene times and file names of AMSR data over “Mongolia” 2. Processing - for each scene of AMSR data • Open file, geolocate pixels, extract 12 pixels over the Mongolia lat/lon locations. 3. Send pixels, with the times of the satellite scenes, to the Final Module at NASDA. CEOS/WGISS, Chiang Mai – September 17, 2003 18

Final Module 1. Extract in-situ data • Find time of satellite scene (e. g. July 1, 2001 at 01: 05). • Find closest time of in-situ data (e. g. July 1, 2001 at 01: 00). • Extract soil moisture values, at 12 ASSH locations, from Mongolia in-situ data. 2. Put satellite scene time, satellite pixel values, in-situ data time and in-situ soil moisture values into Results File (in a convenient format). 3. Send Results to the user. CEOS/WGISS, Chiang Mai – September 17, 2003 19

“Expand the Picture” The Science Picture • • Develop Algorithms Validate algorithms (Produce product – GRID? ) “Virtuous Cycle” of algorithm development CEOS/WGISS, Chiang Mai – September 17, 2003 20

Hydrology Research Soil Moisture Land Surface Scheme Snow Microwave Radiometer Snow Physics Model Precipitation Cloud Physics Model Surface Emissivity & Temp. CEOS/WGISS, Chiang Mai – September 17, 2003 21

CEOS/WGISS, Chiang Mai – September 17, 2003 22

Precipitation Algorithm CEOS/WGISS, Chiang Mai – September 17, 2003 23

Water Cycle – Energy Budget Dry Air-mass Energy Water Vapor Energy Radio Radiation Sonde Profilers Sensors AIRS AMSU HSB SSM/T 2 HIRS GVAP MODIS GLI CERES SRB ISCCP ERB CEOS/WGISS, Chiang Mai – September 17, 2003 convergence/divergence Eddy Corr. Bowen Ratio AMSRE TMI MODIS (ASAR) (ETM) +4 DDA Model Output Rt Top of atmosphere conv. H+conv. LQ Air column Hs 24 Rn LEs land surface

Water Cycle – Water Budget Radio Sonde Profilers AIRS AMSU HSB SSM/T 2 HIRS GVAP convergence/divergence Rain Gauge Model Output Radar Eddy Corr. Bowen Ratio AMSRE TMI SSM/I PR GPCP MODIS (ASAR) CEOS/WGISS, Chiang Mai – September 17, 2003 +4 DDA Tropopause Pre-GPM div. Q Air column P Es Land surface 25 div. Q

Geolocation and Temporal Integration 60 …………………………. ……………. 59 58 57 100 CEOS/WGISS, Chiang Mai – September 17, 2003 101 102 26 103

CEOP Data Integration • Common formats • Consistent data types • Consistent variable names With CEOP “Data Mining” • Single user interface (menus) • Geolocated Integration (in-situ, satellite, model) • Temporal Integration (in-situ, satellite, model) • Delivery of exact pixels required (validation) CEOS/WGISS, Chiang Mai – September 17, 2003 27

Add Interoperability WGCV • Land products for validation? Core Site WTF • Satellite or land products? NOMADS, Earth System Grid, NERC Data Grid CEOS/WGISS, Chiang Mai – September 17, 2003 28

Are we finished yet? Problems? Yes!! Example – cloud cover. • Put cloud cover selection in menus (“No scenes with cloud cover greater than xx%”)? • What cloud metadata do we have? • Cloud masks? • “Bad Data” values (e. g. 999. 99 is bad data) • What is acceptable to scientists? • What can we do with the metadata? CEOS/WGISS, Chiang Mai – September 17, 2003 29

Don’t Need Everything! WTF-CEOP is enough! (For a great Plenary Demo) • Algorithm development • Algorithm validation • Climate Research model output validation (!!) • “Data Integration”/”Data Mining” + GRID – Produce product, other? + WGCV – Data mining of land cover products? + Future missions - GPM CEOS/WGISS, Chiang Mai – September 17, 2003 30

Way Forward 1. Do we agree to support this process? 2. Agencies determine resources. – NASA, ESA, NASDA, Other – WTF-CEOP server! 3. Develop a CEOP science scenario – CEOP scientists + CEOS team (choose sat. data to be supported) 4. Science Scenario [balance] CEOS – Types of data (MODIS is difficult, promote AIRS? ) 5. Schedule (CEOS Plenary in 2004? ) 6. Write a Project Plan 7. Telecons/Detailed Schedule/Develop in stages? /. . . CEOS/WGISS, Chiang Mai – September 17, 2003 31