Discovery of Patterns in the Global Climate System using Data Mining Vipin Kumar Army High Performance Computing Research Center Department of Computer Science University of Minnesota http: //www. cs. umn. edu/~kumar Collaborators: G. Karypis, S. Shekhar, M. Steinbach, P. N. Tan (AHPCRC), C. Potter, (NASA Ames Research Center), S. Klooster (California State University, Monterey Bay). This work was partially funded by NASA and Army High Performance Computing Center © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining
Research Goals: l Find global climate patterns of interest to Earth Scientists A key interest is finding connections between the ocean and the land. l l © V. Kumar Global snapshots of values for a number of variables on land surfaces or water. Monthly over a range of 10 to 50 years. Discovery of Patterns in the Global Climate System using Data Mining 2
Sources of Earth Science Data l Before 1950, very sparse, unreliable data. l Since 1950, reliable global data. – Ocean temperature and pressure are based on data from ships. – Most land data, (solar, precipitation, temperature and pressure) comes from weather stations. l Since 1981, data has been available from earth orbiting satellites. – FPAR, a measure related to plants and greenness l Since 1999 TERRA, the flagship of the NASA earth observing system, is providing much more detailed data. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 3
Importance of Global Climate Patterns l The climate of the Earth’s land surface is strongly influenced by the behavior of the Earth’s oceans. – El Nino is the anomalous warming of the eastern tropical region of the Pacific. – Associated with droughts in Australia and Southern Africa and heavy rainfall along the western coast of South America. El Nino Events Sea Surface Temperature Anomalies off Peru (ANOM 1+2) © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 4
Importance of Global Climate Patterns and NPP l Net Primary Production (NPP) is the net assimilation of atmospheric carbon dioxide (CO 2) into organic matter by plants. – NPP is driven by solar radiation and can be constrained by precipitation and temperature. l Keeping track of NPP is important because it includes the food source of humans and all other organisms. – Sudden changes in the NPP of a region can have a direct impact on the regional ecology. l NPP is impacted by global climate patterns. – Precipitation and temperature are directly affected by global climate patterns such as El Nino. – Solar radiation is affected indirectly by cloudiness. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 5
Role of Statistics and Data Mining l Previously Earth scientists have relied on statistical techniques. – Hypothesize-and-test paradigm is extremely laborintensive. l Data mining provides earth scientist with tools that allow them to spend more time choosing and exploring interesting families of hypotheses. – By applying the proposed data mining techniques, some of the steps of hypothesis generation and evaluation will be automated, facilitated and improved. l However, statistics is needed to provide methods for determining the “statistical” significance of results. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 6
Patterns of Interest l Zone Formation – Find regions of the land or ocean which have similar behavior. l Teleconnections – Teleconnections are the simultaneous variation in climate and related processes over widely separated points on the Earth. l Associations – Find relations between climate events and land cover. l River Discharge – Relationship between water discharged from a river and precipitation, climate, and man. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 7
Clustering for Zone Formation l l l Interested in relationships between regions, not “points. ” For ocean, clustering based on SST (Sea Surface Temperature) or SLP (Sea Level Pressure). For land, clustering based on NPP or other variables, e. g. , precipitation, temperature. – Typically we work with the points. l When “raw” NPP and SST are used, clustering can find seasonal patterns. – Anomalous regions have plant growth patterns which reversed from those typically observed in the hemisphere in which they reside, and are easy to spot. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 8
K-Means Clustering of Raw NPP and Raw SST (Num clusters = 2) © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 9
K-Means Clustering of Raw NPP and Raw SST (Num clusters = 2) Land Cluster Cohesion: North = 0. 78, South = 0. 59 Ocean Cluster Cohesion: North = 0. 77, South = 0. 80 © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 10
K-Means Clustering of Raw NPP and Raw SST (Num clusters = 6) © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 11
Preprocessing l Time series preprocessing issues – Need to remove seasonality u Earth scientists mostly interest in anomalies – Need to remove most of the autocorrelation u Statistical test are affected – Need to remove trends u Normally want to detect patterns and trends separately – Normally interested in similarity once differences in means and scale have been considered. u © V. Kumar Pearson’s correlation coefficient has this property Discovery of Patterns in the Global Climate System using Data Mining 12
Sample NPP Time Series Minneapolis Correlations between time series Atlanta Sao Paolo Minneapolis 1. 0000 0. 7591 -0. 7581 Atlanta 0. 7591 1. 0000 -0. 5739 Sao Paolo © V. Kumar Minneapolis -0. 7581 -0. 5739 1. 0000 Discovery of Patterns in the Global Climate System using Data Mining 13
Seasonality Accounts for Much Correlation Minneapolis Normalized using monthly Z Score: Subtract off monthly mean and divide by monthly standard deviation Correlations between time series Atlanta Sao Paolo Minneapolis 1. 0000 0. 0492 0. 0906 Atlanta 0. 0492 1. 0000 -0. 0154 Sao Paolo © V. Kumar Minneapolis 0. 0906 -0. 0154 1. 0000 Discovery of Patterns in the Global Climate System using Data Mining 14
Removing Seasonality Removes Most Autocorrelation © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 15
Preprocessing: Removing Trends A slight linear trend added to two random time series increases their correlation dramatically, from 0. 01 to 0. 17. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 16
Ocean Climate Indices: Connecting the Ocean and the Land l An OCI is a time series of temperature or pressure – Based on Sea Surface Temperature (SST) or Sea Level Pressure (SLP) l OCIs are important because – They distill climate variability at a regional or global scale into a single time series. – They are well-accepted by Earth scientists. – They are related to well-known climate phenomena such as El Niño. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 17
Ocean Climate Indices – ANOM 1+2 l ANOM 1+2 is associated with El Niño and La Niña. El Nino Events l Defined as the Sea Surface Temperature (SST) anomalies in a regions off the coast of Peru l El Nino is associated with – Droughts in Australia and Southern Africa – Heavy rainfall along the western coast of South America – Milder winters in the Midwest © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 18
Connection of ANOM 1+2 to Land Temp OCIs capture teleconnections, i. e. , the simultaneous variation in climate and related processes over widely separated points on the Earth. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 19
Ocean Climate Indices - NAO l The North Atlantic Oscillation (NAO) is associated with climate variation in Europe and North America. Iceland Azores l Normalized pressure differences between Ponta Delgada, Azores and Stykkisholmur, Iceland. l Associated with warm and wet winters in Europe and in cold and dry winters in northern Canada and Greenland l The eastern US experiences mild and wet winter conditions. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 20
Connection of NAO to Land Temp © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 21
Influence of OCI on Land – Area Weighted Correlation l Correlation of an OCI with a land variable is a standard way to evaluate its “influence. ” – Correlation does not imply causality. – Temperature and precipitation are the typical land variables. l l If relatively many land points have a relatively high correlation, then an OCI is influential. To evaluate whether clusters (or pairs) are potential OCIs we compute their area weighted correlation. – Weighted average of the correlation with land points, where weight is based on area. – May exclude points whose correlation is low and then calculate area weighted correlation. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 22
Evaluation of Known OCIs via Area Weighted Correlation of Known OCIs to Land Temp Overlapping, threshold = 0 © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 23
Evaluation of Known OCIs via Area Weighted Correlation … Area weighted correlation declines as we consider only land points whose temperature correlates with the OCI above a given threshold. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 24
Discovering OCIs via Data Mining l Earth scientists have discovered currently known OCIs. – Observation – Eigenvalue techniques such as Principal Components Analysis (PCA) and Singular Value Decomposition (SVD). l Clustering provides an alternative approach. – Clusters represent ocean regions with relatively homogeneous behavior. – The centroids of these clusters are time series that summarize the behavior of these ocean areas, and thus, represent potential OCIs. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 25
Finding Influential Ocean Regions l l Not all points on the ocean correlate well with land variables such as temperature and precipitation. Best points are those which have a high “density” – Dense points are relatively homogenous with respect to their neighboring points. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 26
Discovery of Ocean Climate Indices l Use clustering to find areas of the oceans that have high density, I. e. , relatively homogeneous behavior. – Cluster centroids are potential OCIs. – For SLP pairs of cluster centroids are potential OCIs. l Evaluate the “influence” of potential OCIs on land points. l Determine if the potential OCI matches a known OCI. l For potential OCIs that are not well-known, conduct further evaluation. – Are there land points that have higher correlation for the potential OCI than for known indices? © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 27
SST Clusters © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 28
Evaluating Cluster Centroids as Potential OCIs l Evaluation will be based on area weighted correlation – Ignore clusters who area weighted correlation is low. l Three cases: – Clusters are highly similar to known OCIs (corr > 0. 4) u u u May represent a known OCI Clusters may be “better, ” i. e. , higher coverage Clusters may cover different area, i. e. , some points for which the new OCI is a better predictor – Clusters are moderately similar to known OCIs – © V. Kumar ( 0. 25 < corr < 0. 4 ) u Again, new OCIs may be better predictors for some points. Clusters are not similar to known OCIs (corr < 0. 25) u These clusters may represent as yet undiscovered Earth Science phenomena. Discovery of Patterns in the Global Climate System using Data Mining 29
SST Clusters Highly Correlated to Known Indices Area Weighted Correlation of Cluster Centroids to Land Temp Overlapping, threshold = 0 © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 30
SST Clusters Highly Correlated to Known Indices … © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 31
SST Clusters that Correspond to El Nino Climate Indices 75 78 67 94 El Nino Regions Defined by Earth Scientists SNN clusters of SST that are highly correlated with El Nino indices, ~ 0. 93 correlation. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 32
SST Clusters Highly Correlated to Known Indices … Examples of some SST clusters that are highly correlated to known OCIs and have high area weighted correlation with land temperature. These indices have a significant correlation with El Nino indices. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 33
SST Clusters Highly Correlated to Known Indices However, there areas (yellow) where these clusters correlate better. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 34
SST Clusters Highly Correlated to Known Indices © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 35
SST Cluster Moderately Correlated to Known Indices © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 36
Comments from our NASA collaborators “Ocean cluster results based on SST correlations with land surface temperature suggest that – New areas of the ocean may be identified that are unknown as being highly representative of the El Nino Southern Oscillation (ENSO) and the Arctic Oscillation (AO). ” – New predictive indices for land climate over the past 40 years can be identified that will improve upon predictions using any known ocean climate index to date, including SOI and AO. ” © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 37
Issues in Mining Associations from Earth Science Data is continuous rather than discrete. l Data has spatial and temporal components. l Data can be multilevel – time and spatial granularities. l Observations are not i. i. d. due to spatial and temporal autocorrelations. l Data may contain noise, missing information and measurement errors – historical SST data between 1856 -1941 is measured using wooden buckets. l Data may come from heterogeneous sources – Calibration issues. l © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 38
Mining Associations in Earth Science Data: Challenges l How to transform Earth Science data into transactions? – What are the “baskets”? – What are the “items”? – How to define “support”? © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 39
Mining Associations Patterns in Earth Science Data: Challenges 1 FPAR-HI PET-HI PREC-HI SOLAR-HI TEMP-HI ==> NPP-HI (support count=145, confidence=100%) 2 FPAR-HI PET-HI PREC-HI TEMP-HI ==> NPP-HI (support count=933, confidence=99. 3%) 3 FPAR-HI PET-HI PREC-HI ==> NPP-HI (support count=1655, confidence=98. 8%) 4 FPAR-HI PET-HI PREC-HI SOLAR-HI ==> NPP-HI (support count=268, confidence=98. 2%) … l How to efficiently discover spatio-temporal associations? – Use existing algorithms. – Develop new algorithms. l How to identify interesting patterns? – Use objective interest measures. – Use domain knowledge. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 40
Event Definition Items are events abstracted from time series. l Events of interest include: – Temporal events: l Anomalous temporal events such as warmer winters and droughts. u Changes in the periodic behavior such as longer growing seasons or earlier month of onset of greenup. u – Spatial events: Large percentage of land areas in a certain region having below-average precipitation. u – Spatio-temporal events: u © V. Kumar Changes in circulation or trajectory of jet-streams. Discovery of Patterns in the Global Climate System using Data Mining 41
Example of Anomalous Event Definition If threshold for Z = 1. 5, on average, there are ~20 events per time series. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 42
Transaction and Support Definitions l Convert the time series into sequence of events for each spatial location. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 43
Examples of Association Patterns l min support = 0. 001%, min confidence=10% 1 FPAR-HI PET-HI PREC-HI SOLAR-HI TEMP-HI ==> NPP-HI (support count=145, confidence=100%) 2 FPAR-HI PET-HI PREC-HI TEMP-HI ==> NPP-HI (support count=933, confidence=99. 3%) 3 FPAR-HI PET-HI PREC-HI ==> NPP-HI (support count=1655, confidence=98. 8%) 4 FPAR-HI PET-HI PREC-HI SOLAR-HI ==> NPP-HI (support count=268, confidence=98. 2%) 5 FPAR-HI PET-HI PREC-HI SOLAR-LO TEMP-HI ==> NPP-HI (support count=44, confidence=97. 8%) 6 FPAR-LO PET-LO PREC-LO SOLAR-LO ==> NPP-LO (support count=216, confidence=96. 9%) 7 FPAR-LO PREC-LO SOLAR-LO TEMP-HI ==> NPP-LO (support count=152, confidence=96. 2%) 8 FPAR-LO PET-LO PREC-LO SOLAR-LO TEMP-LO ==> NPP-LO (support count=47, confidence=95. 9%) 9 FPAR-LO PREC-LO SOLAR-LO TEMP-LO ==> NPP-LO (support count=49, confidence=94. 2%) 10 FPAR-LO PREC-LO SOLAR-LO ==> NPP-LO (support count=595, confidence=93. 7%) … 75 FPAR-HI ==> NPP-HI (support count = 216924, confidence = 55. 7%) NPP = Solar * FPAR * * Temperature * Moisture © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 44
Example of Interesting Association Patterns FPAR-Hi ==> NPP-Hi (sup=5. 9%, conf=55. 7%) Shrubland areas Rule has high support in shrubland areas © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 45
Land Cover Types Shrublands/
Using Land Cover as Additional Features 1. FPAR-HI PET-HI PREC-HI SOLAR-HI TEMP-HI ==> NPP-HI (support count=145, confidence=100%) 2. FPAR-HI PET-HI PREC-HI SOLAR-HI TEMP-HI GRASSLAND ==> NPP-HI (support count=145, confidence=100%) 3. FPAR-HI PET-HI PREC-HI SOLAR-HI TEMP-HI FOREST ==> NPP-HI (support count=44, confidence=100%) 4. FPAR-HI PET-HI PREC-HI SOLAR-HI TEMP-HI CROPLAND ==> NPP-HI (support count=44, confidence=100%) 5. FPAR-HI PET-HI PREC-HI SOLAR-HI FOREST ==> NPP-HI (support count=75, confidence=100%) 6. FPAR-HI PET-HI PREC-HI SOLAR-HI CROPLAND ==> NPP-HI (support count=81, confidence=100%) 7. FPAR-HI PREC-HI SOLAR-HI TEMP-HI CROPLAND ==> NPP-HI (support count=58, confidence=100%) 8. FPAR-HI PET-HI PREC-HI TEMP-HI GRASSLAND ==> NPP-HI (support count=376, confidence=99. 5%) 9. FPAR-HI PET-HI PREC-HI TEMP-HI CROPLAND ==> NPP-HI (support count=170, confidence=99. 4%) 10. FPAR-HI PET-HI PREC-HI CROPLAND ==> NPP-HI (support count=277, confidence=99. 3%) …. . • Produce multiple rules that have the same form: • {A} ==> {B}, {A, Grassland} ==> B, {A, Cropland} ==> {B}, etc. • Some of the support counts could be missing if itemsets fall below the minimum support threshold. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 47
Finding Interesting Earth-Science Patterns l A pattern is interesting if it occurs relatively more frequently in some homogeneous regions. l If the relative frequency of a pattern is similar in all groups of land areas, then it is less interesting. If the pattern occurs mostly in a certain group of land areas, then it is potentially interesting. l © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 48
Filtering Patterns using Land Cover Types For each pattern p: • Actual coverage for land cover type i = si /S • Expected coverage for land cover type i = ni /N • Ratio of actual to expected coverage for land cover type i, ei = si N / ni S • Interest Measure • If pattern occurs in arbitrary regions, interest measure will be low. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 49
Interesting Spatial Association Pattern FPAR-Hi ==> NPP-Hi • FPAR-Hi NPP-Hi tends to occur in shrubland grassland regions. • Possible explanation: this type of vegetation can take advantage of periodically high precipitation more quickly than forests. • New hypothesis: FPAR-HI events in these regions could be related to unusual precipitation conditions. Forest (0. 1%), Croplands (4. 2%), Grasslands (25. 9%), Desert (3. 6%) FPAR-Hi Prec-Hi ==> NPP-Hi © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 50
Interesting Spatial Association Pattern… Land Cover • Prec-Hi NPP-Hi tends to occur in grassland cropland regions. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 51
Other Interesting Spatial Association Patterns Support Count Land Cover • Temp-Hi NPP-Hi tends to occur in the forest and cropland regions in the northern hemisphere (Forests (33. 5%), Grassland(8. 7%), Cropland (24. 5%), Desert (0. 4%) ) © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 52
Global River Discharge Data • Global River Discharge Data • 30 rivers, 0. 5 degree resolution • Two measurement stations: mouth and source of river system/basin • Minimum of ten continuous years of monthly station discharge records • Interesting associations • e. g. , Amazon discharge is highly correlated with ANOM 3. 4(r = -0. 5) © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 53
Relationship Between River Basin PREC and OCI Amazon Parana • Correlation between PREC aggregation on river basins and OCI is shown in left figure • Interesting Observations: • Amazon and Parana are nearby, however, the signals to OCI are almost reverse © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 54
Discharge Data: Amazon vs. Parana © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 55
Relationship Between River Basin PREC and OCI. . Petchora • Interesting Observations: • Petchora and Pacific Decadal Oscillation (PDO) are highly correlated. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 56
Correlation between PREC and DISCHARGE Danube Nile © V. Kumar Amazon Yenisei Discovery of Patterns in the Global Climate System using Data Mining 57
Correlation between OCI and DIS (r = 0. 3) Amu-Darya Brahmaputra Colorado © V. Kumar Amazon Discovery of Patterns in the Global Climate System using Data Mining Columbia 58
Proposed Framework of River Analysis © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 59
Conclusions l Association rules can uncover interesting patterns for Earth Scientists to investigate. – Challenges arise due to spatio-temporal nature of the data. – Need to incorporate domain knowledge to prune out uninteresting patterns. l By using clustering we have made some progress towards automatically finding climate patterns that display interesting connections between the ocean and the land. – Need to further evaluate candidates for new climate indices. l Correlation analysis on river discharge data can be used to evaluate the effects of climate and man. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 60
Case Studies: Earth Science Data l Michael Steinbach, Pang-Ning Tan, Vipin Kumar, Chris Potter, Steven Klooster, Alicia Torregrosa, “Clustering Earth Science Data: Goals, Issues and Results”, Workshop on Mining Scientific Data, KDD 2001, San Francisco, CA, 2001. l Pang-Ning Tan, Michael Steinbach, Vipin Kumar, Steven Klooster, Christopher Potter, Alicia Torregrosa, “Finding Spatio-Termporal Patterns in Earth Science Data: Goals, Issues and Results, ” Temporal Data Mining Workshop, KDD 2001, San Francisco, CA, 2001. l Vipin Kumar, Michael Steinbach, Pang-Ning Tan, Steven Klooster, Chris Potter, Alicia Torregrosa, “Mining Scientific Data: Discovery of Patterns in the Global Climate System”, Joint Statistical Meetings, Atlanta, GA, 2001. l Michael Steinbach, Pang-Ning Tan, Vipin Kumar, Chris Potter, Steven Klooster, “Data Mining for the Discovery of Ocean Climate Indices”, Workshop on Mining Scientific Data, SDM 2002. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 61
Statistical Issues l Temporal Autocorrelation – Makes it difficult to calculate degrees of freedom and determine significance levels for tests, e. g. , non-zero correlation. – Moving average is nice for smoothing and seeing the overall behavior, but introduces additional autocorrelation. – Removal of seasonality removes much of the autocorrelation (as long as not performed via the moving average). l Measures of time series similarity – Detecting non-linear connections – Detecting connections that only exist at certain times. u Sometimes only extreme events have an effect. – Automatically detecting appropriate time lags. – Statistical tests for more sophisticated measures. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 62
Statistical Issues … l Detecting spurious connections. – We are performing many correlation calculations and there is a chance of spurious correlations. u Given that we have ~100, 000 locations on the Earth for which we have time series, how many spuriously high correlations will we get when we calculate the correlation between these locations and a climate index? – Because of spatial autocorrelation, these correlations are not independent. u Again we have trouble calculating the degrees of freedom. © V. Kumar Discovery of Patterns in the Global Climate System using Data Mining 63
Скачать презентацию Discovery of Patterns in the Global Climate System
0f0f3675f4fa2e269857c2b9a8ffd161.ppt