Regularized Double Nearest Neighbor Feature Extraction for Hyperspectral Image Classification Hsiao-Yun Huang Department of Statistics and Information Science, Fu-Jen University
Hyperspectral Image Introduction 1 (image credit: AFRL)
Hyperspectral Image Introduction 2 (image credit: AFRL)
Applications of Hyperspectral Image n Military: military equipment detection. n Commercial: mineral exploration, agriculture and forest production. n Ecology: chlorophyll, leaf water, cellulose, lignin. n Agriculture: illness or type of the plants.
Classification of Hypectral Image Pixels n How to distinguish different land cover types precisely and automatically in the hyperspectral images is an interesting and important research problem. n Generally, each pixel in a hyperspectral image is consisted of about hounds or even thousands of bands. This makes the discrimination among pixels a highdimensional classification problem.
High-Dimensional Data Analysis n “We can say with complete confidence that in the coming century, high-dimensional data analysis will be a very significant activity, and completely new methods of high-dimensional data analysis will be developed; …” n (Lecture on August 8, to the American Mathematical Society ‘Math Challenges of the 21 st Centure’ by David L. Donoho (2000))
Blessing: The Power of Increasing Dimensionality 0. 35 x 1 0. 35 x 2 0. 3 x 3 0. 25 0. 2 0. 15 0. 1 0. 05 0 -5 0 5 x 2 10 0 -5 0 10 x 3 x 1 5 x 1 0 -5 0 5 10 x 3 x 2 x 1
Curse: Hughes Phenomenon
The Curse of Dimensionality n In statistics, it is about the situation that the convergence of any estimator to the true value of a smooth function defined on a space of high dimension is very slow. That is, we need an extremely large amount of observations. n n (Bellman, 1961 ) http: //www. stat. ucla. edu/~sabatti/statarray/textr/node 5. html
The Challenge n Unfortunately, in hyperspectral image classification, the p > N case is the usual situation due to the access of training samples (ground truth data) can be very difficult and expensive. n The large dimension but few samples problems might cause the accuracy rate of the hyperspectral image classification to be unsatisfied.
Dimensionality Reduction n One common way to deal with the curse of dimensionality is to reduce the number of dimensions. n Two major reduction ideas: Feature Selection n Feature Extraction n
n Feature selection: select l out of p measurements n Feature extraction: map p measurements to l measurements x 1 xp f 1 f 2 xp
Feature Extraction v. s. Feature Selection 150 6 Selection 4 100 2 50 0 0 -5 150 -2 Extraction -4 0 5 100 50 -6 -5 0 5 0 -5 0 5
Basic Ideas of Feature Extraction n Feature extraction consists of choosing those features which are most effective for preserving class separability. n Class Separability depends not only on the class distributions but also on the classifier to be used. n We seek the minimum feature set with reference to the Bayes classifier; this will result in the minimum error for the given distributions. Therefore, the Bayes error is the optimum measure of feature effectiveness.
One Consideration n A major disadvantage of the Bayes error as a criterion is that an explicit mathematical expression is not available except for a very few special cases, therefore, we cannot expect a great deal of theoretical development.
Practical Alternatives n Two types of criteria which have explicit mathematical expressions and frequently be used in practice: Functions of scatter matrices (do not relate to the Bayes error) n Conceptually simple and give systematic algorithms. n Bhattacharyya distance type of criteria (give upper bounds of the Bayes error) n Only for two-class problem, and based on normality assumption. n
Discriminant Analysis Feature Extraction (DAFE or Fisher’s LDA) Is the number of class Sb in pairwise structure Note: The number of extracted features are min{p, L-1} where p is the dimension of the mean vector
DAFE v. s. PCA DAFE
Drawbacks of the Fisher’s LDA (1) n In some situations, is not a good measure of class separability n n Share the same mean: No scatter of M 1 and M 2 around M 0 Multimodal: more than L-1 features are needed Multimodal and share the same mean Multimodal Unimodal share the same mean
Drawbacks of the Fisher’s LDA (2) n The unbiased estimate S (pooled covariance estimate) of the within-class scatter matrix is adopted in LDA. If it is singular, the performance will be poor. n When dim>>n, S will loose its full rank as a growing number of eigenvalues become zero. So, it is not positive definite and can not be inverted. Eigenvalue (dim/n=10) _: true eigenvalues - - : Sw eigenvalues 0 dim 100
Feature Extraction Methods with Other Measure of Separability n Nonparametric Discriminant Analysis (NDA; Fukunaga and Mantock, 1983). n Nonparametric Weighted Feature Extraction (NWFE; Bor-Chen Kuo and Landgrebe, 2004) n Regularized Double Nearest Proportion Feature Extraction (RDNP; Hsiao-Yun Huang and Bor. Chen Kuo, submitted)
The idea of Nonpaprametric Discriminant Analysis (NDA; Fukunaga and Mantock, 1983) Try to separate the boundary Instead of separating the means like LDA
Nearest Neighbor Structure Xik k NN for class i k NN for class j
Pairwise Between-Class Scatter Matrix Large weight Xik Xih Small weight
NDA
The Properties of NDA n The between-class scatter matrix Sb is usually full rank. So, the restriction about only min(#class-1, dim) features can be extracted can be liberated. n Since the parametric nature of the Sb is replaced by the nonparametric Sb which leads to preserve important boundary information for classification, NDA is more robust.
Some Considerations about NDA When Overlap Occurs (1) n Based on the definition of the boundary of NDA (the focus portion of the distribution), the points with similar distance among the considered two groups are regarded as the boundary points. n This definition of boundary will fail when overlap occurs, because the points around and within overlap region will tend to have the same weight.
Then Boundary of NDA When Overlap Occurs + + +) (i xl + ++ ? Projection direction
Some Considerations about NDA When Overlap Occurs (2) n In NDA, the k. NN is adopted for measuring the ‘local’ between-class scatter, so the selected k is a very small integer as k. NN people usually do (All the experiments in the paper and book shown by Fukunaga use either k=1 or 3). n This setting of k might cause the data point j and its local mean are very similar (close). The consequence is that the entries of Sbj will be very close to zero and thus cancels out the effect of the weight or makes the Sbj even with less influence among the overall Sb.
Some Considerations about NDA When Overlap Occurs (3) n Also, in the Sb of NDA only one data point is used to represent one group and used the k. NN mean to represent the other local group. This makes the Sb may not measure the scatter between “groups” very well and be easily influenced by the outliers.
One Another Consideration n In the NDA, the boundary is estimated based on the sample. Even when the sample distributions are not overlapped, based on the setting of NDA, the estimated boundary might be too close to the edge (since small k and only one xj in one group is used in Sb). n Like what happened in the hard SVM, extremely clear cut support vectors (boundary) estimated from the sample might have unsatisfied performance due to the over fitting.
The Singularity Problem n In NDA, the unbiased covariance estimate S is still adopted , thus, the singularity problem still exist in NDA.
Nonparamentric Weighted Feature Extraction (NWFE) Large Weight Light Weight
Nonparametric Weighted Feature Extraction (NWFE; Kuo & Landgrebe, 2002, 2004)
Double Nearest Proportion Structure Class j other-class nearest proportion ＊ ＊ Class i Weight reference self-class nearest proportion
Robust Against the Overlap Class j larger weight ＊ ＊ ＋ ＋ smaller weight Class i
The Improvement of the Estimation of Sw (1) n In Regularized Discriminant Analysis (RDA) (Friedman, 1989), an extension of LDA, also proposed a improvement version of the Sw in LDA. The generalized version of that estimate is ∑ˆ= λ ∑ˆ +(1 - λ) (σˆ)^2 I λ is between 0 and 1. The question is how to choose λ? (Friedman suggested using cross-validation. )
The Improvement of the Estimation of Sw (2) n In NWFE, different way to get the local mean and weight in NDA were proposed. But, the most influential effect on the performance improvement is its proposed estimation of Sw n Why 0. 5?
The Shrinkage Estimation of Sw n Let Ψ denote the parameters of the unrestricted high-dimensional model, and Θ the matching parameters of a lower dimension restricted submodel. Also, let U be the estimate of Ψ and T be estimate of Θ. Then the shrinkage (regularized) estimate U* = λ T +(1 -λ )U where λ is between 0 and 1. n λ can be determined analytically by Ledoit and Wolf lemma (2003). Once the T (target) is specified, the λ can be calculated.
Some Targets n J. Schafer and K. Strimmer (2005) proposed six targets for the shrinkage estimate of the Sw.
RDNP Feature Extraction n The feature transformation matrix of RDPN is composed of the eignvectors of where ,
The Properties of RDNP (1) n RDNP is more likely to figure out the boundary when overlap occurs. n Use proportion mean in each group, so the between groups scatter could be measured more properly, the entries of the Sb will not be so close zero, the influence of the outliers will be reduced, and the estimated boundary will not too close to the edge.
The Properties of RDNP (2) n When NPi=Ni and NPj=Nj, then it can be easily shown that the features extracted by the RDNP is exactly the same as the features extracted by the Fisher’s LDA. Thas is, LDA is a special case of RDNP.
Washington DC Mall Image
Indian Pine Site Image
Experiment Result 1 (Washington # of training Samples DC Mall , Classifier: 1 nn, Features 6) LDA NDA RDA NWFE RDNP 0. 5771 0. 5825 0. 8564 0. 8851 0. 9217 0. 8122 0. 8160 0. 8840 0. 9231 0. 9420 0. 8897 0. 8979 0. 9206 0. 9347 0. 9688 20 40 100
Experiment Result 2 (Washington # of training Samples DC Mall , Classifier: SVM, Features 6) LDA NDA RDA NWFE RDNP 0. 5809 0. 5990 0. 8441 0. 8933 0. 9266 0. 8244 0. 8067 0. 8799 0. 9243 0. 9385 0. 8902 0. 8922 0. 9302 0. 9330 0. 9701 20 40 100
A color IR image of a portion of the DC data set RDA with 1 nn NWFE with 1 nn 1 NN-NS (191 bands) RDNP with 1 nn
Experiment Result 3 (Indian Pine Site, Classifier: 1 nn, Features 8) # of training Samples LDA NDA RDA NWFE RDNP 0. 5512 0. 5825 0. 7662 0. 8012 0. 8377 0. 5729 0. 6060 0. 7911 0. 8331 0. 8503 0. 6345 0. 6495 0. 8180 0. 8452 0. 8910 20 40 100
Experiment Result 4 (Indian Pine Site, Classifier: SVM, Features 8) # of training Samples LDA NDA RDA NWFE RDNP 0. 5512 0. 5825 0. 7662 0. 8012 0. 8377 0. 5729 0. 6060 0. 7911 0. 8331 0. 8503 0. 6345 0. 6495 0. 8180 0. 8452 0. 8910 20 40 100
Other Applications n Microarray Data Discrimination n Quality Control n EEG Signal Classification
The End Thank you for your listening.
Скачать презентацию Regularized Double Nearest Neighbor Feature Extraction for Hyperspectral
84f880db396d97f6c1557acb2313568f.ppt