Unsupervised and Transfer Learning Challenge Isabelle Guyon Clopinet, California 1 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
CREDITS Data donors: Handwriting recognition (AVICENNA) -- Reza Farrahi Moghaddam, Mathias Adankon, Kostyantyn Filonenko, Robert Wisnovsky, and Mohamed Chériet (Ecole de technologie supérieure de Montréal, Quebec) contributed the dataset of Arabic manuscripts. The toy example (ULE) is the MNIST handwritten digit database made available by Yann Le. Cun and Corinna Costes. Object recognition (RITA) -- Antonio Torralba, Rob Fergus, and William T. Freeman, collected and made available publicly the 80 million tiny image dataset. Vinod Nair and Geoffrey Hinton collected and made available publicly the CIFAR datasets. See the techreport Learning Multiple Layers of Features from Tiny Images, by Alex Krizhevsky, 2009, for details. Human action recognition (HARRY) -- Ivan Laptev and Barbara Caputo collected and made publicly available the KTH human action recognition datasets. Marcin Marszałek, Ivan Laptev and Cordelia Schmid collected and made publicly available the Hollywood 2 dataset of human actions and scenes. Text processing (TERRY) -- David Lewis formatted and made publicly available the RCV 1 -v 2 Text Categorization Test Collection. Ecology (SYLVESTER) -- Jock A. Blackard, Denis J. Dean, and Charles W. Anderson of the US Forest Service, USA, collected and made available the (Forest cover type) dataset. Web platform: Server made available by Prof. Joachim Buhmann, ETH Zurich, Switzerland. Computer admin. : Thomas Fuchs, ETH Zurich. Webmaster: Olivier Guyon, Mister. P. net, France. Protocol review and advising: • David W. Aha, Naval Research Laboratory, USA. • Gideon Dror, Academic College of Tel-Aviv Yaffo, Israel. • Vincent Lemaire, Orange Research Labs, France. • Gavin Cawley, University of east Anglia, UK. • Olivier Chapelle, Yahoo!, California, USA. • Gerard Rinkus, Brandeis University, USA. • Yoshua Bengio, Universite de Montreal, Canada. • David Grangier, NEC Labs, USA. • Andrew Ng, Stanford Univ. , Palo Alto, California, USA • Graham Taylor, NYU, New-York. USA. • Andrew Ng, Stanford Univ. , Palo Alto, California, USA. • Yann Le. Cun, NYU. New-York, USA. Beta testing and baseline methods: • Gideon Dror, Academic College of Tel-Aviv Yaffo, Israel. • Vincent Lemaire, Orange Research Labs, France. Unsupervised and Transfer Learning Challenge 2 http: //clopinet. com/ul
What is the problem? 3 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Labeling data is expensive $$ $$$$$ 4 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Examples of domains • • Chemo-informatics Handwriting & speech recognition Image & video processing Text processing Marketing Ecology Embryology 5 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Scenarios Active Learning Semi-supervised Learning Transfer Learning Unsupervised Learning 6 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Setting Billions of images unlabeled or with different class labels Philip and Thomas Philip Anna Martin Solene Bernhard Anna, Thomas and GM Omar, Thomas and Philip Thomas Unsupervised and Transfer Learning Challenge Personal data, only a few labeled examples 7 http: //clopinet. com/ul
Datasets 8 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Datasets 9 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Difficulties • • Sparse data Unbalanced class distributions Noisy data Large datasets No categorical variables No missing values Must turn in results on ALL datasets. 10 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Protocol 11 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Data Split Development data Validation data Final evaluation data Unsupervised and Transfer Learning Challenge (phase 1) Competitors Evaluators Type 2 Labels Type 3 Labels 12 http: //clopinet. com/ul
Data Split Development data (phase 2) Competitors Type 1 Labels Validation data Final evaluation data Unsupervised and Transfer Learning Challenge Evaluators Type 2 Labels Type 3 Labels 13 http: //clopinet. com/ul
On-line feed-back For each dataset in {Avicenna, Harry, …} 1. Download the (P x N) development data matrix and the (4096 x N) validation & final data matrices. 2. Create transformed data matrices (4096 x M), M 4096, or similarity matrices (4096 x 4096), for the validation and/or final data. 3. Submit on the website. 4. Retrieve the learning curves on validation data. 14 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Two phases • Phase 1: Unsupervised Learning – Only unlabeled data available. – Deadline: February 28, 2011. • Phase 2: Transfer Learning – A limited amount of transfer labels available (labels on examples of the development set of classes not used for evaluation). – Deadline: April 15, 2011. Unsupervised and Transfer Learning Challenge 15 http: //clopinet. com/ul
Evaluation 16 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
AUC score For each set of samples queried, we assess the predictions of the learning machine with the Area under the ROC curve. 17 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Area under the Learning Curve (ALC) Linear interpolation. Horizontal extrapolation. 18 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Classifier used • Linear discriminant: f(x) = w. x = Si wi xi • Hebbian learning: X = (p, N) training data matrix Y {– 1/p– , +1/p+}p target vector w = X’ Y = (1/p+)Sk pos xk –(1/p–) Sk neg xk 19 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Kernel version • Kernel classifier: f(x) = Sk ak k(xk , x) with a linear kernel k(xk , x) = xk. x and with ak = – 1/p– , if xk neg ak = +1/p+ , if xk pos • Equivalent linear discriminant f(x) = (1/p+)Sk pos xk. x – (1/p–) Sk neg xk. x =w. x with w = (1/p+)Sk pos xk – (1/p–) Sk neg xk Unsupervised and Transfer Learning Challenge 20 http: //clopinet. com/ul
Justification • Simple classifier • Robust against overfitting • Puts emphasis on learning a good data representation • Easily kernelized 21 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Getting started… 22 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Phase 1: no labels • No learning at all: – Normalization of examples or features – Construction of features (e. g. products) – Generic data transformations (e. g. taking the log, Fourier transform, smoothing, etc. ) • Unsupervised learning: – Manifold learning to reduce dimension (and/or orthogonalize features) – Clustering to construct features – Generative models and latent variable models 23 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
PCA • The canonical example of manifold learning • Diagonalize: X X' = U D U' • The eigenvectors U constitute a set of orthogonal features: U'U=I. • Select a few U corresponding to the largest eigenvalues as new feature vectors. • Similar effect as regularization (for example ridge regression). 24 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Other manifold algorithms • • • ICA Kernel PCA Kohonen maps Auto-encoders MDS, Isomap, LLE, Laplacian Eigenmaps Regularized principal manifolds 25 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
K-means clustering • Start with random cluster centers. • Iterate: o Assign the examples to their closest center to form clusters. o Re-compute the centers by averaging the cluster members. • Create features, e. g. fk= exp –g ||x-xk|| Clusters of ULE valid after 5 it. 26 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Other clustering algorithms • Overlapping clusters (Gaussian mixtures, fuzzy C-means) • Hierarchical clustering • Graph partitioning • Spectral clustering 27 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Deep Learning Greedy layer-wise unsupervised pre-training of multi-layer neural networks and Bayesian networks, including: • Deep Belief Networks (stacks of Restricted Boltzmann machines) decoder • Stacks of auto-encoders encoder 28 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Resources • Unsupervised Learning. Zoubin Ghahramani. http: //www. gatsby. ucl. ac. uk/~zoubin/course 04/ul. pdf • Nonlinear dimensionality reduction. http: //en. wikipedia. org/wiki/Nonlinear_dimensionality_reduction • Data Clustering: A Review. Jain et al. http: //citeseerx. ist. psu. edu/viewdoc/summary? doi=10. 1. 1. 18. 2720 • Why Does Unsupervised Pre-training Help Deep Learning? Erhan et al. http: //jmlr. csail. mit. edu/papers/volume 11/erhan 10 a. pdf 29 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Phase 2: transfer learning • This challenge: No transfer labels available for the primary/target task(s). Some labels available for secondary/source tasks. • Upcoming challenge on Inductive Transfer Learning: A few labels available for the primary/target task(s) and many more labels available for secondary/source tasks. 30 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Transfer learning taxonomy Adapted from survey of Sinno Jialin Pan and Qiang Yang 31 30 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Cross-task Learning • Similarity or kernel learning: – Siamese neural networks – Graph-theoretic methods • Data representation learning: – Deep neural networks – Deep belief networks (re-use the internal representation created by the hidden units and/or output units) 32 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Resources • A Survey on Transfer Learning. Pan and Yang. http: //www 1. i 2 r. astar. edu. sg/~jspan/publications/TLsurvey_0822. pdf • Signature Verification using a "Siamese" Time Delay Neural Network. Bromley et al. http: //citeseerx. ist. psu. edu/viewdoc/summary? doi=10. 1. 1. 28. 4 792 • Fast Graph Laplacian Regularized Kernel Learning via Semidefinite–Quadratic–Linear Programming. Wu et al. http: //books. nips. cc/papers/files/nips 22/NIPS 2009_0792. pdf • Transfer Learning Techniques for Deep Neural Nets. Gutstein thesis. http: //robust. cs. utep. edu/~gutstein/sg_home_files/thesis. pdf 33 http: //clopinet. com/ul Unsupervised and Transfer Learning Challenge
Dec 2010 -March 2011 http: //clopinet. com/ul Competitors Development data Type 1 Labels Validation data Final evaluation data Unsupervised and Transfer Learning Challenge • Prizes: $6000 + free registrations + travel awards • Dissemination: Workshops at ICML and IJCNN; proceedings in JMLR W&CP. Type 2 Labels Type 3 Labels Evaluators 34 http: //clopinet. com/ul
July 2011, ICML - Dec 2011, NIPS http: //clopinet. com/tl Competitors Development Data Primary and Secondary tasks July, 2011 Labels Validation Data Primary task Sept, 2011 Challenge data Primary task Unsupervised and Transfer Learning Challenge Two domains of tasks: - Synthetic, Real-world - Supervised training examples - Concept (binary class) tasks - 5 -10 secondary tasks, 1 primary - Impoverished primary task data - Diversity of tasks with varying degree of relatedness to primary task Labels Evaluators 35 http: //clopinet. com/ul
Challenge June 2011 -Nov. 2011 http: //clopinet. com/gs (in preparation) STEP 1: Develop a “generic” sign language recognition system that can learn new signs with a few examples. STEP 2: At conference: teach the system new signs. STEP 3: Live evaluation in front of audience. 36 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul
Скачать презентацию Unsupervised and Transfer Learning Challenge Isabelle Guyon Clopinet
b28030d2f28d8c21ddb8f963f1fd1c61.ppt