Similarity Scores based on Background Samples 讲解人阚美娜

Similarity Scores based on Background Samples 讲解人：阚美娜日期： 2010 -04 -09

Similarity Scores based on Background Samples l 该方法在LFW数据库上目前是性能最好的 Zhimin Cao , Qi Yin , Xiaoou Tang , Jian Sun. Face Recognition with Learning-based Descriptor. CVPR 2010

文章信息 l 相关文章 One-shot Similarity Ø Lior Wolf, Tal Hassner, and Yaniv Taigman. Descriptor Based Methods in the Wild. Workshop in ECCV, 2008. Ø L. Wolf, T. Hassner, and Y. Taigman. The One-Shot Similarity Kernel. ICCV 2009. 扩展 1 Ø Yaniv Taigman, Lior Wolf, and Tal Hassner. Multiple One-Shots for Utilizing Class Label Information. BMVC, 2009. 扩展 2 Ø Lior Wolf, Tal Hassner, and Yaniv Taigman. Similarity Scores based on Background Samples. ACCV, 2009.

提纲 l 作者信息 l 文章信息 l One-Shot Similarity Ø One-Shot Similarity (ECCV 08 workshop) Ø One-Shot Similarity Kernel (ICCV 09) l One-Shot Similarity 扩展 1 Ø Multiple One-Shot Similarity (BMVC 09) l One-Shot Similarity 扩展 2 Ø Similarity Based on Background Samples (ACCV 09)

作者1 – Lior Wolf l Lior Wolf Ø faculty member at The Blavatnik School of Computer Science at Tel Aviv University (以色列特拉维夫大学) Ø postdoc working with Prof. Poggio at CBCL, MIT Ø Ph. D working with Prof. Shashua at the Hebrew U, Jerusalem(耶路撒冷，希伯来大学) l Awards: Ø ICCV workshop on e. Heritage 2009, Best paper award Ø ICCV 2001, Marr prize honorable mention Ø ECCV 2000, Best paper award Amirim excellence program scholarship 1996 -199 l Publications: Ø Lior Wolf, Rotem Littman, Naama Mayer, Nachum Dershowitz, R. Shweka, and Y. Choueka. Automatically Identifying Join Candidates in the Cairo Genizah Post ICCV workshop on e. Heritage and Digital Art Preservation, 2009. Best paper award winner. Ø Lior Wolf and A. Shashua. On Projection Matrices P^k -> P^2, k=3, . . . , 6, and their Applications in Computer Vision. ICCV 2001. Honorable Mention for the 2001 Marr Prize. Ø A. Shashua and Lior Wolf. Homography Tensors: On Algebraic Entities That Represent Three Views of Static or Moving Planar Points. ECCV 2000. Best paper award winner. Ø CVPR 2010(2) , EUROGRAPHICS 2010(2) , ACM SIGGRAPH Asia 2009(1) , ICCV 2009(3) l Homepage: http: //www. cs. tau. ac. il/~wolf/#personal

作者2 – Tal Hassner l Tal Hassner Ø senior faculty member at the Open University of Israel Ø Ph. D - The Weizmann Institute of Science, Israel(以色列魏茨曼科学学院) l Research Interests: Ø Computer Vision and Computer Graphics Ø Face recognition / Subspace representations / Single view 3 D reconstruction / Image similarity / Image descriptors / Non-parametric methods / New view synthesis (image based rendering) / Triangulated mesh composition / 3 D model decoration l Publications: Ø Ronen Basri, Tal Hassner and Lihi Zelnik-Manor*, A General Framework for Approximate Nearest Subspace Search. ICCV 2009 Workshop on Subspace Methods. Ø 4 papers on LFW cooperation with Lior Wolf. l Homepage: http: //www. openu. ac. il/home/hassner/

作者3 – Yaniv Taigman l Yaniv Taigman Ø Co-Founder and CTO at Face. com, 2007~present Ø Head of Algorithms at Princeton Video Image (Scidel Technologies), June 2005 ~ March 2008 Ø Software Engineer at Princeton Video Image (Scidel Technologies), May 2000 ~ June 2005 Ø Education： Tel Aviv University Ø M. Sc , Computer Science , 2005 — 2006 Ø B. Sc , Mathematics(Major) and Computer Science , 2002 — 2005 l Websites: Øhttp: //il. linkedin. com/in/taigman Øhttp: //face. com/

提纲 l 作者信息 l 文章信息 l One-Shot Similarity Ø One-Shot Similarity (ECCV 08 workshop) Ø One-Shot Similarity Kernel (ICCV 09) l One-Shot Similarity 扩展 1 Ø Multiple One-Shot Similarity (BMVC 09) l One-Shot Similarity 扩展 2 Ø Similarity Based on Background Samples (ACCV 09)

Abstract l One-Shot Similarity Ø The One-Shot Similarity measure has recently been introduced as a means of boosting the performance of face recognition systems. Given two vectors, their One-Shot Similarity score reflects the likelihood of each vector belonging to the same class as the other vector and not in a class defined by a fixed set of “negative” examples (background samples ) that is, examples which do not belong to any of the classes being learned. Ø Also we show that: (1) when using a version of LDA as the underlying classifier, this score is a Conditionally Positive Definite kernel and may be used within kernel-methods (e. g. , SVM) (2) that it is effective as an underlying mechanism for image representation. l Multiple One-Shot Similarity Ø An appealing aspect of One-Shot Similarity（OSS） approach is that it does not require class labeled training data. However, when labeled information is available, the OSS score does not benefit from it. Here, we explore how the One-Shot Similarity may nevertheless benefit from label information by computing the One-Shot Score multiple times. l Similarity Score based on Background Samples Ø First, we define and evaluate the "Two-Shot Similarity" (TSS) score as an extension to the recently proposed "One-Shot Similarity" (OSS) measure. Both these measures utilize background samples to facilitate better recognition rates. Ø Second, we examine the ranking of images most similar to a query image and employ these as a descriptor for that image.

摘要 l One-Shot Similarity Ø 最近提出了一种提高人脸识别性能的One-Shot Similarity measure。给定两个向量，One-Shot Similarity反映的是一个向量和另外一个向量属于同一类而不是属于反例样本集的可能性。反例样本集(也称作Background Sample) 是不属于被比较(学习)的类别的样本组成的一个集合。 Ø 同时我们还给出: (1)当使用Free-Scale LDA作为隐含的分类器时，OSS是一个条件正定的 Kernel。(2) OSS作为图像表示是很有效的。 l Multiple One-Shot Similarity Ø OSS很吸引人的一个特点是它不需要标注训练样本。因此当label信息已知时，OSS是无法利用这些信息的，这里，我们提出一种能够利用label信息的Multiple OSS，它将反例样本集按照 label信息分为多个子集，计算多个OSS。 l Similarity Score based on Background Samples Ø 通过对OSS进行扩展，提出Two-Shot Similarity。OSS和TSS都利用了background sample，可以得到更好的性能。 Ø 同时提提出了一种利用背景样本的相似度度量方法：用背景样本集中与查询图像最相似的n个样本的ranking来描述该查询样本。 l 在LFW的 Image-Restricted测试下， One-Shot Similarity及其扩展的方法目前是性能最好的.

提纲 l 作者信息 l 文章信息 l One-Shot Similarity Ø One-Shot Similarity (ECCV 08 workshop) Ø One-Shot Similarity Kernel (ICCV 09) l One-Shot Similarity 扩展 1 Ø Multiple One-Shot Similarity (BMVC 09) l One-Shot Similarity 扩展 2 Ø Similarity Based on Background Samples (ACCV 09)

One-Shot Similarity* l One-Shot learning-learn from single or few samples Ø 是物体分类问题，基于machine learning的物体分类算法通常需要很多的训练数据，One-shot learning的目标是从一个或者少量的训练数据中学习物体分类信息。 Ø One-shot learning与single object recognition、standard recognition区别在于， One-shot learning更强调知识迁移，融合已经学过的类别 identity信息，只需要很少量的训练样本即可进行新的学习。 l One-Shot Similarity Ø Draw its motivation from one-shot learning which learn from one or few training exampls Ø 核心思想：对于给定的两个样本，学习一个专门针对这两个样本的相似度计算方法 *Lior Wolf, Tal Hassner, and Yaniv Taigman. Descriptor Based Methods in the Wild. Workshop in ECCV, 2008.

One-Shot Similarity* l 要解决的问题 Ø pair matching(或者确认问题)：给定两幅图像I和J，判断他们是否属于同一类 l 传统方法 Ø 通过一个训练集预先训练一个模型，用该模型对图像I和J计算相似度 l One-Shot Similarity Ø 对待比较的图像I和J，专门训练一个模型计算相似度 Ø 如果相似度大于某个阈值就判断为同一类，反之，不同类。

One-Shot Similarity* l 基于One-Shot Similarity的训练和测试过程 Ø 训练过程：学习一个相似度阈值 Ø 对于给定训练集B中的所有图像对，计算one-shot similarity，用SVM学习一个相似度阈值t，使得相似度大于t的图像对是来自同一个人，相似度小t的图像对是来自不同人的。 Ø 测试过程： Ø 给定图像I, J，计算图像I和J的One-Shot Similarity，如果相似度大于阈值t则认为是同一个人，否则认为是不同人

One-Shot Similarity的计算：分类器可以选择各种不同的分类器 LDA ：可以解析求解 SVM：迭代求解 I J Model 1 A Model 2 Score 1 One-Shot Similarity Score 2

基于LDA 的OSS l 对于一个正例样本和一个反例样本集，做两类的 LDA建模，得到的分类器(投影向量)为： μp: 正例样本的均值 μn: 反例样本的均值 l Score： Ø v 0：bias term，投影之后两点连线的中点 l 样本xi和xj的OSS： VT

基于Free-Scale LDA 的OSS* 基于标准LDA的OSS： OSS(xi, xj)= 基于Free-Scale LDA： OSS= CPD Kernel *L. Wolf, T. Hassner, and Y. Taigman. The One-Shot Similarity Kernel. ICCV 2009.

基于Free-Scale LDA 的OSS l One-Shot Similarity通常并不是一个PD或CPD Kernel l 基于Free-Scale LDA的OSS是CPD Kernel l 基于Free-Scale LDA的OOS，它的指数是PD Kernel： CPD Kernel：

OSS Kernel证明(1/2) 基于Free-Scale LDA的OSS： OSS= = K 1 + K 2 K 1是PD Kernel，K 2是CPD Kernel

OSS Kernel证明(2/2) 可以证明K 2是CPD Kernel： K 1是PD Kernel，K 2是CPD Kernel，因此OSS = K 1+K 2是CPD Kenerl 如果u. A=0，OSS可以看做是在一个变换后的空间中，xi和xj的内积减掉xi 模的平方和xj模的平方

One-shot Similarity 实验 l LFW(Labeled Face in the Wild) Ø a database of face photographs designed for studying the problem of unconstrained face recognition. Ø 13, 000 images, 1680 of the people pictured have two or more distinct photos in the database. Ø 2 views Ø view 1： Model selection and algorithm development Ø view 2： Performance reporting Ø Training Ø Image-restricted： only know whether a pair of images is matched or mismatched. Ø Image-unrestricted：image’s label is provided

One-Shot Similarity 实验结果欧式距离的结果 One-shot Similarity的结果

实验结果 l Image Representation on LFW Ø从图像上随机选取 1000个patch，计算 1000个 oss作为图像特征，用linear SVM进行确认 Feature：LBP

实验结果 l Insect Species identification Gallery训练SVM Feature：Bags-of-Feature, Hessian-Affine extractor, SIFT

Visualization of OSS distances l LFW中随机选取 5个人的图像 l LBP特征 l 通过计算距离的2 D Multidimensional-Scaling显示在二维图像上

One-Shot Similarity优势 l Unlabeled Negative example set ‘A’ 不需要标注反例样本集 l Use discriminative learning Explicitly build models which underscore the difference between them* 判别式的模型显示地强调了待比对图像的差异 l Models are produced per the vectors being compared and so are often better suited to comparing them* 针对每对图像训练模型，更适合对他们进行比较 l OSS score reflects the likelihood of each vector belonging to the same class as the other vector and not in a class defined by a fixed set of ‘negative’ example. ▲ OSS 反映的是一个向量和另外一个向量属于同一类而不属于反例样本集的可能性 l 传统的similarity measure是通过一个预先学好的模型直接进行比较，而 OSS是专门针对待比较的图像学习的判别式模型 *L. Wolf, T. Hassner, and Y. Taigman. The One-Shot Similarity Kernel. ICCV 2009. ▲Yaniv Taigman, Lior Wolf, and Tal Hassner. Multiple One-Shots for Utilizing Class Label Information. BMVC, 2009.

提纲 l 作者信息 l 文章信息 l One-Shot Similarity Ø One-Shot Similarity (ECCV 08 workshop) Ø One-Shot Similarity Kernel (ICCV 09) l One-Shot Similarity 扩展 1 Ø Multiple One-Shot Similarity (BMVC 09) l One-Shot Similarity 扩展 2 Ø Similarity Based on Background Samples (ACCV 09)

Multiple One-Shot Similarity* l One-Shot Similarity Ø 反例样本集A可以包含多个类，没有使用label information l Multiple One-Shot Similarity Ø 利用label information，将反例样本集A分成n个子集，每个子集只包含一类样本。对于每个子集Ai计算一个OSS Muliple OSS *Yaniv Taigman, Lior Wolf, and Tal Hassner. Multiple One-Shots for Utilizing Class Label Information. BMVC, 2009.

Multiple One-Shot Similarity* l Multiple One-Shot Similarity Ø 子集可以用identity，pose或其它条件进行划分 Ø Reason： Ø 集合A包含很多因素，姿态，表情，身份，在计算OSS时，I作正例， A作反例训练分类器时，由于I只包含一种特定的姿态、表情、身份信息，得到的分类器可能会基于任何一种因素，不一定是身份信息。 Ø Multiple OSS将A分成很多子集，每个子集包含一个人，但可以有多种姿态、表情等因素，因此训练到的分类器更可能是基于身份信息的 *Yaniv Taigman, Lior Wolf, and Tal Hassner. Multiple One-Shots for Utilizing Class Label Information. BMVC, 2009.

系统描述

One-shot Similarity 实验 l LFW(Labeled Face in the Wild) Ø a database of face photographs designed for studying the problem of unconstrained face recognition. Ø 13, 000 images, 1680 of the people pictured have two or more distinct photos in the database. Ø 2 views Ø view 1： Model selection and algorithm development Ø view 2： Performance reporting Ø Training Ø Image-restricted： only know whether a pair of images is matched or mismatched. Ø Image-unrestricted：image’s label is provided

实验结果 ITML：Information Theoretic Metric Leaning technique

实验结果 Single descriptor: LBP

提纲 l 作者信息 l 文章信息 l One-Shot Similarity Ø One-Shot Similarity (ECCV 08 workshop) Ø One-Shot Similarity Kernel (ICCV 09) l One-Shot Similarity 扩展 1 Ø Multiple One-Shot Similarity (BMVC 09) l One-Shot Similarity 扩展 2 Ø Similarity Based on Background Samples (ACCV 09)

扩展 2：Similarity Scores based on Background Samples* l Background Samples Ø called Negative Sample Sets before(‘A’) Ø 为什么 background samples 对定义相似度函数是有用的呢？ Ø 样本是位于一个向量空间的，这个空间可以使用多种metric，为了确定哪种metric是最适合当前任务的，我们需要对样本所在流形的内在结构进行分析。 Ø 有监督学习的方法需要很多label的信息。对于大量的没有label 信息的background sample也可以提供一些信息 *Lior Wolf, Tal Hassner, and Yaniv Taigman. Similarity Scores based on Background Samples. ACCV, 2009.

扩展 2： Similarity Scores based on Background Samples l 对于大量的没有label信息的background sample也可以提供一些信息如： Ø 样本更接近某个样本还是更接近background set ？ (one-shot) Ø 两个样本可以很好的与background set分开吗？(twoshot) Ø 两个样本在background set中有相似的近邻吗？ (ranking similarity) 扩展 2

扩展 2. 1: Two-Shot Similarity One-Shot Similarity Two-Shot Similarity 算法 Function One-Shot-Similarity (I, J, A) Model 1 = train (I, A) Score 1 = classify (J, Model 1) Model 2 = train (J, A) Score 2 = classify (I, Model 2) return ½ * (Score 1 + Score 2) Function Two -Shot-Similarity (I, J, A) score = train ({ I, J }, A) return score

扩展 2. 2: Ranking Based Background Similarity l 给定background sample set A，计算图像I和J的ranking similarity Ø计算I(J)与A中每幅图像的相似度，得到两组排序向量r. I (r. J)和πI(πJ) Ør. I (r. J) Ø r. I(k) – A中第k副图像与图像I的相似度排在第几位 ØπI(πJ) ØπI (k) – A中与图像I第k个最相似的图像的序号 Ø图像I和J的ranking相似度：

实验结果 l Feature Ø LBP、Gabor、TPLBP、FPLBP、SIFT以及这些特征的平方根共 10种特征 l DB：LFW Methods accuracy 欧式距离 0. 7521 OSS 0. 8207 TSS 0. 6593 Ranking descriptor 0. 6918 Methods accuracy 欧式距离 0. 7521 欧式距离+OSS 0. 8398 欧式距离+OSS+TSS 0. 8513 欧式距离+OSS+TSS+Ranking descriptor 0. 8557 单独的TSS和Ranking相似度效果并不好，但与其他方法结合会有提升

实验结果特征： 10种特征，相似度：欧式距离 + OSS-LDA + TSS-LDA + Ranking descriptor + OSS-SVM + TSS-SVM 相似度融合：SVM

总结 l One-Shot Similarity Ø对于给定的两幅测试图像，设计一个专门针对这两幅图像的判别模型 Ø不需要标注background sample l 扩展 1：利用label信息 ØMultiple One-Shot Similarity l 扩展 2：利用background sample ØTwo-shot similarity ØRanking Similarity

Thanks for Attentions ！

LFW Result

Similarity Scores based on Background Samples 讲解人 阚美娜

Similarity Scores based on Background Samples 讲解人阚美娜