Multi-Group Tracking with Adaptive Target Model Ph D

Multi-Group Tracking with Adaptive Target Model Ph. D thesis proposal Loris Bazzani, Ph. D student (XXIV cycle) University of Verona Department of Computer Science

Objectives Compare existing Multi-Target Tracking methods, studying the sampling technique Propose a new tracking method: Group Tracking Multi- Model robustly and adaptively the target Integrate target model with Multi-Group tracking 2

Outline Introduction Multi-Target Tracking State of the art Multi-Group Tracking Open issues Target Modeling Proposed ideas Conclusions 3

Introduction (1) Tracking: spatial and temporal localization of a mobile object in an environment monitored by sensor(s) Multi-target (MTT): keeping the identity of different targets Reliable: insensible to noise and occlusions Application to Automated Surveillance

Introduction (2) Multi-Group Tracking (MGT): Spatial and temporal localization of groups of objects Motivations: Humans prefer to stay in group rather than alone High-level representation of the relations among the targets MGT is simpler than MTT in a crowded scenario MGT can help MTT when occlusions occur 5

Introduction (3) Multi-Group Tracking (MGT): why is MGT a hard task? 6

Introduction (4) Target Model: A general and representative example that summarizes any possible changing of the target intrinsic variations: pose variation and shape deformation extrinsic variations: illumination changes, camera movement, and occlusions Not considering the above variation causes the failure of the tracking [Ross 08] Fit with the re-identification problem 7

Multi-Target Tracking (1) Abstract Formulation [Arulampalam 02] State Space Approach for modeling discrete-time dynamic systems State: abstract nature of the target Measurement: “visible” dimensions of the state space Filtering We observe the real world events as a state by the measurement process Objective: estimating the state of the system at each instant given the measurements 8

Multi-Target Tracking (2) Data Association [Bar-Shalom 87] The observer has at his disposal a huge amount of measurements Finding the correct correspondences between measurements and states of the system 9

Multi-Target Tracking (3) 10

Multi-Target Tracking (4) Particle Filter (PF) [Isard 01] 11

Multi-Target Tracking (5) State Space Conformation [Isard 01] (+) Efficient sampling (-) No interaction modeling [Mac. Cormick 00] (+) Implicit interaction modeling (-) Curse of dimensionality [Lanz 06] (+) Efficient sampling (+) Implicit interaction modeling

Multi-Target Tracking (6) - HJS vs. MHT [Bazzani 09] - HJS pros: 1) One track is kept for each target 2) Partial occlusions are handled; 3) Deal with non-linearity of people motion. MHT cons: 1) Multiple tracks cause proliferation in the number of tracks 2) Occlusions generate new tracks 3) Not robust to non-linear people motion

Multi-Target Tracking (7) Open issues of PF-based MTT (and MGT): Sampling Method Dynamic Model Linear-Gaussian model Observation Model State Estimation Maximum-A-Posteriori or Weighted Mean 14

Multi-Target Tracking (8. 1) Sampling Sequential importance sampling/re-sampling [Arulampalam 02]: classical PF + degeneracy problem avoiding by re-sampling Regularized PF [Arulampalam 02]: resamples applying a Kernel to the particles MCMC [Andrieu 03]: defines a Markov chain over the state space, such that the stationary distribution of the chain is equal to the sought posterior Reversible-Jump MCMC [Khan 05]: switches between variable dimensional state spaces Rao-Blackwellizing PF [Schindler 05]: analytically computes a portion of the distribution other the state space

Multi-Target Tracking (8. 2) Observation model Likelihood: compare an observation z given a hypothesis of state of the system x Usually defined in the Gibbs form: metric where d is a x and z MUST be represented in the same feature space ISSUES: feature space (for x and z) and metric definitions occlusion handling 17

Multi-Target Tracking (9) - Proposed Research Occlusion Handling Study and implement RJ-MCMC particle filter Propose a set of jumps in order to cope with tracking a variable number of objects Propose an observation model in RJ-MCMC framework 18

Multi-Group Tracking (1) (-) Social interactions cannot be caught defines a group as the moving regions, by the foreground analysis extracted from a foreground yields a loss (-) The inter-group dynamic analysis of appearance informations infers the MGT from MOT tracks (-) Direct dependence from the estimation carried out by thenot reliable (-) The MOT estimation is MOT (e. g. tracksocclusions occur when clustering) uses the out the above problems and (+) Cancel foreground information MOT to detect groups, buttask tracks (-) Model creation is a hard then them as different entities 19

Multi-Group Tracking (2) Foreground-based MGT Tracking at three levels of abstraction [Mc. Kenna 00] : Regions: connected component that have been tracked for T frames People: one or more regions grouped together Groups: one or more people grouped together, if they share a region (+) Simplicity (-) heuristic FG analysis 20

Multi-Group Tracking (3) MGT from MOT MCMC PF for group tracking [Pang 07]: Track groups as ensemble of targets analyzing : group variable Treated as Bayesian estimation problem Group structure model: captures the relations among objects 22

Multi-Group Tracking (4) - Proposed Research Problems: Definition of “Group”: an entity containing targets with similar characteristics (e. g. motion, interactions, . . . ) Deterministic/formal definition as an ensemble of objects Add non-deterministic component into the tracking method Intra-group occlusions: if we know that the objects hasn’t left the group, we infer that it is still into the group Inter-group occlusions: tracking of groups 23

Multi-Group Tracking (5) - Proposed Research - 24

Multi-Group Tracking (6) - Proposed Research Use a MOT method Create a MGT method (track only groups) Definition of group dynamics -> sociological studies Definition of a group observation model Define a collaborative probabilistic framework in order to share MGT and MOT informations 25

Target Modeling (1) Train the model using the appearance data available before tracking begins Adapt the model to account for its changes in appearance, using an online learning method Open Issues: Representation of the target: feature space Leaning technique 27

Target Modeling (2) Feature Color Histogram: which is the best color space for tracking? [Sebastian 08] (-) No spatial information Color correlogram [Huang 99], spatiogram [Birchfield 05], multi-resolution histogram [Hadjidemetriou 01] (+) Add the spatial information (-) Increase the computational burden 28

Target Modeling (3) Feature Covariance descriptors [Porikli 06] Spatial and appearance attributes (+) Natural way of fusing multiple features (-) Computationally expensive -> integral images 29

Target Modeling (4) Fixed target models Ensemble of localized features [Gray 08] Define a feature space and let machine learning approach find the best representation Ada. Boost extracts the object representation: the most discriminative set of features, and the similarity measures: the most discriminative set of likelihood ratio test Used for re-identification problem 30

Target Modeling (5) Adaptive target models Incremental learning of Covariance-based descriptors [Porikli 06] Principal Component Analysis (PCA) incremental learning [Ross 08] Convex combination of models using a learning rate Feature-based model Parametric model 31

Target Modeling (6) Patch-based (local) updating [Kwon 09] Evolve photometric and geometric appearance Local Patches can be added, deleted or moved to different position Examining the patches by landscape analysis Bad patches are modified on-line: background patches and patches in regions with high density of patch are deleted Good patches are moved Appearance model is updated with a convex combination 32

Target Modeling (7) - Proposed Research Part-based multiple-features: Invariances: view point, partial occlusions Maximally Stable Color Regions Global illumination, view point, deformations, partial occlusions HS(V) histograms Local Recurrent high-structured patches Temporal updating: Delete the non-stable features Cover the Variability of the stable features 33 partial occlusions, illumination

Conclusions Compare existing Multi-Target Tracking methods, studying the sampling technique Propose a new tracking method: Group Tracking Multi- Model robustly and adaptively the target Integrate target model with Multi-Group tracking, using HJS and RJ-MCMC 34

The beginning Now, we “just” put the proposed ideas into practice Thanks for attention Questions? 35

References [Ross 08] D. A. Ross, J. Lim, R. S. Lin, and M. H. Yang. Incremental learning for robust visual tracking. International Journal of Computer Vision, 77(1): 125– 141, 2008. [Arulampalam 02] M. S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp. A tutorial on particle filters for online nonlinear/non-gaussian bayesian tracking. IEEE Trans. on Signal Processing, 50(2): 174– 188, 2002. [Bar-Shalom 87] Y. Bar-Shalom. Tracking and data association. Academic Press Professional, Inc. , San Diego, CA, USA, 1987. [Isard 01] M. Isard and J. Mac. Cormick. Bramble: A bayesian multipleblob tracker. In IEEE Int. Conf. on Computer Vision, 2001. [Mac. Cormick 00] John Mac. Cormick and Andrew Blake. A probabilistic exclusion principle for tracking multiple objects. Int. J. Comput. Vision, 39(1): 57– 71, 2000. [Lanz 06] O. Lanz. Approximate bayesian multibody tracking. IEEE Trans. on Pattern Analysis and Machine Intelligence, 28(9): 1436– 1449, 2006. [Andrieu 03] Christophe Andrieu, Nando de Freitas, Arnaud Doucet, and Michael I. Jordan. An introduction to mcmc for machine learning. Machine Learning, 50(1): 5– 43, January 2003. 36

References [Khan 05] Z. Khan, T. Balch, and F. Dellaert. Mcmc-based particle filtering for tracking a variable number of interacting targets. IEEE Trans. on Pattern Analysis and Machine Intelligence, 27(11): 1805– 1819, 2005. [Schindler 05] G. Schindler and F. Dellaert. A Rao-Blackwellized Parts-Constellation Tracker. In ICCV Workshop on Dynamical Vision; International Conference on Computer Vision, 2005. Springer [Mckenna 00] Stephen J. Mckenna, Sumer Jabri, Zoran Duric, Harry Wechsler, and Azriel Rosenfeld. Tracking groups of people. Computer Vision and Image Understanding, 2000. [Pang 07] Sze Kim Pang, Jack Li, and Simon Godsill. Models and algorithms for detection and tracking of coordinated groups. In Symposium of image and Signal Processing and Analisys, 2007. [Sebastian 08] Sebastian, P. ; Yap Vooi Voon; Comley, R. , "The effect of colour space on tracking robustness, " Industrial Electronics and Applications, 2008. ICIEA 2008. 3 rd IEEE Conference on , vol. , no. , pp. 2512 -2516, 3 -5 June 2008 [Huang 99] J. Huang, S. Ravi Kumar, M. Mitra, W. J. Zhu, and R. Zabih. Spatial color indexing and applications. International Journal of Computer Vision, 35(3): 245– 268, 1999. [Birchfield 05] ST Birchfield and S. Rangarajan. Spatiograms versus histograms for region-based tracking. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. CVPR 2005, volume 2, 2005. 37

References [Hadjidemetriou 01] E. Hadjidemetriou, MD Grossberg, and SK Nayar. Spatial information in multiresolution histograms. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, volume 1, 2001. [Porikli 06] F. Porikli, O. Tuzel, and P. Meer. Covariance tracking using model update based on lie algebra. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 1, 2006. [Gray 08] D. Gray and H. Tao. Viewpoint Invariant Pedestrian Recognition with an Ensemble of Localized Features. In Proceedings of the 10 th European Conference on Computer Vision: Part I, pages 262– 275. Springer, 2008. [Kwon 09] J. S. Kwon and K. M. Lee. Tracking of a non-rigid object via patch-based dynamic appearance modeling and adaptive basin hopping monte carlo sampling. pages 1208– 1215, 2009. 38