A Probabilistic Framework for Video Representation Arnaldo Mayer, Hayit Greenspan Dept. of Biomedical Engineering Faculty of Engineering Tel-Aviv University, Israel Jacob Goldberger, CUTe Systems, Ltd.
Introduction • In this work we describe a novel statistical video representation and modeling scheme. • Video representation schemes are needed to enable segmenting a video stream into meaningful videoobjects, useful for event detection, indexing and retrieval applications.
PACS: Picture Archiving & Communication Systems Query/Retrieve Storage Tele-Medicine Internet Visual Information Query/Retrieve Database Management
What are interesting events in medical data? Spatio-Temporal Segmentation of Multiple Sclerosis Lesions in MRI Spatio-Temporal Tracking of Tracer in Digital Angiography
Introduction • Analysis of a video as a single entity Vs analysis of video as a sequence of frames • Inherent Spatio-temporal tracking • Gaussian Mixture Modeling in color & spacetime domain t y x
Learning a Probabilistic Model in Space-Time t x y Expectation Maximization (EM) Feature Vectors [L, a, b, x, y, t] (6 - dimensional space) Gaussian Mixture Model
Video Representation via Gaussian Mixture Modeling • Each Component of the GMM Represents a Cluster in the Feature Space (=Blob) and a Spatio-temporal region in the video • Pd. F For the GMM : With the Parameter set
• Given a set of feature vectors and parameter values, the Likelihood … expresses how well the model fits the data. • The EM algorithm: iterative method to obtain the parameter values that maximize the Likelihood
The EM Algorithm Expectation step: estimate the Gaussian clusters to which the points in feature space belong Maximization step: maximum likelihood parameter estimates using this data
Initialization & Model selection • Initialization of the EM algorithm via K-means: – Unsupervised clustering method – Non-parametric • Model selection via MDL (Minimum Description Length) – Choose k to maximize: – lk = #free parameters for a model with k mixture components
Video Model Visualization Static space-time blob Dynamic space-time blob The GMM for a given video sequence can be visualized as a set of hyper-ellipsoids (2 sigma contour) within the 6 dimensional color-space-time domain.
Detection & Recognition of Events in Video L a b x y t Correlation coefficient : C Cxt Cyt Ctt - Duration of space-time blob Static/Dynamic blobs - thresholds on Rxt (Hor. motion) & Ryt (Ver. motion) Direction of motion - sign of Rxt, Ryt
Detection & Recognition of Events in Video L a b x y t Cxt Cyt Ctt Blob motion (pixels per frame) via linear regression models in space & time : Horizontal velocity of blob motion in image plane is extracted as the ratio of cov. parameters. Similar formalism allows for the modeling of any other motion in the image plane.
Probabilistic Image Segmentation A direct correspondence can be made between the mixture representation and the image plane. Each pixel of the original image is now affiliated with the most probable Gaussian cluster. Pixel labeling: Probability of pixel x to be labeled:
Original Segmentation Model
Original Segmentation Model Dynamic Event Tracking
Limitations of the Global Model • How can we represent non-convex spatio-temporal regions? • All the data must be available simultaneously Inappropriate for live video - Model fitting time increases directly with sequence length -
Piecewise Gaussian Mixture Modeling • Modeling the Video sequence as a succession of overlapping blocks of frames. • Obtain a succession of GMMs instead of a single global model. • Important issues: initialization; matching between adjacent segments for region tracking. (“gluing”)
Piecewise GMM : “Gluing” / Matching at Junctions Ex: Frame J 5 Blob matching Frame J 5 blobs via GMM 5 Frame J 5 blobs via GMM 6
Original Sequence Dynamic Event Tracking Model Sequence
Horizontal Velocity in function of Block of Frame Index Pix / frame Bo. F #
Vertical Velocity in function of Block of Frame Index Pix / frame Bo. F #
Original Sequence Segmentation Map Sequence Pix / frame Vertical Velocity Horizontal Velocity Sweater Trousers BOF #
Spatio-Temporal Segmentation of Multiple Sclerosis Lesions in MRI Sequences
Methodology Ti me GMM for Luminance Frame by frame Segmentation Maps K >= 4 1) CSF 2) White Matter 3) Gray Matter 4) Sclerotic Lesions 3 D (x, y, t) Connected Components Blobs in [L x y t] Feature Space
Original Sequence Dynamic Event Tracking Segmentation Maps Sequence
Time Evolution Area (in Pixels) Time point
Conclusions • The modeling and the segmentation are combined to enable the extraction of video-regions that represent coherent regions across the video sequence, otherwise termed video-objects or sub-objects. • Extracting video regions provides for a compact video content description, that may be useful for later indexing and retrieval applications. • Medical applications: lesion modeling & tracking Acknowledgment Part of the work was supported by the Israeli Ministry of Science, Grant number 05530462.
Скачать презентацию A Probabilistic Framework for Video Representation Arnaldo Mayer
70f3a0845b613fda4eec6e23a28c5eae.ppt