Dynamic Scenes by Image Sequence Analysis Jun Shen 2004 1
Presentation scheme l l l General presentation Dynamic scene analysis (DSA): a general view Motion detection by background subtraction & by orthogonal moments l 3 D-model-based vehicle pose determination & tracking l Face tracking l Gait tracking (Model-based tracking) l Automatic gait recognition Learning & recognition of activity patterns by fuzzy self-organizing Kohonen net Demonstration of results l l 2
I. General Presentation 3
General framework of visual surveillance Camera 1 P R O C E S S I N G . . . Camera n Environment modeling Motion segmentation Object classification Tracking Behavior understanding Personal and description identification P R O C E S S I N G Fusion of Information from multiple cameras I. General presentation 4
II. Dynamic Scene Analysis (DSA): A general view 5
II. Dynamic Scene Analysis (DSA): A general view Low-level analysis Motion detection l Pose determination l Hidden effect processing l Moving object classification l Tracking l II. DSA: A general view 6
l Motion detection methods – Background subtraction – Temporal difference between successive frames – Optic flow – Matching: correlation, etc – Frequency domain methods l Pose determination II. DSA: A general view 7
Moving object classification l Classification based on geometric and radiometric properties of object – Shape – Size – Height-width ratio – Color – Texture – Features l Classification based on motion II. DSA: A general view 8
Tracking l Tracking based on regions l Tracking based on moving contours l Tracking based on features l Tracking based on object models II. DSA: A general view 9
Behavior understanding and description l Finite state automate l Non-deterministic automate l Hidden Markov process model l Neural nets l Syntactic methods …. II. DSA: A general view 10
Person identification by gait analysis for video surveillance Model-based methods l Statistical methods l Characteristic-parameter-based methods l Temporal-Spatial-motion-based methods l Combination of gait analysis with other biometrics methods l II. DSA: A general view 11
Fusion of information from multiple cameras Positioning of cameras l Calibration of cameras l Matching of objects from multi-camera l Switching of cameras l Fusion of information from multi-camera l Hidden effect processing using multiple cameras l II. DSA: A general view 12
III. Motion detection 13
III. Motion detection Background subtraction l Temporal Gaussian-Hermite moments – Moments – Orthogonal moments – Gaussian-Hermite moments – Motion detection by Gaussian-Hermite moments l III. Motion detection: 1. Background subtraction 14
III. 1. Motion detection in color (or gray value) image sequence by background subtraction 15
System overview Based on background subtraction Input image sequence Moving objects detected Filtering Labeling Background image creation Moving pixel detection Shadow elimination Illumination change elimination III. Motion detection: 1. Background subtraction 16
Filtering & background image (c'tnd) l. A mobile object stops during a period > half the temporal W. size, ·It would be considered as static object and backgr'd updating will take moving object color. ·When it begins to move again, backgr'd image thus updated would disturb the detection of its motion (double moving objects detected). False moving object III. Motion detection: 1. Background subtraction 17
Filtering & background image (c'tnd) l Solution – Color of moving pixels not taken into account in backgr'd updating. – Distinguishing stopped “mobile” objects from real static objects. – Comparison of present & preceding positions tells in motion or a stopped mobile object. III. Motion detection: 1. Background subtraction 18
Motion detection by background subtraction for color images l Difference between current frame & backgr'd Current image R, B, G Channels of Difference Image Background image Difference image Diff III. Motion detection: 1. Background subtraction 19
Motion detection by b'gd subtraction (c'tnd) Difference between current frame & backgr'd l Segmentation of the difference color image – Fuzzy segmentation of R, B, G channels separately l Automatic determination of threshold T, l Fuzzy set “mobile pixels” by non-sym. p m'ship function. – Fuzzy segmentation with 3 channels together l III. Motion detection: 1. Background subtraction 20
Motion detection by b'gd subtraction (c'tnd) hi Threshold by "Max. Distance" Difference between current frame & backgr'd l Segmentation of the difference color image – Fuzzy segmentation of R, B, G channels Immobile Fuzzy set of mobile separately l Automatic determination of threshold T l i T III. Motion detection: 1. Background subtraction 21
Motion detection by b'gd subtraction (c'tnd) Difference between current frame & backgr'd p(x) l Segmentation of the difference color image l – Fuzzy segmentation l 2= R, B, G channels separately of + l Automatic determination of threshold T, set “mobile pixels” by non-sym. p m'ship function. x l Fuzzy T c= dmax+ k/( dmax- dmin), (k>0) dmax and dmin, max. & min. intensities. – Fuzzy segmentation with 3 channels together III. Motion detection: 1. Background subtraction 22
Actual color frame Background image Difference Image R channel B channel ATD (Automatic Threshold by ATD G channel ATD max. distance) Fuzzy M’ship fn Fuzzy deduction Automatic threshold by moment conservation method Mobile pixel image Fuzzy Segmentation of Difference Image III. Motion detection: 1. Background subtraction 23
Elimination of false motion due to illumination change l Problem - Bg'd image update using preceding frames not fast adapted to illumination v. Rapidness of bg'd adaptation depends on temporal window size & bg'd adaptation method. Even auto-adaptation used, bg'd adapted to illumination change only after an accumulation of frames - - III. Motion detection: 1. Background subtraction 24
Diagram of false motion elimination Mobile pixels detected by variation in successive frames Mobile pixels detected in preceding frame Mobile pixels detected by background subtraction for the current frame OR AND Validated mobile pixels III. Motion detection: 1. Background subtraction 25
Shadow Elimination Problem: Shadows of moving objects being of almost the same motion as moving objects l Importance of shadow elimination l Obtaining more precise description of moving objects Center of gravity III. Motion detection: 1. Background subtraction 26
III. 2. Motion detection by orthogonal moments 27
III. 2. Motion detection by orthogonal moments l Moments – Geometric, Legendre & Hermite moments – Behavior in space & frequency domains – Gaussian-Hermite (G-H) moments l Motion detection by G-H moments l Comparison with other methods l Concluding remarks III. Motion detection: 2. G-H moments 28
Geometric, Legendre & Hermite moments and their calculation l Geometric moments and their calculation l l 1 D geometric moments Mn(x) at point x: Mn(x)= S(x+ t) tn dt n= 0, 1, 2, . . . l 2 D geometric moments of a 2 D image I(x, y): – Mm, n(x, y)= l l l I(x+ u, y+ v) um vn du dv Fast algorithms, such as Pascal Triangle. Explicit statistical signification. Functional analysis viewpoint: Signal projected onto polynom. space, taking monomial functions as bases. III. Motion detection: 2. G-H moments 29
Orthogonal Legendre moments l Using orthogonal bases: – Calculation could be reduced, – Error easier to estimate when limited proj. used, – Reconstruction simpler. l Orthogonal Legendre polynomials: (dn/ dxn) (x 2 - 1)n / (2 n. n!) Pn(x) = { 0 for xÎ [-1, 1], otherwise. III. Motion detection: 2. G-H moments 30
![l Scaled Legendre polynomials: [(dn/ dxn) (x 2 - w 2)n ]/ [(2 w)n.](https://present5.com/presentation/fcfa8c02df4afc3b0652753d5aeeb3df/image-31.jpg "l Scaled Legendre polynomials: [(dn/ dxn) (x 2 - w 2)n ]/ [(2 w)n.")
l Scaled Legendre polynomials: [(dn/ dxn) (x 2 - w 2)n ]/ [(2 w)n. n!] for xÎ [-w, w] Ln(x) = { 0 l otherwise. n-th order moment: Mn(x) = orthogonal Legendre S(x+ t) Ln(t) dt = <L 0(t), S(x+ t)> (inner product). III. Motion detection: 2. G-H moments 31
Recursive calculation of Legendre moments l The nth order orthogonal L. moments, calculated from window [x- w, x+ w], can be computed from (n- 1)th & (n- 2)th order L. M. : M 0(x) = <L 0(t), S(x+ t)> = S 1(x+ w) - S 1(x- w) M 1(x) = <L 1(t), S(x+ t)> = [S 1(x+ w) + S 1(x- w)] - <L 0(t), S 1(x+ t)> / w Mn(x) = <Ln(t), S(x+ t)> = <Ln-2(t), S(x+ t)> - [(2 n- 1)/ w] <Ln-1(t), S 1(x+ t)> , for n> 1 with S 0(t)= S(t) and Si(t)= Si-1(y) dy for i= 1, 2, 3, … l Si(t) easily calculated from Si-1(t) by recursive sum-box tech. 32 III. Motion detection: 2. G-H moments
2 D Legendre moments In 2 D cases: k x ky Mp, q(x, y)= ò ò I(x+ t, y+ v) Lp(t) Lq(v) dt dv -kx -ky l Separable, decomposed into cascade of 1 D calculation, by recursive algo. III. Motion detection: 2. G-H moments 33
Hermite moments l Scaled Hermite polynomial Pn(t)= Hn(t/ s) with Hn(t)= (-1)n exp (t 2) (dn/ dtn) exp (-t 2). 1 D n-th order Hermite moment: Mn(x, S(x))= Pn(t) S(x+ t) dt n= 0, 1, . . . l 2 D Hermite moments of an image I(x, y): l Mp, q(x, y, I(x, y))= Hp, q(t/s, v/s) I(x+t, y+v) dt dv with Hp, q (t/ s, v/ s)= Hp(t/ s) Hq(v/ s). l Separable, calculated by cascade of 1 D. III. Motion detection: 2. G-H moments 34
Behavior of geometric, Legendre & Hermite Moments in space & frequency domains l Importance of behavior analysis l Behavior in space domain l Behavior in frequency domain III. Motion detection: 2. G-H moments 35
l Geometric moment base functions – Graphs of similar shapes, – Moments considered as projections onto base function space, not efficient for diff. spatial modes. l Hermite & Legendre mnt. base functions – Many oscillations, depending on the order, – Extract efficiently characteristics of diff. spatial modes (orthogonal polynomial of order n has n diff. zerocrossings). Oscillations in Hermite bases much less important than Legendre ones (because the Hermite bases are not really orthogonal). l Same conclusion holds in 2 D cases. l III. Motion detection: 2. G-H moments 36
l l Geometric moment base functions: – low-pass kernel, FT monotonically decreased. Hermite moment base functions: – as order increased, max. FT position moves to right, and more similar to a band-pass kernel. Legendre moment base functions: – best band-pass characteristics except for very low orders. The higher the order is, the more to the right the pass-band moves. L. moments separate characteristics in different frequency bands better than H. moments, which are in turn better than geometric ones. III. Motion detection: 2. G-H moments 37
Gaussian-Hermite Moments III. Motion detection: 2. G-H moments 38
Property of G-H moments III. Motion detection: 2. G-H moments 39
Comparison l G-H moments better separate diff. bands. l Larger quality factor Q= (Center freqency)/ (Effective bandwidth). l G-H moments & G. -filtered deriv. : – G-H moments: linear combinations of Gaussfiltered derivatives of signal. – Construct orthogonal features from Gaussianfiltered derivatives. III. Motion detection: 2. G-H moments 40
G. -H. moments & wavelet analysis l l Derivatives of Gaussians widely used as mother wavelets, Different order derivatives of Gaussian filters define different wavelets, Derivatives filtered by Gaussian filters of different s represent the decomposition of signal into wavelets. Smoothed orthogonal Gaussian-Hermite moments offer a solution to construct orthogonal features from the wavelet analysis results. III. Motion detection: 2. G-H moments 41
2 D orthogonal G-H moments III. Motion detection: 2. G-H moments 42
Performance comparison: Sensibility to noise l Noise-free images and noisy ones with additional random noise, l Moment vectors (m 0, 0, m 0, 1, …, m 0, 5, m 1, 0, m 1, 1, …, m 1, 5), l Normalized distances between noisefree images and noisy ones. III. Motion detection: 2. G-H moments 43
Orthogonality equivalence To better understand the good performance of orthogonal moments in both spatial and frequency domains, we have l Orthogonality equivalence theorem - Orthogonal moment base functions are not only orthogonal in spatial domain but also in frequency domain. III. Motion detection: 2. G-H moments 44
Experimental verification l Three different reference shape images: quadrilateral, hexagon and octagon. l Noisy images: adding random noises of diff. standard deviations. l Each shape characterized by 12 moments of orders (0, 0), . . . , (0, 5), (1, 0), . . . , (1, 5). Geometric, H. and L. moments are tested. l Classification by comparing moment vector of noisy shape with 2. the 3 ref. 45 III. Motion detection: G-H moments
Motion detection by Gaussian-Hermite moments l Why using G-H moments Motion detection using G-H moments l Results and comparison l – Comparison with differential methods – Comparison with background subtraction – Comparison with adaptive background subtraction III. Motion detection: 2. G-H moments 46
l Why using G-H moments? – Methods of motion detection in image sequence Background-subtraction-based, including stochastic estimation of activation Difficulty – Frame-to-frame illumination changes, – Slowly moving and/or uninterested moving objects – Calculation of adaptive background images demanding accumulation of a large number of images. l Based on temporal variation in successive images Difficulty – Sensibility to noise l III. Motion detection: 2. G-H moments 47
l Advantages of using orthogonal G-H moments for motion detection – G-H moments: linear combinations of image derivatives, permitting to detect image changes – Much smoother than other moments, therefore much less sensitive to noises, facilitate moving object detection in noisy image sequences. – Odd-order G-H moment base functions: linear combinations of odd order derivatives of Gaussian functions. – Temporal G-H moments: composed of temporal image derivatives to detect moving objects in image sequences. III. Motion detection: 2. G-H moments 48
Detecting moving targets using G-H moments of different orders Given an image sequence l l Calculation of temporal G-H moments M 1, M 3 and M 5 Fuzzy motion detection by moment image segmentation, using threshold by improved invariable -moment-method, using non-sym. p Mship function for each point in moment images. Membership function update by fuzzy relaxation: spatial relation between pixels in single and successive frames Moving pixel decision III. Motion detection: 2. G-H moments 49
Comparison with other methods Comparison with differential methods III. Motion detection: 2. G-H moments 50
Comparison with background subtraction l Test image sequence: illumination changed in some frames l Background subtraction method fails for illum. changed frames l G-H moments succeed III. Motion detection: 2. G-H moments 51
Comparison with adaptive background subtraction l Adaptive back’d subtr’n improving motion detection l Problem: back’d updating para. value choice, depending on motion velocities l G-H moments: problems much better solved. III. Motion detection: 2. G-H moments 52
Example: an image sequence III. Motion detection: 2. G-H moments 53
Motion detection result by G-H moment III. Motion detection: 2. G-H moments 54
Moving car trajectory (Spline) III. Motion detection: 2. G-H moments 55
IV. 3 D-model-based Pose determination & Tracking of vehicles IV. Pose and tracking 56
System configuration Image sequence Camera model Low-level video tracking 3 D vehicle model High-level behavior analysis IV. Pose and tracking 57
System framework Image sequence Camera calibration Modeling Low-level processing Motion detection New target s? Y Initialization N 3 D pose estimation Pose optimization Behavior analysis and semantic description Obstacle hiding analysis Model projection Pose quality evaluation High-level processing IV. Pose and tracking 58
Vehicle pose determination Problems: l. Detection of region of interest containing a vehicle b. Motion detection b. Classification of moving objects l. Determination of 3 D pose of the vehicle IV. Pose and tracking 59
Known data =ROI containing the vehicle on the image =3 D model of the vehicle =Camera intrinsic and extrinsic parameters =Road surface plane constraint =Initial vehicle pose estimation IV. Pose and tracking 60
Pose quality evaluation l Model features: Selected straight edge segments of the 3 D model l Image features: Edge points detected on the image l Quality based on PLS (Point to Line Segment) distance IV. Pose and tracking 61
Pose optimization l Make move 3 D model in 3 D space, from the initial pose estimation to optimal pose l 3 D model moves on the road surface plane (Plan motion constraint): l. Translation on the road plane l. Rotation around axis normal to road plane & passing through vehicle’s center of gravity l Weak projective projection hypothesis: Translation and rotation above independent on the image plane Decomposition Translation optimization + Rotation optimization IV. Pose and tracking 62
Translation optimization § For the projection Lp of the pth segment of 3 D model, define a subset of I: § Pose error function IV. Pose and tracking 63
Determination of rotation ZM ZC OM XM YM ZW OW XW XC OC YW M: Reference system on the model object W: Reference system in 3 D world C: Reference system on the camera IV. Pose and tracking YC 64
Rotation optimization § After translation optimization § Searching in a small interval centered at the estimated rotation angle § Take the angle that minimizes the pose error function IV. Pose and tracking 65
Hidden effect detection and visible region determination § Different types of hidden effect ü Case 1: Moving object hidden by background ü Case 2: Moving object leaving or entering in the vision field ü Case 3: Moving object hidden by other moving objects § Hidden effect detection § Visible region determination IV. Pose and tracking 66
Experimental results IV. Pose and tracking 67
Principle of Tracking § Non-deformable solid object tracking (Vehicles, …) For the entire object: – Shape, size, color, . . . – Estimation from motion § Solid objects with joints (Human body, etc) For each part of the object: – Sub-model: shape, size, color, . . . – Estimation from the motion IV. Pose and tracking 68
V. Face Tracking 69
An input sequence Template Updating The first frame in an image sequence Template Confidence Measure > Threshold N Face Detection Template Initialization Y Motion Detection H Profile Body-part Constraints Template Matching Set Search Region V Profile Overview of Algorithm V. Face tracking 70
VI. Gait Tracking (Model-based tracking) 71
Gait Tracking (Model-based tracking) Initialization Human body model Pose estimation Pose optimization Motion constraints Human body pose in the preceding frame or the initial pose Motion Model Current frame Tracking result Application Dynamic model Search strategy Motion synthesis Motion model Human body model Gait recognition Motion constraints Pose evaluation function VI. Gait tracking 72
Model representation & Learning - Geometric model of human body • Generalized cylinder model • Motion parameters: VI. Gait tracking 73
VII. Automatic Gait Recognition 74
VII. Automatic Gait Recognition Gait: – Useful biometric feature for recognition – Attractive modality of human identification at a great distance, for surveillance l Application: automated person identification for surveillance or monitoring systems in securitysensitive environments such as banks, parking lots and military bases. l Method based on Statistical Shape Analysis l VII. Gait recognition 75
Advantages and Disadvantages l Advantages – The only perceivable biometric at a distance; – Not requiring proximal contact; – Easy to capture; – Difficult to conceal. l Disadvantages – A large amount of data; – Intermediate recognition accuracy; – Subject to some physical conditions such as drunkenness, pregnancy, and injuries involving joints. VII. Gait recognition 76
Monitoring Area Gait image Camera Sequence Tracking Gait Feature Extraction Classifier Background image creation Database Motion Detection Recognition Results General framework of gait recognition VII. Gait recognition 77
VIII. Learning & Recognition of Patterns of Activity by Fuzzy Self-Organizing Kohonen Network 78
VIII. Learning & Recognition of Patterns of Activity by Fuzzy Self. Organizing Kohonen Network • Activity understanding in particular, • Learning of activity patterns • Anomaly detection • Activity prediction VIII. Activity patterns 79
General Schema 1. Moving target tracking 2. Trajectory coding 3. Activity recognition – Data acquisition – Recognition structure – Learning – Recognition VIII. Activity patterns 80
• Recognition structure • Why Self-Organizing Kohonen Network? • Classical activity recognition systems? • Depending on predefined activity patterns • Non adaptable to changing environments èHighly desirable to establish general approach of activity recognition able to automatically generate activity models. l. Kohonen self-organizing topological map ‘Winner takes all’ èUsing Fuzzy self-organizing Kohonen net VIII. Activity patterns 81
Learning • Training data: Set of training trajectories VIII. Activity patterns 82
Anomaly Detection • Detecting abnormal trajectory Given a trajectory: • We first look for the neuron that best matches it, which gives the class to which it is classified. • If the Euclidean distance between the input trajectory code and the best matched neuron is greater than a threshold q, the activity represented by the trajectory is considered as unusual (abnormal). ( VIII. Activity patterns 83
Prediction of Activity Given a part of a motion trajectory: • Sampling this part to get a "sub-sample" vector V. • Mismatching score between the sub-sample and each neuron i by the Euclidean distance • The probability of each possible future motion trajectory along which the object • According to the probabilities thus determined, several future trajectories can be predicted with probability. VIII. Activity patterns 84
IX. Demonstrations of results 85
Thank you! Address: Jun Shen Institut EGID - Bordeaux 3 1, Allée Daguin 33607 Pessac cedex FRANCE Email: shen@egid. u-bordeaux. fr Phone: Fax: (+33) 5 57 12 10 26 (+33) 5 57 12 10 01 86
Скачать презентацию Dynamic Scenes by Image Sequence Analysis Jun Shen
fcfa8c02df4afc3b0652753d5aeeb3df.ppt