Department of Communication Systems Engineering Ben Gurion University

Department of Communication Systems Engineering Ben Gurion University of the Negev Be’er-Sheva, Israel Computational Controlled Mode Selection for H. 264/AVC Ariel Kit & Amir Nusboim Supervised by: Dr. Ofer Hadar Mr. Evegeny Kaminsky Computational Controlled Mode Selection for H. 264/AVC June 2006 1

Presentation Content • • H. 264 Advanced Video Coding Overview Project objectives Introduction to mode selection Method 1: Cost map estimation Method 2: Fast Mode Selection The FMS Algorithm Results Future work Computational Controlled Mode Selection for H. 264/AVC June 2006 2

H. 264/AVC Overview • Written by the JVT (Joint Video Team) – A Joint effort of the ITU-T and ISO. • Previous standards: MPEG 1 ISO, MPEG 2 JVT, H. 261 ITU, MPEG 4 ISO, H. 263 ITU. • Substantially lower bit rates (half of less). • Project’s related Features: Ø Multi-picture motion compensation using up to 32 reference pictures (frames). Ø Variable block-size motion compensation with block sizes from 16 x 16 to 4 x 4. Ø Weigh-scaled predictors based on motion vectors are used to enhance compression. Ø Quarter-pixel precision for motion prediction and compensation. Computational Controlled Mode Selection for H. 264/AVC June 2006 3

H. 264/AVC Overview (cont. ) Computational Controlled Mode Selection for H. 264/AVC June 2006 4

Motivation • The performance gain of H. 264 comes at a price of increased computational complexity. Therefore there is a need to develop a low complexity implementation of H. 264 that can offer performance and flexibility without excessive computational cost. “Time is Money!” Computational Controlled Mode Selection for H. 264/AVC June 2006 5

Project’s Goals • Implementing a fast mode selection algorithm that is based on different macroblock’s properties. • The algorithm is computationally controlled for usage in limited resources application such as cellular phones and other portable video equipment. Computational Controlled Mode Selection for H. 264/AVC June 2006 6

Motion Estimation/Compensation + Video Input T - Motion Estimation/ Compensation Intra Prediction + + Q T-1 Lossless Compression 10010101… Q-1 Deblocking Filter Memory (Reference Frames) H. 264 Encoder Computational Controlled Mode Selection for H. 264/AVC June 2006 7

Mode Selection Motion Estimation/ Compensation • In the mode selection process the encoder must decide which mode is best used for a specific macroblock (16 x 16 pixels). • Decision making is critical for the inter prediction, if a macroblock is involved in a precise movement of various objects, we need to split up the macroblock according to the movement of the object to get an accurate motion compensation based on motion vectors. Computational Controlled Mode Selection for H. 264/AVC June 2006 8

Mode Selection (cont. ) • Mode selection is based on RDO (Rate Distortion Optimization) of the cost function per each mode: J=Distortion + MODE x Rate q Distortion – difference between reconstructed macroblock to the original macroblock. q λ – Lagrange multiplier q Rate – Bits used to encode the macroblock. Best Mode=Smallest Computational Controlled Mode Selection for H. 264/AVC June 2006 9

Method 1: Cost Map Estimation • Estimating cost using motion vectors and predictors to the reference frame. • Cost information for a reference frame is saved as a cost map (table) with quarter pixel resolution. • Cost is estimated using RDO and Best Cost is saved to the map. • The algorithm was simulated using Matlab; based on information received from the Joint Model Reference software (C code) used to simulate the H. 264 standard encoderdecoder. Matlab 7 Computational Controlled Mode Selection for H. 264/AVC JM v 10. 2 June 2006 10

Method 1: Algorithm Steps • Step 0: Create a cost Map of the reference image based on RDO chosen modes (quarter pixel resolution). • Step 1: Calculate motion vectors for the current mode. • Step 2: Use calculated motion vectors to find macroblock’s cost in the reference Cost Map. • Step 3: If cost is larger than the previous Cost go to Step 5. • Step 4: Repeat step 2 for all modes (different partitions). • Step 5: Choose Best mode with the lowest cost. 8976 Computational Controlled Mode Selection for H. 264/AVC June 2006 11

Method 1: Results Foreman, QP=28 News, QP=28 Hall Monitor, QP=28 Improvement mean - Cost Estimation vs. skip prediction Test sequence Improvement Rate Foreman 2% Foreman (QP=0) 3% Hall Monitor 1% News 1. 5% Foreman, QP=0 Computational Controlled Mode Selection for H. 264/AVC June 2006 12

Method 1: Conclusions • Results are compared to skip prediction which is the case where cost is predicted using motion vectors predicators using macroblock’s neighbors and motion copy. • Improvement is achieved only for mode 16 x 16 prediction. • The improvement is much smaller than expected due to the quarter resolution averaging made in the Joint Model software. • The method is not applicable for fast mode selection in the current constellation. Computational Controlled Mode Selection for H. 264/AVC June 2006 13

Method 2: Fast Mode Selection • Method uses macroblock’s spatial and temporal properties in order to reduce computational complexity. Ø Temporal Stationarity – macroblocks with zero motion and small differences from the reference measured by threshold. Ø Spatial Homogeneity – homogenous macroblocks doesn’t need to be sub-partitioned; homogenous is measured by applying a threshold to the macroblock Sobel edge map. Based on Reference: K. P. Lim, S. Wu, D. J. Wu, S. Rahardja, X. Lin, F. Pan, Z. G. Li, “Fast INTER Mode Selection, ” ITU-T, Doc. I 020, Sep. [2003]. Computational Controlled Mode Selection for H. 264/AVC June 2006 14

Method 2: Temporal Stationarity Frame i-1 Image Difference Frame i Computational Controlled Mode Selection for H. 264/AVC June 2006 15

Method 2: Spatial Homogeneity 121121 - 0 0 0 1 0 11 0 1 - 2 0 22 0 2 - • Sobel Masks are used for edge detection 1 2 1 1 0 11 0 1 - Absolute Value >Th Horizontal Sobel Absolute Value Edge Map Vertical Sobel Computational Controlled Mode Selection for H. 264/AVC June 2006 16

Method 2: Algorithm Steps Edge Detection Zero Motion? Yes MB Difference Yes RDO (16 x 16) No No Yes 16 x 16 Homogenous? RDO (16 x 16, 16 x 8/8 x 16) No RDO (16 x 16, 16 x 8, 8 x 16) 8 x 8 Homogenous? Similar? Yes • Homogeneity and Similarity are decided by using statistical thresholds. • After RDO, mode with the lowest cost Is chosen as the best mode. • Skip mode is always checked. RDO (8 x 8) No RDO (8 x 4, 4 x 8, 8 x 8) Computational Controlled Mode Selection for H. 264/AVC June 2006 17

Method 2: Experiments • The algorithm was applied into the JM v 10. 2 reference software (Code was written in Microsoft Visual C v 6). • 4 test sequences were used: * Sequences Length – 150 Frames * Video Rate – 30 Frames per second Computational Controlled Mode Selection for H. 264/AVC June 2006 18

Method 2: Results Test Sequence Complexity PSNR Luma Bit Rate Reduction (%) Attenuation (d. B) Reduction (%) 48. 264 0. 08 0. 777 47. 911 -0. 09 23. 0974 53. 509 0 33. 658 34. 427 0. 04 9. 099 * Configuration: Fast Motion Estimation + Early Skip + 4 Reference Frames MV Search +/- 32, Hadamard Transform, GOP structure: IPPP Computational Controlled Mode Selection for H. 264/AVC June 2006 19

Method 2: Results (cont. ) Test Sequence Complexity PSNR Luma Bit Rate Reduction (%) Attenuation (d. B) Reduction (%) 26. 067 0. 13 -2. 89 34. 555 0. 1 1. 945 32. 636 0. 07 -1. 89 33. 626 0. 13 -2. 12 * Configuration: RDO High Complexity (Regular) MV Search +/- 32, Hadamard Transform, GOP structure: IPPP Computational Controlled Mode Selection for H. 264/AVC June 2006 20

Method 2: Results (cont. ) Computational Controlled Mode Selection for H. 264/AVC June 2006 21

Future Work • • Adding a Complexity parameter. Case of High Complexity – FMS (Rate Control) Case of Medium Complexity – FMS Case of Low Complexity – FMS advanced stages are skipped when application resources run out. • Complexity parameter is calculated from average macroblock encoding time. Computational Controlled Mode Selection for H. 264/AVC June 2006 22

Computational Controlled Mode Selection for H. 264/AVC June 2006 23