Advanced Video Compression Standards Gary Sullivan Gary Sull microsoft

Advanced Video Compression Standards Gary Sullivan (Gary. Sull@microsoft. com) Microsoft Corp. Software Design Engineer ITU-T Rapporteur of Advanced Video Coding ITU-T Recommendation H. 263 Editor Stanford, February 15, 2001

Overview • Video Standardization Concepts • History • Recent events – – – Standardization projects H. 263 v 1 & H. 263++ MPEG-4 v 1, v 2, v 3 H. 26 L Progress Microsoft Windows Media Video • Future Stuff – H. 26 L Finalization – MPEG Video Plans – Trends

Formal Standards • Specification available to all at little or no cost • Anyone allowed to implement • Agreement officially by consensus, not decided by a single organization’s interests • Relatively open committee with variety of participants (including hostile competitors, with no contract to support a common agenda, often meeting with formal government approval) • In practice, each standards organization tends to have its own “personality”

Video Coding Standardization Organizations • Two organizations dominate video compression standardization: – ITU-T Video Coding Experts Group (VCEG) International Telecommunications Union – Telecommunications Standardization Sector (ITU-T, a United Nations Organization, formerly CCITT), Study Group 16, Question 6 – ISO/IEC Moving Picture Experts Group (MPEG) International Standardization Organization and International Electrotechnical Commission, Joint Technical Committee Number 1, Subcommittee 29, Working Group 11

Dynamics of the Video Standardization Process • VCEG is older and more focused on conventional (esp. low-delay) video coding goals (e. g. good compression and packet-loss/error resilience) • MPEG is larger and takes on more ambitious goals (e. g. “object oriented video”, “synthetic-natural hybrid coding”, and digital cinema) • Sometimes the major organizations team up (e. g. ISO, IEC and ITU teamed up for both MPEG-2 and JPEG) • Relatively little industry consortium activity (DV and organizations that tweak the video coding standards in minor ways, such as DVD, 3 GPP 2, SMPTE, IETF, etc. ) • Growing activity for internet streaming media outside of formal standardization (e. g. , Microsoft, Real Networks, Quicktime)

The Scope of Picture and Video Coding Standardization • Only the Syntax and Decoder are standardized: – Permits optimization beyond the obvious – Permits complexity reduction for implementability – Provides no quality guarantees – only interoperability Source Destination Pre-Processing Encoding Post-Processing Decoding Scope of Standard

H. 261: The Basis of Modern Video Compression • ITU-T (ex-CCITT) Rec. H. 261: The first widespread practical success – First design (late ‘ 90) embodying typical structure that dominates today: 16 x 16 macroblock motion compensation, 8 x 8 DCT, scalar quantization, and variable-length coding – Key aspects later dropped by other standards: loop filter, integer motion comp. , 2 -D VLC, header overhead – v 2 (early ‘ 93) added a backward-compatible high-resolution graphics trick mode – Operated at 64 -2048 kbps – Still in use, although mostly as a backward-compatibility feature – overtaken by H. 263

Typical MC+DCT Video Coder DCT, Quantize, Entropy Encode Input Frame Motion Compensated Prediction (Dotted Box Shows Decoder) Encoded Residual (To Channel) Entropy Decode, Quant. Recon. , Inverse DCT Approximated Input Frame (To Display) Prior Coded Motion Comp. Frame Approx Frame Buffer Predictor (Delay) Motion Estimation Motion Vector and Block Mode Data (To Channel)

Video Coding Efficiency PSNR [d. B] 38 TMN-10 Half-pel Variable motion block size compensation motion (MPEG-1 1993) compensation (H. 263 1998) 36 ? 34 Integer-pel motion compensation (H. 261 1991) 28 26 0 100 Foreman 10 Hz, QCIF 100 frames encoded 67 % 32 30 Frame difference coding (H. 120 1988) 200 300 Intraframe DCT coding (DCT 1974, JPEG 1992) 400 500 Bit-Rate [kbps]

MPEG-1: Practicality at Higher Bit Rates • Formally ISO/IEC 11172 -2 (‘ 93), developed by ISO/IEC JTC 1 SC 29 WG 11 (MPEG) – use is fairly widespread, but mostly overtaken by MPEG -2 – Superior quality to H. 261 when operated a higher bit rates ( 1 Mbps for CIF 352 x 288 resolution) – Can provide approximately VHS quality between 1 -2 Mbps using SIF 352 x 240/288 resolution – Technical features: Adds bi-directional motion prediction and half-pixel motion to H. 261 design

MPEG-2/H. 262: Even Higher Bit Rates and Interlace • Formally ISO/IEC 13818 -2 & ITU-T H. 262, developed (‘ 94) jointly by ITU-T and ISO/IEC SC 29 WG 11 (MPEG) – Now in wide use for DVD and standard and high-definition DTV (the most commonly used video coding standard) – Primary new technical features: support for interlacedscan pictures and scalability – Essentially the same as MPEG-1 for progressive-scan pictures, and MPEG-1 forward compatibility required – Not especially useful below 4 Mbps (range of use normally 5 -30 Mbps)

H. 263: The Next Generation • ITU-T Rec. H. 263 (v 1: 1995): The next generation of video coding performance, developed by ITU-T – the current best standard for practical video telecommunication (has overtaken H. 261 as dominant videoconferencing codec) – Superior to H. 261 at all bit rates – Wins by a factor of two at very low rates – Versions 2 (late 1997/early 1998) & v 3 (2000) later developed

MPEG-4: Baseline H. 263 and Many Creative Extras • MPEG-4 (v 1: early 1999), formally ISO/IEC 14496 -2: Contains the H. 263 design and adds all prior features and various creative new extras – Includes segmented coding of shapes, zero-tree wavelet coding of still textures, coding of synthetic and semi-synthetic content, etc. – v 2 (early 2000) & v 3 (early 2001) later added

MPEG-4 and H. 263 Standardization Dynamics • MPEG-4 project launched soon after H. 263 completed • MPEG-4 project was very ambitious and was planned to be significantly different from H. 263 • Compatibility with H. 263 was not initially planned in MPEG-4 (although it eventually turned out to be significantly compatible!) • ITU-T decided to extend its H. 263 quickly and compatibly rather than join up with longer, more ambitious, potentially -incompatible MPEG-4 effort for the features the ITU wanted • Much cross-fertilization of ideas and people in projects

Detailed Recent History In Video Coding Standardization • ITU-T Events – – H. 263 v 1 completed late ‘ 95 H. 263+ project (H. 263 v 2) technically final Sept ‘ 97 H. 263++ project (H. 263 v 3) technically final July ‘ 00 H. 26 L project underway (test version available) • ISO/IEC Events – – MPEG-4 v 1 completed early ’ 99 MPEG-4 v 2 completed early ’ 00 MPEG-4 v 3 completed early ‘ 01 Potential for new work under evaluation

H. 263++ New Version 3 Features Part 1 of 2 • Annex U: Fidelity enhancement by macroblock and block-level reference picture selection – a significant improvement in compression quality • Annex V: Packet Loss & Error Resilience using data partitioning with reversible VLCs (roughly similar to MPEG-4 data partitioning, but improved by using reversible coding of motion vectors rather than coefficients)

H. 263++ New Version 3 Features Part 2 of 2 • Annex W: Additional Supplemental Enhancement Information – IDCT Mismatch Elimination (specific fixed-point fast IDCT) – Arbitrary binary user data – Text messages (arbitrary, copyright, caption, video description, and URI) – Error Resilience: • Picture header repetition (current, previous, next+TR, next-TR) • Spare reference pictures for error concealment – Interlaced field indications (top & bottom)

H. 263++ Annex U Rate Distortion Performance H. 263 combined with Long-Term Memory Prediction 50 10 3 reference pictures 36 TMN-10 PSNR [d. B] 34 17 % 32 Foreman 10 Hz, QCIF 100 frames encoded 30 28 26 0 20 40 60 80 100 120 140 160 Bit-Rate [kbps]

Average Bit Rate Savings 40 Average Bit-Rate Savings in [%] @ 34 d. B 30 Average Tempete Container Silent Foreman Mobile & Calendar Mother & Daughter Stefan 20 13. 5 % 10 0 1 2 3 5 10 17 33 65 100 Number of Reference Frames

MPEG-4 Version 3 Just Completed (part 1 of 2) • “Studio Profile” – Various additions oriented toward professional use of video within specialized studio environments – Adds 4: 2: 2 and 4: 4: 4 sampling structures – Adds more MPEG-2 elements to MPEG-4 • “Fine Granularity Scalability Streaming Video Profile”, a new form of scalable video coding – Uses a scalable enhancement layer – Temporal prediction in enhancement layer is stopped to prevent temporal error propagation – Enhancement layer coded by bit-planes to form a “progressivetransmission” bitstream

MPEG-4 Version 3 Just Completed (part 2 of 2) • “Advanced Simple Profile”, a combination of v 1 features, containing: – – – – “Simple Profile” features B pictures MPEG-2 -style quantization Interlace features (at higher levels only) ¼-pel motion Global motion comp Single stream support in new “level 0”

ITU-T VCEG H. 26 L Project Goals (Completion 2002) • • • Compression beyond capability of H. 263 v. N Real-time low-cost complexity Delay reduction Enhanced error and packet loss resilience Bit-rate adaptivity (e. g. scalability & BR reduction) • Spatio-temporal resolution adaptivity • Robustness to source material behavior

H. 26 L Status • Test Model Long-Term Number 6: Designed January ’ 01 (Eibsee), description and software soon available on the ‘net • TML-5 software and spec availalbe (Geneva, November ’ 00) • Gain goal over 1999 standards: 50% savings in bits for same fidelity! (at all bit rates)

The H. 26 L TML-6 Design Part 1 of 4 • Still using a hybrid of DPCM and transform coding as in prior standards. Common elements include: – 16 x 16 macroblocks – Conventional sampling of chrominance and association of luminance and chrominance data – Block motion displacement – Block transforms (not wavelets or fractals) – Scalar quantization – Variable-length coding

The H. 26 L TML-6 Design Part 2 of 4 • Motion Compensation: – – – – Multiple reference pictures (per H. 263++ Annex U) B picture support (per several prior standards) Multihypothesis concept being evaluated 1/4 sample accuracy motion (sort of per MPEG-4, could possibly go to 1/8 pel) 6 x 6 tap filtering to 1/2 sample accuracy, bilinear filtering to 1/4 sample accuracy Various block sizes and shapes for motion compensation (7 segmentations of the macroblock) “Funny position” with heavier filtering Affine motion under consideration

The H. 26 L TML-6 Design Part 3 of 4 • Intra Coding Structure: – Directional spatial prediction (6 types for luma, one for chroma) – Alterations under consideration • Transform – Variable block size for intra (16 x 16, 8 x 8, 4 x 4) – Technically not exactly a DCT, but an integer transform closely approximating a DCT – Based primarily on 4 x 4 transform size (all prior standards used 8 x 8) – Expanded to 8 x 8 for chroma by 2 x 2 DC transform – Adaptive block size under consideration

The H. 26 L TML-6 Design Part 4 of 4 • • Two inverse scan patterns Logarithmic step size control Smaller step size for chroma (per H. 263 Annex T) Universal variable-length coding (configurability under consideration) Adaptive arithmetic coding under strong consideration In-loop deblocking filter Distinct Network Adaptation Layer (NAL) design for network transport Inter-sequence transitional pictures under consideration

Future Work in MPEG • MPEG to assess new video technology and address digital cinema needs – Calls for proposals issued – Tests to be conducted in next few months – ITU-T VCEG bringing H. 26 L as reference – Exploring potential future cooperative work between VCEG and MPEG

Digital Media World u u u Many devices Wired or wireless Access from anywhere Software Integration Personalized delivery Rich Services

Windows Media Live Content License Server Download & Play Streaming Live Feed UNICAST, MULTICAST On-demand Content Windows Media Encoder Stored Content Authoring Windows Media Services Server Streaming from a WM Server (or Web Server) Distribution Windows Media Player PC, Hand-held, STB Playback

WM Video Coding • 2 main video codecs: – Standard MPEG-4 Video – WM Video v 8 ~ 40% bit rate savings over MS MPEG-4 • Examples of WM Video : • “Near-VHS to VHS quality” Ø 320 x 240 -480, 24 -30 Fps (250 – 500 Kbps) Ø PII 300 encoder, P 5 200 decoder • “Near-DVD to DVD quality” Ø 640 x 480, 24 -30 fps (500 Kbps – 1. 5 Mbps) Ø Dual PIII 700 encoder, PII 400 decoder • WM Player also supports other video codecs such as MPEG-1

Windows Media Technologies Video-Related Features • WMV 8 Codec: A big step forward in compression performance • Screen Codec: Outstanding compression – (near) Lossless ! – 640 x 480, 10 Fps < 20 Kbps (modem) – 800 x 600, 15 Fps < 45 Kbps (ISDN/LAN) • Advanced Streaming Format (ASF) file format • Digital Rights Management (DRM): Critical for Content Providers

Future Trends Prediction is difficult - especially of the future. – Bohr (1885 -1962) If we do not succeed, then we run the risk of failure. – Quayle (Phoenix Rep. Forum, 1990)

Principles of Rate-Distortion Theory Errors using inadequate data are much less than those using no data at all. – Charles Babbage (1792 -1871) A little inaccuracy sometimes saves tons of explanation. – Saki (H. H. Munro, 1870 -1916, The Comments of Moung Ka)

On Rate-Distortion Optimization • Rate-distortion optimization and searching techniques will increase in importance – Most enhancements take the form of an expanded range of choices – More choices implies more need for searching and optimization – Lagrange multiplier optimization provides an understandable, straightforward framework – Recent understanding of coupling of step size and Lagrange multiplier makes it straightforward

Some Future Projections • Coding Efficiency will continue to improve (Proof by existence): – – – 4 x 4 coding Long-term memory Enhanced motion accuracy Enhanced motion models Enhanced intra coding • People continue to come up with good ideas (and relatively predictable ones!)

What Area will Yield the Most Improvement? • Although “prediction is difficult”, it is the area that will yield the most performance improvement • Today’s coded motion model is primitive • Several motion model improvement areas have yet to be fully exploited • Waveform difference coding gain is limited

Won’t This be Unnecessary when Megabits become free? • The need for better compression will not be reduced – Got more bits? Give me higher resolution. – Got more bits? Give me more channels. – Improving worth effort? 20% of a lot is a lot. – Bit rates have a slower doubling time than computing power.

Increasing “Layers” of Standardization • In olden days: Design a system for a network with a video coder as part of that system design. • Now: – Standardize a “language” of syntax with maximum flexibility and a rich feature set – Standardize how to configure the standard – Standardize how to encapsulate the standard data on a network – Standardize digital rights management for the data – Standardize the system to carry the data

Other Kinds of Layers • Continuing interest in Layered coding: – Scalability in MPEG-2, H. 263+, and MPEG-4 – Layered coding ongoing work (Microsoft, MPEG Enhanced FGS) – Mixed success toward products • Motivation 1: The bit-rate scalability dream • Motivation 2: The limitations of resolution • Motivation 3: The error resilience need

Conclusions • There will be plenty of need for further work. • There will be plenty of need for more processing power. • There will be plenty of need for more bits. • There will be plenty of need for good ideas. • And those good ideas will come. Dream no small dreams, for they have no power to move the hearts of men. - Goethe (1749 -1842)