The Past Present and Future of Video Telephony

The Past, Present, and Future of Video Telephony: A Systems Perspective Dave Lindbergh Stanford Networking Seminar 27 January 2005 1

Thanks to Athina for inviting me I’m going to take advantage of the opportunity to present some opinions about video telephony I hope at least they’re educated opinions Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 2

Contents A little bit about my perspective Where we’ve come from Where we are now – – – What is succeeding What is not succeeding (yet) Current problems & challenges The mass-market barrier – Expectations vs. reality – What it will take to succeed Where we go from here Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 3

A little bit about my perspective Engineering background – – Modems & data communications Protocols, real-time systems, image processing 1993: Picture. Tel, largest vendor of video conferencing gear – ISDN, H. 320, H. 261 128+ kbit/s minimum Soon got sucked into standardization work – Mid-90 s: Chaired H. 324 Systems Experts Group • Edited ITU-T Rec. H. 324 • Basis of today’s 3 G-324 M system • Precursor to H. 323 (yes, I take some of the blame) Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 4

What I’ve been doing lately H. 264 video compression standardization – – Profiles/Levels Applications Editor, ITU-T Rec. H. 239 – Role management – – Live = People Presentation = Content Editor, ITU-T Rec. H. 241 – Video signaling Editor, H. 324 (again) Rapporteur, ITU-T Q. 23/16 (“Media Coding”) Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 5

Video telephony system 18 frames/second Progressive scan Plasma display Pixel aspect ratio 3: 2 Image quality described as “excellent” End-to-end latency 1 millisecond (great!) Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 6

April 7, 1927 – Bell Labs Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 7

New York – Washington DC Walter Gifford President, AT&T New York Herbert Hoover US Sec’y of Commerce Washington DC Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 8

“Television” = Telephone + Vision 50 x 50 pixel display, neon bulbs Camera: Scanning arc lamp beam Optional projection to 2 x 3 feet – But “results were not so good” Edna Mae Horner Operator Chesapeake and Potomac Telephone Company Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 9

AT&T Picturephone 1957 “Experimental Model” Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 10

Early 1960 s Mirror Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 11

AT&T was very serious Plenty of smart business people! 1964 Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 12

Framing Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 13

Did it “cost too much”? AT&T finally gave up in the early 1970 s Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 14

1980 s – Still image picture phones Mid-1980 s: Japanese consumer electronic firms introduced still-image picture phone – – – Used existing regular analog phone line POTS modem ~ 5 seconds to send 1 black & white frame No audio during picture transmission ~$200 each Very few takers Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 15

1992 – AT&T Videophone 2500 “Predicting that 10 years from now video phones will be as popular as cordless phones and fax machines, last week AT&T introduced the first full-color motion video phone that operates over regular phone lines…AT&T officials say the picture quality was acceptable to test-market consumers” – Newsweek, January 20, 1992 10 frames/second, $1500 Marconi, others, had similar products Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 16

Many more videophones since then Mostly based on ITU-T standards H. 320 (ISDN) H. 323 (IP) H. 324 (POTS) and SIP They all worked Siemens T-View H. 320 (ISDN) Phone 1997 Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 17

More videophones Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 18

And more… Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 19

And more Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 20

FOMA experience in Japan FOMA = H. 324/Mobile, 64 kbit/s channels Video calls cost ~2 x price of voice calls 3 million phones deployed (as of Sept. 2004) Average monthly video usage = 2 minutes Top 20% of users do 20 minutes/month Most users young – Show where they are, who they’re with – Don’t point camera at themselves Do. Co. Mo is hopeful that usage will increase when penetration > 1 phone/family Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 21

Still-image camera phones 2 nd generation – Camera is on back of phone Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 22

Did they all cost too much? Many had good video quality Most were reliable & easy to use Many < $50 PC cameras with videophone apps MS Net. Meeting & Messenger are free Clearly, people do want video phones – Witness all the attempts, user excitement But they don’t buy or use them when offered – For some reason people are disappointed – We need to understand why before we can fix this Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 23

What is succeeding? The real killer app: TELEVISION But TV is doing fine without help from me… Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 24

What else is succeeding? Video conferencing – – $2 B/year industry, profitable Top vendors: Polycom, Tandberg, Sony Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 25

Video conferencing today Most use is in large organizations – – – Industry Government Education Most use is internal – Between sites of the same organization Most use is scheduled – Planned meetings, not spontaneous Only a few meeting rooms have VC equipment – Much talk about ubiquitous access, but not real yet Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 26

Situations where VC works well With people you already know – Already introduced, not strangers Not too many people on screen at once – Need to see facial expressions clearly Good lighting Good room & furniture layout People & Content at same time How I use it every week – Offices in Boston, California, Texas, Atlanta, Israel… Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 27

Why is VC successful? Relatively big picture size, high resolution – Less restriction on where people are in the frame Good lighting High-value application Work environment, pre-scheduled meetings – People come dressed & prepared to meet others – Reduces discomfort with “being on camera” Yet, VC is still in < 2% of conference rooms – Lots of room for growth – Similar problems as stopped video telephony – It works, but not nearly as well as we want it to! Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 28

How are we doing? We’re doing an excellent job on the classical technical challenges – – Video and audio coding Cost: $250 K (1989) to $2000 (2004), less for PCs Bandwidth is getting cheaper all the time Simplicity, reliability have improved greatly Some immediate challenges – Standards and network issues – being worked Longer-term challenges for video telephony – Expectations vs. reality – Human factors Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 29

Standards Wonderful thing: So many to choose from! Religion: H. 323, SIP, MGCP, proprietary… – No real differences from user perspective – Some want to start over…again Every standard is unnecessarily complex – – – Over-reaction to past mistakes, too little experience The POTS network was also incredibly complex Limits of human complexity management ability Directory services – ENUM/DNS, H. 350/LDAP, UMMAP… – This will settle out with time Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 30

Latency Lots of denial – this is not helpful ITU-T G. 114 gives 150 ms as an upper limit – For total end-to-end latency • • Including propagation over distance This is about right, but difficult to achieve IP networks inherently have latency issues – Usually make ARQ, backchannel schemes impractical Low frame rates make things worse Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 31

Quality of Service on IP Lots of solutions in theory – Diff. Serv, MPLS, IP Precedence, etc… Zero penetration on public Internet – There is no pricing model Most private networks provide Qo. S with massive over-provisioning – This is often cheaper than “clever” schemes Qo. S will remain a problem on the public Internet until there is a way to charge for it Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 32

More network issues NATs and Firewalls – – IP is effectively unusable between organizations Virtually all inter-organization calls are still ISDN Network fragmentation – – IP, ISDN, POTS, 3 G, 4 G… Lack of public/automatic gateways These are all being worked; will get solved – Some things will take time to shake out A “killer app” could force more rapid change – But this hasn’t happened yet Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 33

Video compression coding Ideal lossy video compression system – Every possible bit sequence decodes as something meaningful to human perception – On various time scales – Might this be the way the brain works? • Markov chain text generators sound a lot like dreams… Lots of room for improved coding Past: Biggest challenge was reducing bitrate – Bandwidth and storage were expensive Today: Computational efficiency a challenge – Bits are getting cheaper faster than computes are Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 34

Beyond video coding Most video research focused on coding – Without compression video was unmanageable Between improved compression and cheaper bandwidth and storage, things can now change – Computation has gotten much cheaper Fast, cheap video DSP means we can do more – – Stitching, warping, perspective correction… Searching, indexing, processing, recognizing content Analogous to audio & still-image DSP – There will be new apps unique to video Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 35

The mass-market barrier Video conferencing is a successful niche Video telephony hasn’t succeeded yet – Yet, clearly there is a market desire! Current issues don’t explain past failures – Standards, directories were solved for videophones – Latency was not a problem in the analog world – Qo. S, NAT/FW issues didn’t exist for switched circuits Then what will it take for success? – Why have users not yet embraced video telephony? Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 36

What will it take to succeed? Cost? – PC apps are nearly free, yet very little used Reliability? – Current systems work well on private networks – Video telephones were quite reliable Complexity? – Video phones are easy to use, as are modern VCS Video quality? – VC systems provide TV-like quality, yet haven’t broken into the mass-market Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 37

Expectations – Fiction Metropolis (Fritz Lang, 1926) Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 38

From Jetsons to Star Trek Countless science-fiction films & TV shows – Perfect framing, perfect lighting – People look straight into the camera – Nobody is nervous “being on camera” 2001: A Space Odyssey (Stanley Kubrick, 1968) Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 39

Expectations – Film & TV experience We’ve all grown up with film & TV – – Professional cinematography / videography Studios with proper lighting, layout Multiple camera angles Directors to choose the best shots With video telephony we get – – – Single camera viewpoint Bad lighting Bad or no framing Often, poor resolution and video quality Sometimes, tiny pictures Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 40

What it will take: Human factors Framing – Keeping people in the picture Camera viewpoint & perspective – Psychological factors, geometrical distortion Eye contact & gaze direction – And “camera shyness” Peripheral vision – Sense of space – close-up vs. wide views Attention requirements – Tradeoffs of different media Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 41

Framing Keeping people in the picture – – – Close enough to see faces clearly Far enough for freedom of movement Consciousness of framing, control can be distracting – Scenes in movie Mother (Albert Brooks, 1996) – Rob Morrow and Debbie Reynolds on videophone Automatic speaker-following not ideal – – – Often used when multiple people are in the room Want to see listener reactions (not just talker) Want to see VIPs (even if they’re listening) Close-ups can lose sense of relative position Still, often better than doing nothing Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 42

Camera viewpoint & perspective If the camera is too close – Geometric distortion – big noses, etc. If your camera is above eye-level – They’re looking down on you – you look submissive If your camera is below eye-level – They’re looking up at you – you look dominant This is why royal thrones are tall There is no single “right” position – People can either stand or sit Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 43

Eye contact What happens when you stare at someone? Does it happen if you stare at video? What happens when someone stares at you? – Do you feel comfortable? Eye contact is a form of innate, highly evolved non-verbal communication – A deep part of human nature – Lots of emotional charge – Not present in video telephony - unnatural Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 44

Eye contact & gaze direction People can detect eye contact at great distances – – They can tell when they’re being observed They may respond with a glance, or return contact Cooperation or liking = more direct gaze Disagreement or dislike = less direct gaze Gaze and emotional signals – – – Unwavering gaze - dominance or threat Gaze avoidance – submission or fear Gaze can signal sincerity, discomfort, challenge… Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 45

Eye contact and video We need to solve eye contact on video – I think this will reduce “camera shyness” Need to know who is looking at you Need to know if/when you’re stared at Need to allow natural feedback response Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 46

Other “naturalness” issues Peripheral vision – Noticing what other people are doing Many people prefer to see but not be seen – At least with current video systems Attention demand & media – – Text: Least – can carry on several IMs at once Audio: More – one at a time, can do other things Video: Most – can’t do other things Not a flaw, just something to take into account Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 47

Why is this all so complicated? Voice telephony doesn’t have these problems Because people are evolved to talk in the dark – This is why telephones “work” Because video is not “just another channel” – But that’s how engineers usually think about it – It’s something very different than audio The video telephony experience needs to feel more natural and intuitive – I think this is the real reason it hasn’t succeeded yet – This is where research needs to focus Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 48

We are still in the “mainframe” era Video telephony is not unique in facing this challenge – Automobiles: Benz Motorwagen to Ford Model T • Mass production, simplicity – Aviation: Wright Flyer to Douglas DC-3 • Efficiency, safety – Computing: Mainframes to PCs • VLSI microprocessors From possibilities in theory to useful practice – – – High-value niche applications come first These teach us about what is missing When technology matures, the mass-market arrives Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 49

Where we go from here “Show-me” 3 G video phones can succeed now – – See where I am See who I’m with Human factors issues not a problem in this model Limited usage compared to voice minutes Video conferencing, other high-value apps will continue to mature & expand Human factors improvements needed – For “talking heads” video telephony to succeed A fertile field for research – please work on it! Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 50

It may take a visionary individual ? Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 51

Thank you! Polycom has an opening for a video DSP researcher to work on these topics Send CVs to lindbergh@92 F 1. com Past, Present, and Future of Video Telephony – Lindbergh – January 2005, Stanford University 52