Extensible 3 D X 3 D Graphics Requirements

Extensible 3 D (X 3 D) Graphics Requirements for Video on the Web Don Brutzman W 3 C Workshop, Video on the Web 12 December 2007 Web 3 D Consortium www. web 3 D. org Naval Postgraduate School Monterey California USA

Topics X 3 D Overview Functional Requirements • Video in X 3 D Scenes • Converting 3 D Models to Video Recommendations

X 3 D Overview

www. web 3 D. org The Web 3 D Consortium is a public-private partnership of industry, agencies, universities and individuals working on open standards for real-time 3 D communication on the Web 3 D develops, implements, evaluates and writes the Extensible 3 D (X 3 D) Graphics specification.

Today’s www. web 3 D. org page

Community rules Thanks to steady innovation by Web 3 D members and users, new features continue to evolve and grow into X 3 D capabilities Working groups identify requirements and write implementable specifications Individuals can directly extend specification capabilities without formal “permission”

ISO Formal annual review and ratification by the International Standards Organization (ISO) have made X 3 D an approved standard for real -world use, both on and off the Web Experts from 12 -15 nations review our specs Immediate adoption by other governing bodies helps to increase deployment

W 3 C Further collaboration by Web 3 D Consortium with the World Wide Web Consortium (W 3 C) has made X 3 D a "first-class citizen" on the Web, providing excellent (and growing) interoperability with other XML standards Current joint work includes efforts on Efficient XML Interchange (EXI) for compression and processing improvements.

Intellectual Property Rights (IPR) Web 3 D and W 3 C have similar policies Any known patented technology must be declared by members prior to consideration by working groups Any patented technology contributions must be licensed on a royalty-free (RF) basis for inclusion in an openly used Web standard http: //www. web 3 d. org/membership Caveat: any legal problem can be solved, but only in advance

Open Source Open for any use, without license fees Free = freedom to innovate • Not necessarily free cost (more like “free puppy”) Common shared example implementation(s) • Not a reference implementation – the specification/standard hopefully provides that – but required nevertheless Open source implementations are necessary to break logjams when company participants can’t resolve technical issues Also provides self-sustaining business model for continued activity, improvement

IPR summary Royalties incompatible with Web architecture Open standards + open source are both needed • Open source by itself is necessary but insufficient • Proprietary implementations are great too Standards organizations and IPR agreements provide a stable playing field for long term

X 3 D Profiles for Extensibility Different levels of content complexity Browsers can support increasing levels of capability Authors can use the proper palette for intended delivery Movie. Texture node is Immersive Profile

Example syntax <Shape> <Box/> <Appearance> <Movie. Texture/> <Texture. Transform/> </Appearance> </Shape>

Family of X 3 D specifications Abstract specification describes “how it works” Equivalent encodings XML. x 3 d, Classic. VRML. x 3 dv, Binary. x 3 db Scene Access Interface (SAI) Consistent programming in Ecma. Script, Java

X 3 D Specifications honeycomb diagram

The key challenge is scalability Because the only information systems capable of scalably growing to match global scope are the Internet and the World Wide Web, X 3 D deliberately follows the architectural principles of World Wide Web. Architecture of the World Wide Web, Volume One http: //www. w 3. org/TR/webarch

Functional Requirements Video within X 3 D Scenes 3 D Geometry from Video

Linkability • Video content needs to be referencable by URI • X 3 D approach allows ordered list of equivalent URI addresses for reliable portability, access “Bookmark” additions are desirable for video, similar to using Viewpoints as X 3 D bookmarks • Further annotation of location or time as part of URI is under consideration

Planet 9 Raygun demos raygun. planet 9. com

Playability Internal playability • Video is often used as a user-facing “billboard” within the X 3 D scene • Can be special display or match a particular view • Mixed rendering is challenging but feasible • Performance requirements are demanding External playability • Anchor links or scripts may launch external video displays for multimedia experience

Interoperability Small number of required formats: • PNG JPEG GIF images, WAV audio, MPEG-2 video • Motivated by performance and royalty-free (RF) • Alternate formats are allowed but might not be supported, url list can alleviate this problem • X 3 D browsers not (yet? ) plugging in video codecs, but plugin interoperability continues to improve Adoption of imagery, audio and video formats supported by W 3 C is shared goal for Web 3 D

Planet 9 Ray. Gun scene 1 Webcam stream as Movie. Texture Billboard

Animation Need synchronization of video playback with 3 D -centric event-based behavior model • Necessary for consistent playability • Connect simple X 3 D animation interpolators, sequencers • Unlocks both Ecmascript and Java, either inside X 3 D Script node or externally in HTML page Future work assignment • investigate possible alignment of X 3 D event model with W 3 C’s Synchronized Multimedia Integration Language (SMIL)

Metadata overview X 3 D included document metadata (similar to html meta tags) and typed Metadata nodes • Also Humanoid Animation (H-Anim) joint naming Some metadata conventions are emerging • SAVAGE Modeling and Analysis Language (SMAL) No metadata vocabularies yet formalized by Web 3 D • Won’t reinvent other unencumbered standards

Metadata requirements Video metadata of interest • duration, loopability, whether file or streamed • time, geographic location for precise positioning in virtual environment • Camera direction (and perhaps settings) • Ability to add other metadata extensions Metadata needs to be inspectable by browser • Use cases available for video embedded within, or external to, video content stream

Construction of 3 D geometry Current research demonstrating ability to automatically generate 3 D models from carefully registered/recorded video streams • Variety of techniques are being developed Applications include city modeling, tracking human pedestrians, classifying produced objects, querying associated databases, etc.

Planet 9 Ray. Gun scene 2 Numerous efforts generate matching 3 D geometry using video sources

Security Scene authors often want to encrypt or digitally sign 3 D content Content is king, but creation is hard W 3 C recommendations for XML Security appear appropriate: encryption, signature, PKI Security needs to be compatible with digital rights management (DRM) scheme Stream security needed, but not yet addressed

Digital rights management X 3 D’s XML and Compressed Binary encodings allow use of W 3 C’s Security recommendations XML Encryption XML Digital Signature (for authentication) Public key infrastructure DRM appears to be feasible, but not used yet More uses exist than Hollywood-commercial See Sun’s DRea. M project http: //www. openmediacommons. org

Streamability Numerous historic examples of streamed video in 3 D environments Streamed video in X 3 D browsers is feasible but infrequent • Likely due to performance and format permissions Real Time Streaming Protocol (RTSP) might be mapped to distribute X 3 D event model among multiple players • Exemplar is Distributed Interactive Simulation (DIS) protocol Clear shared standards strategy might motivate Web 3 D member support for necessary next-step efforts

Planet 9 Ray. Gun scene 3 Browsers have ability to stream video within scenes

Synthesis Merging recorded or live video with 3 D geometry • Motion tracking for human avatars • View from a window Likely future capability of broad interest

Transparency Goal: superposition of selected video pixels of interest without box artifacts • Example: video talking head or human avatar without confounding background Ability to encode some pixels as transparent • Red Green Blue Alpha (RGBA) pixel encoding is not practical for video streams • Designating certain color as transparent is feasible, e. g. “green screen”

Export Recording of animated activity in immersive environment, as seen from one or more animated viewpoints • Practice becoming more commonplace, (e. g. video -game players save victorious sessions) • Different from playing back events within 3 D • Hollywood cliché “What I really want is to direct” now becoming feasible • Similar requirements likely (e. g. video metadata)

Recommendations

Recommendations X 3 D graphics requirements for Web video are technically and politically feasible • Interactive 3 D performance is biggest challenge • Plethora of oddball non-RF codecs unsustainable • Royalty free model for video content thus essential Consider X 3 D technical requirements for video when building the foundation for emerging multimedia Web

Contact Don Brutzman brutzman@nps. edu http: //web. nps. navy. mil/~brutzman Code USW/Br, Naval Postgraduate School Monterey California 93943 -5000 USA 1. 831. 656. 2149 voice 1. 831. 656. 7599 fax