Image-Based Rendering Computer Vision CSE 576, Spring 2005 Richard Szeliski CSE 576, Spring 2005 Image-Based Rendering
Today’s lecture Image–Based Rendering • Light Fields and Lumigraphs • Panoramas and Concentric Mosaics • Environment Matting • Image-Based models CSE 576, Spring 2005 Image-Based Rendering 2
Today’s lecture Video-Based Rendering • Facial animation • Video matting and shadow matting • Video Textures and Animating Stills • Video-based tours CSE 576, Spring 2005 Image-Based Rendering 3
Readings • S. J. Gortler , R. Grzeszczuk , R. Szeliski and M. F. Cohen, The Lumigraph, SIGGRAPH'96. • M. Levoy and P. Hanrahan, Light field rendering, SIGGRAPH'96. • H. -Y. Shum and L. -W. He. Rendering with concentric mosaics, SIGGRAPH’ 99. CSE 576, Spring 2005 Image-Based Rendering 4
Readings • D. E. Zongker et al. Environment matting and compositing, SIGGRAPH'99. • Y. -Y. Chuang et al. Environment matting extensions: Towards higher accuracy and real-time capture. SIGGRAPH'2000, pp. 121 -130, 2000. • P. E. Debevec , C. J. Taylor and J. Malik, Modeling and rendering architecture from photographs: …, SIGGRAPH'96. CSE 576, Spring 2005 Image-Based Rendering 5
Readings • Y. -Y. Chuang et al. Video matting of complex scenes. ACM Trans. on Graphics, 21(3): 243 -248, July 2002 • Y. -Y. Chuang et al. Shadow matting. ACM Transactions on Graphics, 22(3): 494 -500, July 2003. • A. Schödl et al. , Video textures. SIGGRAPH'2000, pp. 489 -498, 2000. • M. Uyttendaele et al. Image-based interactive exploration of real-world environments. IEEE Comp. Graphics and Applications, 24(3), May/June 2004. CSE 576, Spring 2005 Image-Based Rendering 6
Lightfields and Lumigraphs (with lots of slides from Michael Cohen) CSE 576, Spring 2005 Image-Based Rendering
Modeling light How do we generate new scenes and animations from existing ones? Classic “ 3 D Vision + Graphics”: • • • take (lots of) pictures recover camera pose build 3 D model extract texture maps / BRDFs synthesize new views CSE 576, Spring 2005 Image-Based Rendering 8
Computer Graphics Output Image Synthetic Camera CSE 576, Spring 2005 Image-Based Rendering Model 9
Computer Vision Output Model Real Scene Real Cameras CSE 576, Spring 2005 Image-Based Rendering 10
Combined Output Image Synthetic Camera CSE 576, Spring 2005 Model Image-Based Rendering Real Scene Real Cameras 11
But, vision technology falls short Output Image Synthetic Camera CSE 576, Spring 2005 Model Image-Based Rendering Real Scene Real Cameras 12
… and so does graphics. Output Image Synthetic Camera CSE 576, Spring 2005 Model Image-Based Rendering Real Scene Real Cameras 13
Image Based Rendering Output Image Real Scene Synthetic Images+Model Camera Real Cameras -or. Expensive Image Synthesis CSE 576, Spring 2005 Image-Based Rendering 14
Ray Constant radiance • time is fixed 5 D • 3 D position • 2 D direction CSE 576, Spring 2005 Image-Based Rendering 15
All Rays Plenoptic Function: • all possible images • too much stuff! CSE 576, Spring 2005 Image-Based Rendering 16
Line Infinite line 4 D • 2 D direction • 2 D position • non-dispersive medium CSE 576, Spring 2005 Image-Based Rendering 17
Ray Discretize, then interpolate Distance between 2 rays • Which is closer together? CSE 576, Spring 2005 Image-Based Rendering 18
Image What is an image? All rays through a point • Panorama CSE 576, Spring 2005 Image-Based Rendering 19
Panoramic Mosaics Convert panoramic image sequence into a cylindrical image + … + CSE 576, Spring 2005 Image-Based Rendering = 20
Image plane 2 D • position in plane CSE 576, Spring 2005 Image-Based Rendering 21
Object Light leaving towards “eye” 2 D • just dual of image CSE 576, Spring 2005 Image-Based Rendering 22
Object All light leaving object CSE 576, Spring 2005 Image-Based Rendering 23
Object 4 D • 2 D position (on surface) • 2 D direction CSE 576, Spring 2005 Image-Based Rendering 24
Object All images CSE 576, Spring 2005 Image-Based Rendering 25
Lumigraph / Lightfield Outside convex space Empty Stuff 4 D CSE 576, Spring 2005 Image-Based Rendering 26
Lumigraph Inside convex space Stuff Empty 4 D CSE 576, Spring 2005 Image-Based Rendering 27
Lumigraph How to ? • organize • capture • render CSE 576, Spring 2005 Image-Based Rendering 28
Lumigraph - Organization 2 D position 2 D direction q CSE 576, Spring 2005 Image-Based Rendering s 29
Lumigraph - Organization 2 D position s u 2 plane parameterization CSE 576, Spring 2005 Image-Based Rendering 30
Lumigraph - Organization 2 D position s, t t u, v s, t v u, v 2 plane parameterization CSE 576, Spring 2005 Image-Based Rendering s u 31
Lumigraph - Organization Hold s, t constant Let u, v vary An image CSE 576, Spring 2005 Image-Based Rendering s, t u, v 32
Lumigraph - Organization Discretization • higher res near object – if diffuse – captures texture • lower res away – captures directions CSE 576, Spring 2005 Image-Based Rendering s, t u, v 33
Lumigraph - Capture Idea 1 • Move camera carefully over s, t plane • Gantry – see Light Field paper CSE 576, Spring 2005 Image-Based Rendering s, t u, v 34
Lumigraph - Capture Idea 2 • Move camera anywhere • Rebinning – see Lumigraph paper CSE 576, Spring 2005 Image-Based Rendering s, t u, v 35
Lumigraph - Rendering For each output pixel • determine s, t, u, v • either – find closest discrete RGB – interpolate near values CSE 576, Spring 2005 Image-Based Rendering s, t u, v 36
Lumigraph - Rendering q For each output pixel • determine s, t, u, v • either • use closest discrete RGB • interpolate near values CSE 576, Spring 2005 Image-Based Rendering s u 37
Lumigraph - Rendering Nearest • closest s • closest u • draw it Blend 16 nearest • quadrilinear interpolation CSE 576, Spring 2005 Image-Based Rendering s u 38
Lumigraph - Rendering Depth Correction • • closest s intersection with “object” best u closest u CSE 576, Spring 2005 Image-Based Rendering s u 39
Lumigraph - Rendering Depth Correction • quadralinear interpolation • new “closest” • like focus [Dynamically Reparameterized Light Fields, Isaksen, SG’ 2000] CSE 576, Spring 2005 Image-Based Rendering s u 40
Lumigraph - Rendering Fast s, t, u, v finding • scanline interpolate • texture mapping • shear warp CSE 576, Spring 2005 Image-Based Rendering s u 41
Lumigraph - Ray Space 3 D space ray space surface depth slope in ray space CSE 576, Spring 2005 Image-Based Rendering 42
Lumigraph - Ray Space Image effects: • parallax • occlusion • transparency • highlights CSE 576, Spring 2005 Image-Based Rendering 43
Lumigraph - Demo Lumigraph • Lion, Fruit Bowl CSE 576, Spring 2005 Image-Based Rendering 44
Complex Light Field acquisition Digital Michelangelo Project – Marc Levoy, Stanford University – Lightfield (“night”) assembled by Jon Shade CSE 576, Spring 2005 Image-Based Rendering 45
Unstructured Lumigraph What if the images aren’t sampled on a regular 2 D grid? • can still re-sample rays • ray weighting becomes more complex [Buehler et al. , SIGGRAPH’ 2000] CSE 576, Spring 2005 Image-Based Rendering 46
Surface Light Fields Turn 4 D parameterization around: image @ every surface pt. Leverage coherence: compress radiance fn (BRDF * illumination) after rotation by n CSE 576, Spring 2005 Image-Based Rendering 47
![Surface Light Fields [Wood et al, SIGGRAPH 2000] CSE 576, Spring 2005 Image-Based Rendering](https://present5.com/presentation/a500a1612213d848c0d60068c82e5475/image-48.jpg "Surface Light Fields [Wood et al, SIGGRAPH 2000] CSE 576, Spring 2005 Image-Based Rendering")
Surface Light Fields [Wood et al, SIGGRAPH 2000] CSE 576, Spring 2005 Image-Based Rendering 48
3 D Representations Image (and panoramas) are 2 D Lumigraph is 4 D What happened to 3 D? • 3 D Lumigraph subset • Concentric mosaics CSE 576, Spring 2005 Image-Based Rendering 49
3 D Lumigraph One row of s, t plane • i. e. , hold t constant CSE 576, Spring 2005 Image-Based Rendering s, t u, v 50
3 D Lumigraph One row of s, t plane • i. e. , hold t constant • thus s, u, v • a “row of images” s [Sloan et al. , Symp. I 3 DG 97] CSE 576, Spring 2005 Image-Based Rendering u, v 51
![Concentric Mosaics Replace “row” with “circle” of images [Shum & He, SIGGRAPH’ 97] CSE](https://present5.com/presentation/a500a1612213d848c0d60068c82e5475/image-52.jpg "Concentric Mosaics Replace “row” with “circle” of images [Shum & He, SIGGRAPH’ 97] CSE")
Concentric Mosaics Replace “row” with “circle” of images [Shum & He, SIGGRAPH’ 97] CSE 576, Spring 2005 Image-Based Rendering 52
Concentric Mosaics CSE 576, Spring 2005 Image-Based Rendering 53
Concentric Mosaics Rendering ( as seen from above ) CSE 576, Spring 2005 Image-Based Rendering 54
Concentric Mosaics Depth correction CSE 576, Spring 2005 Image-Based Rendering 55
Concentric Mosaics q Demo CSE 576, Spring 2005 Image-Based Rendering 56
2. 5 D Representations Image is 2 D Lumigraph is 4 D 3 D • 3 D Lumigraph subset • Concentric mosaics 2. 5 D • Layered Depth Images • Sprites with Depth (impostors) • View Dependent Surfaces (see Façade) CSE 576, Spring 2005 Image-Based Rendering 57
Layered Depth Image 2. 5 D ? Layered Depth Image CSE 576, Spring 2005 Image-Based Rendering 58
![Layered Depth Image q Rendering from LDI [Shade et al. , SIGGRAPH’ 98] •](https://present5.com/presentation/a500a1612213d848c0d60068c82e5475/image-59.jpg "Layered Depth Image q Rendering from LDI [Shade et al. , SIGGRAPH’ 98] •")
Layered Depth Image q Rendering from LDI [Shade et al. , SIGGRAPH’ 98] • Incremental in LDI X and Y • Guaranteed to be in back-to-front order CSE 576, Spring 2005 Image-Based Rendering 59
Layered Depth Image q Rendering from LDI • Incremental in LDI X and Y • Guaranteed to be in back-to-front order CSE 576, Spring 2005 Image-Based Rendering 60
Sprites with Depth Represent scene as collection of cutouts with depth (planes + parallax) Render back to front with fwd/inverse warping [Shade et al. , SIGGRAPH’ 98] CSE 576, Spring 2005 Image-Based Rendering 61
Environment matting and compositing D. E. Zongker, D. M. Werner, B. Curless and D. H. Salesin. SIGGRAPH'99 CSE 576, Spring 2005 Image-Based Rendering
Environment Matting Capture the reflections and refractions of a realworld object Composite object over a novel background CSE 576, Spring 2005 Image-Based Rendering 63
Environment Matting - examples Movie CSE 576, Spring 2005 Image-Based Rendering 64
Environment Matting Capture the mapping from each image pixel to a real-world ray direction(s) CSE 576, Spring 2005 Image-Based Rendering 65
Acquisition setup Use several monitors with stripes CSE 576, Spring 2005 Image-Based Rendering 66
Environment matting equation Captures foreground color and background directions CSE 576, Spring 2005 Image-Based Rendering 67
Environment matting - result CSE 576, Spring 2005 Image-Based Rendering 68
![Environment matting extensions [Chuang et al. , SIGGRAPH’ 2001] • accurate (multiple refractions): •](https://present5.com/presentation/a500a1612213d848c0d60068c82e5475/image-69.jpg "Environment matting extensions [Chuang et al. , SIGGRAPH’ 2001] • accurate (multiple refractions): •")
Environment matting extensions [Chuang et al. , SIGGRAPH’ 2001] • accurate (multiple refractions): • soft stripes • fast (video rate): • color ramp CSE 576, Spring 2005 Image-Based Rendering 69
Image-Based Modeling CSE 576, Spring 2005 Image-Based Rendering
Image Based Models CSE 576, Spring 2005 Image-Based Rendering 71
Image-Based Modeling Create 3 D model (and texture maps) from images • automated • (structure from motion, stereo) • interactive • Façade system CSE 576, Spring 2005 Image-Based Rendering 72
Façade 1. Select building blocks 2. Align them in each image 3. Solve for camera pose and block parameters (using constraints) CSE 576, Spring 2005 Image-Based Rendering 73
View-dependent texture mapping 1. Determine visible cameras for each surface element 2. Blend textures (images) depending on distance between original camera and novel viewpoint CSE 576, Spring 2005 Image-Based Rendering 74
Model-based stereo Compute offset from block model Some more results: CSE 576, Spring 2005 Image-Based Rendering 75
Image-Based Faces Estimate shape from images Match metrics to shape Project video onto shape • Texture map Animate [Z. Liu et al. , MSR-TR-2000 -11] CSE 576, Spring 2005 Image-Based Rendering 76
![Hierarchy of Light Fields [Levoy] 8 D: Refractive/reflective environment 5 D: Plenoptic Function (Ray)](https://present5.com/presentation/a500a1612213d848c0d60068c82e5475/image-77.jpg "Hierarchy of Light Fields [Levoy] 8 D: Refractive/reflective environment 5 D: Plenoptic Function (Ray)")
Hierarchy of Light Fields [Levoy] 8 D: Refractive/reflective environment 5 D: Plenoptic Function (Ray) 4 D: Lumigraph / Lightfield 4 D*: Environment Matte (single view) 3 D: Lumigraph Subset 3 D: Concentric Mosaics 2. 5 D: Layered Depth Image 2. 5 D: Image Based Models 2 D: Images and Panoramas CSE 576, Spring 2005 Image-Based Rendering 77
Graphics/Imaging Continuum Concentric mosaics Image centric Geometry centric Fixed geometry Viewdependent texture Viewdependent geometry Polygon rendering + texture mapping CSE 576, Spring 2005 Sprites with depth LDI Warping Image-Based Rendering Lumigraph Light field Interpolation 78
What lies beyond Image-Based Rendering? CSE 576, Spring 2005 Image-Based Rendering
Video-Based Rendering Image-Based Rendering: • render from (real-world) images for efficiency, quality, and photo-realism Video-Based Rendering • • use video instead of still images for dynamic elements and source footage generate computer video instead of computer graphics CSE 576, Spring 2005 Image-Based Rendering 80
VBR Examples Facial animation • Video Rewrite, … Layer/matte extraction • Video Matting, … Dynamic (stochastic) elements • Video Textures, … 3 -D world navigation • Image-Based Realities, … CSE 576, Spring 2005 Image-Based Rendering 81
![Facial animation Modeling from still images • [Pighin et al. , SG’ 98] Lip-synching](https://present5.com/presentation/a500a1612213d848c0d60068c82e5475/image-82.jpg "Facial animation Modeling from still images • [Pighin et al. , SG’ 98] Lip-synching")
Facial animation Modeling from still images • [Pighin et al. , SG’ 98] Lip-synching from video • • Video Rewrite [Bregler et al. , SG’ 97] [Ezzat et al. , SG’ 02] CSE 576, Spring 2005 Image-Based Rendering 82
Matting and Compositing CSE 576, Spring 2005 Image-Based Rendering
http: //grail. cs. washington. edu/projects/digital-matting/ CSE 576, Spring 2005 Image-Based Rendering 84
Video Matting Pull dynamic -matte from video with complex backgrounds [Chuang et al. @ UW, SIGGRAPH’ 2002] CSE 576, Spring 2005 Image-Based Rendering 85
Video Matting CSE 576, Spring 2005 Image-Based Rendering 86
Shadow Matting Transfer a shadow from one background to another: • Extract and model photometry (darkening) • Extract and model geometry (deformation) CSE 576, Spring 2005 Image-Based Rendering 87
Shadow Matting CSE 576, Spring 2005 Image-Based Rendering 88
Shadow Matting CSE 576, Spring 2005 Image-Based Rendering 89
Shadow Matting Video CSE 576, Spring 2005 Image-Based Rendering 90
Video Textures CSE 576, Spring 2005 Image-Based Rendering
Video Textures How can we turn a short video clip into an amount of continuous video? • • dynamic elements in 3 D games and presentations alternative to 3 D graphics animation? [Schödl, Szeliski, Salesin, Essa, SG’ 2000] CSE 576, Spring 2005 Image-Based Rendering 92
Video Textures Find cyclic structure in the video (Optional) region-based analysis Play frames with random shuffle Smooth over discontinuities (morph) CSE 576, Spring 2005 Image-Based Rendering 93
Region-based analysis CSE 576, Spring 2005 Image-Based Rendering 94
Crossfading and morphing Jump Cut Crossfade Morph CSE 576, Spring 2005 Image-Based Rendering 95
Video portrait CSE 576, Spring 2005 Image-Based Rendering 96
Dynamic scene element Live waterfall in static panorama Demo CSE 576, Spring 2005 Image-Based Rendering 97
Interactive fish CSE 576, Spring 2005 Image-Based Rendering 98
A complete animation CSE 576, Spring 2005 Image-Based Rendering 99
Video-Based Tours CSE 576, Spring 2005 Image-Based Rendering
Video-Based Walkthroughs Move camera along a rail (“dolly track”) and play back a 360 video Applications: • Homes and architecture • Outdoor locations (tourist destinations) CSE 576, Spring 2005 Image-Based Rendering 101
Surround video acquisition system Omni. Cam (six-camera head) CSE 576, Spring 2005 Image-Based Rendering 102
Omni. Cam Built by Point Grey Research (Ladybug) Six camera head Portable hard drives, fiber-optic link Resolution per image: 1024 x 768 FOV: ~100 o x ~80 o Acquisition speed: 15 fps uncompressed CSE 576, Spring 2005 Image-Based Rendering 103
Acquisition platforms Robotic cart Wearable CSE 576, Spring 2005 Image-Based Rendering 104
Demo A Virtual Home Tour CSE 576, Spring 2005 Image-Based Rendering
Open issues How to best sample and interpolate Light Field • (sub-? ) pixel accurate stereo • reflections, refractions, … Compositing • how to insert Light Field into new environment • relighting • …? CSE 576, Spring 2005 Image-Based Rendering 106
Summary Image–Based Rendering • Light Fields and Lumigraphs • Panoramas and Concentric Mosaics • Matting: natural, environment, and shadows • Image-Based models Video-Based Rendering • Facial animation • Video Textures and Animating Stills • Video-based tours CSE 576, Spring 2005 Image-Based Rendering 107
Summary Image–Based Rendering • Light Fields and Lumigraphs • Panoramas and Concentric Mosaics • Environment Matting • Image-Based models Video-Based Rendering • Facial animation • Video matting • Video Textures and Animating Stills • Video-based tours CSE 576, Spring 2005 Image-Based Rendering 108
Скачать презентацию Image-Based Rendering Computer Vision CSE 576 Spring 2005
a500a1612213d848c0d60068c82e5475.ppt