Statistical and numerical techniques for photorealistic image synthesis

Statistical and numerical techniques for photorealistic image synthesis Kartic Subr

Who am I? • Born in India – Bangalore University (Bachelor of Engineering) 2001 – Hewlett Packard, India/Singapore • 6 years in USA – Ph. D, University of California Irvine, 2008 – Advisor: Jim Arvo (Ph. D Yale University), pioneered methods in light transport • 2 years in France – Post doctoral researcher, ARTIS, INRIA-Grenoble (2008 -2010)

My goal: Generating realistic visuals Gustave Courbet, Stone-Breakers, 1849. Realism in art Wilhelm Oswald Gustav Achenbach, Abendstimmung in der Campagna, 1850.

My goal: Generating realistic visuals Photograph: Nicéphore Niépce, 1826 Gustave Courbet, Stone-Breakers, 1849. Realism in art Wilhelm Oswald Gustav Achenbach, Abendstimmung in der Campagna, 1850.

Notion of “realism” depends on technology Pedro Campos Gerhard Richter, 1983 Hyperrealism

Realistic image synthesis

Image synthesis involves light transport Light sources Digital models of scene (geometry + materials) Virtual camera Image ?

Image synthesis adds visual impact Digital model Captured video + digital model Avatar

Applications of image synthesis Entertainment Advertising Virtual prototyping Defense Biomedical imaging

Multidisciplinary nature of the problem • Physically based optical simulations • Mathematical tools for analysis • Numerical techniques for light transport solution • Understanding biological processes eg. Perception et cognition

Reflection of light is an integration

Light transport: multi-domain integration • Combinatorial explosion from sampling each domain Image space Exposure time Aperture Visible spectrum Reflectance Direct illumination Indirect illumination [Efficient sampling strategies for Monte Carlo integration (my Ph. D thesis)]

Talk outline • Recent contributions – Simulating defocus – Rendering translucent materials • Research plan – Core problems in image synthesis – Model representation and abstraction

My contributions: Fourier depth of field

Defocus blur is important in photography

Defocus is due to aperture integraion Image Pixel p Lens Aperture

Defocus Pixel p Image Pixel p Lens Aperture Scene

Monte Carlo estimation of aperture integral NA primary rays per pixel Image Integrate at p Aperture

Aperture integration is very costly NP x NA Primary rays Image Aperture NP pixels NA Aperture samples

Paradox: Blurry image is costlier to compute! 64 x #primary rays of the pinhole image

Observation 1: Image Blurry regions should not require dense sampling of the image

Observation 2: Lens Regions in focus should not require profuse sampling of the lens for diffuse objects

Fourier depth of field • Fourier domain analysis of finite aperture cameras • Adaptive sampling • Speedup of around 20 over existing algorithms [ACM Transactions on Graphics 2009. Presented at ACM SIGGRAPH 09] Collaborators: MIT

My contributions: Translucent materials

Translucency: Sub-surface scattering Opaque Translucent • Brute force Monte Carlo: prohibitively expensive • Diffusion approximations: severe constraints on geometry

Finite difference method on new domain • Approximation: diffusion equation Diffuse flux • Domain: Dual graph of tetrahedralization

Rendering translucent materials • • Arbitrary geometry Heterogenous materials Dynamically deforming shapes In real-time! [Computer Graphics Forum 2010. To be presented at Eurographics 2010] [Collaborators: Microsoft Research, Tsinghua University]

Research program Realistic image synthesis Model representation and abstraction

1. Realistic image synthesis • Bandwidth driven sampling Short-term – Transport of local light field spectrum – Derive spatial / angular sampling rates – co-advising Ph. D student Laurent Belcour (ARTIS) • Importance vs radiance – Tracing from eye vs tracing from light – Monte Carlo matrix chain multiplication Long-term

Importance vs radiance Radiance

Importance vs radiance Importance

Importance vs radiance

MC matrix-chain product estimator

MC matrix-chain product estimator Related to optimal matrix chain multiplication using dynamic programming?

2. Model representation and abstraction • Abstracting detail in geometry Short-term – First step: images (published at SIGGRAPH Asia 09) • Alternate representation – Voxel data to represent geometry and materials Long-term

Detail = oscillations between extrema Local maxima Input Local minima

Image multiscale decomposition Intensity 1 D Input Fine + Medium + Coarse Pixels

Allows smoothing high-contrast detail Input Smoothed

Thank you! • Collaborators – Established • MIT, USA • Microsoft Research • Tsinghua University, China • University of California, Irvine – Current • Cornell University, USA • University of California, Berkeley – Potential • Indian Institute of Information Technology • International journal publications – Computer Graphics Forum 2010: Translucent materials. 4 th author of 6 – TOG 2009: Multiscale image decomposition. 1 st author of 3 – TOG 2009: Fourier Depth of Field. 2 nd author out of 5 • Refereed international conference papers – Pacific Graphics 2007: Statistical hypotheses. 1 st author of 2 – Interactive raytracing 2007: Steerable importance sampling. 1 st author of 2 – ICIAP 2005: Contrast enhancement. 1 st author of 3

Merci ! • Collaborators • Teaching – Columbia University, USA (120 h) – Established – University of California, Irvine (360 h) • MIT, USA • Microsoft Research • Tsinghua University, China • Industry • University of California, Irvine – Rhythm and Hues Studios • LJK Grenoble – NVIDIA Corporation – Current – Hewlett Packard • Cornell University, USA • University of California, Berkeley – Potential • Indian Institute of Information Technology • International journal publications – Computer Graphics Forum 2010: Translucent materials. 4 th author of 6 – TOG 2009: Multiscale image decomposition. 1 st author of 3 – TOG 2009: Fourier Depth of Field. 2 nd author out of 5 • Refereed international conference papers – Pacific Graphics 2007: Statistical hypotheses. 1 st author of 2 – Interactive raytracing 2007: Steerable importance sampling. 1 st author of 2 – ICIAP 2005: Contrast enhancement. 1 st author of 3