Reconstruction of 3 D Face Surface from Slices

Reconstruction of 3 D Face Surface from Slices: A Literature Survey Mahmudul Hasan CPSC 601. 20: Biometric Technologies Department of Computer Science, University of Calgary 2500 University Drive NW, Calgary, AB T 2 N 1 N 4, Canada mhasan@cpsc. ucalgary. ca

Table of Contents Introduction & Background 3 D Face Surface Reconstruction Applications of 3 D Face Surface Reconstruction Existing Methods of 3 D Face Surface Reconstruction Statistical Approach to Shape from Shading Area Matching Based on Belief Propagation Analysis-by-Synthesis Technique Based on 3 D Morphable Model Shape Estimation Based on a Set of Feature Point Locations Minimum Variance Estimation of 3 D Face Shape Comparative Study & Performance Analyses Findings & Conclusions

Introduction The main objective of this study was to perform a comparative analysis of the existing 3 D face surface reconstruction algorithms in terms of their basic methodologies and performance issues. In addition, this study also focuses on some general 3 D surface reconstruction algorithms which can contribute in the reconstruction of 3 D faces. This study has categorized the existing algorithms based on their requirement of prior knowledge about the class of solutions. A detailed comparative study is presented based on the advantages, limitations and areas of application of the studied 3 D face surface reconstruction techniques.

Introduction (cont. ) Face recognition has recently received significant attention as one of the most successful applications of image analysis and understanding, especially during the past several years. At least two reasons account for this trend: the first is the wide range of commercial and law enforcement applications, and the second is the availability of feasible technologies after 30 years of research [1]. The problem of machine recognition of human faces continues to attract researchers from disciplines such as image processing, pattern recognition, neural networks, computer vision, computer graphics, and psychology [1]. Even though current machine recognition systems have reached a certain level of maturity, their success is limited by the conditions imposed by many real applications. For example, recognition of face images acquired in an outdoor environment with changes in illumination and/or pose remains a largely unsolved problem [1].

Introduction (cont. ) One of the major findings of Face Recognition Vendor Test (FRVT) 2002 [2] was that the three-dimensional morphable models and normalization increase the performance of face recognition. 3 D model-based methods [3, 4, 5, 6] provide potential solutions to pose invariant face recognition. 3 D face models are usually derived from laser scanned 3 D heads (range data) or reconstructed using shape from shading [7].

Introduction (cont. ) Realistic looking facial modeling and animation is one of the most interesting and difficult problems in computer graphics [8]. So far, in most of the popular commercially available tools, the 3 -D facial models are obtained not directly from images but by laserscanning of people’s faces [8]. These scanners are usually expensive and a number of hours of work is required prior to animating the model [8]. To avoid the shortcoming of laser-scan based face modeling, image based face modeling methods have received significant attentions in the past several years [8]. Some of these methods reconstruct the 3 D face from one or more 2 D face images (slices).

Background: Existing Methods In 1996, J. J. Atick et al presented a technique for recovering 3 D face shape from a single 2 D image using only the shading information i. e. solving the shape-from-shading problem [3]. I n 2 0 0 4 , D. O n o f r i o et al proposed a method that determines correspondences between surface patches on different views of a face through a modeling of disparity maps based on Markov Random Fields (MWFs) [8]. In 2004, V. Blanz et al presented an algorithm based on a set of feature point locations which produces high-resolution shape estimates of the 3 D face from a 2 D face image [9, 10]. In 2006, V. Blanz et al presented an algorithm based on an analysis-bysynthesis technique that estimates shape and pose by fully reproducing the appearance of the face in the image [10]. In 2006, Z. Zhang et al proposed a minimum variance estimation framework for 3 D face reconstruction from multiple views and a new 3 D surface reconstruction algorithm based on deformable subdivision mesh [11].

Background: General 3 D Surface Reconstruction Algorithms In 1994, D. Shiwei et al proposed a method where the range image is segmented into regions corresponding to the surface patches on objects. Then, algebraic surfaces are fitted to the range points in these regions by solving a generalized eigenvector problem [12]. In 1996, G. Barequet et al presented an algorithm which reconstructs a solid model given a series of planar cross-sections. The main contribution of this work was the use of knowledge obtained during the interpolation of neighboring layers while attempting to interpolate a particular layer [13]. In 2002, S. F. Frisken et al presented an efficient method for estimating 3 D Euclidian distance field from 2 D range images which can be used by many existing algorithms that reconstructs 3 D models from range data [14]. Few other 3 D surface reconstruction exist which are based on given sample points [15, 16, 17] and labeled image regions [18].

3 D Face Surface Reconstruction The goal of 3 D face surface reconstruction is to reconstruct a 3 D face given one or more 2 D face images. The key approach which has been used to solve this problem is to use a 3 D template which is then deformed to represent the target face in the database. For most of the existing algorithms, the template is built such a way that it is deformable to represent all the existing faces in a particular database. A great challenge is to deform the reconstructed 3 D face to represent the target face in the database when the input 2 D images have variations in illumination, pose, and facial expression.

Applications of 3 D Face Surface Reconstruction The reconstructed 3 D face can be used to register a face in the database or for the purpose of face recognition. 3 D face surface reconstruction has some interesting applications in animation and face recognition where a single view of a person can be used to generate the new views to any pose [3]. It also potentially has some biomedical applications; for example, it can be used to design custom masks for facial burn victims from pre-burn photos. These masks are mostly designed using laser scans of person’s face which are expensive, not convenient, and not always feasible for burn victims [3].

Statistical Approach to Shape from Shading (A 1) This technique can recover 3 D face shape from a single 2 D face image using only the shading information [3]. Shading is the variation in brightness from one point to another in an image [3]. Shading carries information about shape because the amount of light a surface patch reflects depends on its orientation (surface normal) relative to the incident light. So, in the absence of variability in surface reflectance properties (surface material), the variability in brightness can only be due to changes in local surface orientation and hence conveys strong information about shape [3]. The statistical technique, principal component analysis (PCA) has been used to derive a low dimensional parametrization of head shape space [3]. The ideal diffuser model or Lambertian model for surface reflectance is used under this technique [3].

Statistical Approach to Shape from Shading (A 1) (cont. ) Lambertian surfaces have two basic properties: firstly, they reflect light diffusely or equally in all directions; secondly, their brightness at any point is proportional to the cosine of the angle between the surface normal at that point and the incident light ray [3]. This algorithm, although idealized, turns out to be a fairly realistic approximation to many surfaces including human skin [3].

Area Matching Based on Belief Propagation (A 2) This method that determines correspondences between surface patches on different views of the face through a modeling of disparity maps based on Markov Random Fields (MWFs) [8]. Under this technique, images were acquired by trinocular calibrated cameras and correspondences between the three views were determined [8]. To deal with the problems of occlusions and textureless regions, disparity maps were modeled with Markov Random Fields (MRFs), in order to propagate information from textured to textureless regions [8]. The Belief Propagation algorithm is applied to obtain the maximum-aposteriori estimation of the disparity maps [8]. In order to reduce false matching due to occlusions, outliers were eliminated by epipolar constraint check [8].

Area Matching Based on Belief Propagation (A 2) (cont. ) In the above mentioned trinocular calibrated camera system, one of the cameras, taken as a reference (master) has reasonable frontal, occlusions free view, while the others (slaves) show some occlusions [8]. The proposed algorithm computes two dense disparity maps between the master and the other two slaves views, and each map is modeled by one pairwise MRF [8]. The marginal probability is estimated performing Belief Propagation (BP) iterations on the MRF [8]. At the end of the process, the MRF results are coupled in order to satisfy epipolar constraint on the triplet of images and hence to eliminate outliers [8].

Area Matching Based on Belief Propagation (A 2) (cont. ) Fig. 1. Face image triplet [8] Fig. 2. Reconstructed 3 D face [8]

Analysis-by-Synthesis Technique Based on 3 D Morphable Model ( A 3) (cont. ) In order to solve the ill-posed problem of reconstructing an unknown shape with unknown texture from a single image, the morphable model approach uses prior knowledge about the class of solutions [10]. In case of 3 D face reconstruction, this prior knowledge is represented by a parametrized manifold of face-like shapes embedded in the high-dimensional space of general textured surfaces of a given topology [10]. More specifically, the morphable model captures the variations observed within a dataset of 3 D scans of examples by converting them to a vector space representation [10]. For surface reconstruction, the search is restricted to the linear span of these examples [10]. Under this technique, estimation of 3 D shapes, texture, pose and lighting are done simultaneously in an analysis-by-synthesis loop [10]. The main goal of the analysis is to find suitable parameters for the morphable model that make the synthetic image as similar as possible to the original image in terms of pixel wise image difference [10].

Analysis-by-Synthesis Technique Based on 3 D Morphable Model ( A 3) (cont. ) Fig. 3. The top row shows the reconstructions of 3 D shape and texture. In the second row, results are rendered into the original images with pose and illumination recovered by the algorithm. The third row shows novel views [10].

Shape Estimation Based on a Set of Feature Point Locations ( A 4) From a small number of 2 D positions of feature points, the algorithm can recover 3 D shape of human faces at high resolution, inferring both depth and the missing vertex coordinates [9]. The system is based on a morphable model that has been built from laser scans of 200 faces, using a modified optical flow algorithm to compute dense point-to-point correspondence. Each face is represented by the coordinates of 75972 vertices at a spacing of less than 1 mm. 140 most relevant principal components have been used [9]. For shape reconstruction, the user clicks on feature points in the image and the corresponding points on the 3 D reference model. Good results are achieved with 15 to 20 points [9]. Due to the automated 3 D alignment, no estimate of pose, position and size is required. The system successfully compensates for rotation, scaling and translation [9]. The color values of the image are mapped as a texture on the surface, and missing color values are reflected from visible parts or filled in with the average texture of the morphable model [9].

Shape Estimation Based on a Set of Feature Point Locations ( A 4) (cont. ) Fig. 4. From an original image at unknown pose (top, left) and a frontal starting position (top, right), the algorithm estimates 3 D shape and pose from 17 feature coordinates, including 7 directional constraints (second row). 140 principal components and 7 vectors for transformations were used. The third row shows the texturemapped result. Computation time is 250 ms [9, 10].

Minimum Variance Estimation of 3 D Face Shape (A 5) This 3 D face surface reconstruction method is based on deformable mesh [11]. The developed system uses six synchronized cameras to capture face images from six different views [11]. Then, a minimum variance estimation framework for 3 D face reconstruction is applied to reconstruct a personalized 3 D model of the face [11]. Next, a 3 D surface reconstruction algorithm based on deformable subdivision mesh is applied to the images captured from different views to get more observations of the 3 D face, especially the depth information, which could not be obtained from a single image directly [11]. This algorithm continuously deforms a triangular mesh to minimize an energy function that measures the matching cost of input images [11]. Finally, the minimum variance estimation is again used to refine the result of the 3 D surface reconstruction algorithm [11].

Minimum Variance Estimation of 3 D Face Shape (A 5) (cont. ) Fig. 5. Synchronized images captured from six views [11] Fig. 6. (a) 2 D face alignment; (b) Initial 3 D shape estimated from the 2 D facial feature points; (c) The deformable mesh; (d) Result of 3 D surface reconstruction; (e) The 3 D shape estimated from 3 D points [11]

Comparative Study Algorithm Number of 2 D images used Depends on the registered faces in the database A 1 1 YES A 2 3 NO A 3 1 YES A 4 1 YES A 5 6 YES

Comparative Performance Analyses Algorithm Advantages Limitations Suitable areas of application Suitable for various pose generation. Not optimized for speed or output quality. Albedo is assumed to be constant. Requires prior knowledge about the class of solutions. Face recognition, animation, 3 D transformation etc. Capable to handle occlusions and textureless regions. Generates very accurate 3 D face model. Computational cost grows linearly. Belief Propagation is highly parallelizable. The regularization parameter is determined heuristically. Uses calibrated cameras i. e. it cannot work with unknown camera parameters. Face recognition, and applications that require 3 D face reconstruction without any prior knowledge about the class of solutions. A 1 A 2

Comparative Performance Analyses (cont. ) Algorithm Advantages Limitations Suitable areas of application Requires prior knowledge about the class of solutions. Face recognition, 3 D transformation etc. A 3 Uses morphable models of 3 D faces which provide a promising technique for face recognition under uncontrolled imaging conditions. Works well even on a wider ethnic variety of faces. Capable to handle occlusions. Estimation of illumination is not handled. Requires prior knowledge about the class of solutions. Face recognition, and computer aided design of non-uniform 3 D surfaces. A 4 Performs comparatively faster 3 D reconstruction from feature points. Computation time for the example presented in [9, 10] is 250 ms. Uses morphable models of 3 D faces which provide a promising technique for face recognition under uncontrolled imaging conditions.

Comparative Performance Analyses (cont. ) Algorithm A 5 Advantages The overall procedure requires about half a minute of computation time. Reconstructs the 3 D face very precisely. Limitations Requires prior knowledge about the class of solutions. Uses calibrated cameras i. e. it cannot work with unknown camera parameters Suitable areas of application Face recognition, animation, 3 D transformation etc.

Findings & Conclusions A brief survey of existing 3 D face surface reconstruction techniques has been conducted under this study. In addition, this study also focused on some general 3 D surface reconstruction algorithms which can contribute in 3 D face surface reconstruction. Along with the description of the methodologies of five 3 D face surface reconstruction algorithms, a detailed comparative analyses of their characteristics, advantages, limitations and the areas of application has been presented under this study. The comparative study found two broad categories of 3 D face surface reconstruction techniques; one of which requires prior knowledge about the class of solutions and other works independently based on the input 2 D images.

Findings & Conclusions (cont. ) The focus of research in 3 D face surface reconstruction is shifting more towards uncontrolled imaging conditions. The techniques that employ 3 D morphable models for faces seem to handle the uncontrolled imaging conditions most promisingly. 3 D face surface reconstruction under different pose, facial expression and illumination is still a great challenge to the researchers.

Thank You. Questions or Comments?