A  Real Time  3D  Face  Reconstruction

A Real Time 3D Face Reconstruction

Shahzadi Amber Ijaz1, Muhammad Sajid Khan2 Mazdaq Iqbal3,Rafay Tariq4
2
College of Computer Science, Sichuan University, Chengdu . China
[email:sajidpk48@yahoo.com]
1,3,4
Army Public College of Management & Sciences, Rawalpindi Pakistan
[email:shahzadiamber512@gmail.com]
*Corresponding author:

Abstract
In this Paper, we present reconstruction of 3DMMs using DEMAP (detection and mapping)
algorithm that are strong priors on 3D face, shape and texture that could be leveraged in fitting
algorithms to accurate and robust the reconstructed complete 3D representations of faces from data
sources like in-the-wild 2D images or live input video. The process of reconstruction of 3DMMs
includes two major sections. First robust cascaded regression landmark detection from Random
Cascaded-Regression Copse(R-CR-C) for robust facial Landmark detection in order to generate
3D mesh frame models to convert that mesh frame into morphable model from live camera input.
It loads a face landmark’s mesh frame model trained with RCR-training, detects a face using Haar
cascade classifier and then runs the landmark detection algorithm. Second understanding the
annotated training data for face geometry algorithms that covers a large variety of pose estimation
and face parameters position. It proceeds furthers to mapping of 3D mesh frame with face
(detected from live camera) using landmark mapper localization algorithm of OpenCV or DLB
resulting the reconstruction of 3D morph able model. As a results 3DMM will be generated whose
resolution could vary with system specifications. In contrast to many existing fitting algorithms
our system achieves real time performance.

Keywords: Random Cascaded Regression Copse, unsupervised learning, 3D morphable

Model, Frontalization, Open Source Computer Version, Dynamics Load Balancing

1. Introduction
                                                                      A  Real Time  3D  Face  Reconstruction

Modelling and synthesizing faces is an active research field in computer graphics and
computer vision. Characters modelling and animation techniques are instrumental in
increasing the level of realism in modern real time applications.
Many applications of virtual environments (VE) require the modelling of human-like
characters with high visual and physical accuracy. Many attempts have been made to
model, animate, and interact with human-like characters in VE. Modelling of these
characters, however, can become a non-trivial task when it comes to dealing with real-
time constraints [1][3][4].
The 3D modelling is used in a many industries example: films, animation and
gaming. A large number of software are being used in modelling a digital view of
character and objects models before their manufacturing. These kinds of software not
only generate the different part of an object but also assemble their functionality with
respect to their specifications.
Our main objective in this paper is to generate a human head model by assembling
the proposed features information via camera. The features of modelled character will be
as similar as in real.
Our approach is basically reconstruction of 3D Face Model using 2D video stream
that will be fix from the camera. Reconstruction of 3D face model from a 2D video is an
open problem in the field of computer vision. As requirement for 3D face reconstruction
is still growing, however, an accurate reconstruction of a 3D face model from 2D face
images still remains challenging due to training algorithm complexity [2][3] .
The proposed method of our 3D face reconstruction, belongs to the type of face
reproducing, in which, given a 2D input from live camera, its corresponding 3D face
version is generated. The reconstruction process of 3D facial models is a wide topic that
is considered hot area within the research community. This is an example of 3D face
frontalization, where approaches of Computer Vision and Computer Graphics are used to
solve a problem related to reconstruction of faces.
For the face reconstructing, many methods of reconstruction and 3D facial animation
have been proposed. The approaches considered in our approach is. i) based on the
location of facial landmarks, where we used to do mapping between the 2D input face
and the face model obtained used in the training stage, and ii) application of face
frontalization algorithm via face detection algorithm of open Cv.

2. Related Work
Research Many researchers have approached lot of techniques with variation results with
different prospectus. The work that we highlighted here, has multiple domain [5].Our
main concern is 3D face reconstruction with input obtained through live camera. But
many existing systems have been developed like face modeling from a base shape, face
deforming, face texturing and use of Gaussian filters to improve face modeling
techniques and to produce more like real face [5][6] .
KSII TRANSACTIONS ON INTERNET AND INFORMATION SYSTEMS

The work we have proposed is not very much new since it has work already done
from various perspectives yet the solution carries a different technique that is far cheaper
and easy to access as compared to existing ones like a research has covered face modeling
via Infrared Scan based face detection that carries 6 monochromatic cameras which is an
expansive set up. More ever use of 3D morph able model to map input face on an output
with a predefined set of landmarks provides far better and accurate results than most of
systems [7] .

3. Proposed Approach

Section A

3 .1 Landmark detection from Random Cascade Regression Copse( R-CR-

C)

A face shape that is an area used to detect landmark on is represented by the coordinates
of P landmarks
s = [x1, y1, · · ·, xP, yP ]T while s is set of coordinates of landmark. Given a facial image I
and an initial face shape estimate s0, the aim of CR- based approach is to find a shape
updater U:

U: f(I, s0) → δs (1)

||s0 + δs − ˆs||2 (2)

where f (I, s0) is a shape-related feature mapping function δs is the shape update and ^s is
the ground truth shape.
U:f represents dependency of U on f.
U here is a strong regressor that is used to estimated data carrying landmark position.
Above mentioned example is a general representation of how cascaded regression of
landmarks will be calculated [11].

3.2 R-CR-C implementation

In R-CR-C based regression basic key component is to design multiple cascaded

regressors and fuse their estimation of landmark detection to obtain better face shape
estimate.
In our propose algorithm we illustrate a copse with 3 CR regressors to obtain a 3D
shape each of them labelling as 1D, 2D, and 3D respectively and saving their estimated
data in from of Eigen vector. Given a training dataset with N labelled facial images can
set Sf to either the distance between the pupils, or the distance between the mean of the
                                                                      A  Real Time  3D  Face  Reconstruction

two outer mouth corners and the mean of two outer eye corners, or the maximum of these
two distances. Here S f is calculated by updating face shape directly, but this attempt is not
very accurate after the first regressor’s estimated data, due to the rough initial shape
estimate from the face bounding box. However, the estimate becomes more accurate as
the current shape gets closer to the defined deep value and data of 3 regressors have been
obtained.
Technical view of this training can be seen in following diagram.

Fig. 1. Regression Training

For instance, when we set Sf to the pupil distance and pick the shrinking parameter K =
2 for a 5-deep (D-deep: number of week regressor) CR copse, the patch size of area to put
landmark on decreases from half size of the inter landmark distance for the 1st regressor
to a quarter for the last one. Here we resize these patches to a fixed size (25 × 25 in our
case) and then extract local features.
The shape initialization and training data rescaling for landmark detection were
performed in the same way as mentioned above. The initial shape estimate was obtained
by putting the mean shape at the centre of the detected face bounding box. The training
data was increased in size by randomly fazing the initially recorded shape estimates. The
parameters of R-CR-C were adjust by cross validation where the width is set to 3 depth of
facial landmark is set to 5 and the regularization weight in order to provide additional
information for landmark accurate detection.

Section B:

3.3 Face Detection

Face detection has been improved in terms of speed with the application of haar-features
whose detailed data training supported by open CV’s algorithm haar Cascaded.
Implementations of this framework, such as OpenCV, provide different face classifiers
created by authors that used different datasets into their training. The performance and
KSII TRANSACTIONS ON INTERNET AND INFORMATION SYSTEMS

reliability of these classifiers varies a lot evaluating the performance of some classifiers
and also tested their accuracy. This paper focuses on evaluating facial classifiers
regarding facial features contained in the found face.

Fig.2.Structure of Haar Cascade

In this paper we used Haar cascaded algorithm a complete build in package integrated
with open CV. Simple method of using .xml file that contains coded haar like feature
based face detection can be represented:

("facedetector,f",po::value<fs::path>(&facedetector)-
>required()>default_value("../share/haarcascade_frontalfac alt2.xml"),
(haarcascade_frontalface_alt2.xml)")

3.4 Face Frontalization

Reconstruction of a frontal view of a face from a 2D live video camera with deep pose
variation is very challenging because recovering the 3D information from 2D projections
is equivocal and there exists self-occlusion that is object’s parameters overlapping each
other.
                                                                      A  Real Time  3D  Face  Reconstruction

Face Frontalization is a Straight-forward method to build 3D models for faces and

directly rotate the 3D face models.
So Frontalization from 2D video projection is a most focused problem. A totally
data-driven approach might be possible given sufficient data and an appropriate training
sequence, however it is non-trivial. Therefore, we propose to implement a prior on the
generation process, in the form of a 3D Morphable Model (3DMM).This reduces the
training complexity and leads to better performance with limited data[12] .
Following equation explain the 3DMM represents faces in 2D projected space:

A=¯A+Xidαid +Xexp αexp, (3)

V=¯V+Xtexαtex (4)

Where A are the 3D shape coordinates computed as the linear combination of the mean
shape ¯A the shape basis Xid and the expression basis X exp, while V is the texture that is
the linear combination of the mean texture ¯V and the texture basis X tex. The coefficients
{αid, αexp , αtex} defines a unique 3D face.
Let p={m, αid, αexp , αtex} denotes the 3DMM coefficients. The target of the
reconstruction module R is to estimate p=R(x), given an input image x. Since the intent is
for R to also be trainable with the rest of the framework, we use a CNN model based on
CASIA-Net [3] for this regression task. We apply z -score normalization to each
dimension of the parameters before training. A weighted parameter distance cost similar
to [4] is used:

minLR= (p−pg)T>W(p−pg) (5)

Where W is the importance matrix whose diagonal is the weight of each parameter.
KSII TRANSACTIONS ON INTERNET AND INFORMATION SYSTEMS

4. Proposed System Diagram

Fig. 3. Our proposed system complete diagram

                                                                      A  Real Time  3D  Face  Reconstruction

5. Comparison with Existing System

Table 1. Comparison of our System vs. Existing System

Approaches Our System Existing systems

Facial Landmark Detection R-CR-C Base landmark Deep-Learning base

detection
Input LIVE Video Camera Image Sequence, Video based

Tool obtained Resolution 649x480(via VGA sensors) Based on video resolution captured
from the camera

Face Detection techniques: openCv Face detection Haar Cascaded

algorithm

Algorithms involved:: DEMAP

(Detection and mapping)

Pose Estimation Technique Gold Standard Algorithm of Hartley &

Zusserman

Texture Representation 100millioseconds parallel Compression at 400 kbites/sec

estimated time with pose estimation and (mostly cylindrical texture mapping)
landmark detection( measured
in i7-47000MQ)
Linear texture mapping

Output 3D face reconstruction 3D facial expression reconstruction

6. Measurements Based Analysis

Table 1.Existing Systems Efficiency
Measures [3] [4] [5] [6] [7]
Resolution  512x256 512x512 640x480 960x520 800x600
Frames per  30 30 20 15 30
sec
Image  105bit/s 240 k bit/s 14bit/s 7.5 bits/s 20  bits/s
KSII TRANSACTIONS ON INTERNET AND INFORMATION SYSTEMS

processing 
rate 
Landmarks  28 With the  Over all  22 View Based, 
considered help of  1257 point  Modular 
Fiducials  160 points Eigenspace 
3D points  Semi  methods of 
have been  automatic  tracking eyes, 
selected  deature  head, ,nose
which acta point 
as control  extraction 
points to  Snake 
wrap  contouring 
cyber  method by 
ware scan  Kass
mesh of 
head
Face  DDMs(Discrete Cyberware Free­Form  Features  FEM(finite 
geometry  Deformable  scan of  Deformatio tracking by   elements 
model face  n  kalman  method) mesh
filter
LMTC(loca
l maximum 
curvature 
tracing) 
algorithm
Resulted  Functional face 3D 3D 3D Noval  FACE like 
Dimension  model poor  avatar model
resolution 
frontalization 
Input type  Texture map  Video of  2D pictures  Video frame 2D recoded 
via relaxation  Live  video 
method actor’s 
face  
                                                                      A  Real Time  3D  Face  Reconstruction

6.1 Measurements based Line Graph

Fig. 4.Frames vs Landmarks vs processing rate of existing face frontalization systems

[15]

6.2 Landmarks variation Graph

KSII TRANSACTIONS ON INTERNET AND INFORMATION SYSTEMS

Fig. 5.Landmarks variation Difference [17]

6.3 Output based analysis

Table 3. Algorithms and perspective Results

Approach Resulted Faces

Discrete Deformable
Models [3]

Deformable polygonal 3D
face models with changing
texture [4]

Free Form Deformation [5]

1. Infrared (IR) based

facial features
tracking.[6]
2. 3D face avatar
based animation.[6]

FEM(finite elements
                                                                      A  Real Time  3D  Face  Reconstruction

method) of head modelling

[7]

Our System: DEMAP

Approach

5. Conclusion

The 3D Morphable models of the human face have prepared myriad of application in computer
graphics and computational photography. However, the acquisition, registration, modelling and
reconstruction of morphable models are currently point of interest that demands major efforts.
The 3DMMs have been used in face recognition and also introduces in face analysis, face
texture tracking, landmark and facial parameters and pose estimation based on 3D face data.
However, due to the type, amount and complexity of training data, as well as the landmark
mapping fine detailing bases, the representation power of 3DMM can be limited. In this Paper, we
present reconstruction of 3DMMs using DEMAP (detection and mapping) algorithm that are
strong priors on 3D face, shape and texture that could be leveraged in fitting algorithms to accurate
and robust the reconstructed complete 3D representations of faces from data sources like in-the-
wild 2D images or live input video.

References

[1] D. Yi, Z. Lei, S. Liao, and S. Z. Li. Learning face representation from scratch. arXiv
preprint:1411.7923, 2014.
[2]. X. Zhu, Z. Lei, X. Liu, H. Shi, and S. Z. Li. Face alignment across large poses: A 3D
solution. In CVPR, 2016.
[3]. Lee, Y., Terzopoulos, D., & Waters, K. (1995, September) Realistic modeling for facial
animation In Proceedings of the 22nd annual conference on Computer graphics and interactive
techniques (pp. 55­62). ACM.
[4]. Prag, A. J. N. W. (1997). Making faces
[5]. Lee, W. S., & Magnenat­Thalmann, N. (2000) Fast head modeling for animation Image
and Vision Computing, 18(4), 355­364.
[6]. Wei, X., Zhu, Z., Yin, L., & Ji, Q. (2004, July). A real time face tracking and animation
KSII TRANSACTIONS ON INTERNET AND INFORMATION SYSTEMS

system In Computer Vision and Pattern Recognition Workshop, 2004. CVPRW'04. Conference on
(pp. 71­71). IEEE.
[7]. Essa,   I.,   Basu,   S.,   Darrell,   T.,   &   Pentland,   A.   (1996,   June).   Modeling,   tracking   and
interactive   animation   of   faces   and   heads   using   input   from   video.   In   Computer   Animation'96.
Proceedings (pp. 68­79). IEEE
[8]. Chen  Cao*  YanlinWeng*   Stephen  Lin*  Kun Zhou*  at   State  Key  Lab  of  CAD&CG,
Zhejiang University †  3D Shape Regression for Real­time Facial Animation Microsoft Research
Asia.
[9]. Patrik   Huber1,   Guosheng   Hu2,   Rafael   Tena1_,   Pouria   Mortazavian3,   Willem   P.
Koppen1,   William Christmas1, Matthias R¨atsch4 and Josef Kittler1 (2015) A Multi resolution
3D Morphable Face Model and Fitting Framework.
[10]. Rafael  Padilla, Marly Costa, cicero    Evaluation of Haar  Cascade Classifiers  for Face
Detection Conference paper April 2012.
[11]. Zhen­Hua Feng, Patrik Huber,  Josef Kittler, Life Member, IEEE, William  Christmas,
Xion  Random Cascaded­Regression Copse for Robust Facial Landmark Detection.

[12]. Xi   Yiny,   Xiang   Yuz,   Kihyuk   Sohnz,   Xiaoming   Liuy   and   Manmohan   Chandraker
Towards  Large­Pose  Face   Frontalization  in  the Wild  Michigan  State   University University  of
California, San Diego NEC Laboratories America.
[13]. Jesús   P.   Mena­Chalco,   Luiz   Velho,   Roberto   M.   Cesar   Junior   3D   human   face
reconstruction   using   principal   components   spaces   1   Institute   of   Mathematics   and   Statistics,
University   of   São   Paulo,   São   Paulo   –   Brazil     2   National   Institute   of   Pure   and   Applied
Mathematics,   Rio   de   Janeiro   –   Brazil­jmena@vision.ime.usp.br,lvelho@impa.br,
cesar@ime.usp.br [2016].
[14]. Sandipan Banerjee, Joel Brogan, Janez Kriˇzaj, Aparna Bharati, ‘‘To Frontalize or Not To
Frontalize: Do We Really Need Elaborate Pre­processing To Improve Face Recognition?’’Dept. of
Computer Science & Engineering, University of Notre Dame, USA
[15]. Aaron S. Jackson, Adrian, Vasileios ,Georgios Tzimiropoulos1 ‘‘Large  Pose 3D Face
Reconstruction from a Single Image via Direct Volumetric CNN Regression’’ 1 The University of
Nottingham, UK 2 Kingston University, UK
[16]. Elad   Richardson,   Matan   Sela,   Roy   Or­El,     Ron   Kimme,     Learning   Detailed   Face
Reconstruction   from   a   Single   Image,   Department   of   Computer   Science   and   Engineering,
University of Washington.
[17]. Christos, Yannis, Stefanos , Maja , Robust Statistical Face Frontalization ICVV paper.   
                                                                      A  Real Time  3D  Face  Reconstruction