3D Person Tracking in Retail Stores
Russell Kaplan and Michael Yu
Stanford University
450 Serra Mall, Stanford, CA 94305
{rjkaplan, myu3}@stanford.edu
Abstract
In this project, we sought track the movement of mul-
tiple people in 3D given security footage that is represen-
tative of what would be available in retail stores without
modifying existing camera deployments. More specifically,
this involves using a single, possibly fisheye-distorted view
to track people in 3D space and model where they are in a
room or store. This involved bridging various papers across
the computer vision literature, looking at radial distortion
resolution for images from an uncalibrated camera; cali-
bration techniques from single-view metrology, affine ap-
proximation, and a simplified special-case mathematical
model for object detections on a ground plane; and deep
region-based convolutional networks for 2D person detec-
tion.
We define an approach that uses this single security cam-
era view to track people on a ground plane, relying on some
assumptions about the geometry of the space, but no ad-
ditional hardware, giving our work an advantage over ex-
isting companies and processes that rely on more sophisti-
cated sensors or sensor networks for person-tracking. The
assumptions we make are realistic in the retail store context,
and our quantitative results are compelling: significantly
more accurate than WiFi-based tracking solutions that are
already being deployed [9]. Our approach can be deployed
in most retail spaces without any hardware modifications to
existing security setups. We are excited by the real-world
applicability of our work.
1. Introduction
As we see much of retail moving online from brick-
and-mortar, opportunities to analyze consumer shopping
behavior are rapidly growing and becoming commercial-
ized. Recommendation engines, product positioning on
webpages, and sales funnels are all relentlessly A/B tested
and optimized to seduce the consumer into clicking Buy.
Much as the abundance of data makes optimization and it-
eration easy in the e-commerce space, brick-and-mortar re-
1
tail stores are looking to obtain and harness similar data to
shelve products, auction shelf space, and strategically place
discounts and information.
There are numerous current approaches being taken to
obtaining real-time consumer data in retail stores. Many
of these involve tracking peoples movement throughout the
store, with Bluetooth or wifi tracking, or with photogates.
All of these require significant hardware and deployment
costs, which has hindered their scalability.
Computer vision is a promising tool to address this busi-
ness need. As weve seen significant progress in the field as
of late (notably from convolutional neural networks), cam-
eras are an attractive option to track users in a store and
obtain data on customer retail behavior. Cameras are a par-
ticularly attractive method to obtain this data because most
retail stores already have the necessary hardware in place
for the purposes of security.
In this project, we track the 3D positions of multiple peo-
ple throughout a store in real-time, given a camera source
and certain assumptions about the store layout (namely, that
only one floor is visible to the camera, and that people’s
feet are visible in the image frames). We show a live 3D
bounding box whose coordinates are relative to the world
frame that tracks each person as they move, with error lev-
els within 55cm in outdoor settings and 30cm indoors, much
better than the existing WiFi- and Bluetooth-based tracking
solutions [9]. To do this, we tie together numerous concepts
from computer vision in a single approach - we design an
entirely new approach for this business need by integrat-
ing various approaches. To each subproblem, we tried var-
ious options, picked the best, and identified optimizations
for this particular case where applicable - for instance, we
found that affine approximation of tiled floor grids worked
far better than vanishing points given the configuration of
many security cameras with respect to the floor.
In this paper, we beginning by examining the problem
statement, and related work, both work that we read to gain
background knowledge, and also that we read to learn from
and build on their approaches to solving problems of dis-
tortion, calibration, and object detection. We then dive into
our approaches to each of these three problems, and how
they tie together into an end-to-end approach that could be
deployed into retail stores. Finally, we use a larger scale
dataset to obtain some quantitative metrics with which to
evaluate the success of our approach, and we leave space
for future work, such as integrating Extended Kalman Fil-
ters to enforce temporal consistency.
2. Problem Statement
Our objective is to accurately predict the 3-D position
of a person based on their location in a security camera.
If the person’s true location (we use the location of their
feet) can be given by (x∗, y∗, Z∗), and we estimate loca-
tion (x0 , y 0 , z 0 ) for them in 3-D space, then we are trying to
minimize:
p people in our image frame. Cutting edge research in object
d= (x ∗ −x0 )2 + (y∗ = y 0 )2 + (z ∗ −z 0 )2
for each person in each image. Since we use people’s
feet, we can constrain z 0 = 0, enabling us to use a single
view to predict position - this creates errors when people
jump, but this is not typical behavior.
We use two coordinate systems in this paper. The pri-
mary system is standard, where x spans the width of the
image, increasing to the right, and y spans the height, in-
creasing downwards, and the origin (0, 0) in the upper left
hand corner of the image. We also use a unique coordi-
nate system when undistorting images, where (0, 0) is at
the optical center of the image, and x and y increase right
and downwards respectively. This coordinate system is nec-
essary to model radial distortion parameters by expressing
points in polar coordinates from the optical center.
3. Related Work
There is a significant amount of work that has been done
in the space of retail analytics via camera, but this work has
been done almost exclusively by startups which protect their
methods as intellectual property. These include Prism Sky-
labs, Brickstream, and RetailNext. These are all dependent
on custom hardware, or sensors to augment the surveillance
feed.
On the technical front, we had to integrate work from
various frontiers in computer vision. Solving this business
problem required us to solve a number of technical prob-
lems. One was correction of barrel distortion - we leaned
heavily on Sing Bing Kangs work in Semiautomatic Meth-
ods for Recovering Radial Distortion Parameters from A
Single Image, in which he defined an algorithm by which
a user to draws snakes on a distorted image, with each ap-
proximately corresponding to a projected straight line in
space [8]. In his paper, he outlines how these snakes can
2
be used consistently with a model of radial image distor-
tion to solve for the radial distortion parameters and thus
undistort the image. For a given snake, the algorithm fits it
to the line of best fit, rotates this line to be horizontal, and
estimates constant distortion parameters that fit all of these
snakes/lines.
Another problem that was clear was recovering both in-
trinsic and extrinsic camera parameters from a single view.
To do this, we used the affine calibration approximation
taught in class, and covered in R. Hartley and A. Zissermans
textbook, Multiple View Geometry in Computer Vision [7].
In particular, we used calibration from a checkerboard with
the direct linear transformation algorithm, with tiled floors
as our checkboard. We had also tried single view metrology
with three sets of parallel lines, but this left us estimating
extrinsics.
Finally, we had the problem of object detection, to find
detection suggests that deep convolutional nets is the best
way to do this. Scalable Object Detection using Deep Neu-
ral Networks by Erhan et al. at Google demonstrated that
convolutional neural nets are very powerful for finding re-
gions of interest, while also having an effective recognition
path that categorizes the object of interest. The two steps
take a while though, and are not necessarily fast enough to
build real-time bounding boxes on video - Ren et al.s Faster
R-CNN: Towards Real-Time Object Detection with Region
Proposal Networks takes this work a step further by folding
the localization and recognition path into the same convolu-
tional neural networks, training the weights with the local-
ization and recognition cost functions alternately [10]. We
use this work directly as a component of our solution.
Finally, after proving our concept on some footage via
YouTube, we were able to find a much more expansive
dataset with ground truths from A new Dataset for Peo-
ple Tracking and Reidentification via the Video Surveil-
lance Online Repository [11]. This dataset was also pre-
calibrated and undistorted for us, via the methodology out-
lined in Cooperative Object Tracking with Multiple PTZ
Cameras, presented by Everts, Jones, and Sebe [3].
4. Technical Approach
4.1. Distortion Correction
While there are a number of approaches to fixing this
issue, such as un-distorting the image with projections of
area, or computing radial distortion coefficients, most meth-
ods depend on knowing the intrinsics of the camera pre-
distortion. Sing Bing Kangs work, however, suggests a
method to manually pick points on a line and accordingly
fit distortion parameters, as referred to above [8]. In par-
ticular, it tries to fit all points that should be collinear (as
denoted by a person) so that they are, while moving those
points as little as possible, and only moving all points radi- for a retail security video that has been corrected for barrel
ally by adjusting radius parameters. distortion), we can constrain ω to:
Radial distortion, of which barrel distortion is a type, can
be modeled by imposing a polar coordinate system on the  
ω1 0 ω2
image. From the center of the image, each pixel has an an-
ω=0 ω1 ω3  (1)
gle and distance from the optical center. Changes in this
ω2 ω3 ω4
distance create radial distortion. The distortion at a point,
which we call ∆r, is the change in distance from the op- This matrix has four unknowns, but it is only known up
tical center from the undistorted distance. We model this to scale. This means there are effectively three unknowns
distortion with the equation: if we set one of the unknown variables to 1 and scale the
rest accordingly. As a result, we can solve for the matrix ω
∞
X p by using our three vanishing points, and exploiting the fact
∆r = C2i+1 r2i+1 where r = x2 + y 2 that because they are mutual orthogonal, for each vi , vj with
i=1 i 6= j, vi> ωvj = 0. Thus we have three scalar equations in
three unknowns:
Then, the approach is to find values of C such that all
points we manually constrain to be collinear as collinear,
while also minimizing the distance we move them. This v1> ωv2 = 0 (2)
is a common radial distortion correction algorithm, and we v1> ωv3 =0 (3)
found that Photoshop actually provides a very effective im-
plementation which can be used to adjust entire videos, and v2> ωv3 =0 (4)
imposes the same distortion parameters on each frame. We
went this route to manually undistort video, rather than im- It is known that ω = (KK > )−1 , where K is the 3x3
plement the polar geometry and solver for the parameters matrix of camera intrinsics. So we can find K using the
from scratch. Cholesky decomposition of ω. We did this with our retail
video and found the camera  intrinsics.
 Unfortunately after
4.2. Calibration this process the extrinsics R|T are still unknown, so we 
could not recover the entire camera matrix P = K R|T .
Being able to map pixel coordinates to world coordinates
Setting the camera to be the origin in world coordinates
is a central component of understanding shoppers’ 3D lo-  
is not helpful, because even though it resolves the R|T
cations from images. Thus it is necessary to find a robust 
parameters (they would simply be I|0 , we still need to
camera calibration for any given video feed. We considered
know where the ground plane is in world coordinates to re-
two approaches to solving this problem for our unlabeled
solve the projective ambiguity
 ofmapping a pixel to a world
retail store data. For our 3DPeS data, calibration parame-
point. We tried estimating R|T by hand through trial and
ters were included with the dataset. The parameters were
error, but the results were very unreliable.
given in a third type of calibration formulation, which we
also explain below. Because retail cameras only need to be
calibrated once, it is practical to do these calibrations by
hand in a real-world context; thus we did not invest time in
automating the calibration process.

4.2.1 Single View Metrology

Given that we are constrained to one camera, in an environ-
ment with multiple sets of mutually orthogonal lines, it is
natural to first try an approach based on single-view metrol-
ogy to calibrate the camera. In our retail video, we labeled
three sets of mutually orthogonal lines by selecting two
Figure 1. Sets of mutually orthogonal lines used for single view
points on each of 6 lines. For each pair of parallel lines, we
metrology calibration in the retail camera frame. The line intersec-
found the corresponding vanishing point vi , i ∈ {1, 2, 3} by
tions give us three vanishing points, which we use in the equations
computing the intersection of the lines in image coordinates. above to solve for K.
We then considered the matrix ω, the projection of the abso-
lute conic Ω∞ into image coordinates. By assuming a cam-
era with square pixels and 0-skew (a reasonable assumption wher e

3
4.2.2 Affine Calibration 4.2.3 PTZ Calibration for Ground Plane Object Detec-
tion
Our problems with a calibration based on single view
metrology could be resolved by finding point correspon- Affine calibration worked well for our retail video data. But
dences and solving for the camera matrix directly. For our we used a different approach when working with the 3DPeS
retail video, we labeled 15 points by hand in the scene. We dataset, because that dataset already included parameters
place the origin at the bottom left corner of the bottom- for a different type of calibration. Due to the difficult po-
leftmost tile that is fully visible, we let each tile be 1 × sition of the cameras in the dataset, the publisher used a
1 in width and height in world coordinates, and we say that simpler type of calibration [3] designed specifically for Pan,
all tiles lie on the ground plane z = 0. We model the cam- Tilt, and Zoom (PTZ) cameras, which are commonly used
era matrix P as affine, which is a desirable approximation in surveillance. The methodology is fully described in the
even though the true camera matrix is projective, because source paper; we briefly summarize it here for convenience.
the lines in the scene are nearly parallel and solving for The calibration assumes that objects are only detected
fewer unknowns is preferred with only 15 point correspon- along a ground plane of Z = 0. Let U, V, H be the dis-
dences. That is, we let: placement of the camera coordinate system relative to the
world; ∆i = i − i0 , ∆j = j − j0 are the pixel positions
relative to the image’s optical center (i0 , j0 ); αxf and αyf are
 
a1,1 a1,2 a1,3 a1,4
P = a2,1 a2,2 a2,3 a2,4  (5) the horizontal and vertical scales between the image and im-
0 0 0 1 age plane; t is the tilt angle of the camera; and p0 = p + p0
is the pan angle after the camera is aligned with the world
Then, we use our n = 15 points to solve the following coordinate system. An object’s world coordinates X, Y are
over-constrained system of 2n equations: then given as:
 f 
Ax = b (6)
  αx ∆j  
X H U
= f R  αyf ∆i  + (9)
Where the world coordinates of point i are (xi , yi , zi ), Y αy ∆i sin t + cos t V
−1
the image coordinates are (ui , vi ), and:
where
 
x1 y1 z1 1 0 0 0 0 
cos p0 sin p0 cos t sin p0 sin t

 x2 y2 z2 1 0 0 0 0 R= (10)
 
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . sin p0 − cos p0 cos t − cos p0 sin t
 
xn yn zn 1 0 0 0 0
A=
  (7) 4.3. Person Detection
0 0 0 0 x1 y1 z1 1 
0 0 0 0 x2 y2 z2 1
 
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
0 0 0 0 x1 yn zn 1
 
u1
 v1 
 
 u2 
 
 v2 
b= (8)

. .
 
un 
vn
And x is a column vector of the eight unknowns in P ,
arranged in order with unknowns from the first row of P
before unknowns from the second. We solve the system
of equations in the standard way, by rearranging so that the
right hand side is 0 and the matrix A has additional columns
(and x additional rows) to maintain the constraints imposed
in the original equation by the values in b; then taking the
SVD of this augmented left-hand matrix, and using the last Figure 2. Our ConvNet can detect multiple pedestrians with high
column of the third output from SVD as the parameters of confidence, especially in clear environments such as this one. This
P. image is from Camera 3 in the 3DPeS dataset.

4
 Person detection in 2D images is a well-studied problem
[12] [1] [6] with several existing solutions, offering various
tradeoffs between speed, accuracy and simplicity. There
are essentially two parts to the problem: generating regions
of interest (RoIs) where a person might be, and classifying
those regions to determine if the region does indeed contain
a person.
Recently, approaches that utilize deep ConvNets have
been shown to perform exceptionally well at object detec-
tion, and person detection specifically. For this part of our
problem, we use a deep Convent object detection architec-
ture proposed by Girshick et. al. known as Faster R-CNN
[10]. Faster R-CNN is an improvement on Fast R-CNN [4],
which is itself an improvement of the original R-CNN ar-
chitecture [5]. Faster R-CNN works as a single, unified
ConvNet that uses shared convolutional layers to output fea-
ture maps that then get sent to a Region Proposal Network
(RPN) and a classifer head. The network is trained end-to-
end with back propagation and stochastic gradient descent,
with a multi-task loss function. The full details can be found
in [10].
We use a pretrained version of Faster R-CNN that we
modified to only output person detections (the original ver-
sion outputs detections of 20 types of objects).
Figure 3. The Faster R-CNN architecture. (Image from [?]ref3))
4.4. Putting it Together
Given a neural network that can detect 2D bounding
boxes around people and a calibrated camera, how do we
put the system together? Our approach takes as input a
5
stream of video data. For each frame in the stream, we run
our person detector over the image and get as output a set
of bounding boxes. In the general case, the mapping be-
tween the pixels defining the bounding box and the world
coordinate system is ambiguous, thanks to the ambiguity of
the 3D to 2D transformation of the camera. But because
we know these bounding boxes are people, we can make an
assumption that each person’s feet rest on the ground plane
(Z = 0). This is a reasonable assumption in nearly all re-
tail environments; the only test videos we encountered in
which this is not the case is when the camera watches over
an escalator or can see multiple floors at once.
Once we have bounding boxes for each person, we take
the bottom center pixel of each box (call its image coor-
dinates cx , cy ) and find the 3D coordinates associated with
that pixel, assuming it lies on the Z = 0 plane. In the affine
calibration case, this means finding the intersection of the
ray from the camera along which any point in 3D would
project to (cx , cy ), and the Z = 0 plane. By construction
this intersection must resolve to a unique point.
In the PTZ calibration case, the ground plane assumption
is built into the calibration model and so there is nothing
else that must be done besides converting cx , cy to offsets
from the optical center and plugging the results into equa-
tion (9).
In both cases, once we have obtained the world coordi-
nates of each person, we plot 3D voxels representing each
person in 3D graphing environment modeled after the room,
to visualize the positions of the people in 3D.
5. Experimental Setup and Results
We performed experiments on two data sources for this
project. The first was from a video clip of a retail security
system demo on YouTube [2]. This clip was useful for us
to understand real world surveillance video conditions. For
example, before looking for real surveillance video online
we had not considered the fact that we might encounter bar-
rel distortion: upon encountering that problem we realized
a practical implementation would need to correct for this
(which we now do). It was also helpful as a qualitative as-
sessment tool of our system’s performance. Unfortunately
YouTube videos don’t have ground truth labels, so we could
not evaluate our results quantitatively with this data source.
The lack of truth labels led us to search for other data
sources. We found that the 3DPeS Video Surveillance
dataset was quite useful in this regard. This dataset contains
outside surveillance footage, where there are often fewer
occlusions and people are farther away than the in-store en-
vironment, so it is not exactly representative of a retail sys-
tem. Nonetheless the dataset offered numerous advantages,
including ground-truth labels and pre-computed calibration
parameters. It gave us the chance to address the same fun-
damental task, predicting where people are in 3D given a
single 2D camera, in a cleaner environment.
5.1. Environment
We performed all tests on late-2013 Macbook Pro with
16GB RAM and a 2.3GHz processor. Due to lack of hard-
ware, we ran all code on the CPU, even though the Con-
vNet runs much faster on the GPU. This resulted in an aver-
age execution time of 3.19s per frame, a roughly 15X slow-
down in prediction speed compared to the results reported
by [10] on better hardware. The time spent outside of our
ConvNet’s forward pass was negligible. From these num-
bers it is clear that a real world deployment of our work
should have a dedicated GPU.
5.2. Results and Error Analysis: Retail Clip Exper-
iments
Figure 4. Example bounding box predictions for the retail video
data. These were generated without doing image distortion correc-
tion, although in our final implementation we were sure to correct
distortion first if it was present.
The goal of our experiments on the retail clip data was
to verify qualitatively that our approach was sound, and to
produce for each frame a 3D visualization of the scene ge-
ometry with people accurately tracked throughout. In our
affine calibration step, we hand-labeled 15 point correspon-
dences, shown here in figure 5. The root-mean-square error
(RMSE) of the calibration matrix we found on the data used
to create it was 32.7084 pixels, less than the width of one
tile almost everywhere in the frame. Ad-hoc measurements
of the final 3D voxels outputs showed they were generally
within two thirds of a tile to the true position of each person
when the bounding box was correct, or roughly 20cm. We
did not analyze this rigorously, as we performed most of our
quantitative analysis on the second dataset.
A common failure mode of our solution on the retail clip
data was occlusions. Occlusions cause problems in two
ways. The first is that they sometimes prevent our Con-
vNet from finding a person in the frame. Even if the Con-
vNet does find a bounding box, however, occlusions can
6
Figure 5. The point correspondences we labeled for the retail cam-
era video clip, vantage point 2. It is important that some of the
points are off the ground plane, or the calibration would be degen-
erate.
still cause problems if the feet of the person are not visible
in the image. This is because our pipeline assumes that the
bottom of the bounding box is where a person’s feet are, and
thus where the ground plane is. When that assumption is vi-
olated (e.g. because the bounding box ends at the person’s
waste), then the output is noticeably inaccurate.
Another typical failure occurred when people were in
rapid motion. In the video clip, there is a point at which
the two women sprint out of the store. For most of these
frames, the system loses track of them because no bound-
ing boxes are predicted. We hypothesize two reasons for
this failure. One is that the rapid and blurry stills of a hu-
man sprinting do not look very much like a typical person,
and these types of images are likely underrepresented in the
dataset on which our ConvNet was trained. The second is
that due the underlying architecture of the ConvNet, it has
a receptive field size of 228 pixels. This is suitable for most
purposes but when the people in this video clip are sprinting
with arms extended on both sides, their width in the image
easily exceeds 400 pixels. This makes it nearly impossi-
ble for the ConvNet to have a chance at detecting the entire
bounding box.
5.3. Results and Error Analysis: 3DPeS Dataset
Experiments
We also evaluate our pipeline on the 3D People Surveil-
lance Dataset provided by [5]. In general our people detec-
tion ConvNet works much more reliably on this dataset be-
cause of the reduced occlusions, better lighting and higher
definition of the images. In our run of the pipeline on a
live stream of 17 frames from the same camera, we detect
51 of 53 total person bounding boxes when the person is
more than halfway in the scene (i.e. not majority cut off
by an edge of the image). Across all frames we tested, the
root-mean-square error of our position predictions in world
coordinates was 554 millimeters. This is about 2x higher

Figure 6. Bounding boxes found for a sample frame and the cor-
responding 3D scene model that we generated. In the 3D model,
the origin is marked by the blue plus sign below the left voxel. It
corresponds to the tile in the frame found right below the ”i” in
”DixonSecurity.com”. We can see here that the model is rather ac-
curate considering the low number of calibration points, the origi-
nal fisheye distortion, and partial occlusions in the scene.
than our ad-hoc estimate of our performance on the retail
dataset, due mostly to the vastly greater field-of-view of the
camera we used in this dataset. (This is an outdoor cam-
era which overlooks more than 200 square meters of space,
much more than can be seen by the indoor camera. So be-
ing off the same number of pixels will translate to a much
larger increase in RMSE.)
We can glean several interesting insights from the pre-
diction error graph. For example, we see that in general
7
Figure 7. Our prediction errors in millimeters in the world frame
for each person in each image we evaluated, shown collectively.
Each point is the difference between the predicted x, y of a person
in world coordinates and the true x, y of the person.
Figure 8. The same graph as before but with the outlier (Y offset
> 3000) removed. Note the different scales of the X and Y axes.
there is more error along the X axis than the Y axis, but
most of the Y axis error that does occur is in the same di-
rection: consistently slightly positive. This is because we
use the bottom of the bounding box as the intersection point
of the person with the ground, when in reality the ground
truth label for the person’s position in 3D considers the cen-
ter of the person overall. (Imagine a circle on the ground
around the person’s feet. The centerpoint of this circle is
the ground truth x, y label. It will consistently be slightly
offset from a point at the edge of one foot, which is what
we get with the bounding box method.)
The X axis error is also because of our bounding-box-to-
intersection-point methodology. As people walk they swing
their arms and stride their legs. The bounding box produced
by our ConvNet will generally capture all of these extremi-
ties, so any time they are not displaced from the person’s
center symmetrically, the bottom center of the bounding References
box will not be an accurate representation of where the feet
[1] N. Dalal, B. Triggs, and C. Schmid. Human detection us-
intersect. Finally, the bounding boxes are in general imper-
ing oriented histograms of flow and appearance. Computer
fect, and random noise is surely a factor as well. Vision ECCV, 2006.
[2] T. Dixon. Fight caught on cctv security camera. https:
6. Conclusions and Future Work //www.youtube.com/watch?v=Kla8W8IIAtk.
[3] I. Everts, G. Jones, and N. Sebe. Cooperative object tracking
We’ve successfully developed and outlined an end-to- with multiple ptz cameras. Image Analysis and Processing,
end approach to turning raw security footage into a 3D 2007.
model of customer movement throughout a retail store. This [4] R. Girshick. Fast r-cnn. IEEE International Conference on
involves a one-time calibration of distortion and camera pa- Computer Vision (ICCV), 2015.
rameters, and then the usage of Faster-RCNN to find peo- [5] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich fea-
ple in the frame. The aforementioned parameters are used ture hierarchies for accurate object detection and semantic
segmentation. CoRR, 2013.
to related their location on frame to their real location in-
[6] I. Haritaoglu, D. Harwood, and L. S. Davis. W4s: A real-
side the store. While scaling difficulties arise in the once-
time system for detecting and tracking people in 2 1/2d.
per-deployment cost of manually determining the camera’s Computer Vision — ECCV, 1998.
distortion, intrinsic, and extrinsic parameters, this method [7] R. Hartley and A. Zisserman. Multiple View Geometry in
seems to be accurate enough to effectively provide data to Computer Vision. Cambridge University Press, 2003.
retail environments. [8] S. B. Kang. Semiautomatic methods for recovering radial
With typical error in the range of 20cm or so in real space distortion parameters from a single image. Technical Report
in indoor settings, this could very plausibly be used to track CRL, 1997.
the location of shoppers in a retail space - information such [9] F. Manzella and I. T. Teije. The truth about in-store analytics:
as aisle choice, for instance, is easily determined at this level Examining wi-fi, bluetooth, and video in retail. 2014.
of granularity. Back projecting the person’s location into 3- [10] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: To-
D space is very important for these businesses, and it’s ex- wards real-time object detection with region proposal net-
citing that a simple and practical assumption about position works. arXiv, 2015.
(that feet are on the ground) is so effective. [11] V. S. O. Repository. 3dpes: A new dataset for people
tracking and reidentification. http://imagelab.ing.
There is, however, a major obstacle to usage of this ap- unimore.it/visor/3dpes.asp.
proach in practice. This is occlusion of the feet - it’s not [12] Z. Zivkovic and B. Krose. Part based people detection using
uncommon for shelves or other objects to block the feet of 2d range data and images. IEEE/RSJ International Confer-
subjects, making it impossible for our existing algorithm ence on Intelligent Robots and Systems, 2007.
to guess their position. We had to carefully select datasets
because of this limitation, but real retail stores will not be
able to do this, instead having to work with whatever their
camera sees. One promising way to handle this would be to
use temporal consistency (i.e. relate similar bounding boxes
across timeframes) to estimate foot position now based on
foot position in previous frames. This could be done with
Extended Kalman Filters, which allow us to integrate a
physical model of the world alongside noisy measurement
data (the person detector) to produce an output that is over-
all more robust. We can also use a Faster-RCNN architec-
ture ConvNet trained specifically to look for feet, and when
feet are not detected in a bounding box (due to occlusions)
we can instead use the position of the face and extrapolate
downward based on assumptions about human proportions.
The problem of consistent offsets in one direction caused
by the edge-of-feet point from the bounding box v.s. the
between-the-feet ground truth point can be resolved with a
simple addition of a mean error vector x∆ , y∆ that can be
learned from training data. Overall, we think this is a com-
pelling first step towards real-time 3D person detection in
retail and that the remaining obstacles are surmountable.

8
 AUGMENTED REALITY IN LIVE VIDEO STREAMS USING POST-ITS

FINAL REPORT
JUNE 6TH, 2016

BOJIONG NI, JORIS VAN MENS

bojiong@stanford.edu, jorisvm@stanford.edu

Abstract Another frequently used method for object de-

In this paper, we set the goal to use generic Post-it
notes as fiducial markers for real-time augmented re-
ality applications. We compare detection using SIFT,
Template Matching and a custom designed “Color-
Shape” method, and find that the latter approach de-
livers high quality results, allowing for robust and ac-
curate projective pose estimation at low processing
time.
1. Introduction
Augmented reality is an exciting experience where
the virtual world meets the real. In a typical setup,
virtual imagery is projected into live, real world video
streams. Popular implementations of augmented re-
ality often use fiducial markers that are specifically
designed to be easily detected in video streams. In
our paper, we aim to create a robust augmented real-
ity experience that works with a colored Post-it note
as fiducial marker, and which can run sufficiently fast
on a single thread on a general processor.
With this approach, we hope to bring the aug-
mented reality experience to larger audiences. A spe-
cific example would be the classroom, where teachers
presenting to their students on a generic laptop could
enrich their teaching with virtual experiences using
only a Post-it note and our software application.
2. Review of previous work
On the specific problem of locating Post-it notes
in real time, we were not able to find any literature
(nor on Post-it note detection in general). There has
however been extensive research focused on designing
and finding fiducial markers that are easy to detect
in video frames, using a variety of methods[9, 16].
Template matching is one of such commonly used
methods for detecting markers in augmented reality.
In Template Matching approaches, the marker (or
template) tends to be carefully designed to enhance 2.1. Contribution of our work. We provide a
recognition and camera estimation[12]. comparison of several methods to solve the problem
1
tection and tracking is SIFT. Previous work shows
that it delivers strong results for a wide range of de-
tection goals, yet it is a computationally expensive
approach[2].
Literature on color detection has often focused on
the problem of skin color detection[13]. Detection in
Red Green Blue (RGB) space is common, as is de-
tection in Hue Saturation Value (HSV) space. It is
noted that a potential downside of the latter space is
that it can cause hue discontinuities. In addition, the
value (brightness) dimension does not relate well to
human perception of brightness. An alternative Tint
Saturation Lightness space is similar in nature. A
YCrCb space, where Y relates to brightness and Cr
and Cb relate to the color hue, is also found to ex-
hibit favorable color detection properties for various
purposes[3].
A typical, fast approach to filtering color is done by
defining an explicit cuboid bounding box in the cho-
sen three dimensional color space[5]. This method
can be augmented by normalizing the image for
brightness[6]. The boundaries can be found empir-
ically by taking samples of the object in different
scenes. Elliptical boundaries have also been tested[7],
as have various probability-based models[17].
For edge detection, Canny edge detection is often
regarded as one of the most accurate methods, with
decent performance, although various other edge de-
tection algorithms, such as Laplacian of Gaussian,
may be less computationally expensive[1, 10].
For line detection, typical methods are the Radon
and Hough transform. The two methods are similar
in nature, where the Hough transform can be consid-
ered a discretized version of the more general Radon
transform[19]. Various speed optimized Hough trans-
form have also been proposed[18, 14, 4].
Augmented Reality in Live Video Streams using Post-its

of robust, accurate and fast Post-it detection for aug-
mented reality, which had not yet been done in pre-
vious literature.
In addition, we show that it is indeed possible
to use generic Post-it notes as fiducial markers for
augmented reality, robustly obtaining full pose es-
timation at high accuracy and high speed. We do
so using a combinatory approach of various low-level
algorithms (color filtering, noise reduction, edge de-
tection, line detection and several logical elements)
specifically tailored to the use case, providing results
superior to general methods such as SIFT and Tem-
plate Matching.
3. Technical approach
Using Post-its for augmented reality imposes two
important constraints. First, the Post-it is an ob-
ject with very few distinguishing features, implying
some general feature detection methods might not
work well. Second, the speed requirement implies
a further restriction on the methods being available
to use, and creates a focus on minimizing execution
speed. We aim to consistently render 30 frames per
second, which implies the full algorithm must take
less than 33 milliseconds to execute on our hardware.
We will test a manual ”Color-Shape” approach, a
SIFT approach and a Template Matching approach
and compare their applicability to solving our prob-
lem.1
(5) For bright colored variants, the saturation
level is high compared to most surrounding
scenes
(6) For various color variants (e.g. pink), the hue
is uncommon in most scenes
The Color-Shape method aims to make use of all
of these properties to achieve an optimal solution.
While creating this method and choosing param-
eters, we aim to optimize several factors. First, we
aim for a high detection rate, which we define as the
percentage of frames that return valid vertices (as op-
posed to frames that return no vertices). Second, we
aim for high accuracy, defined as the percentage of
detected vertices that are accurate. Third, we aim
for speed, as measured in milliseconds of execution,
while also keeping the standard deviation in mind.
High standard deviation can result in video stutter
even when mean speed is low, as a single frame with
high processing time will halt the video until process-
ing is completed.
We use a combination of color masking, binary
noise reduction, edge detection, line detection and
various logic steps to estimate the Post-it’s location
and calculate the transformation matrix. We rely on
Python Opencv3.0.0 implementations of mentioned
algorithms, and NumPy for other image-wide calcu-
lations, given both run optimized machine code to
provide optimal performance. Figure 1 shows the
main elements of the Color-Shape pipeline.

3.1. Experimental setup. The hardware we use for

recording and processing is a MacBook Pro 2013.
The sticky notes we use are the original Post-it brand,
in various color variants. For testing accuracy and
speed we wave a Post-it in circular motion at 2 feet
distance from the webcam and capture 100 pose es-
timations. We test our methods in rooms well lit by
either daylight or artificial light.

3.2. Color-Shape Approach. In the color-shape

approach, we make use of a number of properties of
the colored Post-it under projective transformation.
These assumptions are as follows:
(1) It has 4 exact edges and vertices
(2) In the absence of radial distortion, the edges
are straight
(3) Due to its square shape and small size com-
pared to the camera distance for expected
scenes, the opposing edges will be of similar
size (near-affine transformation)
(4) It is a solid, non-porous object Figure 1. Color-Shape pipeline
1
Code can be found at github.com/Bojiong/cs231a
2 of 10
Augmented Reality in Live Video Streams using Post-its

3.2.1. HSV filter. We create a mask on the image by

filtering for specific hue, saturation and value ranges hmin = min(Hsamples ) − stdev(Hsamples )
as such:
 hmax = max(Hsamples ) + stdev(Hsamples )
 [hmin < H < hmax ]



1 ∀ [s 3.2.2. Erode-Dilate. We use erosion and subse-
min < S < smax ] quently dilation to remove small patches (noise and
HSV mask =


 [v min < V < vmax ] false positives) from our mask while keeping larger
0

otherwise patches intact. For erosion, we move a pixel kernel K
(10x10 matrix of ones) over the mask M (which holds
To find relevant hmin , hmax , smin , smax , vmin and vmax all non-zero pixels). We then keep only those pixels p
values, we sampled Post-it camera captures under for which all surrounding pixels covered by the kernel
various lighting conditions (figure 2), and captured at that pixel (K ) are also part of the mask:
p
their mean HSV values (figure 3).
M K = {p|Kp ⊆ M }
This returns pixels surrounded by patches of ones,
while removing any smaller patches for which at least
one pixel covered by the kernel was zero. For dilation
the process is similar, but reverse: any mask pixel for
which at least one of the surrounding pixels within
the kernel is one, we return one. This effectively
”grows” single pixels into patches of 10x10, undoing
the ”shrinking” caused by the erosion. The final ef-
Figure 2. Post-it color samples un- fect can be seen in figure 4.
der various lighting conditions

Figure 4. Effect of erosion and

subsequent dilation (right) on noisy
mask (left)

3.2.3. Canny edge detector. We apply the Canny

edge detection algorithm to find edges in the mask.
The edge detector first applies a Gaussian filter to
the mask:
1 (i − (k + 1))2 + (j − (k + 1))2
Mij = e(− )
2πσ 2 2σ 2
Figure 3. Mean hue-saturation The size of the filter is given by (2k + 1) in both x
(left) and hue-value (right) distri- and y direction. Subsequently, it finds the gradients
bution of samples. Green markers throughout the image as such:
correspond to pale yellow Post-its, q
blue to bright yellow and pink to S = Sx 2 + Sy 2
pink.

Θ = atan2 Sy , Sx
We take minimum and maximum values for every Where S represents the size of the gradient and
dimension as below (example for hue H), and apply theta the angle. As a next step, non-maximum sup-
the mask. pression is applied to remove multiple signals for the
3 of 10
Augmented Reality in Live Video Streams using Post-its

same line (”thin the edges”). After this, a threshold- transform. We subsequently erase all pixels on the
ing mechanism is applied to find the most likely true edge image corresponding to this line with a 3-pixel
edges and remove ones more likely caused by noise. boundary radius. On the new edge image, we re-
Given the relatively simple mask provided by the apply the Hough transform. We iterate through this
previous steps, the edge detector provides good re- method 4 times. The result on the edge image after
sults as expected. Experimentation with the thresh- 2 iterations can be seen in figure 6. As an alternative
old values within reasonable bounds caused no signif- to the iterative approach, we have also tested an ap-
icant difference in the results. proach that aims to filter out multiple line detection
per Post-it edge by filtering lines with similar rho and
3.2.4. Iterative Hough transform & line erase. On theta values. While this gave decent results (see ex-
the edge detected output, we will apply a Hough periments section), we found this to be less robust
transform to find lines. The Hough transform trans- than our iterative erase-line approach.
lates Euclidean x-y coordinates into curves in the 3.2.5. Intersection finder. To find intersections, we
polar space, representing lines with different dis- use the lines’ polar coordinates and solve the follow-
tances from the origin (r) and angles (θ). Cells in ing equation:
Hough space with votes above a certain threshold
will be accepted as lines and converted back into
    
cos(θ1 ) sin(θ1 ) x ρ
x-y coordinate space, where points at extreme x-y = 1
cos(θ2 ) sin(θ2 ) y ρ2
coordinate values on the specified line will be used as
endpoint estimates. Figure 5 shows the initial output. 3.3. Nearest group filter. Given the detected in-
tersections, we find all intersections that lie within
(or just outside of) the image. Of those, we filter for
the 4 intersections in the closest group by using only
the 4 intersections with minimum distance to the full
group’s geometric center (filtering out intersections
at largest distance from the rest).
3.3.1. Validity decision. Given the resulting inter-
sections, we perform several validations to verify
whether or not the intersections correspond to a valid
Post-it transformation. Specifically:
(1) There must be exactly 4 vertices
(2) There cannot be 3 vertices on a single line
Figure 5. All Hough lines found (3) Opposing edges must be of similar length
(near-affine transformation)
For the third rule, we apply a minimum to maximum
line length range as such:

Figure 6. Initial edge image (left)
and edge image after 2 Hough and
Erase Line iterations (right)
One problem with the Hough transform is that it
will fit multiple lines on a single Post-it edge. As a
robust method for finding the most promising lines
corresponding to the unique Post-it edges, while re-
moving duplicate Hough matches for a single edge,
we use only the highest-voted line from the Hough
4 of
µ = l1 + l2 /2
lmin = µ ∗ (1 − δ)
lmax = µ ∗ (1 + δ)
3.3.2. Find perspective. Subsequently, we will find
the projection matrix M that allows us to project
pixels of the overlay image into the position of the
Post-it: Ptrans = M ∗ Poverlay . Projective matrix M
has 8 degrees of freedom, and every matching pair of
points gives us 2 equations, so we can find the matrix
using 4 matching points. For the Poverlay coordinates
(x & y), we use the 4 vertices of the square image we
want to overlay. For the Ptrans coordinates (x & y),
we use the 4 vertices of the Post-it in the video frame.
We can now solve the linear system (using Direct Lin-
ear Transformation):
10
Augmented Reality in Live Video Streams using Post-its

3.4.1. Find matching key points. As SIFT is fre-

 0 quently used for object detection, it is worth trying
t2 x02 t3 x03 t4 x04

t 1 x1
 t1 y10 to see how SIFT applies to our use case. In order to
t2 y20 t3 y30 t4 y40  =
use SIFT, we first need a reference image and extract
t1 t2 t3 t4
   (1) local features from it. Then we extract the SIFT fea-
h11 h12 h13 x1 x2 x3 x4 tures from each video frame. SIFT features are scale,
h21 h22 h23   y1 y2 y3 y4  rotation and affine invariant[8], thus we do not need
h31 h32 h33 1 1 1 1 to explicitly account for the transformation between
reference camera frame and video camera frame.
Once we have the features in both reference image
Where M is only defined up to scale, allowing to and the video frame, we match the features in the
normalize to h33 = 1. two images using a K-Nearest Neighbors approach.
Here we will find the closest two matching points for
3.3.3. Output. Typical frames are shown in figure 7. each feature in reference. Each of the two matching
The top frame shows a noisy environment (with a points will have a distance value indicating how close
similar-colored object in background) and the bot- it is to the corresponding feature in the reference im-
tom frame shows occlusion of one corner. The left age. To eliminate false positives, we will only accept
images show all Hough lines (thin) with the 4 chosen a match if the closest distance is 70% or less than that
Hough lines plotted thick. The quadrilateral corners of the second closest point. The rationale behind this
are identified by white circles, and a projected ani- is that for a true match, the second closest matching
mation (globe) is shown on the original input on the point’s distance will be much larger than the closest
right.2 (true) one’s. When there is a false positive, various
closely matching points may have similar distances.
We used the Python Opencv3.0.0 2D feature li-
brary to extract features from images and find the
closest neighbors. We have experimented with both
colored images and gray scale images for detection.
For the reference picture, we tested with plain Post-
its, Post-its with patterns drawn on them, and a rec-
tangle directly drawn from an array. We will compare
the results in the experiments section.

3.4.2. Find transformation matrix. After the key

points are matched, we estimated the transformation
of the Post-it in video from the original feature lo-
cation and the matched feature location. Since we
know that SIFT is affine invariant, we can assume
the matched points are the source and destination of
an affine transformation. This transformation matrix
Figure 7. Debug images with ap- will later be used for transforming the animation to
plied mask, Hough lines and de- be overlay on the Post-it. Let (x, y) be the key point
tected vertices (left) and origi- in the original template, (x0 , y 0 ) be the matching key
nal captures with projected overlay point in the video frame. We have the following rela-
(right) tion:  0   
tx h11 h12 h13 x
ty 0  = h21 h22 h23  y 
3.4. SIFT approach. Scale-invariant feature trans-
form (or SIFT)[8] uses techniques of Difference of t 0 0 1 1
Gaussians, scale-space pyramid and orientation as- The transformation matrix has 6 unknowns (de-
signments to ensure the features are scale and ro- fined up to scale) and each matched key points will
tation invariant. It also resamples the local image give one two equation, we need at least 3 matching
orientation planes in order to achieve full affine in- points pairs to solve the problem. In case the system
variance. is over ranked (more than 3 pairs of independent key
2A video example can be found at
youtu.be/f16gHGPc3wE
5 of 10
Augmented Reality in Live Video Streams using Post-its

points are found) or rank deficient, we can use least performed Canny edge detection to only work on the
square fitting to find the estimate. edges of the image (see figure 8).
We use the Opencv Python library for Template
3.5. Template Matching approach. In Template Matching. In the Opencv library, the sum of differ-
Matching, a small image is used as a template to see ences can be calculated in different ways, and we will
if a matching version can be detected in the larger compare results.
image. This approach takes the template as the con-
volution mask and perform a convolution with the 3.5.2. Find the rotation and scaling matrix. In this
search image, sliding a window of the same size of case, we can only find the rotation and scaling of
the template over the search image. Then we com- the matched window instead of the full perspective
pare the pixel intensity difference between the search transformation. As we iterate through different scale
image in the window and the corresponding pixel in s and rotation angle θ, the corresponding transfor-
template. We sum the difference over the window. mation matrix is:
The window with lowest difference sum gives the best P 0 = sR(θ)P + p
match.
where p is the location of the left upper corner of
3.5.1. Find the best matching window. Let us give a matched window box.
formal definition[15]. Suppose coordinates (xs , ys )
at the search image has intensity Is (xs , ys ) and 3.6. Overlay. Once we have the applicable homog-
the coordinates (xt , yt ) at the template has in- raphy from either of above methods, we apply it to
tensity It (xt , yt ). Define the absolute difference consecutive frames of an animation to overlay the an-
in the pixel intensities as Dif f (xs , ys , xt , yt ) = imation into the video (as described in the Color-
|Is (xs , ys )It (xt , yt )|. Shape section). We remove pixels with zero alpha
Define Sum of absolute differences measure as values to allow for basic transparency (allowing us to
overlay e.g. a spherical globe instead of only rectan-
TX Tcol
row X
gular pictures).
SAD(x, y) = Dif f (x + i, y + j, i, j)
i=0 j=0
4. Experiments and Results
We loop over the entire search image and calculate
4.1. Experiments using Color-Shape.
the corresponding SAD(x, y). The pixel with lowest
SAD is the best match. 4.1.1. Hough transform parameters. To find the best
Note that Template Matching is not scale invari- rho, theta and threshold values for the Hough trans-
ant. I.e. we do not know how big the object (Post-it) form, we have done various experiments as shown in
is in the search image (video frame). We will resize table 1.
and rotate the original template to create a series of
templates with different sizes and rotations. Then we Threshold 15 30 15
perform a search of all those templates in the search Rho 1 1 2
frame and return the best match. Theta π/45 π/45 π/45
Speed (µ, ms) 4 1.8 3.6
Speed (σ, ms) 3.2 1.2 2.4
Accuracy High High Mid
Detection rate High Low High
Threshold 15 15 15
Rho 5 1 1
Theta π/45 π/90 π/22.5
Speed (µ, ms) 3.6 6.1 3.7
Speed (σ, ms) 2.8 5.3 2.5
Accuracy Mid High Mid
Detection rate Mid High High
Table 1. Hough transform param-
Figure 8. Edge image for Tem- eter results
plate Matching

In order to reduce noise and improve speed, we Per these results, we have chosen 1, π/45 and 15
transform the image from RGB to gray scale and as our respective optimal parameters.
6 of 10
Augmented Reality in Live Video Streams using Post-its

4.1.2. Speed of Color-Shape. An analysis of execu- 4.2.2. Gray scale reference and colored video frame.
tion time for the various elements of the Color-Shape In this experiment, we use a gray scale plain Post-it
method can be seen in figure 9. Total execution time as reference. Different from the above, the key points
is well within our target range (below 33 ms) at 12 in the reference image are greatly reduced and the
ms mean. matching is more accurate. However, the detection
We find that the iterative Hough transform takes is always at the corner of the Post-it and the results
most computation time. This triggered us to also de- still occasionally contain false positives. The result
sign a non-iterative method for finding the four most can be seen in Figure 11.
promising lines using only one Hough transform. This
alternative method filters lines with similar sigma and
rho (as described in the Hough transform section of
the technical approach). While the method saved 3
ms of mean execution time, we found it to be less
robust than the iterative approach (0.91 accuracy vs.
0.98 baseline). Given total execution time is already
low, we decide to trade execution time for higher ro-
(a) Match at corner (b) False positive
bustness.

(c) Key point match-

ing

Figure 11. Colored image with

gray Post-it
Figure 9. Color-Shape element-
wise execution time
4.2.3. Binary rectangle and colored video frame. In
this experiment, we use a rectangle drawn from an ar-
4.2. Experiments using SIFT. ray as reference image, and use colored video frames
for detection.
4.2.1. Colored reference with colored video frame. In
this approach, we take a picture of the Post-it, de-
tect the features using SIFT and try to match the
key points in video frame.

(a) match at center (b) match at corner

(a) False positive (b) Key points match

Figure 10. Colored image with col- (c) false positive (d) key point matching
ored reference
Figure 12. colored image with rectangle
We can see from figure 10 that the colored Post-it
picture contains too many features and key points, The value within the rectangle is 255 and 0 outside.
making the matching process slow and noisy. The This time, the features and key points from the refer-
Post-it is detected inaccurately. ence image are far fewer. From the results in Figure
7 of 10
Augmented Reality in Live Video Streams using Post-its

12, we can see that it can successfully find matches the method described above where we only accept a
between the reference rectangle and the video frame. match if the closest distance is within 70% of the sec-
However, we still see some false positives. ond closest distance. We can see from figure 14 that
when the reference is simple, e.g. a binary rectan-
4.2.4. Post-it with pattern. In this approach, instead gle, the approach without distance ratio gives more
of using a plain Post-it, we use a Post-it with pattern matches within the Post-it. We can see from the im-
drawn. For simplicity, we will use gray scale for both age that there are two key points within the Post-it
reference and video frames. Figure 13 shows a re- (and the binary rectangle). Those two key points
sult. Here the matching key points are within in the do not have much difference in terms of color inten-
Post-it instead of at the corners as in previous cases. sity or texture. The gradients around those two key
The lack of features for plain Post-its gives little in- points are very similar. Thus they might have similar
teresting matching key points within the center of the distance to the same key point within the reference.
Post-it. A patterned Post-it, however, has richer con- This is an example where distance ratio can eliminate
tent and gradient variation within the area of Post-it, false negatives.
making it easier to match local feature inside. However, when the reference image has richer fea-
tures, i.e. with the patterned Post-it, the first ap-
proach generates many more false positive than the
second approach.

4.2.6. Speed of SIFT. Our experiments show that our

SIFT method takes on average 59 ms per frame to de-
tect the location of the Post-It, which is much higher
than our target maximum of 33 ms. This result is
(a) False positive (b) Key point match expected as SIFT is a computational heavy feature
detector.
Figure 13. Post-it with pattern

4.2.7. Accuracy of SIFT. From the above analysis,

we found that SIFT works best for patterned Post-its.
Both the Plain Post-it and digital rectangle template
have too few interesting feature key points, making
the perspective transformation between key points
rank deficient. For SIFT approach, a patterned Post-
It gives best results for our use case. We manually
reviewed 100 video frames to measure how accurate
the algorithm is in finding matches. The result shows
(a) No distance ratio (b) No distance ratio that 94 out of the 100 frames had the Post-it cor-
rectly detected. For the other 6, it either identified
no match or identified a point outside of the Post-it
area (false positive).

4.3. Experiments using Template Matching.

For the Template Matching method, we rotate and
resize the template and do a matching process for
(c) With distance ratio (d) With distance ratio each resized and rotated template. This causes la-
tency to increase as we try finer granularity of size
Figure 14. Distance Ratio Effect and angle rotation. Table 2 shows how processing
time varies depending on the parameters. For pro-
4.2.5. KNN distance ratio effect. We experiment cessing times greater than 70 ms (i.e. surpassing our
with two different schemes for KNN matching. The maximum goal of 33 ms), we experience significant
first is to find the nearest neighbor for key points delay in video rendering.
and directly take it as a matching. The second is
8 of 10
Augmented Reality in Live Video Streams using Post-its

#rotation #sizes processing time (ms)

1 1 6 We reviewed 100 video frames and found 92 to have
2 2 18 the correct bounding box for the best matching re-
2 3 23 gion. For the rest, the boxes also includes a part
3 5 70 (< 50%) of the Post-it, but the majority of the Post-
5 10 195 it fell outside of the box.
Table 2. Template Matching Speeds
4.3.2. Post-It with Pattern. We reviewed 100 video
frames and found 94 to have the correct bounding
box for the best matching region. Some of the false
Similar to the SIFT case, we tested with both plain matches still have part (< 50%) of the Post-it region
Post-Its and Post-Its with patterns on it. In order in the box, while some completely missed the Post-it.
to maintain low latency, we restricted the number of Figure 16 shows one frame where we use Template
sizes to 2 and number of rotations to 3. It turns out Matching with a patterned Post-it.
Template Matching works well on both the patterned
and plain Post-its with such rotation and scale iter-
ation. Here, if the box includes > 50% of the region
of the Post-it, we categorize it as a correct match.
In both figure 15 and figure 16, the three images in
the first row (from left to right) are the original tem-
plate, the template after Canny edge detection and
the actual resized/rotated template giving the best
match.
The three images in the second row (from left to
right) are the video frame with best matched region
(the rectangle enclosed region), the Canny edge de-
tection of the video frame and the visualization of the
convolution between template and the best matched Figure 16. Post-it with pattern
region.
One drawback of the Template Matching approach
is that we cannot estimate the full perspective trans-
4.4. Comparison of methods. Comparing the
formation. We can only estimate similarity: we get
most important metrics for the 3 approaches, we find
scaling and rotation by iterating through different
the manual Color-Shape approach performs much
scales and rotations of the template, and translation
better than both SIFT and Template Matching. Ta-
from the box center location of the best matched re-
ble 3 shows the high-level comparative overview.
gion.
Template
4.3.1. Plain Post-It. Figure 15 shows one frame SIFT Color-Shape
Matching
where we use Template Matching for plain Post-it Time (ms) 59 23 12
detection. 0.94
(pattern)
Accuracy 0.94 0.98
0.92
(plain)
2 scale
Notes Pattern Bright color
3 rotation
Table 3. Comparison of Methods

In addition to processing time and detection accu-

racy, the Color-Shape method provides a more accu-
rate perspective transformation matrix that we use
to transform the animation to overlay on the Post-
Figure 15. Plain Post-it it. For SIFT, the affine transformation can be rank
9 of 10
Augmented Reality in Live Video Streams using Post-its
deficient when insufficient key points are matched.
For Template Matching, we can estimate a similar-
ity matrix but cannot estimate the full perspective
transformation.
SIFT and Template Matching are designed for gen-
eral, rich pattern detection, while the Color-Shape
method we use is designed specifically for our Post-it
use case. SIFT and Template Matching can be used
for a much broader range of use cases, whereas the
Color-Shape method is limited in scope to Post-it de-
tection (and highly similar use cases).
5. Conclusions
We find that typical methods such as SIFT and
Template Matching do not sufficiently meet our goals,
as they do not provide accurate pose finding for the
plain-faced Post-it, and require too much computa-
tion time to operate.
We find the manually designed Color-Shape ap-
proach, exploiting the saturated color and square
shape properties of the Post-it, to work well and meet
our goals. The full Color-Shape method takes an av-
erage of 12 milliseconds to execute (with 4.5 millisec-
onds standard deviation) on a 3-year old MacBook
Pro laptop, indicating the maximum 33 milliseconds
goal should be achievable on a wide range of laptops.
We achieve high accuracy projective pose estimation
with a reasonable robustness to noise and lighting
variation.
6. Future Work
A number of improvements could be considered for
future work:
• Interest region: the Color-Shape approach
could be made significantly faster by search-
ing only in an interest region derived from
the previous location of the Post-it
• Multiple Post-its: the Color-Shape approach
could be generalized to find multiple Post-its
in one image
• Dynamic color filter: the HSV color filter
could be made more accurate by dynamically
adapting parameters to the scene
• Wearables support: the source code could be
ported to relevant augmented reality hard-
ware such as smartglasses
References
[1] Canny, J. A Computational Approach to Edge Detection
IEEE Trans. on Pattern Analysis and Machine Intelligence,
8(6), pp. 679-698, 1986.
10 of 10
[2] Daniel Wagner, Gerhard Reitmayr, Alessandro Mulloni,
Tom Drummond, Dieter Schmalstieg Real-time detection
and tracking for augmented reality on mobile phones IEEE
Transactions on Visualization and Computer Graphics
(Volume:16 , Issue: 3), 2009
[3] Charles A. Poynton Frequently Asked
Questions about Colour Available at
https://engineering.purdue.edu/ bouman/info/Color-
FAQ.pdf
[4] Fernandes, Leandro AF, and Manuel M. Oliveira. Real-
time line detection through an improved Hough transform
voting scheme Pattern Recognition 41.1, 2008
[5] Fleck, Margaret M., David A. Forsyth, and Chris Bregler.
Finding naked people Computer VisionECCV’96. Springer
Berlin Heidelberg, 1996
[6] Fleyeh, Hasan. Color detection and segmentation for road
and traffic signs Cybernetics and Intelligent Systems, 2004
IEEE Conference on. Vol. 2. IEEE, 2004
[7] Lee, Jae Y., and Suk I. Yoo An elliptical boundary model
for skin color detection Proc. of the 2002 International Con-
ference on Imaging Science, Systems, and Technology, 2002
[8] Lowe, David G. Object recognition from local scale-
invariant features Proceedings of the International
Conference on Computer Vision. pp. 11501157.
doi:10.1109/ICCV.1999.790410, 1999
[9] M. Fiala Designing Highly Reliable Fiducial Markers IEEE
Transactions on Pattern Analysis and Machine Intelligence,
vol. 32, no. 7
[10] Maini, R. et al. Study and Comparison of Various Image
Edge Detection Techniques International Journal of Image
Processing (IJIP), Volume (3) : Issue (1), 2009
[11] Matas, J. et al. Robust Detection of Lines Using the Pro-
gressive Probabilistic Hough Transform CVIU 78 1, pp 119-
137, 2000
[12] Nipat Thiengtham and Yingyos Sriboonruang Improve
Template Matching Method in Mobile Augmented Reality
for Thai Alphabet Learning International Journal of Smart
Home Vol. 6, No. 3, July, 2012
[13] P. Kakumanu, S. Makrogiannis, and N. Bourbakis A sur-
vey of skin-color modeling and detection methods Pattern
Recogn. 40, 3, March 2007
[14] Palmer, Phil L., Josef Kittler, and Maria Petrou An opti-
mizing line finder using a Hough transform algorithm Com-
puter Vision and Image Understanding 67.1, 1997
[15] Roberto, B. Template Matching techniques in computer
vision: theory and practice 2009
[16] S. Garrido-Jurado, R. Muoz-Salinas, F.J. Madrid-Cuevas,
M.J. Marn-Jimnez Automatic generation and detection of
highly reliable fiducial markers under occlusion Pattern
Recognition, Volume 47, Issue 6, June 2014
[17] Schumeyer, Richard P., and Kenneth E. Barner Color-
based classifier for region identification in video Photonics
West’98 Electronic Imaging. International Society for Op-
tics and Photonics, 1998
[18] Singh, Chandan, and Nitin Bhatia A Fast Decision Tech-
nique for Hierarchical Hough Transform for Line Detection
arXiv preprint arXiv:1007.0547, 2010
[19] Van Ginkel, Michael, CL Luengo Hendriks, and Lucas J.
van Vliet. A short introduction to the Radon and Hough
transforms and how they relate to each other Delft Univer-
sity of Technology, 2004
 Augmenting Videos with 3D Objects

Andrei Bajenov Darshan Kapashi Sagar Chordia

abajenov@stanford.edu darshank@stanford.edu sagarc14@stanford.edu

Abstract for camera tracking and 3D reconstruction. There is also

We propose a way to automatically augment a video of a
static scene with a 3D object. We use SFM algorithms to es-
timate the position of the camera. In order to properly sup-
port occlusions, we use a novel approach to generate dense
depth maps for each frame of the video. We use a combi-
nation of semi-global block matching, image segmentation
using the watershed algorithm, and planar interpolation to
remove noise and sharpen edges. The final result is a video
augmented with a 3D object that is properly occluded by
objects in front of it.
1. Introduction
The idea of augmenting videos has been around for a
while, and we see it everywhere today. One prominent
example is CGI in movies, which augments reality with
computer generated objects and makes the viewer believe
that these objects are part of the environment.
The process to do this is generally difficult and requires
a lot of specialized software and equipment. In this paper,
we describe a system that, given an object mesh and a
video, allows anyone to place this object seamlessly into
the video without any other external inputs.
quite a bit of research in 3D reconstruction from a set of
2D images.
We have not been able to find a product that specifically
takes a video and augments it while supporting occlusions
(although a number of technologies like Wikitude support
projecting objects based on camera positions). This was our
main motivation for building this system.
2. Problem Statement
We propose a system that puts a 3D object in a video of
a static scene while supporting occlusions. We break up the
problem into two components:
1. Estimate camera matrices for each frame of the video,
so that we can project an object back into the scene.
2. Obtain sharp and accurate depth maps for each frame,
to deal with occlusions.
Below is an example of a scene that we used for testing
our system.

There are a number of interesting applications to this.

For example, it could be used for seeing how a piece of
furniture would look in a room or how a new house would
look in a particular location. If integrated with a smart-
phone’s camera, it could also be used when interacting with
an environment. For example it could provide navigational
pointers, highlight parts of an environment, or even project Figure 1: An example of a static scene
another person into an environment.

There are a growing number of technologies that are 3. Previous Work

being built to support this. Project Tango [1], by Google, is
building a phone that has a built-in depth sensor to make
3D reconstruction and camera tracking easier. Wikitude [2]
is an example of a piece of software that is designed for
the purpose of augmenting smartphone videos. It allows
3.1. Estimating Camera Positions
There has been a lot of research done in the area of Struc-
ture from Motion (SFM), and there are a number of existing
libraries that implement SFM algorithms, including:

1
 • Theia SFM [3]
• Bundler [11]
• Visual SFM [10]
• OpenCV SFM [9]
We were looking for a few things from the libraries:
• Camera position estimation
• Camera parameter estimation
• Reliable and accurate sparse 3D reconstruction
For this project, it was not our goal to try to improve or
optimize any of these libraries. We tried a few of them and
picked the one that was easiest to use. In our case it was the
Theia SFM library.
3.2. Estimating Depth Map
A critical part to solving our problem was obtaining ac-
curate and dense depth maps for each frame of the video.
There were a number of techniques that we considered:
• Reconstructing 3D objects using volumetric stereo and
using these reconstructions to obtain depth maps [12]
• Using a combination of the original images and the
sparse 3D points obtained from SFM to approximate
the 3D surface positions (using segmentation and pla-
nar reconstruction).
• Using a combination of SFM and stereo matching al-
gorithms to obtain a dense 3D reconstruction of the
scene. [16]
While researching volumetric stereo, we found that
it was genereally used to get a 3D reconstruction of a
single object within a scene. For our purposes, we needed
information about the full scene. To extend this algorithm
to work on a full scene, we would have needed very reliable
image segmentation algorithms that were determenistic
between frames. We were not able to find anything that
looked promising in this space, so we abandoned this idea.
We also considered using the sparse 3D points obtained
from SFM to approximate a dense 3D reconstruction. We
though about partitioning the original image into uniform
segments. We would then approximate each segment as a
plane in 3D and use the sparse 3D points to estimate these
planes. Unfortunately what we found was that we did not
have enough points in each image segment to do a planar
reconstruction, so we could not use this approach by itself.
Lastly we found a lot of research about using stereo-
matching algorithms to aid in dense 3D reconstruction [17].
2
Specifically, the Middlebury website [13] contains a lot
of submissions and evaluations of many stereo-matching
algorithms. We ended up pursuing this approach the most
because the research here showed the most promising
results.
To solve our problem, however, we do not need a full 3D
reconstruction of the scene. We just need an approximate
depth map that has good accuracy around the object bound-
aries. What we found while using just stereo-matching al-
gorithms was that they were prone to noise. To overcome
this problem and achieve the desired results, we propose a
novel approach that uses a combination of image segmenta-
tion techniques, stereo-matching, and planar interpolation.
4. Technical Details
Below we describe our solution. We talk about how we
calibrate our camera and run SFM. We then describe our
method for getting accurate depth maps. Lastly, we describe
how we project 3D objects back into the scene.
4.1. Sparse 3D Reconstruction and Camera Matrix
Estimation
As per standard practice in the camera model used in
computer vision, there are 2 parameters:
• Intrinsic matrix K: A 3x3 matrix which incorporates
the focal length and camera center coordinates.
• Extrinsic matrix [R T]: A 3x4 matrix which maps
world coordinates to camera coordinates. R denotes
rotation and T denotes translation.
The camera transformation is given by a matrix,
M = K[R T ]
It transforms a point in homogeneous world coordinates
to homogeneous image coordinates.
The point correspondence problem is defined as follows:
Given n images, find points in the images which correspond
to the same 3D point. There are several well known
algorithms which work reasonably well in practice, for
example, SIFT, SURF and DAISY.
The Structure from motion (SFM) problem is defined
as follows: Given m images and n point correspondences,
find m camera matrices (M) and n 3D points. Solving the
SFM problem for a set of images will give us a sparse 3D
reconstruction of the scene.
To get the intrinsic parameters for our camera we tried a
few different approaches:
 • Computing K using single view metrology with 3 van- 4.2. Depth Map Estimation
ishing points derived from 3 pairs of mutually orthog-
A sparse reconstruction is not enough to get a full depth
onal lines in 3D.
map for each frame.
• Using a checkerboard image to calibrate using
Below we propose a novel approach of using a combi-
OpenCV routines.
nation of stereo-matching, image segmentation (using the
watershed algorithm), and planar interpolation to get dense
• Allowing Structure From Motion (SFM) algorithms to
3D depth maps for each frame.
self-calibrate (which is possible given enough view-
points of a static scene)
4.2.1 Terminology
Each of these approaches gave us a similar K, so we
decided to go with the self-calibration method since it is
automatic and we have plenty of views.

We tried a couple different SFM libraries. The SFM

library bundled with OpenCV didn’t give good results. The
Theia [3] library was easier to work with and was able to
give us fairly accurate sparse reconstructions of a 3D scene,
figures 2, 3.

Figure 4: Stereo setup

1. Disparity map. Disparity refers to the difference in im-

age location of an object seen by the left and right cam-
eras, resulting from different positions of two cameras
Figure 2: Theia’s camera positions and a sparse 3D recon- as seen in figure 4. A disparity map is a mapping for
struction each pixel in the image to the disparity value for that
pixel. The value represents the distance between the
locations of a point in the left and right rectified stereo
images. It indicates the relative distance to the camera.
A higher value means it is closer to the camera.

2. Depth map. A mapping for each pixel in the image to

the depth value for that pixel. It indicates how far a
point is from the camera. A higher value means it is
further from the camera.

Figure 3: Side view of the 3D reconstruction

At this point, we have the camera intrinsic and extrinsic

matrices as well as a small set of 3D points which can be
used to get a sparse reconstruction of the scene. Figure 5: Image rectification

3
 3. Image rectification. A transformation to project two
images onto a single image plane as seen in figure 5.
After rectification, all epipolar lines are parallel in the
horizontal axis. All corresponding points have identi-
cal vertical coordinates.

4.2.2 Semi-global block matching

After obtaining camera matrices for every frame of the
image, we use that information to perform stereo matching
between pairs of frames. We picked pairs of frames with a
good baseline distance between them (in our case around
2.0), and performed stereo matching on rectified versions
of these frames.
We tried different stereo-matching algorithms and
picked Semi-global block matching (SGBM) since it was
readily available in OpenCV and had good performance on
the Middlebury dataset.
SGBM aims to minimize a global energy function E
for the disparity image D, based on the idea of pixel-wise
matching of mutual information and approximating a
global 2D smoothness constraint by combining many 1D
constraints.
It takes as input 2 rectified images, taken from the cam-
era on the left and from the camera on the right. It also takes
the camera matrix K. It produces disparity maps for the left
and right images. Figure 6 shows an example disparity map.
Figure 7: Weighted Least Squares filtering on an SGBM
disparity map
4.2.3 From disparity maps to 3D points
Disparity maps alone don’t help, since they don’t tell us the
exact depth of objects in each frame. To convert between
disparity maps and depth maps, we first need to reproject
the disparity values to 3D.
p = (x, y) is a point in the disparity map. The matrix
Q incorporates the transform between the left and right
cameras which is obtained during image rectification.
We can get the homogeneous coordinate in 3D using this
equation.
[X Y Z W ]T = Q ∗ [x y disparity(x, y) 1]T
And finally get a mapping from 2D to 3D.
3d image(x, y) = (X/W, Y /W, Z/W )

Figure 8 is the dense 3D reconstruction obtained from

the above equation.

Figure 6: SGBM Disparity Map

After obtaining a disparity map, we do a first pass to

remove noise. We found a technique called Weighted Least
Squares filter [9]. The result is in figure 7.
Figure 8: 3D reconstruction from a disparity map

4
4.2.4 From 3D points to a depth map
So far, we are able to obtain 3D points from a disparity
map. These points are not in world coordinates though,
so we need to transform them to world coordinates before
generating depth maps.
Let x be the point in world coordinates. Let p be the
point in the original image. We rectify the image for stereo
matching. Rectification is a homographic transform. Let H
be the inverse of this transform. K, R and T are camera
parameters.
The point x maps to p using the camera transform.
p = KRx + KT
The point pr is the point in the rectified image, which
corresponds to the point p in the original image. xr is the
3D point in rectified coordinates.
pr = Kr Rr xr + Kr Tr
We get p by applying the rectification transform H on pr ,
p = Hpr
Using these equations, we can derive the equation for
point x in original world coordinates.
x = R−1 K −1 HKr Rr xr + R−1 K −1 HKr Tr − R−1 T
We reproject these points back into the original frame
and compute the depth for each pixel. Figure 9 is a depth
map for the frame from which we generated the disparity
map.
Figure 9: Example depth map obtained from a disparity
map
5
4.2.5 Missing depth maps
We don’t have a disparity map for every frame. Frames
where the camera is moving forward, for example, don’t
have a good corresponding stereo frame. Rectification
between such views introduces too much distortion.
As such, we need to be able to reconstruct a depth map
for any frame, using a depth map generated from some other
frame. Fig 10 is an example of a depth map viewed from a
different camera:
Figure 10: Depth map from another camera location - more
occlusions
There are far more missing depths, which are mostly
there due to occlusions. To help fill in the rest of this depth
map, we use a novel technique which combines image
segmentation and planar reconstruction, as described in
subsections below.
4.2.6 Image segmentation
We use the marker controlled watershed algorithm for
image segmentation. The watershed algorithm is based
on the concept of flooding the image from its minima
and preventing the merging of water coming from dif-
ferent sources. This partitions the image into 2 parts:
the catchment basins and the watershed lines. This
approach results in over-segmentation, so we use a vari-
ant which is based on starting to flood from a set of markers.
To find the set of markers, we apply the following set of
transformations to the image. In the process of thresholding
an image, we set the value for each pixel to either 0 or 1
based on a threshold. We use adaptive thresholding to the
image. It considers local variations in intensity and makes
pixels white and black. This is significantly better than
using a global threshold because the lighting in the scene is
not uniform. Then, the transformed image has several small
holes. We use morphological opening to fill in these holes
and have a much smaller set of bigger segments. Next,
we apply a distance transform, followed by a thresholding
transform which gives us a candidate set of markers. Using use this set of points to estimate a plane using SVD decom-
this, we can apply the watershed algorithm to segment the position for the linear system Ax = t. This gives us a plane.
images.

This method has several parameters that can be tuned to ax + by + cz + d = 0

get a segmentation of desired quality. This includes

• Window size of adaptive threshold
• Kernel for the morphological opening
• Threshold for the distance transform
Figure 11 is an example of a segmented image.
This plane gives us the depth for every point on it,
irrespective of whether we had a depth for it previously
from the stereo matching algorithms. This is how we fill
up holes in the depth map. We can now trace a ray which
starts from the camera and hits the approximate plane. We
can compute the length of this line segment and this is the
depth of this image point.

For a point p = (x, y) in the image plane, we can

compute the point of intersection between the plane and the
ray and compute the depth as

depth = −d/(a ∗ (py − cy )/fy + b ∗ (px − cx )/fx + c)

For certain segments, because of measurement noise,

Figure 11: Image segmented using the watershed algorithm
Note that when tuning parameters, our goal is to make
sure that segments don’t spill between objects. We achieve
this by tuning the parameters to generate small segments.
poor segmentation, or non-flat surfaces, it is possible to end
up with a bad estimate of the plane. We found a simple
heuristic to prune these bad planes, ||Ax−t|| > threshold.
This helps to reduce noise in this approach for reconstruc-
tion.

4.2.7 Planar interpolation With the above, we get an depth map that looks some-
thing like:
We have now divided the image into several segments. We
assume that each segment is part of a plane. We have the
depth map for each 2D image point, which means we have
a 3D point corresponding to each 2D point. The camera
matrix K has the following structure:

 
fx 0 cx
 0 fy cy 
0 0 1
For each 2D point, we can compute the corresponding
3D point. z is the depth of this point from the depth map.

p = (z ∗ (py − cy )/fy , z ∗ (px − cx )/fx , z) Figure 12: Planar interpolation of image segments

For each image segment, we collect all the 2D points

for which we know a depth (z is not equal to 0). We get
corresponding 3D points. For n points in a segment, we
construct a nx3 matrix A where each row is a 3D point For areas where the planes couldn’t be reconstructed, we
p = (x, y, z). The matrix t is a nx1 matrix of −1. We can fill those with the original depths, to get:

6

Figure 13: Filling in missing segments with original depths
4.2.8 Combining depth maps from multiple views
So far we’ve dealt with depth maps obtained from a single
pair of rectified images. We observed that using a single
pair of images doesn’t give a full depth map when viewed
from different camera angles. Fortunately, in a video
sequence, we have many pairs of such images that can help
improve results and fill in missing depths.
To find a depth map for the current frame, we take
a number of nearby frames for which we have obtained
high-quality depth maps. We deem a depth map to be high
quality if it was obtained from a disparity map with a good
baseline distance which is not too small and not too large.
We then re-project the depth map from those frames into
the current view. What we end up with is multiple depths
per pixel.
To pick the desired depth, we sort the depths and pick the
first value for which the value between it and the next value
is no higher than 15 percent. This is a rudimentary way to
pick the smallest z-value cluster. We pick the smallest z-
value cluster to eliminate noise and ignore occluded objects.
With the above approach we obtain a depth map as seen
in Fig 14. Notice that there is less noise than in previous
depth maps, and more pixels have a depth value.
Figure 14: Depth map generated by merging depth maps
from multiple viewpoints
7
4.3. Augmenting Video with 3D Objects
The last step of our system is augmenting the video
with a 3D object. At this point we have estimated camera
matrices and depth maps. The location of the object
within the scene is determined manually, i.e. we take 3D
coordinates in the object and translate them such that it
is placed behind one of the boxes in the scene. In a real
application, you can imagine a user interface which lets
you drag and drop the object in the scene. We do not solve
this problem here and focus on the mathematical aspects.
The object (a bird) is a 3D object. It looks like a 2D blob
because we did not add shading to it.
The 3D object is given as a mesh of triangles. An easy
way to augment the frames of the video with this object is
to apply the camera transform on each of the vertices and
fill in triangles with the object texture. It gets slightly more
complex when we want to handle occlusions.
Since we want to be as accurate as possible, we make
sure that the triangles which make up the object are small
enough. If they aren’t, we can split each triangle into
3 smaller triangles using the centroid and the current
vertices. A triangle is visible if all 3 vertices of the triangle
are visible. With small enough triangles, this is a good
approximation.
For each vertex v = [x y z], which is a point in 3d world
coordinates, we transform it into camera coordinates point
p = (x, y, z) using the camera transform.
(x, y, z) = [R T ] v
The depth of the point is the z coordinate in camera co-
ordinates (zp ) and the depth in the frame of the video is
as computed using stereo correspondence and planar recon-
struction (zi ). If the 3d object point is at a greater depth,
zi < zp , it is hidden in the image, otherwise zi > zp , it is
visible.
Figure 15: Bird is completely visible in this view

Figure 16: Bird is half hidden behind the box in this view
5. Evaluation
As discussed in section 4.1, we use existing algorithms
and libraries to estimate camera parameters in each image
of the video. Our main contribution lies in effectively
connecting various computer vision algorithms to augment
a video. The novelty in this paper is about refining depth
maps using image segmentation and approximating a
reconstruction using planes. Hence, we will skip evaluation
for camera calibration and focus on our method of refining
depth maps using planar reconstruction.
For easy evaluation of individual components of stereo
matching algorithms, the computer vision group from
Middlebury have designed a stand-alone flexible C++
implementation. [13] They also provide a collection of
datasets and benchmark it against various state-of-art
algorithms. The evaluation framework is flexible and
supports easy additions of new algorithms. We integrate
our methodology and compare the results with other
stereo matching algorithms already implemented in the
framework.
We describe the quality metrics we use for evaluating the
performance of various stereo correspondence algorithms
and the techniques we used for acquiring our image data
sets and ground truth estimates. [18]
1. RMS (root-mean-squared) error, measured in disparity
units, between the computed disparity map dC (x, y)
and the ground truth map dT (x, y), i.e.,
  12
1 X 2 dlebury evaluation dataset using our method as well as pre-
R= (|dC (x, y) − dT (x, y)| )
N defined algorithms in Middlebury. In the following table
(x,y)
where N is the total number of pixels.
2. Percentage of bad matching pixels,
1 X
B= (|dC (x, y) − dT (x, y)| > δd )
N than normal-SGBM in both metrics we defined earlier.
(x,y)
8
where δd is the disparity error tolerance.
There are various parameters which can be tweaked
while evaluating stereo matching in the Middlebury
framework. We use default values for most parameters
except one parameter. eval bad thresh which controls
thresholding to decide whether a pixel is a bad match or
not was increased from 1.0 to 5.0. We found 1.0f was too
strict and > 90% pixels were marked as bad pixels in most
images. But a value of 5.0 gives good results for most
images.
Figure 17a shows an image in the Middlebury evaluation
dataset. Figure 17b shows the ground-truth disparity map of
figure 17a. Disparity map of figure 17a is computed using
Semi-Global Block Matching (SGBM) algorithm and is
shown in figure 17c. Figure 17d shows image-segmentation
on the original image figure 17a. Figure 17e shows refined
disparity maps obtained by combining figure 17c and plane
fitting on figure 17d. Computed disparity maps figure 17c
and figure 17e are compared to the ground truth disparity
map figure 17b and the above two metrics are computed.
We also compare other algorithms from the Middlebury
evaluation framework to our methodology.
We can control the strictness of plane fitting to refine
disparity maps by norm thresh. When the error of fitting
a plane to points of a given image segment is greater than
norm thresh then we don’t refine the disparity map and
use the original disparity map. In figure 17e we can see
how disparity maps are affected as norm thresh value
is increased. Figure 17e is bad compared to figure 17d in
terms of both metrics and hence it is critical to tune this
parameter correctly.
Sometimes refined disparity maps may not be better than
the original disparity map because of bad image segmen-
tation. Figure 18b is a better approximation to figure 18a
compared to figure 18c. In figure 18c, plane fitting on bad
image segmentation results in a weird disparity map. So
it’s important to control the quality of image segmentation
to ensure refined disparity maps are better.
We compute disparity maps for 5 test images in the Mid-
we report mean values of RMS error and percentage bad
pixels. normal-SGBM refers to our SGBM implementa-
tion of disparity map. planar-SGBM refers to the filtered
disparity map generated by fitting planes using image seg-
mentation. [18] describes the other algorithms used for en-
coding. As seen in the table, planar-SGBM performs better
 (a) Image to be evaluated from Middlebury dataset (b) Ground truth disparity map

(c) Disparity map with Semi-Global Block Matching

(d) Image segmentation using watershed transform
(SGBM)

(e) Filtered disparity map combining SGBM and plane (f) Filtered disparity map combining SGBM and ag-
fitting on image segmentation gressive plane fitting on image segmentation

Figure 17: StereoMatch Evaluation using middlebury dataset

(c) Refined disparity map using image

(a) True disparity map of sawtooth (b) Disparity map by SGBM
segmentation

Figure 18: Another test image from Middlebury evaluation dataset

9
 Algorithm Mean RMS error Mean Bad pixel ratio 7. Conclusion
SSD09bt05 1.559039 0.024049 In this paper, we proposed a system that takes a 3D
object mesh and a video, and augments that video with the
SSD09t20 1.714662 0.030914 object. The system is able to estimate camera positions and
generate depth maps for each frame (to support occlusions).
SADmf09bt05 1.733064 0.031358
SADmf09t02 2.4118 0.051564 We used the Theia SFM library to estimate camera
positions, and proposed a novel method to estimate depths
SAD09t02 2.565821 0.058292
in each frame. To estimate depths, we used a combination
SADmf09t01 3.019650 0.084793 of image segmentation techniques (watershed algorithm),
stereo matching (SGBM), and planar interpolation. When
planar-SGBM 3.177850 0.071215
compared to stereo matching alone, the combination of
normal-SGBM 3.200869 0.072295 these technique allowed us to improve depth map accuracy
while at the same time significantly reducing noise and
SAD09t01 3.717277 0.135821 improving sharpness around object boundaries.

6. Future Work We were able to successfully project a 3D object back

into a video. Our algorithm was fully autonomous, and
There are still a few things that we would have liked to
did not require anything other than specifying the object
try to improve our results.
position.
• Implement state-of-the-art algorithms for stereo-
matching and see how they perform with and without You can find our code by going to:
planar interpolation. We only had time to add planar https://bitbucket.org/bajenov1/cs231a/
interpolation on top of SGBM, but there are better al-
gorithms out there. You can find our final augmented video here:
https://www.youtube.com/watch?v=X37SP4Dihhg
• Look into using the DAISY descriptor instead of
stereo-matching for dense 3D reconstruction. See: ”A
Fast Local Descriptor for Dense Matching” by Tola et.
al. [5]

• Try different image segmentation approaches to re-

move spilling out of objects. We started looking into
using the canny edge detector to seed the watershed
algorithm. [14]

• Exploit the fact that we have a video instead of a set of

photos to run SFM in real-time.

• Exploit the fact that we need a depth map of only a

small region where the 3D object is projected into the
scene (to make our algorithm real-time).

• Use a better rectification algorithm, as described in ”A

simple and efficient rectification method for general
motion” [19]

• [17] describes how to add better depth map merging

techniques - ”Metric 3D Surface Reconstruction from
Uncalibrated Image Sequences”.

• Project more interesting geometry into the scene. Ei-

ther by integrating with a ray-tracing library or using
OpenGL and taking advantage of its built-in shaders.

10
References
[1] Lee, J. C., and R. Dugan. ”Google project tango.”
[2] Perry, Simon. ”Wikitude: Android app with
augmented reality: Mind blowing.” digital-lifestyles.
info 23.10 (2008).
[3] Sweeney, Christopher, Tobias Hollerer, and Matthew
Turk. ”Theia: A Fast and Scalable
Structure-from-Motion Library.” Proceedings of the
23rd Annual ACM Conference on Multimedia
Conference. ACM, 2015.
[4] Ravimal Bandara, Image Segmentation using
Unsupervised Watershed Algorithm with an
Over-segmentation Reduction Technique.
[5] Tola, Engin, Vincent Lepetit, and Pascal Fua. ”A fast
local descriptor for dense matching.” Computer Vision
and Pattern Recognition, 2008. CVPR 2008. IEEE
Conference on. IEEE, 2008.
[6] Mur-Artal, Raul, J. M. M. Montiel, and Juan D.
Tardos. ”ORB-SLAM: a versatile and accurate
monocular SLAM system.” Robotics, IEEE
Transactions on 31.5 (2015): 1147-1163.
[7] Furukawa, Yasutaka, and Jean Ponce. ”Accurate,
dense, and robust multiview stereopsis.” Pattern
Analysis and Machine Intelligence, IEEE Transactions
on 32.8 (2010): 1362-1376.
[14] Canny, John. ”A computational approach to edge
detection.” Pattern Analysis and Machine Intelligence,
IEEE Transactions on 6 (1986): 679-698.
[15] Haris, Kostas, et al. ”Hybrid image segmentation
using watersheds and fast region merging.” Image
Processing, IEEE Transactions on 7.12 (1998):
1684-1699.
[16] Pollefeys, Marc, Reinhard Koch, and Luc Van Gool.
”A simple and efficient rectification method for
general motion.” Computer Vision, 1999. The
Proceedings of the Seventh IEEE International
Conference on. Vol. 1. IEEE, 1999.
[17] Pollefeys, Marc, et al. ”Metric 3D surface
reconstruction from uncalibrated image sequences.”
3D Structure from Multiple Images of Large-Scale
Environments. Springer Berlin Heidelberg, 1998.
139-154.
[18] Scharstein, Daniel, and Richard Szeliski. ”A
taxonomy and evaluation of dense two-frame stereo
correspondence algorithms.” International journal of
computer vision 47.1-3 (2002): 7-42.
[19] Pollefeys, Marc, Reinhard Koch, and Luc Van Gool.
”A simple and efficient rectification method for
general motion.” Computer Vision, 1999. The
Proceedings of the Seventh IEEE International
Conference on. Vol. 1. IEEE, 1999.

[8] Kundu, Abhijit, et al. ”Joint semantic segmentation

and 3d reconstruction from monocular video.”
Computer VisionECCV 2014. Springer International
Publishing, 2014. 703-718.

[9] Bradski, Gary, and Adrian Kaehler. Learning OpenCV:

Computer vision with the OpenCV library. ” O’Reilly
Media, Inc.”, 2008.

[10] Wu, Changchang. ”VisualSFM: A visual structure

from motion system.” (2011).

[11] Snavely, Noah. ”Bundler: Structure from motion

(SFM) for unordered image collections.” Available
online: phototour. cs. washington.
edu/bundler/(accessed on 12 July 2013) (2010).

[12] Eisert, Peter. ”Reconstruction of Volumetric 3D

Models.” 3D Videocommunication: Algorithms,
Concepts and Real-Time Systems in Human Centred
Communication (2005): 133-150.

[13] Scharstein, Damiel, and R. Szeliski. ”Middlebury

stereo datasets.” 2014-04-06]. http://vision,
middlebury, edu/stereo/data (2006).

11
 Classroom Data Collection and Analysis using Computer Vision
Jiang Han
Department of Electrical Engineering
Stanford University
Abstract
This project aims to extract different information like
faces, gender and emotion distribution from human beings
in images or video stream. Based on those collected data we
may be able to obtain some useful feedback or information,
which can be valuable guidance on how we can improve
the class education quality. In this project, a few computer
vision topics were touched like face detection, gender clas-
sification and emotion analysis. Technical details tested in-
clude feature extraction strategies like Bag of Words, HOG,
LBP, key-point detection, feature reduction etc. Machine
learning classifier tested including Nave Bayes, KNN, Ran-
dom Forest and SVM. Classifier parameters are tuned to
achieve the best accuracy. The system is able to achieve
accuracy of 0.8673 for gender classification and 0.5089 for
emotion analysis (0.6073 when we prune particular class).
Real life image and video stream tests also verified the va-
lidity and robustness of the system.
1. Introduction
Motivation of this project came from my personal expe-
rience. One time when I was taking a class at Stanford CE-
MEX Auditorium, I often saw a TA came to the second floor
and count how many students presented at the class. Atten-
dance was not strictly required for this class, so this data is
only used by the instructor to have a better understanding of
current instruction status. At that time, I thought it will be
great if we can do this by simply taking a photo.
When I take CS231A this year, this old idea came to me
so I immediately decided to do some related work. Instead
of only counting number of students, I decided to get more
interesting information like gender and emotion. By de-
tecting faces and doing gender classification, we can have a
rough number of students attendance and their gender dis-
tribution. Emotion analysis maybe more useful in analyzing
the class quality. Later on from Section 4, it is shown that
we are able to get the emotion distribution across time with
a video stream. If some emotion like “Surprise” appears a
lot in the distribution, the class instructions may not be as
1
clear as it should be.
So far most camera system on mobile phone has embed-
ded face detection algorithm, but they have not put all three
topics including face detection, gender analysis and emo-
tion analysis all together. Thus implementing such a demo
system will be very exciting to me.
Good thing is there has already been a lot researches on
the above three areas. As a classical topic in computer vi-
sion, a lot of approaches have been proposed for face detec-
tion. Authors from [1] [2] concluded there are four types
of approaches for face detection: knowledge based method,
feature based method, template matching approach and ap-
pearance based method. In particular, an outstanding face
detection algorithm was proposed by Viola and Jones [3].
Viola-Jones algorithm applys adaboosting and cascading
classifiers and has advantages like fast speed, suitable for
scaling, which makes it as the embedded face detection al-
gorithm by toolbox from OpenCV and Matlab. For gen-
der classification, authors from [4] also roughly divide it as
feature-based and appearance-based approaches. Later on,
Lian etc. [5] used LBP feature with SVM and were able to
achieve very high accuracy. Also, Baluja etc. [6] applied
Adaboost classifier and able to achieve more than 93% ac-
curacy. Similar to gender classification, emotion analysis is
also a classifying problem, but with multiple classes instead
of two, which increases difficulty in accuracy. Authors from
[7] presents an approach using means of active appearance
model to do both gender and emotion classification. SVM
was used to label four emotion types (happy, angry, sad and
neutral). Y Kim etc. [8] applied deep learning techniques to
overcome the linear feature extraction limitation in emotion
detection, which is able to boost performance. In all, gender
classification and emotion analysis have been very hot top-
ics in researches from both feature extraction and learning
model optimization aspects [9].
In this report, Section 2 shows problem statement, which
briefly describes the system framework. Section 3 is tech-
nical content, which introduces data set used, evaluation
metric, and shows different approaches tried related feature
and classifier selection. Multiple numerical simulation re-
sults are also shown in Section 3. Section 4 is experimen-
 Gender Male Female
Training 9,993 10,992
Testing 3,040 2,967

Table 1. Data set for gender classification

Emotion Angry Disgust Fear Happy

Training 3,995 436 4,097 7,215
Testing 958 111 1,024 1,774
Emotion Sad Surprise Neutral
Training 4,830 3,171 4,965
Testing 1,247 831 1,233

Table 2. Data set for emotion analysis

Figure 1. Framework of system.

tal setup and results, which analyzes on class F1 score, also

shows the system performance on real life images and video
stream. Finally, conclusions and future work are shown in
Figure 2. Samples from gender data set (Male: left three images.
Section 5. Female: right three images).

2. Problem Statement
Main framework of system design is shown in Fig.1,
which includes four modules to process the input image.
Image is transformed to gray scale image at the first be- Figure 3. Samples from emotion data set (left to right: Angry, Dis-
ginning since color information is not that important in this gust, Fear, Happy, Sad, Surprise, Neutral).
classification problem. Then face detection is applied for
the image to locate all the human faces positions. Inside
each face box, gender classification and emotion analysis design a system with good trade-off between performance
engine works to generate corresponding labels. As Fig.1 and complexity is one of the project target.
shows, Module-3 and module-4 shares very similar inside
core, which includes: 3. Technical Content
• Image rescale: The subimage inside face box needs to 3.1. Data set and initial analysis
be rescaled for three reasons: (1) This is necessary and
The used data set for gender classification and emotion
will make things much easier to generate consistent
analysis is shown in Table 1 and 2.
feature dimension later on. (2) The source image data
The data set for gender classification is extracted partly
set for training and testing of gender analysis was dif-
from Image of Group (IoG) data set [10], the original data
ferent in scale size. (3) Face box derived from Module-
set includes more properties on each person like: face po-
1 may be different in sizes due to different face size.
sition, eye position, age, gender, pose. In this project, we
• Feature extraction: After rescaling, this step generates only care about the age property. Thus I split the data
features with consistent dimension. Feature extraction into four folders: “Male Training”, “Female Training”,
can use algorithms like Histogram of Gradient (HOG), “Male Testing” and “Female Testing”. From Table I we
Local Binary Pattern (LBP), Bag of Words (BoW), etc. can see that both male and female image number was
roughly balanced to guarantee the best model training
• Model training/classification: Based on the feature
result. With this split, we can use Matlab command
vectors output from feature extraction, we are able
imageSet easily load corresponding images. And the to-
to train the classifier with input image training data.
tal number of images used for both training and testing is
Training step may take long time due to data size. But
26,992. One thing I notice is that source image for genders
once the model is trained, we are able to use it to do
is not scaled to the same size. This is one of the reasons why
classification on the testing image directly.
we add “Image Rescale” before the feature extraction step
From Fig.1 we can see that each module may have multi- to make the input image 48 by 48 gray image. Fig.2 shows
ple algorithm candidates to implement, while being able to six gender sample images, which includes three male and

2

Figure 4. Bar chart of emotion data set.
three female. Also we note that this data set includes peo-
ple from different races and different age.
Furthermore, Table 2 shows the data set for emotion
analysis. The data set is from ICML [11], which has 7 cate-
gories of emotions including: Angry, Disgust, Fear, Happy,
Sad, Surprise and Neutral (both for training and testing).
Fig.3 shows samples of the seven types of emotions. We
notice that this data set includes emotions from different
gender and ages, which is good to train a robust model.
However, emotion of human beings is very complicated and
vague. For example the “Surprise” image of Fig.3 may also
be treated as “Angry” in reality. Furthermore, Fig.4 shows
the bar chart of different emotion types image number (in-
cluding both training and testing). Notice that this data set
is mostly balanced, except that “Disgust” type has signifi-
cantly lower number than other categories, which explains
why “Disgust” class has the lowest F1 score in later testing.
3.2. Evaluation metric
To evaluate the performance of classification, accuracy
(ACC) [12] is sued as the metric, which defines as:
ACC = (T P + T N )/(P + N ) (1)
Here, P and N are the total number of positive and negative
samples. T P and T N are the true positive and true negative
samples number. Therefore, ACC value of 100% indicates
that the system is able to predict labels exactly same with
the ground truth values. In this project, ACC is used to
evaluate different feature extraction and machine learning
algorithms.
In addition, I also used precision, recall and F1 [13] to do
the analysis on each type of the prediction classes, which is
shown in Section 4.
2 · precision · recall
F1 = (2)
precision + recall
Here, precision is defined as true positive (tp) over tp plus
false positive (fp). recall is defined as tp over tp plus false
negative (fn).
3
3.3. Face Detection
For face detection, initially I was using the similar
method from our problem set 3 with HOG + SVM + sliding
window. I also tried the following dynamic boxing method
to adjust window size to fit the face scaling.
size = minSize + 2N × step (3)
Here, size value is with constraint of size ≤ maxSize.
N is the N-th time of window expanding, step is a value
controls the window expanding speed. The advantage of
this strategy is that from minSize to maxSize, we at most
need N = dlog2 maxSize−minSize
step e times of expanding.
And we are giving smaller expanding speed when the win-
dow size is still small. Hence, instead of doing linear time
of expanding and apply SVM at each position, here we have
cost at O(log) level, eventually we choose the face window
size with the biggest prediction score (only if this score is
bigger than SVM threshold).
However the test shows that sliding window scheme
is quite slow because we need to run SVM multiple
times. Considering face detection is not the main part
of this project, I turned to use the MATLAB embedded
vision.CascadeObjectDetector() to do the face detec-
tion, which is using Viola Jones object detection framework.
Viola Jones algorithm is much faster and good at detecting
scaled faces [3].
3.4. Gender Classification and Emotion Analysis
I put gender classification and emotion analysis in the
same subsection since from Fig.1, we see that the two mod-
ules share very similar inside blocks. Therefore in this sub-
section, I’m going to introduce the following three blocks:
image rescale, feature extraction and training/classification.
3.4.1 Image rescale
Figure 5. Face rescale example on testing image (not all faces are
shown here).
Image scale is necessary for both training and classifi-
cation. For the training set, each image of emotion anal-
ysis was originally given with 48 by 48 gray scale, which
is ok to use directly. But for gender classification data, it Paras Cell:8,8. Block:2,2 Cell:4,4. Block:2,2
was come with RGB images with different scale size. Thus Feature Size 900 4356
rescale is necessary to make the training images into fixed HOG ACC 0.8369 0.8500
size and gray scale. This will make much easier in the fea- Paras Cell:8,8. Block:3,3 Cell:16,16. Block:2,2
ture extraction step to obtain feature vector with consistent
Feature Size 1296 144
dimension.
HOG ACC 0.8337 0.7718
For the classification step, there may be multiple faces
marked from the original image, and each comes with dif- Table 4. Gender classification HOG ACC performance with vari-
ferent box size. Thus we do RGB to gray scale and rescale ous cell/block settings.
to 48 by 48 before we apply classifier. Fig.5 shows an ex-
ample of classification rescale inside box. Paras Cell:8,8. Cell:12,12. Cell:16,16. Cell:24,24
Size 2124 944 531 236
3.4.2 Feature extraction (BoW) ACC 0.8638 0.8390 0.8274 0.7864
Note in order to save space, for feature extraction part Table 5. Gender classification LBP ACC performance with various
from 3.4.2 to 3.4.5, the table ACC values are based on cell settings.
testing of gender classification using SVM. However,
emotion analysis data also gives similar conclusion.
For the feature extraction, I started with bag of words. parameters we can tune for HOG, like “Cell Size”, “Block
Matlab provides some embedded functions like “bagOf- Size”, “Block Overlap”, “Number of Bins”. In my test,
Features”, “trainImageCategoryClassifier”, “imageCatego- I kept default value of “Block Overlap” and “Number of
ryClassifier”, etc. to be used. The default feature vec- Bins” since that is the typical settings. “Cell Size” and
tor is based on SURF, I also tried dense SIFT features. “Block Size” will be more important parameters which can
Based on the extracted feature vector from each images, control feature vector size and testing performance. Here,
K-means is applied to the feature space with entire train- “Cell Size” defines the box to calculate histogram of gradi-
ing images. Here, K defines the vocabulary size for the ents. Smaller cell size will give us better chance to catch
histogram. Eventually the image feature is defined as a his- small-scale details. On the other hand, increasing the cell
togram which defines the nearest cluster center distribution size will be able to capture large-scale spatial informa-
for every image. tion. “Block Size” defines number of cells inside the block,
Table 3 shows the ACC of gender classification with both smaller block size may reduce the influence due to illumi-
SURF and dense SIFT features. From later on ACC of SVM nation changes of HOG features [15].
we can see that this performance is even slightly worse than In addition to HOG, LBP is another feature I found out to
Naive Bayes. It seems reasonable to me since the testing ob- be very useful, the performance is no worse than HOG. The
jects are all faces with same structures (eyes, nose, mouth principle of LBP is different from HOG, instead of using
etc.). Clustering those features may lose some details. The gradient information. LBP compares the pixels value with
scenario of gender classification and emotion analysis is dif- its neighbors, and based on the binary comparison result
ferent from scenarios where bag of words are most used to construct the histogram. LBP can be easily extended to
(for example object classification like cups, ships, etc.). In rotation invariant version [16].
addition, BoW is giving me very slow training speed with Table 4 and Table 5 shows the HOG and LBP testing
around 3 million feature vectors to e clustered. Thus BoW ACC with different feature dimension of gender classifica-
was not selected after testing. tion (emotion analysis data testing result is having similar
trend, thus not listed here). By setting cell/block size we
Feature SURF Dense SIFT are able to obtain different feature dimension. Unsurpris-
BoW Test ACC 0.7137 0.7326 ingly higher feature dimension is able to give better ACC
but may also slow down the system significantly due to in-
Table 3. BoW ACC performance of gender classification on SURF creased complexity for machine learning models.
and dense SIFT features (vocabulary size is 300).
3.4.4 HOG and LBP feature combination

3.4.3 Feature extraction (HOG and LBP)
HOG and LBP were tested after BoW. HOG is a very well-
know feature descriptor in computer vision [14], which ac-
cumulates local gradient information . There are several
To get the best trade-off between performance and speed,
and based on the research fact that combination of HOG
and LBP features is able to improve the detection perfor-
mance [17]. I joined HOG feature (Cell:8,8, Block: 2,2,
dimension of 900) together with LBP feature (Cell: 12,12,

4
dimension of 944) to boost ACC. Table 6 shows the combi-
nation feature result.
Paras HOG LBP Joined
Feature size 900 944 1844
Test ACC 0.8369 0.8390 0.8673
Table 6. LBP and HOG feature combination result.
Table 6 shows the ACC performance of combination be-
tween HOG and LBP. From which we can see that the orig-
inal feature dimension for HOG and LBP were both around
900, ACC performance were between 0.83 - 0.84. By join-
ing HOG and LBP together, we are able to get significant
0.03 ACC boosting. Even though we have doubled feature
dimension after combination, but this performance is still
higher than HOG or LBP alone with similar size. Because
LBP and HOG is using different principles to construct fea-
tures, this kind of combination is able to get diversity gain.
3.4.5 Feature dimension reduction
Consider feature size is important to system speed, a reason-
able prune or feature dimension reduction will be very help-
ful. Especially when we use real-time system, we would
rather lose small performance to have more smooth experi-
ence for users.
The way I did to reduce feature dimension was to only
extract HOG/LBP features from areas around key-points.
Initially I tried several famous key-point detection methods
as following:
Harris: detects corners using HarrisStephens algorithm.
SURF: detect blobs using Speeded-Up Robust Features.
MSER: detect regions using Maximally Stable Extremal
Regions.
Figure 6. Key-point detection result (each person left to right: Har-
ris, SURF, MSER, MSER Region)
Fig.6 shows key-point detection result using Harris,
SURF, MSER. Those detection method will return differ-
ent number of points for different images. Since we are not
using BoW, we need to construct a consistent feature dimen-
sion. I tried different ways to do this like doing K-means,
or select strongest K points out of N key-points. However,
the testing result gives significant ACC loss comparing to
sliding window scheme.
Therefore, instead of using corner or blob key-point de-
tections, which returns mostly different physical positions
in the image. I used a fixed key-point feature extraction,
which extracts fixed number of key-points on face. It turns
5
out we are able to use this method reduces feature dimen-
sion with small performance loss.
Specifically, we use Matlab CascadeObjectDetector sys-
tem object to detect nose on the face. If object returns the
nose position successfully, we select K points evenly round
the circle (with predefined radius) with nose as the center.
If the nose is not detected (CascadeObjectDetector may fail
with the 48 by 48 low resolution image), we simply use the
center of image as the circle center. Here, K can be selected
with different values to get the best trade-off between ACC
performance and complexity.
Figure 7. Circle key-point detection (circle center as nose or image
center, K = 5 and K = 10).
In Fig.7, we show the results of circle based key-point
detection. Matlab CascadeObjectDetector is able to return
nose position on left two images, but failed on two images
on the right side (under which situation we use image center
directly). Also, K = 5 and K = 10 detection are shown
here. After this, we can only extract HOG/LBP features
around those key-points.
K 5 10 15 20
Feature size 20% 40% 60% 80%
ACC Loss 5.6% 2.68% 1.41% 1.16%
Table 7. Key-point based feature reduction performance.
Table 7 shows the testing result with different K values.
We can see that with only 40% of feature dimension, we
are only losing 2.68% of the ACC performance. For real
time systems, people may would rather to satisfy this 2.68%
ACC to get more smooth using experience.
3.4.6 Model Training and Classification
After feature extraction method is selected, we can now test
on different machine learning classifiers. In this project,
I tried different models including: Naive Bayes (NB), K
Nearest Neighbors (KNN), Random Forest and Support
Vector Machine (SVM). Also note that to get better sys-
tem speed, I used same features for gender classification
and emotion analysis. However, we need to choose most
suitable learner for gender and emotion classification.
Naive Bayes: NB method is based on Bayes rule to cal-
culate the probability of each classes. NB also naively as-
sumes the independence of each features.
K Nearest Neighbors: KNN is taking the majority of la-
bels from the K number of nearest neighbors. KNN distance
 Gender Classification Emotion Analysis classification and emotion analysis. The advantage of NB
Test ACC 0.7495 0.3589 is that it’s running super fast, which is the fastest model
among all models, but the ACC is poor with only 0.7495
Table 8. Naive Bayes ACC performance. for gender and 0.3589 for emotion. Note that gender has
higher ACC since it only has two labels while emotion has
Neighbor Number 1 5 10 20 7 labels.
Gender-ACC 0.7446 0.8017 0.8062 0.8190 Table 9 shows the testing result of KNN with different K
Emotion-ACC 0.5306 0.4820 0.4727 0.4583 values. For gender classification, we can see that when K =
20, we get gender ACC of 0.8190. But the boost from K =
Table 9. KNN ACC performance. 10 to K = 20 is relatively small, meaning that the neighbors
ranked 10 to 20 are contributing limited influence. However
Tree Number 20 60 100 300 for emotion analysis, ACC is the best when K = 1, and
Gender-ACC 0.7696 0.8022 0.8102 0.8235 performance is reducing significantly when we increase K
Emotion-ACC 0.4388 0.4859 0.4965 0.5074 value. This means that for emotion analysis data, neighbors
outside the first 1 are introducing more noise than positive
Table 10. Random Forest ACC performance. contributions. Note that comparing with NB method, by
using KNN, we are able to boost gender ACC from 0.7495
C=0.0008 C=0.01 RBF Gaussian to 0.8190, and emotion ACC from 0.3589 to 0.5312.
Gender-ACC 0.8673 0.8608 0.4939 0.4950 Table 10 shows the testing result of random forest, I
Emotion-ACC 0.5089 0.5022 0.2064 0.2017 tested on different tree size. Here, we can see that gender
ACC is increasing from 0.7696 all the way to 0.8235 when
Table 11. SVM ACC performance. tree number is 300. Emotion ACC increases from 0.4388
to 0.5074. Also we notice that from tree size of 100 to 300
Pruned 0 1 2 3 the improvement is relatively small, which means the model
SVM-ACC 0.5672 0.5205 0.5911 0.4752 has almost converged. It has been proved that random for-
Pruned 4 5 6 est is able to prevent over-fitting, thus bigger number of tree
SVM-ACC (s) 0.6073 0.5207 0.5558 size should converge to some value. A reasonable tree size
should be chosen to get the best trade-off between perfor-
Table 12. SVM ACC with class selection. mance and complexity.
Table 11 shows ACC performance of SVM. It turns out
Classifier NB KNN Random Forest SVM that SVM is having less improvement in emotion analy-
Gender Time (s) 1.71 63.3 57.1 170.8 sis than gender classification, this mainly because emotion
Emotion Time (s) 4.12 103.2 231.2 457.2 analysis itself is not a binary classification problem as gen-
der. Also, parameter C value seems not influencing ACC
Table 13. Time cost for data set training. that much. I also tried different kernels like RBF and Gaus-
sian. In both gender and emotion problem, RBF and Gaus-
sian kernel are performing very badly, which definitely not
can be calculated with different metric like Euclidean dis-
a good kernel choice. Considering there are some over-
tance, Hamming distance, etc. (K value can be tuned for
laps/similarity between different emotions, and some emo-
KNN).
tion type may have negative influence, i.e., cause some false
Random Forest: random forest is an ensemble model
positive to other emotions. I also tested on pruning class la-
based on decision tree, where decision tree trains and tests
bels. Table 12 shows the SVM ACC result when pruning
based on attribute split, and label with leaf node. (Tree num-
different emotions. We can see that with specific prune, we
ber can be tuned.)
are able to boost ACC to more than 0.60.
Support Vector Machine: well-know method to split
Table 13 shows the training time cost for each models,
samples with minimum distance maximized. Matlab also
which indicates the following training time relation: NB <
provides ClassificationECOC classifier to support multi-
KNN < Random Forest < SVM. However, longer train time
class classification with SVM. (C value can be tuned, which
does not necessarily indicates longer test time.
controls overfitting, different kernels can be tried.)
To select the best model, we need to run and tune each of
4. Experimental Setup and Results
the classifiers. Note that for random guess, gender classifi-
cation will have 50% ACC, and emotion analysis will have Note in section 3, I already showed the majority of nu-
14.28% ACC (7 classes in total). merical testing results (like different feature performance,
Table 8 shows NB ACC testing results of both gender different classifier performance) in multiple tables. In this

6
section, we will shown some experiment results in addition Gender Male Female
to that. Precision 0.8745 0.8602
The simulation tool used for this project was mainly Recall 0.8615 0.8733
Matlab. The total .m files number is around 20. I also F1 0.8679 0.8667
used JAVA and Python for some data/image parsing. I have
several main functions to test on BoW features, data pro- Table 14. Precision, recall and F1 of gender classification.
cessing, gender detection, emotion analysis, etc. Also other
helper functions to do feature extraction, classification, etc. Emotion Angry Disgust Fear Happy
Vlfeat tool was also used in order to test on dense SIFT fea- Precision 0.4004 0.9091 0.3730 0.6789
ture. Recall 0.3779 0.1802 0.2754 0.7627
For convenience, I parsed and split images into the fol- F1 0.3888 0.3008 0.3169 0.7183
lowing format: Emotion Sad Surprise Neutral
emotion-train/test Precision 0.3719 0.6982 0.4405
0-Angry Recall 0.3841 0.6041 0.5345
1-Disgust F1 0.3779 0.6477 0.4830
2-Fear
Table 15. Precision, recall and F1 of emotion analysis.
3-Happy
4-Sad
5-Surprise 2). Note it also has very high precision of 0.9091 and very
6-Neutral low recall of 0.1802, which meas it may be difficult for the
gender-train/test system to retrieve “Disgust” from testing image, but once it
Male is marked, with 90.91% probability that will be correct.
Female However, those numerical results are tested on the testing
Here, each of the classes of emotion analysis and gen- data set, whose resolution was intended to be low and face
der classification has its own corresponding folder, which expression was very complex. My feeling when testing on
makes very easy to load images using Matlab command im- real life image or video stream is that, the system is working
ageSet. far better than the numerical performance on testing data set
(shown in section 4.2 and 4.3).
4.1. Class F1 score analysis
4.2. Real Life Image Test
In Section 3, we already showed majority of numerical
The numerical ACC results from section 3 should al-
results including test on feature extraction, feature reduc-
ready be enough to verify the correctness of the system.
tion, different classifier models etc. In this part, I’m going
But to have a more straightforward view of the perfor-
to show how robust the system is on each type of classes.
mance. This subsection shows some test result on images
Instead of using accuracy, precision, recall and F1 score are
and videos.
used (definition can be found in Section 3.2).
Table 14 shows the precision, recall and F1 of gender
classification. We can see that both of male and female type
have very close performance on the three metrics. Thus we
can conclude the system has no bias, and will have very
similar good performance on male and female.
Table 15 shows the precision, recall and F1 of emotion Figure 8. Emotion image-1 testing result.
analysis. Different from table 14, here we notice each of
the class is highly biased. Among them, “Happy” and “Sur-
prise” have the highest F1 score, indicating those two types
of emotions will have the best performance in real life test-
ing. Some emotions have relatively low F1, like “Angry”,
“Fear” and “Sad”, but this is understandable since those
emotions were essentially kind of vague. As shown later
on in section 4.2 and 4.3, we can tolerate some overlapping Figure 9. Emotion image-2 testing result.
among them. In addition, “Disgust” has the lowest F1 score,
this is also reasonable since we have much less image train- In order to test the emotion classification result, I found
ing data for “Disgust” emotion (as shown in Fig.4 and Table several images (Fig.8 and Fig.9) [18] with human faces of

7
various expressions. Also, the image includes people with
different races and ages. Note the five faces of Fig.8 and
Fig.9 were passed to the system inside one image. Hence,
corresponding back to Fig.1, the process will be:
• Face detection module circles out five face box areas.
• Rescale image inside face box. Then feature extraction
generates feature vectors for each face with consistent
dimension.
• Run gender classifier with the provided feature.
• Run emotion classifier with the provided feature.
• Add face box, gender and emotion label to image.
From Fig.8 and Fig. 9 we can see that we actually have a
very good test result on both gender and emotion classifica-
tion. For gender classification, only one of the 10 faces had
error. For emotion analysis, we marked five different emo-
tions labels, which are: Angry, Fear, Surprise, Happy and
Neutral. I would say almost all the emotion classification
results look reasonable to me. However, human being’s face
expression is very complicated. It’s even vague for us some-
times to judge other’s emotion through face expression. For
example, the third face on Fig.9 could be explained as either
“Surprise” or “Happy” by different people.
Figure 10. Classroom image testing result.
In addition, Fig.10 shows an example image result of
classroom students (original image from [19]). Fig.10 in-
cludes scaling and rotation. Different students may have
different face size in the image, depending on their distance
to the camera in 3D coordinates. This issue can be handled
by Viola-Jones detection algorithm. But we do lose some
details of the face with smaller face scale. Another issue is
rotation, we notice that different from Fig.8/9, where every-
one was looking at the camera center directly. None of the
people in Fig.10 is looking at the camera, we also have sev-
eral rotated faces. However, there were also a few training
data comes with rotation, thus We are able to do the correct
prediction in Fig.10. Almost all the emotion prediction in
Fig.10 seems reasonable to me, except one person marked
with “Sad”, this may due to the scaling of face. And the
8
gender classification only has one error in the image, since
the gender features for the person seems kind of vague.
I also tested the system on a lot of my personal images,
my feeling is that the system is giving much better perfor-
mance than the ACC value showed in table of Section 3. For
gender classification, we have quite high ACC here because
people in real lift images mostly have more clear features
than training data. While for emotion analysis, as shown in
Section 4.1. The system is giving much higher F1 score on
emotions like “Happy”, “Surprise”, which are more com-
mon emotions in real life.
4.3. Video Stream Test
In addition to image, I also tested the system on video
stream to see how well it can handle continuous expression
change. Different from image, which is static, video is a
more flexible method to do the gender and emotion testing,
since we can show different expressions and observe how
the system handles those changes.
Figure 11. Emotion samples from video.
We can use Matlab vision.VideoFileReader to get each
of the video frame. We can also use sparse sampling rate,
instead of doing classification on each frame, do it for each
of the N frames. This may lose some resolution of classifi-
cation but definitely be able to speed up the system.
After training from section 3.4.6, classifiers are ready to
use (which can be saved as .mat format). Thus in real sys-
tem testing, we can load classifier into memory directly, no
training needed.
I recorded a video with length of 94 frames in total,
which lasts for 15 seconds. Fig.11 shows 6 sample frames
from the video stream. Each of the sample has different
emotion type. The result looks quite reasonable to me. One
interesting thing we notice is face box excludes mouth un-
der some situations (like “Fear” and “Surprise”), but we are
still able to get the correct prediction, indicating the system
is robust. Again, emotion prediction itself is kind of sub-
jective or vague in real life. It seems the major difference
between “Angry” image and “Sad” is on the mouth feature,
but both predictions are reasonable to me.
Figure 12. Emotion distribution during video sampling time.
Figure 13. Gender distribution during video sampling time.
Instead just focusing on one single image, a more inter-
esting or useful analysis would be doing this on entire video
stream time. Fig.12 shows the emotion distribution on the
94 frames of this 15 seconds video. We can clearly see the
proportion of each type of emotions. In this demo video,
I was “Angry” for 6% of the time, never been “Disgust”,
“Fear” for 25%, “Happy” for 15%, “Sad” for 10%, “Sur-
prise” for 28% and “Neutral” for 16%. We notice that emo-
tion “Disgust” never appears in this video stream, this is be-
cause training data for “Disgust” was significantly smaller
than others (based on Table 2 and Figure 4). Also from 4.1,
we can see “Disgust” has a very low recall, which means
it’s relatively difficult recognize this emotion from image,
but once it’s recognized, it will mostly be correct (based on
the high precision from 4.1). System detects 6 types of emo-
tions in this short video because I was changing my expres-
sion frequently on purpose. In reality, this kind of emotion
distribution maybe useful to evaluate the quality of a class.
Fig.13 shows the gender prediction distribution during
the video time. From which we can see that most of the time
(93%) the system is able to make prediction with correct
9
gender. The gender prediction engine seems quite accuracy.
It should also be very robust since I did a lot of exaggerated
face expressions during this video. Otherwise we should
look forward to even higher accuracy.
5. Conclusion and Future Work
In this project, we touched topics like face detection,
gender classification and emotion analysis. Different fea-
ture extraction method like BoW, LBP and HOG were
tested. Key-point detection based feature dimension reduc-
tion was considered to reduce complexity. Multiple classi-
fiers like NB, KNN, Random Forest and SVM were tested.
Parameters for each model were tuned to get the best per-
formance. Numerical results indicate that we are able to
get 0.8673 accuracy on gender classification and 0.5089 on
emotion analysis (0.6073 when we prune particular class).
Further analysis was done on precision, recall and F1 for
each classes. Testing on real life images and video stream
also demonstrates the validity of the system.
Personally speaking, this is a very exciting project,
which lets me familiar with different vision algorithms and
how to connect them with machine learning tools. Being
able to develop a demo system that can be used immedi-
ately is very interesting. I had a lot of fun to test on my
different personal pictures and photos.
ACC on emotion analysis is a part that can be improved
in future work. Also current emotion analysis has biased
performance on different type of emotions. Introducing
deep learning concept should be very help to improve this
multiclass problem.
Also, to be better used in real life scenarios like class
quality analysis. More information can be collected, like
human poses, age information, human recognition etc. A
good model to use the collected information generate an
overall summary score (like group analysis) will also be
very interesting.
Code link: Follow this link.
Code link also submitted through Google Form.
References
[1] Yang M H, Kriegman D J, Ahuja N. Detecting faces in images:
A survey[J]. Pattern Analysis and Machine Intelligence, IEEE
Transactions on, 2002, 24(1): 34-58.
[2] Zhang C, Zhang Z. A survey of recent advances in face detec-
tion[J]. 2010.
[3] Viola P, Jones M. Rapid object detection using a boosted
cascade of simple features[C]//Computer Vision and Pattern
Recognition, 2001. CVPR 2001. Proceedings of the 2001
IEEE Computer Society Conference on. IEEE, 2001, 1: I-511-
I-518 vol. 1.
[4] Mkinen E, Raisamo R. An experimental comparison of gender
classification methods[J]. Pattern Recognition Letters, 2008,
29(10): 1544-1556.
[5] Lian H C, Lu B L. Multi-view gender classification using local
binary patterns and support vector machines[M]//Advances
in Neural Networks-ISNN 2006. Springer Berlin Heidelberg,
2006: 202-209.
[6] Baluja S, Rowley H A. Boosting sex identification perfor-
mance[J]. International Journal of computer vision, 2007,
71(1): 111-119.
[7] Saatci Y, Town C. Cascaded classification of gender and facial
expression using active appearance models[C]//Automatic
Face and Gesture Recognition, 2006. FGR 2006. 7th Inter-
national Conference on. IEEE, 2006: 393-398.
[8] Kim Y, Lee H, Provost E M. Deep learning for robust feature
generation in audiovisual emotion recognition[C]//Acoustics,
Speech and Signal Processing (ICASSP), 2013 IEEE Interna-
tional Conference on. IEEE, 2013: 3687-3691.
[9] Fasel B, Luettin J. Automatic facial expression analysis: a sur-
vey[J]. Pattern recognition, 2003, 36(1): 259-275.
[10] Gallagher A, Chen T. Understanding images of groups of
people[C]//Computer Vision and Pattern Recognition, 2009.
CVPR 2009. IEEE Conference on. IEEE, 2009: 256-263.
[11] https://www.kaggle.com/c/challenges-in-representation-
learning-facial-expression-recognition-challenge/data.
[12] https://en.wikipedia.org/wiki/Accuracy and precision.
[13] https://en.wikipedia.org/wiki/F1 score.
[14] Dalal N, Triggs B. Histograms of oriented gradients for hu-
man detection[C]//Computer Vision and Pattern Recognition,
2005. CVPR 2005. IEEE Computer Society Conference on.
IEEE, 2005, 1: 886-893.
[15] http://www.mathworks.com/help/vision/ref/extracthogfeatures.html
[16] Ahonen T, Hadid A, Pietikainen M. Face description with
local binary patterns: Application to face recognition[J]. Pat-
tern Analysis and Machine Intelligence, IEEE Transactions
on, 2006, 28(12): 2037-2041.
[17] Wang X, Han T X, Yan S. An HOG-LBP human detector
with partial occlusion handling[C]//Computer Vision, 2009
IEEE 12th International Conference on. IEEE, 2009: 32-39.
[18] http://www.shutterstock.com/s/emotions/search.html
[19] https://jonmatthewlewis.wordpress.com/2012/02/24/posture-
and-gestures-in-the-classroom-and-on-the-date/

10
 Computer Vision for Food Cost Classification
Dash Bodington
Stanford University
dashb@stanford.edu
Abstract
This project aims to use various computer vision meth-
ods for the novel task of restaurant cost prediction based
on food images. It uses the Yelp dataset, a dataset of
200,000 images tagged by business, which is significantly
pared down to near 15,000 relevant images tagged as food,
and labeled by the cost rating of the business they come
from. These food images are partitioned into training and
testing datasets, and the test dataset has a uniform sample
distribution over the classes: cheap ($-$$ on Yelp), and ex-
pensive ($$$-$$$$ on Yelp).
Once the dataset is defined, several techniques are
trained and applied to make class predictions on the test
dataset. The prediction model is broken into feature extrac-
tion and classification, and significant exploration was done
to come up with optimal features and classifiers for the spe-
cific task. The primary goal of this project was to achieve
the highest possible classification accuracy on the reserved
test dataset.
The four main feature extractors used were: simple pre-
processing (crop and resize images to uniform dimensions),
color histograms (a normalized distribution of color inten-
sity at all pixels), SIFT bag of words (using normalized dis-
tributions of SURF feature common descriptors over each
image), and deep neural network features (using the trained
Alexnet to extract image features).
Each feature extractor was tested with each of the fol-
lowing classification methods: K-Nearest Neighbor (KNN),
Support Vector Machine (SVM), Naive Bayes, and a custom
neural network with softmax loss.
The best results were achieved from the combination of
Alexnet features and color histograms, independently com-
pressed using PCA, and classified with a shallow neural
network, which acheived 70% accuracy on the test set while
most other methods achieved from 50% - 78% accuracy.
1. Introduction
Since various deep learning methods have shown very
impressive results in classification, detection, and localiza-
1
tion, there has been a great deal of focus on understanding
images beyond their still content, mostly in the form of gen-
erating descriptive sentences, and using semantic analysis
to understand language [6, 7] to generate these descriptions.
While generating descriptions is easy for a human to eval-
uate and judge a perceived intelligence, it is only a subset
of what can be understood beyond image content through
computer vision.
This project aims to tackle a subset of beyond-content
image understanding through food images and restaurant
ratings from the Yelp dataset. The dataset contains 200,000
images tagged with descriptors and the businesses they
come from, and businesses are give a cost rating from $
to $$$$. Because of the small number of restaurants in
the two most expensive classes, these cost ratings are com-
pressed into 1-2 $ and 3-4 $, hereafter referred to as cheap
or $ and expensive or $$ By filtering images by ’food’ tag-
ging, and assigning the business cost rating to the image, a
dataset containing food photos and an estimated cost rank-
ing, or fanciness ranking is created. Many computer vision
approaches can be used in an attempt to understand these
food images beyond their straightforward content. The sim-
ple goal of this project is to train algorithms on the training
data, and maximize accuracy on the test dataset.
Of the possible approaches to this problem, this project
focuses on predictive models divided into feature extractors
and classifiers, and explores several options and combina-
tions between the two model parts. Four primary feature
extractors used were: simple preprocessing (crop and resize
images to uniform dimensions), color histograms (a nor-
malized distribution of color intensity at all pixels), SURF
bag of words (using normalized distributions of SIFT fea-
ture common descriptors over each image), and deep neural
network features (using the trained Alexnet to extract image
features). Each feature extractor was tested with each of the
following four classification methods: K-Nearest Neighbor
(KNN), Support Vector Machine (SVM), Naive Bayes, and
a custom neural network.
Very little work similar to this project has been done, but
the project relies heavily on more general, existing com-
puter vision tools and research. The remainder of this re-
port will discuss the dataset, models, and results, and con-
clusions of this project in detail.
2. Implementation
2.1. Experimental Setup
The implementation of this project was completely com-
putational, and involved dataset extraction, data preprocess-
ing, feature extraction, and classification. All code for this
project was written in python 2.7, opencv was used for most
image processing [2], tensorflow [1] was used to write all
neural networks, and scikit provided implementations of
some other classifiers[5].
The project was run on a fast desktop computer. All neu-
ral network models were run on an Nvidia GTX 780 GPU,
and the remainder of the processing was done on a 4.4GHz
quad-core CPU. Even with reasonably fast hardware, fea-
ture extraction was very time consuming, so it was often
done only once, and the features were cached to be used
on-demand.
2.2. Dataset Extraction
Though this project uses a nicely formatted and labeled
dataset, the subset required for training and testing comes
from several extraction steps. The Entire Yelp dataset con-
tains 200,000 color images from business’ Yelp pages. Im-
ages can be uploaded by either businesses or customers, and
are of varying quality and size. These images are tagged
by Yelp’s own computer vision algorithms, and can be cor-
rected by users, as there are sometimes errors in tagging.
Initially, the image database is analyzed, and all images
tagged as ’food’ are extracted along with the ’id’ of the busi-
ness they come from. All images from the same business
are grouped, and are then labeled with the consolidated cost
label (mapped from Yelp’s $ - $$$$ to the binary $ or $$
for this project) of the business. If a business has not been
labeled with a cost rating, which happens infrequently, the
corresponding images are discarded. Next the businesses
ids are binned by their attached cost ratings, each bin is
shuffled, and a predetermined train/test ratio (0.7 in this
case) is used to assign each business id to the training set
or test set. This binning by business and cost rating is used
to ensure that there is a similar distribution of data in the
test and training datasets, and to avoid placing images of
the same item from the same business into the training and
test dataset, which could be considered as mixing the two.
In total, there are 11,530 images available for training, and
4,412 images available for testing, though these counts de-
crease depending on the desired training and testing distri-
butions.
In the test dataset, images in each class are shuffled, and
images are discarded from the class with more images un-
til exacly 50% of the images in the test set are from each
class. Enforcing this 50% distribution cuts the dataset ap-
proximately in half because of the relative rarity of expen-
sive restaurants.
After the training and test datasets are fully defined, each
image is cropped from the center to the largest square area
possible, and is resized to a variable size, depending on the
feature extractor used (sizes range from 64x64 to 227x227).
With this definition of the training and test datasets, there
are several further processing steps which are sometimes
used on the training set to improve training and perfor-
mance.
• For validation or cross validation, which is only used
when tuning feature or classifier parameters, or train-
ing neural networks, 20% of the training dataset is ran-
domly designated as the validation set. All models ex-
cept neural networks are trained on the whole training
dataset before testing.
• Depending on the classifier and loss function being
used, the training set will usually have images dis-
carded in the same fashion as the test dataset to even
the distribution of images across classes.
2.3. Feature Extraction
Several feature extractors were used in this project as in-
puts to the various classification systems.
2.3.1 Images as Features
In some cases, preprocessed (scaled and cropped) images
were used as features themselves. This method is usually
most appropriate for input into a convolutional neural net-
work classifier, which essentially defines its own features
internally, but can also be used with other classifiers with
varying success. These features use images of size 128x128
or 64x64, which are vectorized unless the classifier is a con-
vnet.
2.3.2 Color Histogram Features
As an initial step beyond images as features, it was thought
that the colors in an image could give an indication about
the cost of food. This feature consists of a length 30, nor-
malized (sum 1) vector, which contains a 10-bin histogram
for each color channel. These features use images of size
128x128 or 64x64 as inputs.
2.3.3 SIFT Bag of Words Features
SIFT Bag of Words features consist of frequency distribu-
tions of common feature descriptors. During training, a set
of SIFT descriptors from all training images is extracted,
and N (range: 20-100) ’words’ are extracted with the K-
means algorithm [2, 4]. Next, during training and testing,
each image’s feature vector is calculated as a normalized
N-length histogram of the words, where each feature de-
scriptor from the image is assigned to one word with the
nearest neighbor algorithm. These features use images of
size 128x128 as inputs.
2.3.4 Alexnet Features
Alexnet is a 13-layer pretrained convolutional neural net-
work which previously achieved state of the art perfor-
mance in the Imagenet large-scale image classification chal-
lenge [3]. Originally, the network’s output was a 1000-class
softmax layer, but because the network learns very useful
features for other tasks, swapping the final layers of the net-
work for custom-trained layers is a common practice, espe-
cially for those working without the computational power or
large data volume required to train a model with similar per-
formance from scratch. For this project, ’Alexnet Features’
are considered to be the output of one of the final layers of
the pretrained network.
With Alexnet, multiple feature sets were created from
multiple layers of the convnet. fc8, fc7, and fc6 (the final
fully-connected layers of the network). Because these fea-
tures are effectively sparse (we are feeding the network a
very small subset of the images it was trained to classify),
and very large in size, PCA is often used to reduce the fea-
ture dimensionality before training.
This network requires image inputs of size 227x227,
which are also mean-subtracted.
Using the GPU allows for a significant speedup in fea-
ture generation, over 70x faster than CPU computation on
the hardware for this project, but is not enough to train a net-
work like Alexnet from scratch with limited time and data.
2.4. Classifiers
Because of the many feature inputs used in this project,
classifiers with different properties and strengths are used to
increase the likelihood of good performance with each fea-
ture set. Feature vector lengths range from 30 (Color His-
togram) to 12,288 (cropped and rescaled images), so mod-
els which may overfit in some cases, may perform well with
fewer features.
2.4.1 K - Nearest Neighbor
The K-Nearest Neighbor classifier archives the entire train-
ing feature set, and at prediction time, calculates the eu-
clidean distance from the input feature vector, and makes
a prediction based on a majority vote from the labels of
the K closest training examples (K = 20 for this project).
While storage-inefficient, it is one of the simplest classi-
fiers, and would perform well if images in the training and
testing dataset were similar enough, but fails to generalize
otherwise.
2.4.2 Naive Bayes
Naive Bayes classification assumes feature indepen-
dence and Gaussian probability distribution of fea-
tures to make a maximum likelihood estimate ŷ =
argmaxy P (y)ΠP (y|xi ) of the class, and usually can per-
form well on small training datasets because it has few pa-
rameters.
2.4.3 Support Vector Machine (SVM)
The SVM is a linear classifier which defines a class-dividing
hyperplane f (x) = B0 + B T x, which minimizes 21 ||B||2
subject to y(B0 + B T x) > 1 on the training set. Generally,
SVMs have the advantage that they are less likely to overfit
than other methods because of build-in regularization.
2.4.4 Neural Networks
While neural networks are the most general classifier used
in this project, they are also the most difficult to tune, and
it requires a great deal of data to train networks with many
neurons. Neural networks are a layered structure of inter-
connected linear and nonlinear operations whose parame-
ters can be learned with various gradient descent methods.
In this project, the classification networks presented have
zero or one hidden layers, and all layers are fully connected.
Each fully connected hidden layer (when present) is fol-
lowed by a Rectified Linear Unit (ReLU), and the final layer
is a softmax layer which takes two inputs and computes
pseudo-probabilities for the input on each class according
to
exi
p̂(xi ) = P xi
e
.
Though neural networks have been responsible for many
recent state of the art results in computer vision, they
are among the most difficult models to manage on small
datasets. Because of this, L2 regularization is sometimes
added to the iterative minimization of the cross-entropy loss
X
Loss = − p(xi )log(q(xi )) + λR(W eights)
where q contains the outputs of the softmax layer, and p is
the one-hot label vector for the training sample x. Other
training tricks, such as dropout and projection of sparse
feature vectors into lower dimensions with PCA are also
attempted to increase robustness and decrease overfitting.
Batch gradient descent with momentum was used to train
networks for this project because it provided reliably con-
verging results, especially when changing the class distri-
bution (and size) of the training set.
 Figure 2. These three images of cheesecake, chocolate dessert, and
steak, are the images with the highest estimated probability of be-
ing expensive by the neural network.

Figure 3. These three images of pizza, a taco, and a sandwich, are

the images with the lowest estimated probability of being expen-
Figure 1. Example of classification neural network. A ReLU is
sive by the neural network.
applied to the hidden layer, and a softmax normalization is applied
to the output.

3.2.1 Details of Best Model

3. Results The best performing model, ’multifeature,’ which achieved
an accuracy of 70% on the test dataset was a multifeature
Though estimating restaurant cost-class from food im- model with Alexnet features, color histogram features, and
ages is a difficult problem, reasonably good results are tab- a neural network classifier.
ulated and discussed below. For this model, 1000-dimensional Alexnet features were
extracted from the 8th fully connected layer of the network
3.1. Accuracy (the layer closest to the original softmax layer) and com-
pressed to 100 dimensions and whitened with PCA. Then,
whitened color histogram features were concatenated with
the Alexnet features, leaving feature vectors of length 130
K-NN NB SVM NN
for the classifier.
Images 0.56 0.60 0.50 The classifier in this model was a fully connected neu-
Color Histogram 0.57 0.63 0.54 0.56 ral network with a hidden layer of size 50 (see Figure 1).
SIFT + BoW 0.59 0.62 0.60 0.60 This model was trained for 10,000 iterations of batches of
Alexnet 0.62 0.65 0.66 0.68 200 images, reporting validation error every 500 iterations.
Multifeature 0.55 0.65 0.68 0.70 After full training, the model from the iteration with the
highest validation accuracy was used for testing.
Table 1. Multifeature methods (Alexnet fc8 and color historgrams)
with a neural network classifier perform the best on the test set.
3.2.2 Unsuccessful Models
In addition to the previously presented models, many other
3.2. Example Classifications less successful attempts were made to solve this task. Some
simply performed poorly, were not unique, or were compu-
Though accuracy results are the goal, actual examples of tationally unfeasible.
results provide a more intuitive look at classification. We
can see that desserts appeared frequently in the $$ class, • Convolutional neural networks were implemented, but
which is not surprising, considering that they are often con- failed to train or generalize well. It is thought that with
sidered to be a luxury item, and $ -classified foods are more the large image size required (greater than 128x128) to
everyday items. resolve objects in detail in these images, the training
 dataset was not large enough to train the large filters
also required. At small image sizes (64x63-128x128)
with smaller filters, models did not generalize, likely
due to the lack of recognition at such a pixelized scale.
• Other faster feature extractors were also tested for the
bag of words method, such as SURF and BRIEF, but
since SIFT features were the best performing, only
they were presented.
• Linear Discriminant Analysis was used for classifica-
tion as well, but performed similarly to the SVM, so
the decision was made to focus on fewer, more unique,
classifiers.
• Many attempts were made to train models on imbal-
anced datasets in order to have more training data,
however, with the imbalanced nature of the data (with
75% of the images in the $ class), many models simply
learned to predict $ consistently, since this would lead
to a higher accuracy than varied predictions.
• To combat the problem above, various loss functions
and update methods were tried, such as weighting each
misclassification by (1 - the prior probability), and
only performing updates for misclassified images. Ul-
timately, training on an even dataset was the most suc-
cessful.
4. Conclusion
High performance in abstract computer vision tasks like
this one is very difficult to achieve, which was a lesson
learned during this project. Though accuracy of 70%, sig-
nificantly above random (50%) was achieved, higher per-
formance would certainly be necessary before using a pre-
diction system like this for an unsupervised application.
4.1. Challenges
The greatest challenge in achieving good performance in
this task seemed to be working with unbalanced or small
datasets. Though near 15,000 food images were available,
the rarity of expensive restaurants made training difficult,
and discarding images to even the datasets led to a signifi-
cantly smaller volume for training, though results improved.
Beyond data problems, there may also be issues with the
labeling of the dataset. Yelp uses their own neural network
to tag images, so some images which were tagged as food
(such as salt and pepper shakers, and a hotel key), were not
food. Though the frequency of this was small, it may have
had an effect on performance. Additionally, the $ rating
system is not perfect for this task. A Yelp $$$ restaurant in
an expensive location could serve similar dishes to a Yelp
$$ restaurant elsewhere, so there is certainly some blurring
of the data across the class boundary, unlike most standard
classification tasks.
4.2. Future Work
Moving forward, it is possible that discarding one of the
middle classes $$ or $$$ from the Yelp dataset would create
a set which is better divided between classes. If a dataset of
photos and menu prices were available, it would be inter-
esting to combine this cost category recognition with dish
recognition to allow for prediction of actual dish prices.
70 % accuracy is satisfactory given the time and data
constraints of the project, and every effort was made to
raise this accuracy as high as possible. With more time and
data, it would be interesting to train end-to-end convolu-
tional neural network models, or focus more on fine-tuning
state of the art models such as Alexnet or Google’s Incep-
tion. As the state of the art in computer vision progresses,
it is likely that tools more suited for this problem will arise,
but for now, the best results of 70% have been achieved with
two neural networks in combination with hand-written fea-
tures.
4.3. Replicating Results
Code for this project can be found at the author’s Github
(github.com/DashBodington/cs231aProject), and the Yelp
Dataset is available from Yelp. Code is written in Python
2.7, and requires Tensorflow, OpenCV with the feature ex-
tractors in opencv contrib, NumPy/SciPy, and scikit-learn.
The GPU implementation of Tensorflow is strongly recom-
mended, and some CPU feature extraction methods may
take several hours on the full dataset.
References
[1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen,
C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghe-
mawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia,
R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané,
R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster,
J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker,
V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. War-
den, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. Ten-
sorFlow: Large-scale machine learning on heterogeneous sys-
tems, 2015. Software available from tensorflow.org. 2
[2] G. Bradski. Dr. Dobb’s Journal of Software Tools. 2
[3] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger,
editors, Advances in Neural Information Processing Systems
25, pages 1097–1105. Curran Associates, Inc., 2012. 3
[4] D. G. Lowe. Distinctive image features from scale-
invariant keypoints. International Journal of Computer Vi-
sion, 60(2):91–110, 2004. 2
[5] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel,
B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss,
V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau,
M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Ma-
 chine learning in Python. Journal of Machine Learning Re-
search, 12:2825–2830, 2011. 2
[6] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,
Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg,
and L. Fei-Fei. ImageNet Large Scale Visual Recognition
Challenge. International Journal of Computer Vision (IJCV),
115(3):211–252, 2015. 1
[7] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and
tell: A neural image caption generator. In The IEEE Confer-
ence on Computer Vision and Pattern Recognition (CVPR),
June 2015. 1
 Using Computer Vision to Solve Jigsaw Puzzles
Travis V. Allen
Stanford University
CS231A, Spring 2016
tallen1@stanford.edu
Abstract
Many in the computer vision community have written
papers on solving the “problem” of jigsaw puzzles for
decades. Many of the former papers written cite the po-
tential extension of this work to other more serious endeav-
ors like reconstructing archaeological artifacts or fitting to-
gether scanned objects while others simply do it because
it’s an interesting application of computer vision concepts.
This author falls in the latter camp. Several methods for
constructing jigsaw puzzles from images of the pieces were
considered from a theoretical standpoint before the comput-
ing power and the high-resolution image capturing devices
necessary to employ these methods could be fully realized.
More recently, many algorithms and methods tend toward
disregarding piece shape as a discriminant entirely by us-
ing “square pieces” and rely instead on the underlying im-
age properties to find a solution. The jigsaw puzzle solver
described in this paper falls somewhere in between these
two extremes. The author of this project paper describes
the creation of an “automatic” jigsaw puzzle solving pro-
gram that relies on multiple concepts from computer vision
as well as past work in the area to assemble puzzles from
a single image of the disassembled pieces. While the pro-
gram is currently specifically tailored to solve rectangular
puzzles with “canonical” puzzle pieces, concepts learned
from this work can be used in concert with other computer
vision advances to enhance the puzzle solver and make it
more robust to varying piece and puzzle shape. The puzzle
solver created for this purpose is fairly unique in that it uses
a picture of the disassembled pieces as input, no reference
to the original puzzle image, and is done using the Mat-
lab Image Processing Toolbox. The solver created for this
project was successfully used on five separate puzzles with
different rectangular bounds and dissimilar puzzle images.
These results are similar to others who have created simi-
lar puzzle solvers in the past. Ultimately, the author hopes
that this work could lay the groundwork for a smart phone
application.
1
1. Introduction
This project focuses on the creation of an end-to-end
“automatic” jigsaw puzzle solving program that assembles
a jigsaw puzzle using an image of the disassembled puzzle
pieces. The solver does not use the picture of the assembled
puzzle for matching purposes. Rather, the solver created for
this project attempts to solve the lost the box conundrum by
displaying the assembled puzzle to the user without the need
for the reference image. Needless to say, this program was
created with an eye toward a potential smart phone applica-
tion in the future. It should be noted that the puzzle solver
is created entirely in Matlab and heavily utilizes functions
found in the Matlab Image Processing Toolbox.
Solving jigsaw puzzles using computer vision (CV) is an
attractive problem as it benefits directly from many of the
advances in the field while still proving to be both challeng-
ing and intellectually stimulating. Due to time constraints,
manpower limitations, and the fact that I had little to no
prior experience with many of these concepts and the Mat-
lab Image Processing Toolbox when beginning this project,
several assumptions about the problem were made up front
in order to make it simple enough to solve in the time given.
The primary assumptions and limitations are as follows:
1. The pieces in the source image do not overlap nor do
they touch.
2. The source image is captured in a “top-down” manner
with minimal perspective distortion of the pieces.
3. The pieces in the source image comprise one entire
puzzle solution (you can not mix and match pieces
from other puzzles).
4. The final puzzle is rectangular in shape and all pieces
fit neatly into a grid.
5. The pieces of the puzzle are standard or “canonical”
in shape – this means they are square with 4 distinct,
sharp corners and four resulting sides.
 6. All intersections of pieces in the puzzle will be at the
corners of the puzzle pieces and all internal intersec-
tions will be at the corners of four pieces.
7. Each side is characterized by having a “head,” a “hole,”
or by being “flat.”
In the following paper, I will discuss the previous work
that has been done in this area in section 2, emphasizing
those papers that most influenced the methodology I fol-
lowed for my puzzle solver. I will then describe my tech-
nical approach to the problem and how my puzzle solver
works in section 3. In section 4 I will show some of the
results obtained with the puzzle solver so far and discuss
both the experimentation that has been conducted and ar-
eas where more experimentation could occur. Finally, I will
wrap things up in a conclusion in section 5.
2. Related Work
As one can imagine, such a problem as solving jigsaw
puzzles might attract a good number of people in the com-
puter vision community. And indeed it has. The problem
has been considered for decades, going back to H. Free-
man, and L. Gardner [2] in 1964. They first looked at how
to solve puzzles with shape alone. Then there’s H. Wolf-
son et al. [4] who describes the very matching methodology
that I use. He is able to assemble a 104 piece puzzle us-
ing his method in 1988. While I do not solve a 104 piece
puzzle, their solution requires individual pictures of each
piece, though he does solve it with shape alone. One area
where his method is different than mine is that it can handle
two intermixed puzzles, whereas mine can only handle the
pieces from one at present. D. Goldberg et al. [3] expanded
on Wolfson’s work and developed an even more global ap-
proach to solving jigsaw puzzles – their method allowed for
the solution of puzzles that did not necessarily intersect at
corners. My inspiration for color matching across bound-
aries comes from D. Kosiba et al. [5] who propose methods
for using color in addition to shape in 1994.
Some insight into how to use Matlab to help solve this
problem came from some detailed student papers that were
found with a Google search. A. Mahdi [8] from the Univer-
sity of Amsterdam and N. Kumbla [6] from the University
of Maryland both attempt the problem using methods sim-
ilar to what I end up using, though neither ends up with
quite full and satisfactory solutions and both rely on high
resolution and highly controlled inputs. Finally, my inspi-
ration for creating a smart phone application comes from
L. Liang and Z. Liu [7] from Stanford who do not use the
same matching methodology (they use the actual image of
the fully constructed puzzle and SURF/RANSAC), but who
do try to implement their solution in a real-time smart phone
application. This is a possibility in the future for my puzzle
solver.
2
Lastly, a former student of CS231A, Jordan Davidson
[1], did his project in this very area, though it was in a
slightly different vein. He looked at a genetic algorithm that
could solve large “jigsaw” puzzles with square pieces that
used the information from the pieces to determine if it had
found the correct match. While the algorithm is interesting
and probably applicable on some level to my puzzle solver,
it was not quite what I was looking to do for this project.
Jordan’s work appears to be in an area that is growing with
others attempting to solve larger puzzles of this kind. Per-
sonally, I wanted to solve the puzzles as they are seen and
manipulated in real life. Most of this area of research has
other applications and was not quite what I was looking to
do. Though, as I said, there is definite applicability of some
of the algorithms to my ultimate solver and future iterations
may look at something like the algorithm explored in Jor-
dan’s paper.
3. Puzzle Solver Technical Approach
Creating a program to construct a puzzle using an im-
age of the pieces requires a number of steps, each of which
can be executed in a number of ways. In this section, I will
describe the methods I used in my final code, but will also
discuss alternatives that were either attempted with subop-
timal results or that were not used but could be in future
iterations.
3.1. Image Capture and Segmentation
In order to capture the pieces to be assembled into a
final puzzle, I used a fairly high resolution camera – a
Canon Rebel T4i DSLR with 18.0 Megapixel resolution.
The pieces were placed face up on an easily segmentable
background (i.e. a “green screen”) with great care taken to
ensure they were not overlapping (see figure 1 for an exam-
ple using the Wookie puzzle). The picture was taken from
a “top-dead-center” position looking straight down onto the
pieces in order to reduce perspective distortion. Lighting
was kept as neutral as possible with consideration given to
sources of glare and to the possible disproportionate light-
ing of some pieces over others. With more time, the abil-
ity to compensate for off-axis image capture of the pieces
(i.e. rectification to the ground plane) could be built into the
code, though that was not explored for the current incarna-
tion of the solver.
As discussed earlier, with an eye toward an eventual
smart phone application, the first step I take is to signifi-
cantly reduce the resolution of the input image from that of
the original in order to shrink the memory burden and make
the resulting image more comparable to one that might be
obtained with a smart phone. Once I have resized the image
(960x1440 was the main resolution used for the test cases),
I use a Gaussian filter (with σ = 1) to blend the edges prior
to the actual segmentation.
 Figure 1. Input Image for the Wookie Puzzle
Since segmentation can be done in a multitude of ways,
the approach I use is to put the pieces on a solid green screen
background and segment based on color. This has been
done for ease and efficiency since segmentation is not the
entire focus of the project. However, as with the additional
adaptation of the code to varying image capture angles, it
would be entirely possible to segment the image using other
means, such as the mean shift or normalized-cut methods
used in class. These other methods have the potential to
make the code more robust to varying backgrounds, such
as wood table tops or off-axis image capture. These meth-
ods were considered, but were not completed since the puz-
zle assembly was considered more important to the overall
program completion.
The primary result of the segmentation is a binary mask
that we use to obtain the location and extent of the individ-
ual pieces using the bwconncomp and bwregionprops
Matlab functions. There are provisions in the code for
cleaning up somewhat imperfect masks that may have re-
sulted from additional specks of dirt blocking the green
background or when the green background does not fill the
entire image and there is a segment along an edge where the
true background (i.e. the floor or table) peeks through.
3.2. Puzzle Piece Characterization
Once the pieces have been segmented from the back-
ground and a binary mask created, the individual pieces are
broken down further and many traits are extracted that will
be used in later matching.
The first step is to define the corners. As per the as-
sumptions about the puzzle pieces used, each piece has four,
well-defined corners with an angle approximating 90 de-
grees. Several approaches for extracting either the corners
or the sides were attempted before ultimately landing on
the method used in the puzzle solver. One of the meth-
ods involved using the Hough transform to find the pre-
dominant direction of the points along the border while an-
other found in [5] involved the conversion of the borders
3
Figure 2. Polar Plot of Boundary for One Puzzle Piece
XY-coordinates to polar coordinates and then using this in-
formation to find the predominant direction of the points,
taking the average of the intersection, and then taking the
closest point on the boundary. While these methods may
have eventually been adapted for my purposes, I had is-
sues getting them to work consistently and instead landed
on a method based on the one presented in [6] where I con-
vert the border points to polar coordinates and use the Mat-
lab function imregionalmax to find peaks in the data
(other matlab functions were tried, but did not return as fa-
vorable of results). See Figure 2 for a visualization of what
is returned for an individual piece. Unfortunately, since the
border is somewhat noisy, there are many more peaks than
actual corners or heads. Thus I wrote a script that steps
through the peaks, consolidates the multiples to the true
peak in the θ region, and tries to discriminate whether the
peak is a corner or a head and, if a head, eliminates the peak.
The script then goes in and uses the remaining peaks to de-
termine the corners, which are two sets of peaks separated
by 180 degrees that alternate.
Once the four corners of the piece are found, the sides of
the piece are simply taken to be the points along the bound-
ary between each set of two corners. The next most signifi-
cant task is to assign each side of each piece as a head (+1),
a hole (-1), or flat (0), see Figure 3 for a visual. This can
be done in many ways. The original method I used would
take the points along the edge in XY-coordinates, re-align
them such that the two corners each lie on the x-axis, and
then take the integral. If there was significant area either
above or below the axis created by the two corner points,
then the side could be declared either a head or hole, other-
wise it was considered flat. However, there were issues with
this method that were brought to light when experimenting
with one of the puzzles. Specifically, there was difficulty
setting a specific pixel integral threshold for all of the sides
when the piece was more rectangular than square. One so-

Figure 3. Example Piece with Labeled Sides and Example Color
Patches
lution to this problem was to divide the resulting integral by
the length of the side, but even that did not completely solve
the issue. So instead I developed a new method that uses the
“height” of the midpoint of the side relative to the x-axis as
defined by the rotated side. If the height is beyond a certain
threshold, it could be considered a head or hole. Otherwise
it is considered a flat side.
Once the sides of the piece are determined and the infor-
mation stored, the next step is to gather color information
about the piece that will be used in the matching process.
By using the mask created by the individual piece, I can
use Matlab functions to back out intensity levels for indi-
vidual color channels for the entire piece. At first I grabbed
a lot of information, and much of it is still in the code. This
includes average, minimum, and maximum intensity levels
for each of the RGB color channels as well as each of the
HSV color channels. After rather exhaustive experimen-
tation with the matching algorithm, however, most of this
information proved to be hit or miss when it came to true
color coordination across pieces. Since this is information
about the entire piece, it is primarily used to find regional
likeness in the final puzzle image. After doing some re-
search, one method for finding how similar two colors are
in the spectrum is by calculating the ∆E, which requires
the color values to be in CIE’s L*a*b* color space. So, ul-
timately, the color values for each piece are translated to
this color space and the average “intensity” in each of those
channels is found and stored.
Not only do I store color information about the entire
piece, but also color information along each edge. Using a
4
method similar to [5], I identify small patches of pixels (I
found three, 2x2 patches to work well) distributed evenly
along the edge, grab the average L*a*b* color information
in each of the patches, and store it for the matching process
(see figure 3 for an example). Of course, I only do this
along the edges with holes and heads – this is unnecessary
along the flat edges. As with the piece information before,
I also gather the HSV values for each of the patches, but
no longer use them in the matching process after receiving
mixed results.
3.3. Puzzle Assembly
When discussing the puzzle assembly, I have decided to
break the process down first and foremost into “global” ver-
sus “local” assembly, and then break the “global” assem-
bly down further into two distinct areas of the puzzle: the
border pieces and the inner pieces. This is a method that
is discussed and accomplished in most of the papers I re-
viewed, but specifically discussed in [4] and [3] and seems
like a reasonable approach to the problem. It also seems
like a logical approach given how one might go about solv-
ing a jigsaw puzzle in the physical world. As such, I will be
breaking this section into subsections along those lines, be-
ginning with the discussion on how two pieces are matched.
3.3.1 Local Assembly – Piece-to-Piece Matching
I am beginning the discussion on how the puzzle is as-
sembled by describing the method by which the pieces are
matched to one another. While the algorithm works slightly
differently whether we’re assembling the border or we’re
assembling the inner pieces, the specific match parameters
remain the same. I will describe the general case of two
pieces being matched together and then discuss the specific
nuances for each of the two global situations.
What this algorithm is ultimately trying to find is a
“match” between the edge of one piece and the edge of an-
other piece. A “match” is defined as two pieces that “fit”
together. Ideally, the two pieces that “fit” the best would
also be the “correct match,” so one would think that if the
head of one piece is similar to the hole of another piece,
they would fit and we could move on. However, when try-
ing to do this process with little to no human input, finding
the correct match is not as easy as finding two shapes that
are simply the inverse of one another. Especially when one
throws in measurement noise due to imperfect segmentation
and image distortion. So instead, I try to use additional in-
formation about each piece in order to try and compute a
“score” for a potential match.
After extensive experimentation, the factors that two
neighboring pieces appear to have most in common are side
length, color along the edge, and overall piece color. Ad-
ditionally, I looked at the difference between the overlaid
curves along each edge to determine an average overlap (or
gap) – obviously the smaller the overlap or gap, the bet-
ter the match. With these factors in mind, here is how the
matching algorithm works.
Once the algorithm determines the two pieces and the
side of each piece to be matched, it first checks to make
sure one is a head and one is a hole. If not, the match is
discarded. It then compares the side lengths and the in-
tegrals along each edge. If the difference in side lengths
is significant (beyond approximately 10 pixels for the stan-
dard resolution image), the match is discarded. The integral
of the curve along each edge is compared. If they are not
approximately equal and opposite, the match is discarded
(the threshold for this varies based on image resolution –
turns out it is not the best method for weeding out candidate
matches so it is not the most strict). Next, the overlap is cal-
culated by overlaying the two edges in XY-coordinate space
and taking the difference (using pdist). The result is the
overlap, or gap, between the two pieces. If the average,
minimum, or maximum overlap is beyond specific thresh-
olds based on image resolution, then the match is thrown
out. Once we’ve looked at the basic shape discriminants for
determining whether a match is likely correct, we then look
at the color discriminants and determine the ∆E. We do this
both from a regional perspective (piece to piece) and from
an edge perspective (using the patches along the edge). ∆E
is essentially the “distance” between two colors in the color
spectrum and is determined using the following formula:
∆L∗ = L∗1Avg − L∗2Avg (1)
∆a∗ = a∗1Avg − a∗2Avg (2)
∗
∆b = −b∗1Avg b∗2Avg (3)
p border matching, the only true difference being that they
∆E = ∆L∗2 + ∆a∗2 + ∆b∗2 (4)
The ∆E is found for each pair of patches along the edge
and the average of those patch values is used. Obviously,
the lower the ∆E, the closer the colors are in the spectrum.
Once we’ve computed and captured these shape and
color comparisons and have weeded out obviously bad
matches, we then compute a match “score.” This score
is found using experimentally determined weights that are
multiplied by the four key matching criteria: side length
difference, overlap difference, ∆E along the edge, and ∆E
between the pieces. Since these values tend to vary widely
between matches and puzzles, the standard range of values
found for “correct” matches on one of the test puzzles was
used to develop a set of weights that somewhat normalizes
each parameter so that one specific criteria is not favored too
much more than another. For most correct matches, when
multiplied by the weights, the values should be no greater
than 100. This means that, theoretically, a correct match
could have a total score of up to approximately 400. How-
ever, in reality, most correct matches have low scores in at
5
least two of the categories, so a score threshold of about
280 (again, experimentally determined) can be set in order
to weed out incorrect matches.
Once the local matching algorithm has pared down to a
final set of scored matches, it then returns these matches to
the global algorithm in score-priority order. The primary
nuances that differ between local matching for the border
versus the inner pieces is the orientation of the piece. As I
will soon discuss, the border pieces are aligned according
to the flat edge, so the local matching only considers one
potential edge for each piece. And, because the border is
being matched in the absence of the rest of the puzzle, the
local matching algorithm does not have to consider any ad-
ditional sides from other pieces that may come into play.
Not so for the inner pieces. For the inner portion of the puz-
zle, a piece is being matched to a slot in a grid, and that
slot has neighboring sides. As will soon be discussed, the
local matching algorithm will always have at least two sides
for each internal piece, but could have upwards of three or
four to consider depending on the location of the slot and
what pieces have been matched so far. In the inner case, the
local matching algorithm has to ensure that the heads and
holes all line up first, as before, but then determines all of
the matching metrics per side and takes the average over the
number of sides. The major difference here is that a single
piece could fit into a slot in multiple ways, so piece orienta-
tion must be accounted for and the piece must be rotated in
all valid configurations before the algorithm returns a set of
matches. A single piece could potentially have four possi-
ble “matches” to a single slot, depending on hole/head ori-
entation. The match thresholding and scoring are the same
across pieces and edges in the inner piece matching as in the
are averaged across each piece/edge to which the piece is
matched in the inner matching (which is unnecessary in the
border case).
3.3.2 Global Assembly – the Border Pieces
It is logical to begin the global assembly with the border
because the border pieces are distinct – they each have at
least one flat side. Since the piece matching algorithm is not
perfect and does not always return the “correct” match as
the “best” match, a so-called “greedy” algorithm that sim-
ply places the “best” match between two pieces in the next
available slot will not necessarily result in a coherent solu-
tion (i.e. one might get something that is non-rectangular
or even nonsensical). In order to provide for this possi-
bility while not resorting to a “brute-force” approach that
runs through every possible combination of pieces, I de-
cided to use a so-called “branch-and-bound” algorithm, nor-
mally used in the solution of the Traveling Salesman Prob-
lem (TSP). In the general description of the TSP, a salesman
needs to do business in a number of cities spread out over a
region with defined distances between each. The salesman
wants to find the shortest overall route that goes to every
city only once and returns to his starting point – thus its a
distance minimization problem.
Much like the TSP, each match made between two pieces
along the puzzle border is given a score. Once a solution
is found, the total of all of the match scores that make up
that border solution should reflect how good the solution
is. Ideally, the smallest overall score will be the best and
correct solution. However, that turns out not always being
the case, as will be discussed.
Since there are many ways of reaching a solution, we
need a way of capturing a large number of possible solu-
tions and then finding the best one of those potential so-
lutions. One way of doing this is the branch-and-bound
method. This algorithm will be discussed shortly, but first I
will describe the greater methodology for how the border is
constructed.
The general construction of the border begins with a
corner piece. We orient the piece such that the counter-
clockwise-most flat side is “down,” with the other flat side
to the “left” and a head or hole to the “right.” Matching is
then done to the “right” in a sequential manner. The local
matching algorithm, then, receives a left piece and a list of
possible right pieces, all with the flat side down. It returns a
list of potential right pieces. We then choose one of the right
pieces to be the new left piece and continue the process until
we run out of possible right pieces. As we progress around
the border, when we hit another corner, we rotate the en-
tire puzzle and continue as if the flat side of each piece is
“down.”
Now, since the potential number of solutions is (n − 1)!
where n is the number of border pieces, we want to con-
strain the number of solutions found through whatever
means necessary. The local matching algorithm does a good
job of weeding out very poor matches, but will still return
multiple potential matches that could lead to a nonsensical
full solution. In order to combat this we also place a set of
side length constraints on the puzzle such that the border
solution must have side lengths corresponding to a rectan-
gular puzzle. If we’ve gone too long without a corner piece
or if we get sides defined by corner pieces that do not equal
one another, the solution is thrown out and we search for a
new one.
After all of this preamble, I am now going to describe
the basic branch-and-bound algorithm that is used for the
border matching problem. The algorithm can be described
in the following manner:
1. Choose a starting piece (usually a corner) and call it
piece A.
2. Use piece A to find a set of potential matches – these
6
can be considered “children” or “branches.”
3. Since the local matching algorithm returns a rank-
ordered list of potential matches, start with the first
potential match as piece B.
4. Next, remove piece B from the list of pieces remaining
to be matched and make piece B the new piece A.
5. Use the new piece A to find more potential matches.
6. This process continues until one of the following oc-
curs:
(a) We run out of pieces remaining – in this case we
have found a solution or “leaf.” We store this set
of matches and their scores as a solution, back
up, and see if we can find more solutions.
(b) We run into a border constraint that isn’t satisfied
– in this case we back up and see if another match
does satisfy the border constraint before moving
on.
(c) We do not get any potential matches for the cur-
rent piece A – in this case we need to back up and
see if we can find another path using a different
piece from an older set of potential matches.
The algorithm either runs until it has exhausted the search
space and found all possible solutions based on both the side
constraints and the local matching thresholds, or until it has
obtained the number of solutions requested of it. As can
probably be surmised from the basic description above, one
can visualize this approach as a tree with the first piece at
the root and branches extending upward for each potential
match. If we are able to make it all the way up a branch to
a leaf, then we have found a solution to the problem. If we
get stuck on a branch and can’t expand, we come back down
the tree until we find another path that looks fruitful. In this
way we can reduce the total number of solutions tried to
well below that which would be found through simple brute
force.
As one can see, because the problem is being solved in
a nonlinear fashion, the number of solutions that might be
found before the “correct” solution is highly dependent on
several factors, not least of which are the first piece cho-
sen and how well the first few pieces match. If incorrect
matches are made early in the process, it can take a long
time (and a lot of matches) before the correct solution is
found. And even when the correct solution is found, it may
not be the “best” solution as per the scoring system. Ideally
the “best” and “correct” would be the same, but that is not
always the case.
3.3.3 Global Assembly – the Inner Pieces Puzzle Total Border Border Inner
Name Pieces Pieces Soln Soln
Once a border solution is found, it is passed to the global as-
Wookie 12 10 1st 1st
sembly for the inner pieces. The global assembly algorithm
Storm Troops 24 16 58th 5th
assembles the border pieces into a grid and then grabs the
Droids 12 10 1st 1st
“upper left” open slot as the new “piece A.” This algorithm
Speeder 16 12 1st 1st
also creates a grid with relative orientations for each piece.
Rey Finn 12 10 1st 1st
When the pieces were first characterized, they were each
Kylo Ren 24 16 N/A N/A
oriented with “side 1” being “up.” Once they are placed
in the final puzzle grid, we create a second grid with the Table 1. Test Puzzle Results
same dimensions that provides the relative 90 degree rota-
tion from “up” for each piece (0-3). Because we know we
have the border completed, we can be assured that that first ing Toolbox, it was not completed in a satisfactory man-
slot will have at least two pieces along its edges. The more ner by the end of this project. Instead, I use the func-
pieces along an edge, the more accurate and discriminating tions vision.AlphaBlender and step along with the
the score should be. The inner piece matching algorithm piece masks and the cropped segmented pieces from the
then uses a similar branch-and-bound algorithm as in the original image to create a quasi-final image “grid” that
border case to find potential matches for this first slot. It shows the extracted pieces oriented per the solution. It’s
then removes the piece from those remaining and moves not ideal, but it at least shows how the final pieces should
across the puzzle filling in all available pieces from left to be laid out and arranged. For reference, see figure 4 for an
right and then top down (like reading a book). Unlike in the example of the final solution.
border case, however, the potential matches could involve
the same piece, just oriented differently. For this algorithm, 4. Experimentation and Results
orientation is very important. As with the border assembly
algorithm, once all of the pieces are found, that solution is I ran my puzzle solver in Matlab on both a home PC with
stored and we then back up and see if we can find more until 4 year old hardware (6 GB RAM, Intel i5 processor, AMD
either we run out of solutions or we have found the number Graphics Card) and a 13-inch MacBook Air, 2015 model
of solutions desired. Ideally, the solution with the lowest to- with 4GB memory and Intel Graphics with little difficulty.
tal score (aka the “best” solution) will also be the “correct” It takes about a minute or so to run one of the test puzzles
solution. (it might take more than a minute for the 24 piece puzzles)
One last note about the global assembly algorithms: from image segmentation through to final construction. I
while it might make sense to have the two algorithms sepa- carried a good amount of information in memory through-
rate during developing and while trying to understand where out the process since I was doing a lot of experimentation
each breaks down, the ideal case would be to combine these and wanted the ability plug and play various modules for
algorithms in order to weed out border solutions that do not both fine-tuning and debugging. This could be pared down
provide for full puzzle solutions. This was thought about, for a future implementation.
though not implemented in the final code. Had there been While the puzzle solver created for this project cannot be
more time, this would have helped to bring down the to- used on every puzzle (per the limitations noted earlier), for
tal number of end puzzle solutions. As it was, in the time those it could be used on, I was able to experiment to find
allowed, I was simply able to get both of these algorithms limitations and weaknesses. I also used this experimenta-
working well enough to tweak the various variables to see tion to find the best criteria for matching.
how best to find matches. The next step would be to link For this project I tested the puzzle solver on six Star
these two algorithms and throw out border solutions that Wars themed children’s puzzles that I bought at Target. The
have no potential solutions based on all of the criteria dis- overview of the results using the final matching parameters
cussed above. are in Table 1.
As one can see, most of the puzzles had 12 to 16 pieces,
3.4. Final Image Construction except for two that had 24 pieces. Of all the puzzles that
Once we have assembled all of the pieces into a grid had less than 24 pieces (four of the puzzles), the correct
with their relative orientations, we now have the solution border solution was the best border solution returned, by
to the puzzle. The next step is displaying that solution score (hence the “1st” in the third column). Then, using the
to a user. Ideally, I would like to display a completed correct solution as the lead-in to the inner puzzle algorithm,
and fully stitched together puzzle image using the puzzle those same puzzles found the correct solution to be the one
pieces as segmented from the original image. While this with the best score.
is most likely possible using the Matlab Image Process- For the Storm Trooper puzzle, there were two primary

7
factors that led to it not doing as well. First, there are more
pieces and therefore more potential matches. Still, if the
matches were registering scores that reflected the true “cor-
rectness” of the match, then one would expect the overall
score of the completed border to be better than 58th. And,
even when we fed the correct border solution to the in-
ner puzzle algorithm the correct solution was 5th best by
score. However, to put this into perspective, the global bor-
der solution algorithm returned in excess of 2000 potential
border solutions for the storm trooper puzzle (of a math-
ematically potential 15!, approximately 1.3 trillion, solu-
tions with brute force), of which the correct one was 58th
by score. Which isn’t all that bad. Additionally, the storm
trooper puzzle was by far and away the most homogeneous
in terms of color of all the puzzles. It was very difficult
to discriminate matches based on color using the methods
I described before, especially because of the way the storm
trooper line discontinuities happen to match up along the
border of the pieces, making cross border color matching
very difficult indeed. And finally, the pieces were fairly
square, so all four sides were very even and comparable in
length. If they were instead more elongated with one pair
of sides longer than the other, the side length discriminant
would have knocked down potential matches.
This was a case where the experimentally determined
match scoring algorithm broke down. While it worked in
the other test cases very well, one can see quite clearly that
other methods would need to be pursued in order to get the
storm trooper puzzle to be solved correctly.
The other puzzle in the table that was looked at as a test
case but does not have a rank for a solution is the Kylo
Ren puzzle. This puzzle highlighted the need for a better
corner-finding or edge-finding process. While the code I
developed to find the corners repeatedly on the other puz-
zles worked quite well, the pieces of the Kylo Ren puzzle
were extremely elongated and many of the “heads” were so
small as to be mistaken for corners. Needless to say, the au-
tomatic corner-finding was not able to find the corners, and
without them the rest of the algorithm just doesn’t work as
is.
When experimentally determining the criteria to use for
the piece matching, individual matches were observed with
special attention paid to the values for correct matches. The
Wookie puzzle was used as the baseline case and the val-
ues derived from this puzzle were applied to the others with
general success (except for the Storm Trooper puzzle). Here
were the typical values and the final weights applied:
• Side Distance Difference: 0-8 pixels (wt = 12.5)
• Overlap Average Difference: 0-14 pixels (wt = 7.0)
• ∆E Patches: 7-45 (wt = 2.9)
• ∆E Pieces: 4-35 (wt = 3.2)
8
While it would logically seem, and in most cases it would
actually be, that matching the color patches across the
boundaries should be one of the best ways to discriminate in
order to find a true match, due to the variability in how the
puzzle pieces are carved up, this was not always the case.
For instance one piece was carved almost perfectly along
Wookie’s nose, which is dark, and just on the other side of
the edge there was a bright background. The ∆E in this
case was fairly large even though the match is a correct one.
In fact, this was also a case where the puzzle pieces had a
small ∆E between them, but the patch difference was much
higher. This is not an expected result. And while one might
begin to think that maybe color is too volatile and should
not be considered at all since the perceived variability in the
shape is much smaller among true matches per my above
list, that is not entirely accurate. Due to noise and distor-
tion, the actual length of the sides and the measured overlap
is not exact. And while the correct match is always small,
so are many other matches. These criteria are best for weed-
ing out those pieces whose shape isn’t even close to correct.
It can also help when the head of one piece is bent in one
direction while the hole of the other piece is expecting it to
be bent in another – then the overlap will suffer. For the
most part, however, the size helps get you close. Unfortu-
nately, many of the pieces have differences that fall within
the acceptable ranges above. That is why color is then used
to help with the ranking of those potential matches. And
in most cases, the color does help. There are just a few in
every puzzle where there are large transitions in both the
puzzle region or just along the border that cause the match
scoring to return some interesting values.
Additional parameters that were used early on for color
scoring were RGB and HSV channel averages. While in
some cases there was clear correlation, in many others there
didn’t appear to be any correlation whatsoever. Color vari-
ance was also considered, though it was also disregarded
because, after some thought, I could not see how it would
return a marked improvement. The extreme variability in
the color scoring led to a rethinking about how the colors
were being compared and the eventual use of ∆E.
Additional methods for finding border matches that were
considered but were unable to be implemented before this
report were:
• Segmentation along the border (such as meanshift) –
if we could determine there are a certain number of
segments along one border that coincide with a cer-
tain number of segments along another, then maybe
we could find a potential match.
• Find lines along a border that break at the border, then
look for the continuation of these lines on the other
side. Would have something to do with the flow of
pixels – seems difficult to implement, though would
 really help with the Storm Trooper puzzle.
• Grab features within the head piece or along the edge
and build a Bag of Words model. Then try to find
matches on the other side of the piece around the hole.
This has potential, but is potentially computationally
expensive.
While the matching algorithm used isn’t perfect, it worked
for the test cases considered. And while others may have
been able to solve larger puzzles [4][3] with their algo-
rithms, my code proved to be fairly robust and efficient at
solving the puzzles provided. Since there appear to be no
standardized “jigsaw puzzle metrics” against which to com-
pare by puzzle solver for puzzles with irregular shapes, I
cannot say exactly how my puzzle solver compares to oth-
ers that have been developed. However, it is one of the few
that I’ve seen that takes a raw image of all of the pieces
at once and produces a fully constructed solution. Most of
the puzzle solvers found in official papers and in student
submissions cited earlier produce only partial or theoretical
solutions, or solutions that require even greater initial con-
straints than my own (i.e. the pieces have to lie in a grid at
Figure 4. Solution Created by the Automatic Puzzle Solver
the outset or each piece has to be scanned individually with
a high resolution scanner). Still others rely on the original
image to find the location of the pieces in the final image,
which is not the problem I set out to solve. One example
using the Wookie puzzle can be seen in with the original
image in figure 1 and the final solution as found by my puz-
zle solver in figure 4.

5. Conclusion
I have created an end-to-end “automatic” jigsaw puzzle
solver that uses Matlab and the Matlab Image Processing
Toolbox to piece together a jigsaw puzzle using only an
image of the pieces. I used this puzzle solver on six test
puzzles and proved that it works on five of them quite reli-
ably, but also found where there were weaknesses in the cur-
rent implementation. Certain design decisions were made
early on that simplified the problem such that I could com-
plete the entire project by the deadline. Unfortunately, this
also meant it was hard to go back and try a completely new
method once I had begun going down a certain path.
I learned a lot over the course of this project. I learned
about how to think about a 3-dimensional, physical world
problem in terms of a 2-dimensional perception of that
problem. I learned how to think about manipulating ev-
ery ounce of information I could glean from a single photo- Figure 5. The “Truth” – A Picture of the Assembled Wookie Puz-
graph to help the computer “think” like a human and make zle
matches that would result in a correct solution. I learned
about many functions inherent within Matlab, especially the
Matlab Image Processing Toolbox. As I developed the pro- differently. This knowledge will certainly be helpful in the
gram, I learned new tools and tricks that, had I known them future and could be applied to improving the puzzle solver.
earlier, I may have approached certain parts of the project Several of the papers I read where people have attempted

9
this problem in the past did not make much sense to me un-
til I went and attempted it myself. I had believed it would
be easier to extract the corners of the four-sided pieces, and
therefore decided to go with canonical piece jigsaw puzzles.
However, this then meant I was fairly limited in the types
and numbers of puzzles to which my program could ap-
ply. It also meant that extraction of this information was ab-
solutely essential to everything my program did afterward.
Some of the other methods, like the use of fiducial points
as in [3], may have proven more difficult at first, but could
have paid dividends in its ability to scale.
If my original end goal was to create a program that be-
gan to explore the possibility of creating an automatic jig-
saw puzzle solver smart phone application, which was the
original idea, then I believe I have achieved a pretty great
stride in that direction. However, my code is not yet robust
enough to the kinds of inputs a smart phone might provide,
nor is it efficient enough in both memory allocation and
processor requirements to be feasible for that application.
Many changes would have to be made before I can get to
that end goal, which is something I realized about halfway
through the project. While I believe I have created a solid
and workable solution within the constraints of the problem
as I originally set forth, I see many areas where it could be
improved for future incarnations. All in all, I did what I set
out to do, I learned a lot, and I enjoyed the process.

References
[1] J. Davidson. A genetic algorithm-based solver for very large
jigsaw puzzles: Final report.
[2] H. Freeman and L. Garder. Apictorial jigsaw puzzles: The
computer solution of a problem in pattern recognition. IEEE
Transactions on Electronic Computers, EC-13(2):118–127,
April 1964.
[3] D. Goldberg, C. Malon, and M. Bern. A global approach to
automatic solution of jigsaw puzzles. Comput. Geom. Theory
Appl., 28(2-3):165–174, June 2004.
[4] A. K. Y. L. H. Wolfson, E. Schonberg. Solving jigsaw puzzles
by computer. Annals of Operations Research, 12:51–64, 1988.
[5] D. A. Kosiba, P. M. Devaux, S. Balasubramanian, T. L.
Gandhi, and K. Kasturi. An automatic jigsaw puzzle solver. In
Pattern Recognition, 1994. Vol. 1 - Conference A: Computer
Vision amp; Image Processing., Proceedings of the 12th IAPR
International Conference on, volume 1, pages 616–618 vol.1,
Oct 1994.
[6] N. Kumbla. An automatic jigsaw puzzle solver.
[7] L. Liang and Z. Liu. A jigsaw puzzle solving guide on mobile
devices.
[8] A. Mahdi. Solving jigsaw puzzles using computer vision.

10
 Database-Backed Scene Completion

Alex Alifimoff
aja2015@cs.stanford.edu

Author’s note: I liberally use the third person pronoun, ”we”, in this paper, as I’m used to
working in group projects. Rest assured, I am the only author of this project.

Introduction

Ever have an almost-perfect photo? Maybe it’s that photo of the beach that your dweeb
uncle stepped in front of, or that wedding photo ruined by the donut truck driving in the
background. Scene completion is the task of taking a photo and replacing a particular
region of that photo with an aesthetically sensible alternative. In this work we demonstrate
our implementation of a method originally implemented by Hayes & Efros which produces
interesting scene completions utilizing a very large database of images.

Previous Work

There are many different approaches to the problem of scene completion. One possible
approach is to use multiple images (either from multiple cameras, multiple pictures, or
video) to determine exactly what kind of information was in the masked part of the image,
and then adjust that information appropriately and place it back into the original image.
[2, 3]
A second common approach is to utilize information from the image itself to attempt to
guess what kinds of missing information should be used to fill the masked part of the image.
[3] The majority of these approaches involve utilizing nearby textures and other patterns
from the input image to fill the scene.
This project follows the methodology outlined by Hayes and Efros [1], who differ from previ-
ous approaches in that they try to complete the scene by finding plausible matching textures
from other photographs. In particular, the implemented system searches thousands to (ide-
ally) millions of photographs to find globally matching scenes, and then utilizes texture
patterns from those scenes to fill the missing hole in the input image.

Key Improvements

The significant improvement of the Hayes & Efros system over previous image completion
software is the ability to produce ”novel” scene completions through the use of the large
image database that is searched to find global scene matches.
Additionally, the Hayes & Efros system does not place wholly stringent restrictions on which
pixels must be used from the source image and which pixels must be completed. All of the
masked pixels must be replaced, but through the use of a novel application of min-cost
graph cutting, the system may decide to replace more pixels in the original image if it
makes for a better fit. This allows interesting completions that simply aren’t possible with
more stringent restrictions. We discuss this in depth in the following sections.

1
 Figure 1: An example input image with corresponding mask (black region)

Technical Approach

The input to the algorithm is an image and a corresponding mask, which indicates which
part of the image needs to be filled.
Generally, we then follow three steps:

1. Semantic Scene Matching. Quickly identify possible images that we could use to fill
the hole in our scene.
2. Local Context Matching. Identify the local context and search all of the remaining
scene matches to find the best local matches.
3. Blending. Perform graph cut and poisson blending to merge the two images.

Semantic Scene Matching

The first part of the method involves finding images which represent similar scenes to the
image being filled. However, since there are potentially millions of images to search, any
comparison must be done extremely quickly. To implement this part of the algorithm,
we rely upon a scene descriptor called GIST. GIST descriptors build a low-dimensional
representation of the scene which is designed to capture a couple of perceptual dimensions.
The authors of the original GIST paper describe these dimensions as ”naturalness, openness,
roughness, expansion, ruggedness” [4, 5].
GIST descriptors are pre-generated for every image in the database. Once a masked image
is provided, the GIST descriptor for the masked image is calculated. I use GIST descriptors
with 5 oriented edge-responses at 4 scales aggregated to a 4x4 spatial resolution. These
descriptors are slightly smaller than the original ones in Hayes and Efros, but the slight
reduction did not impact performance while slightly improving the time it took to build
the GIST descriptors for the database. We augment each GIST descriptor with a color
histogram with 512 dimensions to capture color information.
The search is then performed by using the weighted combined GIST and color descriptor
and an l2-distance metric. Each generated descriptor is compared to the masked input
image, and the best 100 images are kept for local context matching.

2
 Figure 2: Best GIST matches for leftmost image

Figure 3: A local context mask and an example of a possible local context

Computational Limitations

One of the main difficulties in pursuing this project was the computational resources nec-
essary to implement it in the same manner as the original authors. The original paper
utilized a network of 15 computers to examine millions of images simultaneously. Since
this computing power wasn’t available to me, I downloaded a pre-filtered subset of closely
matching images from Hayes and Efros’ project site to augment the thousands of images
I downloaded independently. This allowed me to get high quality matches to ensure the
rest of the scene completion pipeline worked appropriately. The graphics in this report
were generated from my own database of 200,000 images and the additional images from
the original project site. My database primarily consisted of photos downloaded from the
image sharing website, Flickr, that were tagged ”outdoors”. I restricted the category for the
purpose of getting quality completions for images within the same category.
It took about 12 hours to generate all of the GIST descriptors for the small dataset with
liberal use of multiprocessing. However, this computational only needs to be performed
once. Performing a single nearest-neighbor search of the dataset takes approximately five
minutes.

Local Context Matching

The next step is to find appropriate patches in semantically similar images to use as the
scene completion content. The first step of this process is to determine exactly what the
local context of a particular image is. To do this, I first dilate the mask. This effectively
produces a second, slightly larger mask. Then the mask is cropped so that it is only the
width and height of the dilated mask. I then subtract away the original mask and are
left with an image patch which corresponds to the local context of a particular image. The
remainder corresponds to the local context that we will be examining in each image. We will
use this patch to find the ”optimal” patch to use in our scene completion for each matching
image.
We illustrate this process. On the left we show an example dilated mask, where the red
corresponds to the area of local context we will be consider. Then, as we move it across the
image, we consider patches like the one on the right.

3
We take each patch and compute a HOG descriptor and a color histogram. We utilize these
as texture and color features and compare them to the local context of our source image.
We use sum-of-squared distances of this feature set to select which patch to use from a
particular image.

Blending

There are two main parts to the blending step. Given a mask and a dilated mask, we have
to use all of the pixels from the patch image for the area of the image covered by the mask.
However, for the dilated mask, we have a decision to make. One particular innovation of
Hayes and Efros’ method is actually choosing to remove more of the original image than the
mask requires.
To determine which part of the original image to keep and which part to patch, we use a
min-cost graph-cut algorithm. We assign each pixel a label, ”original” or ”patch”. We call
the set of all labels L, and we minimize:
∑ ∑
C(L) = Cunary (p, L(p)) + Cpair (p, q, L(p), L(q)) (1)
p p,q

We define the unary cost functions as follows. For any pixel in the space removed
by the original mask, Cunary (p, original) >> 0 (any very large number) and we set
Cunary (p, patch) = 0. For any pixel that is not covered by the mask or the dilated mask,
Cunary (p, patch) >> 0 and Cunary (p, original) = 0. The intuition here is that for the for-
mer category, we must choose pixels from the patch. In the latter category, we must choose
pixels from the original picture. For all of the rest of the pixels, we define

Cunary (p, patch) = (k ∗ ||f (p) − f ′ (mask)||)3 (2)

where f is a function returning the location of a pixel and f ′ is a function returning the
nearest pixel in the original mask. Intuitively, we want to punish choosing pixels that are
not in the original the further away we get from the hole. Like Hayes and Efros, we use
k = 0.002.
The remaining part of the graph cut algorithm is determining how to define Cpair . In our
implementation, each pixel is connected in a four-way neighbor set-up, so for pixels that are
not adjacent, we have zero cost. For other pixels, we use:

Cpair (p, q, L(p), L(q)) = ||h(ppatch ) − h(poriginal )|| + ||h(qpatch ) − h(qoriginal )|| (3)

where h is a function returning the vectorized (RGB) representation of a pixel. The intution
here is that we want to minimize the gradient of the image difference as opposed to the
intensity difference along the seam.
We include a figure demonstrating the change from before the graph-cut is applied and
afterwards. Generally this causes a small expansion in the size of the mask.
Finally, poisson blending is applied to the image and its patch to seamlessly blend the two
images. This ensures that slight differences in color do not ruin the completion attempt.
The poisson solver is allowed to run on the entire domain of the image and not just the
local region it is attempting to patch.

Results
Here are a number of possible good completions from various input masks and input images.
Generally, when I used the pre-seeded database of gist matches compiled from the Hayes &
Efros site, I got reasonable performance. Additionally, when I used images that were ”out-
doorsy” (this was the image category I primarily downloaded from Flickr), I got reasonable

4
 Figure 4: The patch before and after applying graph-cut

Table 1: Runtime comparison to original implementation

Phase Avg. Runtime StdDev Runtime Hayes & Efros

Semantic Scene Matching 5.2 minutes 0.2 minutes 50 minutes 2
Local Context Matching 22.3 minutes 1.2 minutes 20 minutes
Composition 2 minutes 0.1 minutes 4 minutes

completions. However, when trying to complete images that didn’t have particularly good
GIST descriptor matches in the dataset, the matches could be comically bad.
One of the main take-aways from this project is that this method highly relies upon having
a large dataset available to search for completions. Hayes and Efros required a significant
amount of computation power to search their dataset of 2 million images, and even they
largely restricted the semantic categories in which they downloaded images. Utilizing this
method as a production system for image completion would only be reasonable for companies
that have significant computation power and access to many images, like a search engine
provider or photo-sharing website.
Runtime is another issue of concern with this particular algorithm. As discussed previously,
Hayes and Efros required fifteen CPUs to process a single image in five minutes. On a single
CPU, their algorithm took 74 minutes to run. The average runtime for my implementation
across a sample set of 100 photographs was as follows. For this particular experiment, we
chose to use the 200 best matching scenes for local context matching. 1 My implementation
was comparable to Hayes and Efros, despite being implemented in Python.
Quantitatively evaluating the performance of the algorithm in regards to how effectively
it completes images is difficult, as there is no good metric for evaluating the ”realness” of
photographs without doing human evaluation. This evaluation is done by Hayes and Efros,
but sadly I did not have the time or the access to resources to adequately conduct human
trials.

Areas of Improvement

There are numerous situations in which this system fails. We generally classify these errors
into a couple of different categories.

1. The first category are failures of scene matching. These are situations in which the
GIST descriptor identifies scenes that just don’t belong together (i.e. filling a mask
in a tropical scene using an image from the snow)
2. The second category are blending issues. These errors occur typically when a sub-
optimal image patch is chosen that contains superfluous artifacts, or when the
graph-cut algorithm chooses to include something it should not.
1
This was the same number as Hayes and Efros for the purposes of comparison, although for
smaller databases we suspect this number should be reduced as many of the matches beyond the
20th were quite bad

5
Figure 5: Mask and possible completions for grassy/forest scene

Figure 6: Mask and possible completions for ocean scene

6
Figure 7: An example of a high-level semantic issue. Notice the partial rabbit in the grass.

Figure 8: An example of a blending issue

3. The final category of errors includes issues with high level semantics. These are
situations in which partial objects are included in the patch, such a part of a rabbit
filling in a grassy scene because there is otherwise a good local context match. Since
the algorithm has no notion of objects, this happens quite often.

We include some illustrative examples of each error category.

Final Thoughts

In general, I was quite happy with the output of the system. There was generally at least
one reasonable completion for the vast majority of images that I would input, provided I
was using input images that fell into the same semantic category as input images in my
database. Largely, this algorithm is effective with lots of data, but not generally effective
for solving the scene completion problem on a resource-limited budget.

Acknowledgments
I would like to thank Silvio Savarese for an awesome class and to the entire teaching staff
for making a really strong effort to improve the class, even throughout the quarter.

References
[1] Hayes, J. and Efros, A. Scene Completion Using Millions of Photographs. SIGGRAPH, 2007.
http://graphics.cs.cmu.edu/projects/scene-completion/scene-completion.pdf

7
 Figure 9: An example of a scene matching issue. The forest does not belong in the city!

[2] Irani, M., Anandan, P., and Hsu, S. 1995. Mosaic based representations of video sequences and
their applications.
[3] Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D.,
and Cohen, M. 2004. Interactive digital photomontage. ACM Trans. Graph. 23, 3, 294–302.
[4] Oliva, A., and Torralba, A. 2006. Building the gist of a scene: The role of global image features
in recognition. In Visual Perception, Progress in Brain Research, vol. 155.
[5] Oliva, A., and Torralba, A. 2001. Modeling the shape of the scene: a holistic repre-
sentation of the spatial envelope. In International Journal of Computer Vision, vol 42 (3).
https://people.csail.mit.edu/torralba/code/spatialenvelope/

8
 Deep Drone: Object Detection and Tracking for
Smart Drones on Embedded System
Song Han, William Shen, Zuozhen Liu
Stanford University
Abstract
In recent years, drones have been widely adopted for
aerial photography at much lower costs. However, cap-
turing high quality pictures or videos using most advanced
drones requires precise manual control and are very error-
prone. We are proposing Deep Drone, an embedded system
framework, to power drones with vision: letting the drone
to do automatic detection and tracking. In this project, we
implemented the vision component which is an integration
of advanced detection and tracking algorithms. We im-
plemented our system onto multiple hardware platforms,
including both desktop GPU (NVIDIA GTX980) and em-
bedded GPU (NVIDIA Tegra K1 and NVIDIA Tegra X1)
and evaluated frame rate, power consumption and accu-
racy on several videos captured by the drone. Our sys-
tem achieved real time performance at 71 frames per sec-
ond(fps) for tracking and 1.6 fps for detection on NVIDIA
TX1. The video demo of our detection and tracking algo-
rithm has been uploaded to Youtube: https://youtu.
be/UTx2-5a488s.
1. Introduction
Modern drones are be equipped with cameras and are
very prospective for a variety of commercial uses such as
aerial photography, surveillance, etc. In order to massively
deploy drones and further reduce their costs, it’s neces-
sary to power drones with smart computer vision and auto-
pilot. In the application of aerial photography, object detec-
tion and tracking are essential to capturing key objects in a
scene. Object detection and tracking are classic problems
in computer vision. However, there are more challenges
with drones due to top-down view angles and real-time con-
straints. Additionally, a challenging problem is the strong
weight and area constraint of embedded hardware that lim-
its the drones to run computation intensive algorithms, such
as deep learning, with limited hardware resource.
Deep Drone is a framework that intends to tackle both
problems while running on embedded systems that can be
mounted onto drones. For this project, we present our vision
1
system pipeline and its performance along with accuracy on
multiple hardware platforms, and we analyzed the trade-off
between accuracy and frame rate.
2. Related work
Deep neural networks, object detection and object track-
ing are the three major components in our work. We first
present an overview of past work, and then describe our im-
provements.
Deep neural network is the state-of-the-art technique in
computer vision tasks, including image classification, de-
tection, and segmentation. AlexNet [11] is the classic net-
work proposed in 2012 that has 8 layers and 60 million
connections, and won the Imagenet contest in 2012, spawn-
ing a lot of improvements later. After that, VGGNet [16]
was proposed, which has 16 layers and 130 million parame-
ters. Both AlexNet and VGGNet have bulky fully connected
layers which results in huge model sizes. GoogleNet [17]
is a more compact CNN that consists mostly of conv lay-
ers, it uses the inception model that has 1x1, 3x3 and
5x5 convolution kernels with different scale. It also has
multiple loss layers to prevent vanishing gradient problem.
ResNet [6] was proposed in 2015 and greatly improved the
image recognition accuracy, it adds bypass layers to let the
network learn the residual rather than the absolute value.
SqueezeNet [9] was proposed recently that is aggressively
optimized for model size. It has 50x less connections and
half the computation than AlexNet but has higher accuracy.
After Deep Compression [5, 4], the model is only 470KB
and fits well into the last level cache.
Fast R-CNN [3] is a Fast Region-based Convolutional
Network method (Fast R-CNN) for object detection. In this
algorithm, an input image and multiple regions of interest
(RoIs) are input into a fully convolutional network. Each
RoI is pooled into a fixed-size feature map and then mapped
to a feature vector by fully connected layers (FCs). The net-
work has two output vectors per RoI: softmax probabilities
and per-class bounding-box regression offsets. The archi-
tecture is trained end-to-end with a multi-task loss.
Faster R-CNN [15] made further improvements on Fast
R-CNN that introduce a region proposal Network (RPN)
that shares full-image convolutional features with the detec-
tion network, thus enabling nearly cost-free region propos-
als. An RPN is a fully-convolutional network that simulta-
neously predicts object bounding boxes and scores at each
position. RPNs are trained end-to-end to generate high-
quality region proposals. Because of the region proposal
network are fused and could be trained end to end, the net-
work is faster than fast R-CNN.
YoLo Detector [13] is a new approach to do object de-
tection. Prior work on object detection re-purposes classi-
fiers to perform detection. Instead, YoLo use object detec-
tion as a regression problem to spatially separated bounding
boxes and associated class probabilities. A single neural
network predicts bounding boxes and class probabilities di-
rectly from full images in one evaluation. Since the whole
detection pipeline is a single network, it can be optimized
end-to-end directly on detection performance. The unified
architecture is extremely fast but not as accurate as Faster
R-CNN.
KCF[7, 8] is the kernelized correlation filters used for
detection. It uses the characteristic that under some condi-
tions, the resulting data and kernel matrices become circu-
lant. Their diagonalization by the DFT provides a general
blueprint for creating fast algorithms that deal with trans-
lations, reducing both storage and computation by several
orders of magnitude, obtaining state-of-the-art trackers that
run at 70 frames per second on NVIDIA TK1 and is very
simple to implement.
MDNet [12] is the state-of-the-art visual tracker based
on a CNN trained on a large set of tracking sequences, and
the winner tracker of The VOT2015 Challenge. The net-
work is composed of shared layers and multiple branches of
domain-specific layers, where domains correspond to indi-
vidual training sequences and each branch is responsible for
binary classification to identify the target in each domain.
The network is trained with respect to each domain itera-
tively to obtain generic target representations in the shared
layers. Online tracking is performed by evaluating the can-
didate windows randomly sampled around the previous tar-
get state.
However, the drawback of MDNet is that it needs to run
CNN to extract image features, making it very slow. Con-
sidering the frame rate required by real time tracking, we
used the KCF algorithm for tracking, which achieves 70
frames per second on our hardware: a NVIDIA Tegra K1.
3. Contribution
Fast and Faster R-CNN originally used VGGNet for fea-
ture extraction. It is accurate but slow. Drones have lim-
ited hardware resource both in memory and in computation
power, so we need to have smaller network. In order to run
the CNN fast enough on embedded device, we didn’t use
those off-the-shelf network architectures. Instead, we used
2
a relatively shallow and small network to extract image fea-
tures and to detection. The architecture is shown in Fig 1.
Even using our small network architecture, the detection
frame rate is still low on TK1 mobile GPU. To compensate
for the slow speed of detection, we used the cheap KCF
tracker, although it’s less accurate than MDNet, to track on
the bonding box returned by the detection pipeline. Thus
we have the accurate but slow Faster R-CNN for detection,
and have the less accurate but super fast KCF for tracking.
Detection is only called when the confidence of the tracker
is below certain threshold, which is very infrequent. This
architecture makes the pipeline accurate, robust and fast.
Accuracy is not our sole target in this project. We have
a thorough evaluation with respect to accuracy, power con-
sumption, speed, and area of different hardware running de-
tection and tracking algorithm. Balancing these hardware
constraints, rather than only optimizing for MAP, is our top
priority.
4. System Architecture
The software architecture of our vision system consists
of two components. The first component is a detection al-
gorithm running Convolutional Neural Network (CNN) and
the second part is a tracking algorithm using HOG feature
and KCF. These two algorithms are seamlessly integrated
to ensure smooth and real-time performance. The detec-
tion algorithm, e.g. Faster RCNN, is expensive to compute
since CNN-based detection has lots of GOPs per frame and
is only called to initialize a bounding box for key object in
the scene. The tracking algorithm, e.g. KCF, is relatively
inexpensive to compute and can run at a high frame rate
to track the bounding box provided by detection algorithm.
The main algorithm loop is shown in the pseudo code be-
low, and we discuss the detection and tracking details in the
next two subsections.
Algorithm 1 Detection and Tracking Pipeline for Deep
Drone
boxF ound ← false
while true do
f ← new frame
while boxF ound == false do
detection(f) . Invoke detection algorithm
if Box is detected then
boxF ound ← true
end if
end while
tracking(f) . Invoke tracking algorithm
if Tracking is lost then
boxF ound ← false
end if
end while
4.1. Detection
Drones are mainly used to take pictures of human, so
we focus on detecting people as first step. We made fur-
ther assumption in this project that there is only one person
of interest to track, so that the person with the highest de-
tection score is our target. We have used two detectors to
do people detection: Faster RCNN and Yolo detector. We
analyze them both in below sections.
4.1.1 Using Faster R-CNN
We used a 7-layer convolutional neural network on Faster
R-CNN[15] for people detection. The framework takes raw
image frames from a video stream and outputs bonding
boxes and target classes for detected objects.
We used a in-house model trained on the KITTI[2]
dataset. KITTI contains a rich amount of training samples
that include objects such as cars, pedestrians and cyclists
and can easily generalize to our task. In this project, our in-
terested detection target is people so we modified the script
to detect people only. A detailed architecture is shown in
Figure 1.
We measured the accuracy(mAP) and speed of our in-
house model and compare it with the baseline, shown in
table1. Our in house model has slightly worse accuracy than
the baseline, but has 12x faster speed.
Table 1. Accuracy and runtime for our detection network (runtime
is measured on GTX980 GPU)
Model mAP Runtime
Baseline[15] 65.9 % 2s
Ours 62.0 % 0.17s
4.1.2 Using Yolo Detector
We also tried Yolo detector[14], because it’s easier to com-
pile to mobile. It’s also a faster alternative, at the cost of
worse accuracy. In our experiments, we found Yolo de-
tector unable to detect the cases where the people is small
and remote, as shown in Figure 4.1.2, so we didn’t use this
method.
4.2. Tracking
For tracking we chose KCF algorithm over other state-
of-the-art tracking algorithm like MDNet [12], SRDCF [1]
or EBT [18] despite them having a better accuracy and per-
formance on the VOT 2015 challenge [10]. The reason be-
hind this decision is that we want real-time tracking for our
drone so that we could give consistent control command to
the drone while MDNet, SRDCF and EBT all have perfor-
mance under 5 frames per second.
3
4.2.1 KCF
KCF [7][8] is a more old school tracking algorithm than
MDNet, but it’s supposed to be faster and more succinct.
The algorithm uses Discrete Fourier Transform to diagonal-
ize data matrix, which is then processed to train a discrim-
inative classifier through linear regression and kernel trick;
This new approach is called Kernelized Correlation Filter
(KCF).
We found out that KCF runs very fast on video, it takes
on average around 8.8 milliseconds to check per frame on a
Macbook pro CPU.
The downside of KCF is the requirement that the video
has to be continuous. If the video fades to black, the ob-
ject moves very fast or a jump cut occurs, KCF will have
a hard time to recover. When this occur, the peak value of
detection score from running Gaussian kernel on correla-
tion filter will suffer a significant drop; it will also return
negative bounding box value if it can’t find any match. We
leveraged this feature to combine KCF with faster RCNN
or other detection algorithm to solve the problem. Namely,
when KCF is not confident or fail entirely, it will call a de-
tection algorithm in hope of recover.
Table 2. Speed of detection and tracking on different hardware
platforms
Hardware Platform GTX 980 TX1 TK1
Power 150W 10W 7W
Detection 0.17s 0.6s 1.6s
Tracking 5.5ms 14ms 14ms
Tracking
182fps 71fps 71fps
Frames/Sec
4.3. Hardware Platform
Drones are special hardware platform with limited space
and weight, so computation power is not free lunch. we
can’t afford to use desktop GPUs to do the detection com-
putation, although it’s much faster. To compare hardware
platforms, we measured the computation time for detection
and tracking in Figure 1, and we show the power consump-
tion of these hardware platforms in the same table. GTX980
is roughly 10x faster than the TK1, but consumes 20x more
power. TX1 takes roughly the same power consumption,
but is 3x faster than TK1. So, in conclusion, TX1 would be
the ideal hardware platform to use for drone’s detection.
Looks like TX1 is ideal with respect to speed and power
consumption, but it’s form factor limited us to put it on the
drone. See in Figure 4.3, the TX1 development board is
much larger and much heavier than the TK1 board, making
it very hard to put on the drone. However, TK1 is smaller
and DJI provides the Manifold box to hold the TK1, which
makes it very easy to use, and we managed to put our algo-
rithms on the drone.
 Figure 1. The CNN architecture that we used for detection
In order to deal with the large form factor of the TX1
development board, we bought a small carrier board for
TX1. It is shown in Fig 4.3. This carrier board has the
size as small as the heat sink, we can unplug the TX1 cen-
tral board from the full development board and only plug
the central board that contains the actual TX1 chip to this
carrier board. This makes the size even smaller than TK1.
However, there’s no free lunch. The interface, especially
the power supply isn’t compatible with the DJI drone, we
haven’t got a chance to connect the carrier board with the
drone yet, which could be future work. The TK1 is fully
working, if not optimized for speed, the TX1 carrier board
is not the critical path of our project.
5. Implementation and Experiments
5.1. Detection
Faster RCNN builds on top of Caffe, a deep learning
framework that requires multiple dependencies, and has a
4
number of customized region pooling layers. We spent
great effort in installing CUDA and Faster RCNN onto all of
our desktop and embedded platforms. During installation,
we ran into the following issues.
1. We flashed our TK1 with the latest L4T (Linux for
Tegra), however, TK1 doesn’t support CUDA version
higher than CUDA 6.5, so we only installed CUDA 6.5
dev kit on TK1. However, the latest Caffe is not back-
ward compatible with CuDNN v3 and before, if we
revert the Caffe to an ealier branch, it won’t support
the new layers required by faster r-cnn. So we turned
off the CuDNN switch for Caffe installation to bypass
this problem.
2. Some of Faster RCNN libraries were written in python
and compiled into native c++ code using Cython.
When installing these libraries onto embedded sys-
tems ,i.e. TK1 and TX1, we ran into compilation er-
ror on gpu nms.cpp, a GPU implementation for non
Figure 2. Faster RCNN performs really well on detecting people from drones perspective. Even when the object is far away and twisted
(last figure).

Figure 3. Yolo detector performs not as well as Faster RCNN. When the target person is small, the detector fails.

maximum suppression. The root cause was identified as compiler incompatibility in the embedded systems.

5
Figure 4. KCF tracker performs very well on videos that doesn’t have jump cuts. It fails if the video fades to black. However, this is not a
problem since detection will be called on this scenario.
Figure 5. NVIDIA TX1 and TK1 development board we used
Figure 6. In order to reduce the size of TX1, we bought a small
carrier board.
We eventually found a workaround by manually mod-
6
ifying source code in the generated cpp file and suc-
cessfully passed the compilation.
Finally, we trained a 7-layer CNN Caffe model using
GTX 980 and ported the model onto all platforms to evalu-
ate testing performance using both offline and online video
streams.
5.2. Tracking
For tracking, the original algorithm for KCF is imple-
mented in Matlab. However, since the hardware limitation
of TK1 (storage space and computing power), we imple-
ment the algorithm using C++. As the original model pro-
posed, the tracker interface provides an init method that
takes in a frame and the starting bounding box, then it learns
a model using linear regression from sample patches (with
help of cyclic shift) of the frame. Upon receiving a new
frame, the tracker’s update function is called, it first detects
patches of the same size; if the resulting detection score
doesn’t reach a threshold, different patches under different
scales will be detected. We used 110% and 90% re-scaling
factor. Finally we return the one with highest score (the
new positive patches and negative patches are used here for
online learning).
5.3. Handshake between detection and tracking
Since the detection algorithm is implemented in Python,
we uses the Python.h to realize a C++/Python binding be-
tween the two algorithms. The initialization of detection
algorithm (loading neural network under Caffe) is stored in
the C++ main program as a PyObject. Upon receiving a
new frame, C++ main program converts the frame (stored in
byte array) to a Python ndarray object and passes the result-
ing array to a call to the detection method. It then parse the
result and determine if the detection result matches a thresh-
old (how confident the object is a person). An interesting
bug that arise is when importing a Python Module under
sudo(which is needed for activating and using DJI drone’s
camera); this is because some of the PythonThe NVIDIA
TX1 and TK1 development board that we used packages are
installed with root rw permission only, we worked around
this problem by command sudo su, and hacked the privilege
of using the DJI live camera.
5.4. Interacting with DJI camera library
We use two sets of DJI libraries. It only contains
interface code to talk to the camera, we didn’t use
any DJI code for vision algorithms. First, the cam-
era input module provided by djicam.h, which lever-
age libdcam to read in from the built-in camera on the
drone. The library only provides three simple func-
tions (manifold cam init, manifold cam read
and manifold cam exit) which means we need to ma-
nipulate all data from raw pixel arrays; we initialize the
camera with TRANSFER MODE (transferring image input
to controller and mobile app) and GET BUFFER MODE
(store the video input to local buffer byte array in NV12
format, more on format conversion in later subsection). We
use the CAM NON BLOCK mode (which means not wait-
ing the camera to fully initialized) to ensure that we can give
constant control to the drone even if the camera is not set
up. We sleep the program and wait for the camera to exit at
the end when we return. The specifics of the library can
be found here: https://github.com/dji-sdk/
Manifold-Cam/blob/master/djicam.h
5.5. Offline Detection and Tracking
We first tested our detection and tracking module on our
own offline video recordings from different perspectives (a
DJI Inspire recording of Song Han playing nunchaku and a
GoPro recording of Song Han snowboarding) with the TK1
board mounted on the drone. For detection using faster r-
cnn, each frame takes 1.6 second on average (comparing to
0.6s on TX1 which is small enough to mount on the drone
but not yet supported by DJI). For tracking under KCF, each
frame only takes 14ms (71 fps). Running our detection and
tracking module on the two videos exposes several inter-
esting problems to us. First, the tightness of detection’s
7
bounding box result will affect how well tracking algorithm
performs. If the bounding box is not tight enough, it might
incorporate irrelevant subject (in the nunchaku case incor-
porating big chunk of shadow, Figure 2 sub-image 4), track-
ing will then mistakenly think that it’s the irrelevant subject
it wants to track. Second, detection doesn’t work well when
the person is disguised as a bulkier figure. As in the snow-
boarding video, when the viewpoint is on the side of the
person, with the help of helmet, face mask and bulky cloth,
it’s hard to detect the person. We address the two problem
by adjusting the confidence threshold for detection score,
and retrain the neural network with images from different
angles and viewpoints.
5.6. Online detection and tracking on DJI live cam
We then adjust the detection and tracking module to
function with DJI M100’s live cam. As mentions in sec-
tion earlier, the camera read in data as raw byte array in
NV12 format, the format has three components for each
pixel: a luma component (the brightness) Yánd two chromi-
nance (color) components U and V; since the detection and
tracking algorithm are all based on RGB pixel values. We
do the following conversion to obtain a RGB representation
of the frame (clamping means restrict the value within the
0-255 range for RGB):
R = clamp(Y + 1.4075(V − 128))
G = clamp(Y − 0.3455(U − 128) − 0.7169(V − 128))
B = (int)(Y + 1.779(U − 128));
After obtaining the frame encoded in RGB format, we
use similar approach as offline video and detect and track
video frame by frame. A challenge surfaces in this step is
that since we are doing detection on a live stream, the sub-
ject might be moving rapidly while we are detecting, this
means that there is a possibility that the object has already
moved out of the bounding box while detection finishes an-
alyzing a frame from ∆t seconds ago. We addressed this
issue by initialize the tracker with the frame from ∆t sec-
onds ago instead of the current frame. The tracker will then
train it’s positive patches and negative patches with correct
bounding area. This solved the problem unless the object
has deformed too much in the ∆t time frame.
5.7. Controlling the Drone
We are controlling the camera using the OnBoard SDK
provided by DJI. We first need to send an activation data
to the CoreAPI driver. Since DJI recently upgraded their
drone operating system and their Onboard SDK not up-
dated accordingly, this step causes tremendous trouble. We
had to contact DJI’s engineer who wrote their Onboard
SDK to get the new version of encryption key (a magic
number) to successfully activate. Then we can gain con- [10] M. Kristan, J. Matas, A. Leonardis, M. Felsberg, L. Ce-
trol of the camera by sending GimbalAngleData to it. hovin, G. Fernandez, T. Vojir, G. Hager, G. Nebehay, and
A GimbalAngleData contain three import field corre- R. Pflugfelder. The visual object tracking vot2015 challenge
sponding to the three spatial degree of freedom for the cam- results. In Proceedings of the IEEE International Conference
era: yaw, roll and pitch. on Computer Vision Workshops, pages 1–23, 2015.
[11] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
6. Conclusion Advances in neural information processing systems, pages
We present Deep Drone, a detection and tracking sys- 1097–1105, 2012.
tem running real time on embedded hardware, that powers [12] H. Nam and B. Han. Learning multi-domain convolu-
tional neural networks for visual tracking. arXiv preprint
the drones with vision. We presented our software architec-
arXiv:1510.07945, 2015.
ture that combines the accurate but slow detection algorithm
[13] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. You
and the less accurate but fast tracking algorithm, to make only look once: Unified, real-time object detection. arXiv
the system both fast and accurate. We also compared the preprint arXiv:1506.02640, 2015.
runtime, power consumption and size of different hardware [14] J. Redmon, S. K. Divvala, R. B. Girshick, and A. Farhadi.
platforms, and discussed implementation issues and corre- You only look once: Unified, real-time object detection.
sponding solutions dealing with those embedded hardware. CoRR, abs/1506.02640, 2015.
[15] S. Ren, K. He, R. B. Girshick, and J. Sun. Faster R-CNN:
Acknowledgment towards real-time object detection with region proposal net-
works. CoRR, abs/1506.01497, 2015.
We thank Amber Garage for equipment support. [16] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. arXiv preprint
References arXiv:1409.1556, 2014.
[17] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,
[1] M. Danelljan, G. Hager, F. Shahbaz Khan, and M. Felsberg. D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.
Learning spatially regularized correlation filters for visual Going deeper with convolutions. In Proceedings of the IEEE
tracking. In Proceedings of the IEEE International Confer- Conference on Computer Vision and Pattern Recognition,
ence on Computer Vision, pages 4310–4318, 2015. pages 1–9, 2015.
[2] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets [18] G. Zhu, F. Porikli, and H. Li. Tracking randomly
robotics: The kitti dataset. International Journal of Robotics moving objects on edge box proposals. arXiv preprint
Research (IJRR), 2013. arXiv:1507.08085, 2015.
[3] R. Girshick. Fast r-cnn. In International Conference on Com-
puter Vision (ICCV), 2015.
[4] S. Han, H. Mao, and W. J. Dally. Deep compres-
sion: Compressing deep neural networks with pruning,
trained quantization and huffman coding. arXiv preprint
arXiv:1510.00149, 2015.
[5] S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights
and connections for efficient neural network. In Advances in
Neural Information Processing Systems, pages 1135–1143,
2015.
[6] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learn-
ing for image recognition. arXiv preprint arXiv:1512.03385,
2015.
[7] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista. Ex-
ploiting the circulant structure of tracking-by-detection with
kernels. In Computer Vision–ECCV 2012, pages 702–715.
Springer, 2012.
[8] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista. High-
speed tracking with kernelized correlation filters. Pattern
Analysis and Machine Intelligence, IEEE Transactions on,
37(3):583–596, 2015.
[9] F. N. Iandola, M. W. Moskewicz, K. Ashraf, S. Han, W. J.
Dally, and K. Keutzer. Squeezenet: Alexnet-level accuracy
with 50x fewer parameters and¡ 1mb model size. arXiv
preprint arXiv:1602.07360, 2016.

8
 Eigenfaces and Fisherfaces – A comparison of face detection techniques
Pradyumna Desale
SCPD, NVIDIA
pdesale@nvidia.com
Abstract
In this project we compare different subspace
based techniques of face recognition. Face
recognition is considered a relatively mature
problem with decades of research behind it and there
is a lot of interest because face recognition, in
addition to having numerous practical applications
such as access control, Mug shots searching,
security monitoring, and surveillance system, it is a
fundamental human behavior that is essential for
effective communications and interactions among
people.
In the literatures, face recognition problem is
defined as: given static (still) or video images of a
scene, identify or verify one or more persons in the
scene by comparing with faces stored in a database.
We focus on the classification problem which is a
superset of identification problem.
1. Motivation
Face recognition is a mature problem and even
though computers can not pick out suspects from
thousands of people NCIS style but the ability of
computers to differentiate among a small number of
family member and friends is considered better than
humans. Face recognition has additional
applications, including human-computer interaction
(HCI), identity verification, access controls etc.
Feature based face recognition methods rely on
extracting processing of input image to identify and
extract distinctive facial features such as the eyes,
mouth, nose etc., and the geometric relationship
among the facial points, thus reducing the input
facial image to a vector of geometric features.
Standard statistical pattern recognitions are then
1
Angelica Perez
Stanford University
pereza77@stanford.edu
employed to match faces using these measurements.
The distinct disadvantage of feature based
techniques is that since the extraction of feature
points precedes the training and classification, the
implementer has to make an arbitrary decision about
which features are important and that’s why we are
evaluating statistical methods of face recognition in
this work.
We use various methods in our two-stage face
recognition systems: PCA (Principal Component
Analysis), 2D PCA, and LDA (Linear Discriminant
Analysis) for feature extraction and SVM (Support
Vector Machines) for classification.
Specifically, we compare the methods for
accuracy of face identification between different
statistical methods +SVM in presence of following
variations
a. Training size variation.
b. Variations in number of principal components.
c. Presence of noise in image – Gaussian, salt-and-
pepper, speckle noise.
d. Variations in facial expressions of subjects
e. Variations in angle and structural changes to
face such as beard, glasses.
f. Variations in illumination.
Accuracy can be measured for identification or for
classification of a person against training set. We use
the latter as metric of accuracy.
2. Technical solution
We describe PCA, 2D PCA, LDA, and SVM briefly
in this section. The mathematics behind all these
techniques is proven so we don’t include any
derivations. References can be reviewed for
mathematical proofs.
2.1. PCA and illumination, since it considers the global
information of each face image and represents them
Sirovich and Kirby were the first to utilize
with a set of principal components. Under the
Principal Components Analysis (PCA) to
variation of pose and illumination, statistical features
economically represent face images. They argued
will vary considerably from the weight vectors of the
that any face image can be reconstructed
images with normal pose and illumination; hence it is
approximately as a weighted sum of a small
difficult to identify them correctly. On the other hand
collection of images that define a facial basis
if the face images were divided into smaller regions
(eigenimages), and a mean image of the face. Turk
and the weight vectors are computed for each of
and Pentland presented the well-known Eigenfaces
these regions, the weights will be more
method for face recognition.
representative of the local information of the face.
Suppose there are M training face images for each
When there is a variation in the pose or illumination,
of K subjects. Let each face image A(x, y) be a 2-
only some of the face regions will vary and rest of
dimensional N-by-N array of pixel values. The image
the regions will remain the same as the face regions
may also be represented as a vector of dimension N 2.
of a normal image. Hence weights of the face regions
Let’s denote each face image of the training set as fij,
are not affected by varying pose and illumination
and corresponding average of all training images as
will closely match with the weights of the same
g. Principal components per image are eigen values
individuals face regions under normal conditions. We
of wi = eigen values of covariance matrix of (fij – g).
implemented segmented PCA variations of 1D and
Principal components transform input image to
2D PCA methods that assign equal weight to all the
lower dimensional feature vector
sub images.
yk = wiT*fij
2.4. LDA Fisher’s Linear Discriminant Analysis
2.2. 2D PCA
PCA methods reduce the dimension of input data
In the PCA-based face recognition technique, the
by a linear projection that maximizes the scatter of
2D face image matrices must be previously
all projected samples. Fisher’s Linear Discriminant
transformed into 1D image vectors. The resulting
(FLD) shapes the scatter with the aim to make it
image vectors of faces usually lead to a high
more suitable for classification. A computation of the
dimensional image vector space, where it is difficult
transform matrix results in maximization of the ratio
to evaluate the covariance matrix accurately due to
of the between-class scatter.
its large size and the relatively small number of
In choosing the projection which maximizes total
training samples. As opposed to conventional PCA,
scatter, PCA retains some of the unwanted variations
2DPCA is based on 2D matrices rather than 1D
due to lighting and facial expression. The variations
vectors.
between images of the same face due to illumination
Image covariance or scatter matrix is computed
and viewing direction are almost always larger than
from M training images as
M image variations due to change in face identity. Thus
1 T
while PCA projections are optimal for reconstruction
G= ∑
M 1
( A j− Á ) ( A j − Á )
from a low dimension basis, they may not be optimal
Eigenvectors of scatter matrix G corresponding to from a discrimination standpoint.
the d largest eigen values are the 2D principal In this project we implement a variant of LDA
components of image Aj, If we denote the optimal called D-LDA. The basic premise behind the D-LDA
projection vectors as {X1(j), X2(j), …, Xd(j)}, approaches is that the information residing in (or
corresponding features of image are computed as close to) the null space of the within-class scatter
Yk= A*Xk. The matrix of features is called the matrix is more significant for discriminant tasks than
feature matrix and B = [Y1, Y2, .. Yd]. the information out of (or far away from) the null
space. Generally, the null space of a matrix is
2.3. Segmented PCA determined by its zero eigenvalues. However, due to
insufficient training samples, it is very difficult to
The PCA based face recognition method is not
identify the true null eigenvalues. As a result, high
very effective under the conditions of varying pose
variance is often introduced in the estimation for the
2
zero (or very small) eigenvalues of the within-class eyes, smiling / not smiling) and facial details (glasses
scatter matrix. Note that the eigenvectors / no glasses). All the images were taken against a
corresponding to these eigenvalues are considered to dark homogeneous background with the subjects in
be the most significant feature bases in the D-LDA an upright, frontal position (with tolerance for some
approaches. side movement).
The AT&T Face database is good for initial tests,
2.5. Support Vector Machines but it’s a fairly easy database. The Eigen faces
The goal of SVM classifiers is to find a hyperplane method already has a 90+% recognition rate, so we
that separates the largest fraction of a labeled data set didn’t expect to see considerable improvements with
{(x(i), y(i)); x(i) ϵ RN; y(i) ϵ {-1,1}; i=1,2…,N}. other algorithms. The Yale Face database A is a more
appropriate dataset for our experiments, because the
The most important requirement, which the
recognition problem is harder. The database consists
classifiers must have, is that they must maximize the
of 15 people (14 male, 1 female) each with 11
distance or the margin between each class and the
grayscale images sized 320 × 243 pixel. There are
hyperplane. In most of real applications, the data
changes in the lighting conditions (center light, left
cannot be linearly classified. To deal with this
light, right light), facial expressions (happy, normal,
problem, we transform data into a higher
sad, sleepy, surprised, wink) and glasses (glasses, no-
dimensional feature space and assume that our data
glasses).
in this space can be linearly classified.
We select first M images of each subject for
The discriminant hyperplane is defined as the
computing features of that subject class. These
following
N features are then used for training SVM with RBF
y(x) = ∑ α y ( i) K ( x (i ) , x ) + b where K is the kernel and the images not used for training are used
1 for classification testing.
kernel function for SVM. In this paper we use a
radial basis function kernel for SVM classification. Graph 1: PCA method accuracy vs number
of training images
2.6. Algorithm description
PCA or LDA method is used to identify features of
training images. To apply SVM for classification, we
use one-against-all decomposition to transform a
multi-class problem into a two-class problem.
Training set D= {(x(i), y(i)); x(i) ϵ R N; y(i) ϵ
{1,2,3,…,K}; is transformed into a series of D k=
{(x(i), yk(i)); x(i) ϵ RN; y(i) ϵ {k, 0}.
We use the MATLAB function svmtrain to
compute discriminant function of SVM and in the
classification phase, we use the following rule with
4. Results
MATLAB function svmclassify to identify the class
of test probe image x. In this section we present and discuss a comparison
of each of the previously mentioned feature
3. Experimental Setup extraction techniques with SVM classifiers.
We use two databases for our experiments
4.1. Effect of training size variations and number of
- AT&T/ ORL face database
principal components variations
- Yale face database
The AT&T Face database, sometimes also known Training size significantly affects the accuracy of all
as ORL Database of Faces, contains ten different methods but there is a ceiling on the highest level of
images of each of 40 distinct subjects. For some accuracy obtained from any of the methods. ORL
subjects, the images were taken at different times, database of faces achieves 90% accuracy within 3
varying the lighting, facial expressions (open / closed training images for 2D PCA and LDA methods

3
achieve 95% accuracy with 4 training images as
shown in the graphs below.
Graphs 2,3,4: Graph 2:2D PCA against number of
principal components for various training sets from
ORL
Graph 3: Segmented PCA against number of
principal components for various training sets ORL
Graph 4: LDA against number of principal
components for various training sets ORL
Graphs 5,6,7: Graph 5:2D PCA against number of
principal components for various training sets from
Yale
4
Graph 6: Segmented PCA against number of
principal components for various training sets Yale
Graph 7: LDA against number of principal
components for various training sets Yale
It is hard to compare the accuracy of 2DPCA and
LDA methods with just ORL database but Yale
database (graphs above) clearly shows that LDA
method is superior to 2D PCA when in class
variation is large.
Statistical learning methods PCA and LDA-based
ones seem to perform very well with small sample
and test size but often suffer from the so-called
‘‘small sample size’’ (SSS) problem if number of test
samples is very large.
 Looking closely at the inaccurate face detection in Gaussian noise of different variance in the
Yale database provides us insight into this problem. experiment.
When first 4 images of each subject are picked for
training, only subject 15 and 3 has dark glasses while
other subjects don’t have glasses. When test images
are run against such training set, all the probe images
of subjects with glasses are categorized into either
We also use the Wiener filter which is a MSE-
class 3 or class 15. Since all subject images with
optimal stationary linear filter for suppressing the
glasses were not used for training, our training model
degradation caused by additive noise and
is heavily biased towards two classes. That being
blurring. Fourier transforms are unable to recover
said, increasing the number of training images
components for which Fourier transform of point
improves the accuracy and we get around the bias.
spread function is 0. This means they are unable to
undo blurring caused by band limiting of Fourier
4.2. Variations in facial expression of subjects,
transform. We can see that some of the accuracy of
structural changes to the faces, and variations
face recognition is recovered when Weiner filters are
in pose
used for correction of additive noise in Graphs 8 and
Yale database also includes facial expressions of 9.
subjects and graphs 5,6,7 show that LDA method is Weiner filter noise suppressed images look like
more immune to variations in facial expressions but below
PCA method’s accuracy is not substantially lower
than LDA. LDA method is definitely superior when
Overall we are very surprised by how resilient
both sets of algorithms are to significant changes in
facial poses.

4.3. Presence of noise in probe image Graphs 8, 9:

Noise and distortions in face images can seriously

affect the performance of face recognition systems.
Analog or digital capturing the image has come a
long way and very good quality photo captures are
possible even with cell phone camera but biometric
system will need to be resilient to tampering so we
explore noise immunity of different algorithms now.
Noise in probe image degrades the performance of
all algorithms substantially. In this section we only
evaluate the 2D PCA and LDA based methods since
1D PCA has lower accuracy compared to 2DPCA
and segmented PCA methods are essentially just
PCA methods so the effect of noise on their
performance is easily studied by understanding the
effect of noise on PCA methods only. We also restrict
the description of results to only Yale database even
though ORL database was also studied and we found
the effects of noise had less dependency on the
database.

4.3.1 Gaussian noise

Gaussian noise is the most common noise
occurring in everyday life. We use zero mean

5
4.3.2 Speckle Noise
This granular noise occurs in ultrasound, radar and
X-ray images and images obtained from the
magnetic resonance. The multiplicative signal
dependent noise is generated by constructive and
destructive interference of detected signals. The
wave interference is a reason of multiplicative noise
occurrence in the scanned image. The speckle noise
is image dependent. Therefore it is considered hard
to find a mathematical model that describes the
removal of this noise, especially if we expect the
randomness of the input data Structural changes to
the face image such as beard, glasses etc. We had
identified Lee’s filter as the method to counter the
Speckle noise but speckle noise is not greatly
affecting the recognition performance so we
prioritized the study of Gaussian and S&P noise over
correction to Speckle noise.

4.3.3 Salt & Pepper Noise

Salt & pepper noise is perceived as a random
Graphs 10, 11: Additive speckle noise effect on occurrence of black and white pixels in a digital
accuracy for PCA and LDA. image.

It can be caused by incorrect data transmission or by

a damage of already received data. In CCD and
CMOS sensors or LCD displays, the salt & pepper
noise can be caused by permanently turned-on or
turned-off pixels. Remaining pixels are unchanged.
Usually, the intensity (frequency of the occurrence)
of this noise is quantified as a percentage of incorrect
pixels. The median filtering (as a specific case of
order-statistic filtering) is used as an effective
method for elimination of salt & pepper noise from
digital images. We can see that almost all of the
algorithm performance is recovered by use of median
filters.

Graphs 12, 13: Additive S&P noise effect on

6
accuracy for PCA and LDA and corrected with and that were taken in different sessions after longer
Median filter time periods. We also presented recognition results
for noisy images and compared them to results for
non-distorted images with correction for two types of
noises.
We started this project with the intent of
implementing face recognition algorithms with SVM
and definitely succeeded in that goal. Computer
vision and analytics systems performance is far
superior when combined with deep learning models
CNNs, combining deep-learning and multivariate
wide-learning together with improved feature
descriptor models can enable extraction of more
information from facial images such as facial
expression. During the research of this project, we
have stumbled upon papers that study separation of a
true smile from fake smile using PCAs for feature
recognition and CNNs for training and classification.
We would like to build fundamentals of CNNs and
machine learning and combine the power of
statistical models used in this project with deep
learning methods to create more interesting projects
like machine recognition of facial features.

https://pereza77@bitbucket.org/pereza77/cs231a_final_
project_face_recognition.git

5. Summary
We examined different subspace methods of face
recognition in this project. Two-stage recognition
systems include PCA, LDA for feature extraction
followed by SVM for classification. All methods are
significantly influenced by different settings of
parameters that are related to the algorithm used (i.e.
PCA, LDA or SVM).
For methods working in ideal condition both PCA
and LDA achieve greater than 90% accuracy within
three training images.
This project dealt with ‘closed’ image set, so we
did not have to deal with issues like detecting people
who are not in the training set. On the other hand our
two test databases contain images of the same
subjects that often differ in face expressions,
hairstyles, with or without beard, or wearing glasses

7
References
[1] M. A. Turk and A. P. Pentland, "Face recognition
using eigenfaces," in Computer Vision and Pattern
[2] Recognition, 1991. Proceedings CVPR '91., IEEE
Computer Society Conference on, 1991, pp. 586-591.
[3] Y. Jian, et al., "Two-dimensional PCA: a new
approach to appearance-based face representation and
recognition," Pattern Analysis and Machine
Intelligence, IEEE Transactions on, vol. 26, pp. 131-
137, 2004.
[4] A new LDA-based face recognition system which can
solve the small sample size problem. Pattern
Recognition 33, 1713–1726.
[5] Belhumeur, P. N., Hespanha, J., and Kriegman, D.
Eigenfaces vs. fisherfaces: Recognition using class
specific linear projection. IEEE Transactions on
Pattern Analysis and Machine Intelligence 19, 7
(1997), 711–720.
[6] B. Moghaddam and Y. Ming-Hsuan, "Gender
classification with support vector machines," in
Automatic Face and Gesture Recognition, 2000.
Proceedings. Fourth IEEE International Conference
on, 2000, pp. 306-311
[7] P. Phillips, et al., "The FERET evaluation
methodology for face-recognition algorithms," Pattern
Analysis and Machine Intelligence, IEEE
Transactions on, vol. 22, pp. 1090-1104, 2002.
[8] P. Viola and M. J. Jones, "Robust real-time face
detection," International Journal of Computer Vision,
vol. 57, pp. 137-154, 2004.
[9] T. H. Le and L. Bui, "Face Recognition Based on
SVM and 2DPCA," International Journal of Signal
Processing, Image Processing and Pattern
Recognition Vol. 4, No. 3, September, 2011
[10] MATLAB.com
[11] Brunelli, R., and Poggio, T. Face recognition through
geometrical features. In European Conference on
Computer Vision (ECCV) (1992), pp. 792–800
[12] Kanade, T. Picture processing system by computer
complex and recognition of human faces. PhD thesis,
Kyoto University, November 1973.
[13] H. Yu and J. Yang, "A Direct LDA Algorithm for
High-dimensional Data with Application to Face
Recognition," Pattern Recognition, Vol.34, pp.2067-
2070, 2001.
[14] F. Song, D. Zhang, J. Wang, H. Liu, and Q. Tao, "A
parameterized direct LDA and its application to face
recognition," Neurocomputing, Vol.71, pp.191-196,
2007.
[15] ORL face database -
http://www.uk.research.att.com/facedatabase.html
[16] Yale face database -
http://cvc.yale.edu/projects/yalefaces/yalefaces.html

8
 Emotion AI, Real-Time Emotion Detection using CNN
Tanner Gilligan
M.S. Computer Science
Stanford University
tanner12@stanford.edu
Abstract
In this paper, we describe a Convolutional
Neural Network (CNN) approach to real-
time emotion detection. We utilize data
from the Extended Cohn-Kanade dataset,
Japanese Female Facial Expression data
set, and our own custom images in or-
der to train this model, and apply pre-
processing steps to improve performance.
We re-train a LeNet and AlexNet imple-
mentation, both of which perform with
above 97% accuracy. Qualitative analysis
of real-time images shows that the above
models perform reasonably well at classi-
fying facial expressions, but not as well as
the quantitative results would indicate.
1 Introduction
The ability to confidently detect human emotions
can have a wide array of impactful applications,
and therefore emotion recognition has been a core
area of research in computer vision. We wanted
to focus on the issue of emotion recognition, and
build a real-time emotion detection system.
When we began to work on the area of emotion
detection, we quickly realized that there is an in-
nate problem which is that all data sets are based
on ”acted” emotions instead of ”real” emotions.
Many of these data sets such as CK+ ([Lucey et al.,
2010]) and JAFFE ([Lyons et al., 1998]) are col-
lections of actors who demonstrated core emotions
in front of a camera. Therefore the field isn’t
detecting real emotions, but rather detecting the
emotion that the subject is acting or the observer
is perceiving. This problem was also very obvi-
ous while testing our model, as we saw confidence
scores increase as the subject portray very exag-
gerated facial expressions that would be defined
as ”fake” by a human.
When we discussed possible applications of a
successful emotion recognition tool, one applica-
Baris Akis
B.S. Computer Science
Stanford University
bakis@stanford.edu
tion we though of is to use emotion labels and
prediction scores combined with social science
on emotion research led by Paul Ekman [Ekman,
1992] to predict emotion intensities. As indicated
in Frijda et al. [1992] emotion intensity prediction
is a really hard problem and a very valuable in-
sight for the field of psychology. The main rea-
son we didn’t pursue emotion intensity prediction
is that there were no existing data sets or research
that can serve as the ground truth.
Therefore we concentrated on building a suc-
cessful emotion recognition model that can work
in real-time. In this project we built a model that
uses Convolution Neural Networks to successfully
classify faces as one of the core seven emotions:
anger, contempt, disgust, fear, sadness, happiness,
surprise, and neutral [Darwin et al., 1998].
2 Background
2.1 Previous Works
Emotion Recognition in the Wild (EmotiW) Chal-
lenge is the leading academic challenge on emo-
tion recognition and labeling. We concentrated on
the winning papers of the 2014 and 2015 chal-
lenge. Its important to highlight that the papers
demonstrating the best results for the 2015 Image
based Static Facial Expression Recognition Sub-
challenge used CNNs. Yu and Zhang [2015] pro-
poses a CNN architecture specialized on emotion
recognition performance. They propose two novel
constrained optimization frameworks to automati-
cally learn the network ensemble weights by min-
imizing the loss of ensembled network output re-
sponses. Kim et al. [2016] took a different ap-
proach by creating a committee of multiple deep
CNNS. They also created a hierarchical architec-
ture of the committee with exponentially-weighted
decision making process.
There were also a wide variety of other papers
that suggested alternative methods to CNN, but
didnt perform as well. Most of these papers use
support vector machines (SVM) or largest mar-
gin nearest neighbor (LMNN) for classification.
The main difference between these were the fea-
ture descriptors. [Dhall et al., 2011] used a sys-
tem that extracts pyramid of histogram of gradi-
ents (PHOG) and local phase quantization (LPQ)
features for encoding the shape and appearance
information. [Yao et al., 2015] used AU (Ac-
tion Unit) aware features that were generated after
finding pairwise patches that are significant to dis-
criminate emotion categories. Yaos main insight
was that previous research groups neglected to ex-
plore the significance of the latent relations among
changing features resulted from facial muscle mo-
tions. This approach delivers results better than
the winning team of 2014 but falls short compared
to 2015 winners results. [Shan et al., 2009] con-
centrated on person-independent facial expression
recognition and used Local Binary Patterns (LBP)
descriptors. Many of these algorithms that used
feature based models aimed to mimic Ekman’s
suggestions for human emotion at [Ekman et al.,
2013].
2.2 Improvements on Previous Works
Since many of the most successful emotion recog-
nition applications used CNN’s, we also decided
to use CNNs as our model to target the emotion
recognition problem. We already started seeing
over 90% train and test accuracy by only using
CK+ data set. Similar to many other papers shared
below, we added JAFFE data set and this increased
our accuracies. Different than most other papers
in this area we also created data samples on our
own, and added these to our data sample as well.
Surprisingly this increased accuracies even fur-
ther. We were able to use these additional sam-
ples since we were using CNNs, and the accuracy
aren’t dependent on very specific features like AU-
intensities.
It was hard to find accurate benchmarks since a
lot of the papers were data set dependent. There-
fore we concentrated on research that also trained
their models on CK+ data set, and ideally added
JAFFE as well. In these papers, we saw results
ranging from 45% and 97%, which is in line with
our final results of around 97% test-set accuracy.
Chew et al. [2011] used face tracking and con-
strained local models (CLM) with CK+ data set
and had testing accuracy ranging 45.9% (sad) and
93.6% (surprise). Jeni et al. [2013] also used CK+
data set and they were able to reach 86% average
accuracy on continuous AU (Action Unit) inten-
sity estimation. Velusamy et al. [2011] used most
discriminative AUs for each emotion to predict
emotions and reached to 97% with CK+ data set
but only 87.5% with JAFFE. Since this approach
was really dependent on discriminative physical
expression of emotions it didn’t do as strongly in
JAFFE database. Islam and Loo [2014] Utilized
displacement of points on the face between neutral
and expressive emotions and was able to correctly
classify 83% (fear) to 97% (happiness) of the emo-
tions. Again this approach was able to success-
fully classify emotions like happiness and surprise
since large displacements in the mouth region cre-
ated discriminative results.
Our final results are very promising since they
result in accuracies that are less dependent on the
actors or emotions being portrayed.
3 Approach
3.1 Overview
In order to accomplish our task of developing a
real-time emotion detection system, we had to get
several components working independently. We
also had to conduct research to better understand
the basis for our problem, and how we could im-
prove our results once we got things working. Be-
low summarizes our general approach:
• Build Dataset: We collected labeled facial-
expressions data sets from multiple sources,
and processed their labels and images into a
common format. We then introduced custom
images of ourselves and a friend to further en-
rich the data set.
• Pre-process Images: We ran facial-detection
software to extract out the face in each im-
age. We then re-scaled the croppings, and
manually eliminated poor images. As a pre-
processing step for the CNN, we also applied
a Gaussian filter to the images, and subtracted
the mean-image of the training set from each
image. In order to get more out of our limited
training data, we also augmented the images
to include reflections and rotations of each
image, with the hope that this would improve
robustness
• Construct CNN: We utilized pre-trained ver-
sions of AlexNet and LeNet in Caffe on
Page 2 of 9
 AWS, where we re-trained the first and last
layer. We also had to experiment with vari-
ous learning rate methods and parameters in
order to generate a non-divergent model.
• Develop real-time Interface: OpenCV al-
lowed us to get images from our laptop’s we-
bcam. We then extracted the face as before,
pre-processed the image for the CNN, and
sent it to AWS. On the server, a script would
run the image through the CNN, get a predic-
tion, and the results would be pulled back to
local.
3.2 Implementation
3.2.1 Dataset Development
CK+ Dataset The first step in developing our
emotion-detection system was to acquire data with
which to train our classifier. We sought to find the
largest data set we could, and we selected the Ex-
tended Cohn-Kanade (CK+) data set. This data
set is composed of over 100 individuals portray-
ing 7 different labeled emotions: anger (1), con-
tempt (2), disgust (3), fear (4), happy (5), sad (6),
and surprise (7). In addition, we also introduced
an addition class-0 to represent a neutral expres-
sion. One feature we really liked about this data
set is that for each person displaying an emotion,
the directory contains 10 to 30 images demonstrat-
ing that individual’s progression from a neutral ex-
pression to the target emotion. This is good be-
cause it allows us to have multiple degrees of in-
tensity for each emotion represented in our data
set, as opposed to only the most extreme exam-
ples. We originally elected to take the first two
images of each sequence and label them as neu-
tral, and the last three and label them as the tar-
get emotion. We found that this greatly limited
our training set size, however, as we were left with
fewer than 1000 training images. To combat this,
we looked more closely at the images and decided
to take the last third of each sequence as the target
emotion, as opposed to just the last three.
Excluding Contempt Upon testing this 8-class
classifier, we found that it tended to over-predict
”contempt”. This manifested in quantitatively
lower recall and precision scores for the con-
tempt class, as well as qualitative worse predic-
tions when we fed it live images. We conducted
further research on this and found that many pa-
pers on emotion detection ignore the contempt
class, as they say it is merely a combination of fear
and disgust. Taking this into account, we dropped
all instances of contempt from our data set, and
re-split it. Thankfully contempt was the smallest
class in terms of image count, so we didn’t lose a
substantial part of our data set.
JAFFE Data Set After eliminating contempt,
we again tested our model qualitatively and quan-
titatively. We found that we were doing very well
quantitatively, our precision, recall, and accuracy
were all well over 90% on both test and train, but
our qualitative results were still rather poor. Since
the network was doing very well on data it was
given, but was not generalizing well, we decided
to find additional data sources. One of the research
papers we investigated combined the CK+ with the
Japanese Female Facial Expression (JAFFE) data
set, and was able to achieve improved results. Un-
fortunately, the data set only contained around 250
images, but it was still able to boost the model’s
performance by a few percent.
Custom Images Since the real-time interface
was being tested solely by us, we also decided to
add ourselves to the data set to see if it would im-
prove qualitative results. We found a friend to help
us, and the 3 of us proceeded to take an additional
20 images for each class, further increases our data
set size. After this final inclusion, our final data set
sizes were:
Set Size
Train 2104
Val 300
Test 601
Including the images from the JAFFE data set
and the ones we custom-made, we were able to
again boost our quantitative results by a small mar-
gin, and our qualitative results also noticeably im-
proved.
3.2.2 Data Pre-processing
Given the non-homogeneity of the data set, we had
to pre-process the data into a common format. We
first converted all images to grayscale. We then
utilized OpenCV to detect faces within each im-
age, which returned to us a set of bounding boxes
to examine. In cases where no bounding box was
found, we set that image aside and ran it again us-
ing different detection parameters until we were
able to successfully detect the face. In cases where
multiple bounding boxes were returned, we ana-
Page 3 of 9
lyzed the sizes and locations of the boxes, and se-
lected the one that was largest and/or most central
in the image. Using this approach, we were able to
extract the facial component of every image in our
data set. Once extracted, we re-scaled the images
to a common 250-by-250 size.
In order to help the CNN perform better, we
also applied statistical pre-processing to the im-
ages. The first step we took was applying a small
5-by-5 Gaussian filter over the images, which is
meant to help smooth-out noise while still preserv-
ing image’s edges and distinctive features. The
second step we took was subtracting the training-
set’s mean image from every image. This is ben-
eficial because the distribution of pixel values be-
comes centered at 0, and is common practice for
training data fed to any machine-learning model.
Since we only have 2104 distinct training im-
age’s, and Convolutional Neural Networks tend to
perform better with more data, we sought to find
ways to enrich this data set. To do this, we aug-
mented each image in two ways. First, we mir-
rored the image across the Y-axis, which produced
a similar but not identical training point. In ad-
dition, we also introduced slight rotations of 10
degrees in either direction for each image, which
helped to boost our training-set size, and improve
robustness.
3.2.3 CNN Construction
In order to develop our Convolutional Neural Net-
work, we decided to utilize pre-trained models.
We believed that this would lead to better results
for our project, since these pre-trained networks
are much deeper that we could develop, and would
thus have much better feature-detection power. In
researching existing networks, we couldn’t find
any that dealt directly with facial detection or
recognition, so we chose to use networks with
varying initial applications.
The first network we used was LeNet, which
was trained on the MNIST data set. The MNIST
data set is composed of hand-written numbers, and
the objective of the model is to classify each im-
age as a digit. The second network we looked at
was AlexNet, which was developed and trained for
the ImageNet Challenge. This challenge seeks to
classify images into one of 1000 categories, rang-
ing from animals to beverages. Even though nei-
ther of these networks deals directly with faces,
our hope is that the lower level features learned by
these networks, such as edges and curves, can be
transferred from these data-rich environments to
our data-poor environment.
Since LeNet and AlexNet were trained with dif-
ferent intentions than our own, we needed to tweak
the networks slightly. First, since our input images
were neither color nor the 227-by-227 dimension
utilized by these networks, we had to change the
input data-layer and retrain the first convolutional
layer to account for this. Second, since we are
only predicting 7 classes rather than the 1000 orig-
inally used, we needed to retrain the final softmax
layer. As a result of us having far fewer training
images than these networks originally had, we also
had experiment with different learning rate hyper-
parameters in order to induce convergence, as the
original hyper-parameters often diverged. We ul-
timately settled on a ”fixed” learning rate policy,
with base learning rate of 0.001 with a momen-
tum of 0.9 and weight decay of 0.0005. Note that
even though a ”fixed” learning rate policy is used,
the SGD solver of Caffe still uses the momentum
and weight decay to steadily reduce weight up-
dates over time.
3.2.4 Real-Time Interface
In order to create the real-time interface, we
needed to gather local images and run them
through the CNN on AWS. To accomplish this,
we utilized OpenCV to extract the images from
out laptop’s webcam. The images were them pre-
processed in the same manner as our data set: con-
vert it to grayscale, extract the facial component,
and re-scale to 250-by-250. We chose to use 250-
by-250 images because AWS would only allow us
to send at most 65KB in a single file, so the 250-
by-250 images fit within this constraint.
On AWS, we had a server script that has our
trained model loaded into memory, and waits for
an incoming file. When received, the image has
a Gaussian filter applied, and the mean image is
subtracted, just as with the rest of the data set. The
image is then augmented via a mirroring and ro-
tation, and the set of images is fed into the neural
network. A prediction is produced for each image,
and we select our prediction to be the most com-
mon class label among the images. If there is a tie,
we select the class with the highest sum of class
scores among the maximal classes.
One limitation of AWS is that it does not allow
you to send data directly to a local computer, so
our script could not simply send the results back
when the computation was finished, or even sig-
Page 4 of 9
nal to us that it was done. We had our first script Confusion matrix
write the results to a file and have a second socket 149 0 2 1 0 0 1 0
that listens the file. We had another local script 1 61 0 0 0 0 1 0
that called the second AWS server after sending 4 0 5 0 0 0 0 0
images were done to retrieve the results and this 3 0 0 51 0 0 0 0
combinations of four scripts and two sockets cre- 0 0 0 0 27 0 0 0
ated a close to real-time interface. 1 0 0 0 0 82 0 0
0 0 0 0 0 0 33 0
4 Experiments and Results 1 0 0 0 0 0 0 80

4.1 Result Metrics Precision

.94 1.0 .71 .98 1.0 1.0 .97 1.0
For our project, we have two metrics we use to
evaluate the performance of our model. The first Recall
is the obvious quantitative results, such as pre- .97 .98 .56 .94 1.0 .99 1.0 .99
cision, recall, and accuracy, in which we exam-
ine the statistic success of our model in predicted
our labeled data set. The second success measure Dataset Acc.
we use is how well it is able to classify our live- Train 0.988
streamed images. Since there are no labels for Val 0.972
these images, only us knowing which expression Test 0.972
we are trying to portray, it is difficult to quantita-
tively examine these results without have to hand- From the above results, specifically the pre-
label a second data set (which we do in some in- cision and recall of class 2 (contempt), we can
stances). Furthermore, since neither of us are ac- see that this class is clearly performing the worst.
tors, our expressions could also be poor portrayals Upon inspecting the same statistics on both our
of the target emotions, but this is also a more real- training and validation set, we find similar results.
istic application of the system. In the below anal- In addition, our qualitative analysis also indicates
ysis, we describe our results with respect to both that the contempt class was causing issues, as most
of these metrics. images we sent it to predict were classified as con-
tempt, even though it is the minority class. On
4.2 8-class Prediction a sample of 20 images we sent to be predicted,
In our first round of experimentation, we used 4 were classified correctly, 3 were classified in-
the full 8-class CK+ data set including contempt. correctly, but not as contempt, and the remaining
Here, the classes are: 13 were all labeled as contempt. This revelation
is what drove us to investigate literature further,
and found that most emotion detection researchers
0: Neutral
tend to discard the contempt class. We followed
1: Anger
this example, and retrained our model excluding
2: Contempt
this class
3: Disgust
4: Fear
4.3 7-class Prediction
5: Happiness
6: Sadness After retraining out model, we achieved improved
7: Surprise results. Due to the fact that contempt made up
such a small portion of the training data, the
We used only LeNet for our initial exploration, changes in accuracy and precision aren’t very
and after tuning hyperparameters, achieved the much, but they are substantial when considering
following results on our test set: that only about 1% of the data set was altered.
Upon extracting our summary statistics, from the
model, and using the same class labels as above
(dropping contempt), we obtained the following
results:

Page 5 of 9
 Confusion matrix it includes images for the remaining 6 emotions.
128 0 0 1 0 0 1 The inclusion of this data set resulted in the fol-
1 51 0 0 0 0 0 lowing:
3 0 57 0 0 0 0
0 0 0 34 0 0 0 Precision
0 0 0 0 78 0 0 0.98 0.95 0.88 0.97 0.99 0.75 1.0
0 0 0 0 0 32 0
Recall
1 0 0 0 0 0 100
0.97 0.98 0.92 0.91 0.93 1.0 0.92
Precision
0.96 1.0 1.0 0.97 1.0 1.0 0.99
Recall Dataset Acc.
0.98 0.98 0.95 1.0 1.0 1.0 0.99 Train .979
Val 0.969
Dataset Acc. Test 0.945
Train 1.0
Val 0.975 We can see that the JAFFE data set noticeably
Test 0.986 reduced our accuracies across all three splits. In
addition, the precision of class 5 (sadness) took a
It is difficult to directly compare the results from big hit, dropping from 1.0 to 0.75. In an attempt
the 7-class and 8-class case since the data was to better understand why this occurred, we looked
re-segmented when removing contempt. Despite at some of the JAFFE images that were labeled
this, one can clearly see that the precision, recall, as sad. We found that some of these images were
and accuracy for every class increased between rather poor or subtle examples of a sad expression,
the two runs. This indicates that excluding con- and could easily be confused with neutral by just
tempt not only improve performance by not mis- looking at them. Below is an example of an image
classifying contempt images, but also preventing that is labeled as sad, but was incorrectly classified
other images from being confused with contempt, as neutral by our model:
and were thus classified correctly. Our qualitative
results also improved as a result of this change,
and we were able to get more correct predictions.
In a sample of 20 webcam images we sent to the
network, 8 were classified correctly, and we had
100% accuracy on surprise images. An example
of a correctly classified image is below:

We decided to leave these bad images in the

4.4 JAFFE Predictions
Once we removed the contempt class from our
data set, we were able to add the Japanese Female
Facial Expression (JAFFE) data set as well, since
data set because, even though they are poor exam-
ples of the target emotion, they are still valid ex-
pressions of them. By excluding them, we would
be hand-picking our data set to only train on ex-
aggerative expressions that would likely never be
present in the real world. Instead, we elected
to simply add more data to our data set in the
hopes that it would both disambiguate some of the
JAFFE images, as well as improve our overall per-
formance.

Page 6 of 9
4.5 Custom Images network, the results were much better than with
In the final iteration of constructing our data set, our previous models. When portraying extreme
we added a total of 420 images equally split expressions of surprise or sadness, the model cor-
among the 7 classes. In addition to providing addi- rectly classified them with near perfect accuracy.
tional, unique images to the model, it also helped When making expressions that were less exagger-
to balance the class distribution, which is an is- ative, the model was not able to classify the im-
sue we had previously been unable to address. By ages very well, as one might expect. This is be-
adding these images into our data set, we were cause the key features that differentiate the classes
able to achieve results similar to the 7-class model are not readily apparent, so the model can not pre-
in nearly every category, which is significant given dict as well. One class that the model qualitatively
we retain the JAFFE data set which previously de- performs very poorly at is happiness. In our nu-
creased our performance. In addition, this was the merous attempts to elicit a prediction of happiness
first instance where AlexNet outperformed LeNet, from our model, we nearly always failed. Interest-
so below we show AlexNet’s results: ingly, when a friend of ours attempted to do the
same, she was able to consistently get predictions
Confusion matrix of happiness when we couldn’t. We are unsure
136 0 0 0 0 0 0
why this would occur, but it could be caused by
0 74 0 0 0 0 0
an underlying artifact of our data set, such as a
0 1 59 0 0 0 0
woman’s facial features being more highly associ-
1 1 1 40 0 0 1 ated with a happy expression.
0 0 0 0 117 0 0
In an attempt to quantify our relatively qualita-
0 0 0 1 1 54 0
tive results, we tried to classify 50 live-stream im-
1 0 0 1 0 0 112 ages. Of those we sent, the network correctly clas-
Precision sified 28, typically being those with the most ex-
0.99 0.97 0.98 0.95 0.99 1.0 0.99 aggerative expressions. As previously discussed,
Recall surprise and sadness performed the best, while
1.0 1.0 0.98 0.91 1.0 0.96 0.98 happiness performed the worst. In addition to
looking at the predicted class, we also analyzed
the class scores output for incorrectly classified
Dataset Acc. image. In comparing the class scores produced
Train 1.0 by our earlier models to those produced our fi-
Val 0.99 nal model, we noted a respectable increase in the
Test 0.985 score for the correct class. In every case we ex-
amine, the final model produced class scores such
that the correct class was either the second or third
AlexNet Loss
maximal score, while our previous models had no
such guarantee. This shows that even though we
couldn’t correctly classify the images, our predic-
tions were at least closer to being correct. In sum-
mary, we were able to achieve improved results on
our life-streamed images, but not nearly as well as
our quantitative results would indicate.

In addition to quantitative improvements in our
model, we also experienced qualitative improve-
ments results as well. When observing the live-
stream of predictions being returned to us by our
5 Conclusion
In this paper, a CNN-based emotion detection
model is proposed that utilizes facial-detection
software and cloud computing to accomplish its
task. The final model resulted in accuracies com-
parable to the state-of-the-art papers in the field,
reaching as high as 98.5% accuracy on our cus-
tom data set, and 97.2% on the original CK+ data

Page 7 of 9
set. Our code base can be fount at https://
github.com/barisakis/cs231a_eai. In
addition, our model also exhibits more balanced
accuracy results across the emotion spectrum.
Lastly the proposed model still worked signifi-
cantly well with non-actor subjects, especially for
physically expressive emotions like sadness, hap-
piness and surprise.
One future area of work is to create a user in-
terface where users can iteratively train the model
through correcting false labels. This way the
model can also learn more from real world users
who express various emotions in different ways.
In addition, including a layer in the network that
accounts for class imbalance could provide addi-
tion improvements over our results. We attempted
implement latter of these, but were unable to get it
working.
Another area of interest to explore is predicting
on a continuous scale the intensity of emotions be-
ing portrayed. We believe that we already have a
reliable recognition algorithm, so by incorporat-
ing knowledge from the social sciences on emo-
tion, a more powerful predictor could be built. In
order develop and train such a predictor, however,
one would need an annotated data set with which
to work. One possible means for creating such a
data set would be to aggregate people’s opinions
of an image using a service like Amazon Mechan-
ical Turk, and then average the responses together
to produce an intensity measure for each emotion.
Even though many state of art algorithms are
very good at detecting facial expressions, these
images are often exaggerative and un-realistic. As
such, we believe that there is need for better data
sets aimed at understand ’real’ emotions.
References
S. W. Chew, P. Lucey, S. Lucey, J. Saragih, J. F.
Cohn, and S. Sridharan. Person-independent fa-
cial expression detection using constrained lo-
cal models. In Automatic Face Gesture Recog-
nition and Workshops (FG 2011), 2011 IEEE
International Conference on, pages 915–920,
March 2011. doi: 10.1109/FG.2011.5771373.
Charles Darwin, Paul Ekman, and Phillip Prodger.
The expression of the emotions in man and ani-
mals. Oxford University Press, USA, 1998.
Abhinav Dhall, Akshay Asthana, Roland Goecke,
and Tom Gedeon. Emotion recognition using
phog and lpq features. In Automatic Face &
Gesture Recognition and Workshops (FG 2011),
2011 IEEE International Conference on, pages
878–883. IEEE, 2011.
Paul Ekman. An argument for basic emotions.
Cognition & emotion, 6(3-4):169–200, 1992.
Paul Ekman, Wallace V Friesen, and Phoebe
Ellsworth. Emotion in the human face: Guide-
lines for research and an integration of findings.
Elsevier, 2013.
Nico H Frijda, Andrew Ortony, Joep Sonnemans,
and Gerald L Clore. The complexity of inten-
sity: Issues concerning the structure of emotion
intensity. 1992.
Md Nazrul Islam and Chu Kiong Loo. Geo-
metric feature-based facial emotion recognition
using two-stage fuzzy reasoning model. In
Neural Information Processing, pages 344–351.
Springer, 2014.
L. A. Jeni, J. M. Girard, J. F. Cohn, and F. De La
Torre. Continuous au intensity estimation us-
ing localized, sparse facial feature space. In
Automatic Face and Gesture Recognition (FG),
2013 10th IEEE International Conference and
Workshops on, pages 1–7, April 2013. doi:
10.1109/FG.2013.6553808.
Bo-Kyeong Kim, Jihyeon Roh, Suh-Yeon Dong,
and Soo-Young Lee. Hierarchical committee of
deep convolutional neural networks for robust
facial expression recognition. Journal on Mul-
timodal User Interfaces, pages 1–17, 2016.
Patrick Lucey, Jeffrey F Cohn, Takeo Kanade, Ja-
son Saragih, Zara Ambadar, and Iain Matthews.
The extended cohn-kanade dataset (ck+): A
complete dataset for action unit and emotion-
specified expression. In Computer Vision
and Pattern Recognition Workshops (CVPRW),
2010 IEEE Computer Society Conference on,
pages 94–101. IEEE, 2010.
Michael Lyons, Shota Akamatsu, Miyuki Ka-
machi, and Jiro Gyoba. Coding facial expres-
sions with gabor wavelets. In Automatic Face
and Gesture Recognition, 1998. Proceedings.
Third IEEE International Conference on, pages
200–205. IEEE, 1998.
Caifeng Shan, Shaogang Gong, and Peter W
McOwan. Facial expression recognition based
on local binary patterns: A comprehensive
study. Image and Vision Computing, 27(6):803–
816, 2009.
Page 8 of 9
S. Velusamy, H. Kannan, B. Anand, A. Sharma,
and B. Navathe. A method to infer emotions
from facial action units. In 2011 IEEE In-
ternational Conference on Acoustics, Speech
and Signal Processing (ICASSP), pages 2028–
2031, May 2011. doi: 10.1109/ICASSP.2011.
5946910.
Anbang Yao, Junchao Shao, Ningning Ma, and
Yurong Chen. Capturing au-aware facial fea-
tures and their latent relations for emotion
recognition in the wild. In Proceedings of
the 2015 ACM on International Conference on
Multimodal Interaction, pages 451–458. ACM,
2015.
Zhiding Yu and Cha Zhang. Image based static fa-
cial expression recognition with multiple deep
network learning. In Proceedings of the 2015
ACM on International Conference on Multi-
modal Interaction, pages 435–442. ACM, 2015.

Page 9 of 9
End-to-end learning of motion, appearance and interaction cues for multi-target
tracking
Amir Sadeghian Khashayar Khosravi
Stanford University Stanford University
amirabs@stanford.edu khosravi@stanford.edu
Abstract
The task of Multi-Object Tracking (MOT) largely con-
sists of locating multiple objects at each time frame, and
matching their identities in different frames yielding to a
set of object trajectories in a video frame. There are sev-
eral cues used for representing the individuals in a crowded
scene. We demonstrate that in general, fusing appearance,
motion, and interaction cues together can enhance the level
of performance on the MOT task. In this paper we combined
appearance, motion and interaction cues in one deep uni-
fied framework. An important contribution of this work is
a generic scalable MOT method that can fuse rich features
from different dynamic or static models.
1. Introduction
Multiple Object Tracking (MOT), or Multiple Target
Tracking (MTT), plays an important role in computer
vision and is a crucial problem in scene understanding.
The objective of MOT is to produce trajectories of objects
as they move around the image plane. MOT covers a
wide range of application such as pedestrians on the street
[26, 19] sports analysis (e.g. sport players in the court
[14, 17, 24], bio tracking (birds [15], ants [9], fishes
[20, 21, 5], cells [16, 12], and etc), robot navigation, or
autonomous driving.). In crowded environments occlu-
sions, noisy detections (false alarms, missing detections,
non-accurate bounding), and appearance variability (Same
person, different appearance or different people, same
appearance) are very common. As a result, multi object
tracking has become challenging task in computer vision.
Recent works have proven that tracking objects jointly
and taking into consideration their interaction in addition
to their appearance can give much better results in crowded
scenes. The focus of this paper is to marry the concepts of
appearance model, object motion, and object interactions
to obtain a robust and scalable tracker than works in
1
Alexandre Robicquet
Stanford University
arobicqu@stanford.edu
crowded scenes. We propose an online unified deep neural
network tracker that jointly learn to reason on a strong
appearance model, strong individual motion model, and
object interactions (dynamic scene knowledge). In this
project we will not study the interaction model and only
focus on appearance and motion model.
Our strong appearance model is a Siamese convolutional
neural network (CNN) that is able to find occlusions and
similarity of objects in different time frames in addition to
object bounding box prediction in next time frame. We also
use two seperate Long Short-Term Memory (LSTM) model
for our motion prior and interactions model that tracks
the motion and trajectory of objects for longer forecasting
period (suited in presence of long-term occlusion). These
models extract appearance cues, motion priors, and inter-
active forces which are critical parts of the MOT problem.
We then integrate these parts into a coherent system using
a high-level LSTM that is responsible to reason jointly
on different extracted cues. We show our model is able
to fuse and use different data modalities and get a better
performance. The magic is this scalability, one can add
another cue component (e.g. exclusion model) to the model
and finetune the model to reason jointly on the new cue and
previous cues.
2. Related Work
In recent years tracking has been successfully extended
to scenarios with multiple objects [18, 11, 8, 23]. Differ-
ent from single object tracking approaches which have been
constructing a sophisticated appearance model to track sin-
gle object in different frames, multiple object tracking does
not mainly focus on appearance model. Although appear-
ance is an important cue but in crowded scenes relying only
on appearance can lead to a less accurate MOT system. To
this end, different works have been improving only the ap-
pearance model [6, 3], some works have been combining
the dynamics and interaction between targets with the tar-
get appearance.
2.1. Appearance model
Technically, appearance model is closely related to vi-
sual representation features of objects. Depending on how
precise and rich the visual features are, they are grouped
into three sets of single cue, multiple cues, and deep cue.
Because of efficiency and simplicity single cue appearance
model is widely used in MOTs. Many of single cue mod-
els are based on raw pixel template representation for sim-
plicity [25, 2, 22, 19], while color histogram is the most
popular representation for appearance modeling in MOT
approaches [4, 11, 28]. Other single cue approaches are
using covariance matrix representation, Pixel comparison
representation, or SIFT like features. The multi cues ap-
proaches combines different kinds of cues to make a more
rebust appearance model. The final appearance cue used in
tracking is the deep visual reperesentation of objects. These
high-level features are extracted by deep neural networks
mostly convolutional neural networks trained for a specific
task [7]. Our model shares some characteristics with [7],
but differs in two crucial ways: first, we are learning to han-
dle occlusion and solve the re-identification task in addition
to David’s work that is bounding box regression only. We
output the similarity score (same object or not) and bound-
ing box. Second, there are differences in the overall archi-
tecture, e.g. the number of fully connected layers on top
of two networks for fusing, loss function, inputs and out-
puts and hence the training and testing procedure is different
since we want to address re-identification as well as bound-
ing box to help tracking.
2.2. Motion model
Object motion model describes how an object moves.
Motion cue is very important for multiple object tracking
since knowing the potential position of objects in the fu-
ture frames will reducing search space and help the appear-
ance model on better detectation of similar objects. Popular
motion models used in multiple object tracking are divided
into linear motion models and Non-linear motion models.
As the name ”linear motion” indicates objects following the
linear motion model move with constant velocity. This sim-
ple motion model is the is the most popular model in MOT
[3]. There are many cases that linear motion models can not
deal with, in this cases non-linear motion models are pro-
posed to produce a more accurate motion model for objects
[27]. We present a new Long Short-Term Memory (LSTM)
model which jointly reasons based on the past movements
of an object and predicts the future trajectorys of that object
[1].
2
3. Multi Object Tracking Framework
As shown in Figure 1, MOT involves three primary com-
ponents. Our model includes modeling of appearance, mo-
tion, and interaction. These components will be described
in more details.
MOT
Components
Appearance Motion Interaction
Figure 1. MOT components
3.1. Appearance
In this section, we now describe the appearance model
that we integrate into our framework for multi-object track-
ing. As we recall, our problem is fundamentally based on
addressing the challenge of data association: that is, given a
set of targets Tt at time step t, and a set of candidate detec-
tions Dt+1 at timestep t + 1, we would like to compute all
of the valid pairings that exist between members of Tt and
Dt+1 .
The idea underlying our appearance model is that we can
compute the similarity score between a target and candi-
date detection based on purely visual cues. More specif-
ically, we can treat this problem as a specific instance of
re-identification, where the goal is to take pairs of bound-
ing boxes and determine if their content corresponds to the
same person. We thus desire our appearance model to rec-
ognize the subtle similarities between input pairs, as well as
be robust to occlusions and other visual disturbances.
To approach this problem, we construct a Siamese Con-
volutional Neural Network (CNN), whose structure is de-
picted in Figure 2. Let BBi and BBj represent the two
bounding boxes we wish to compare – in our case, BBi
might be a target bounding box at frame t, and BBj would
be a candidate detection at frame t+1. We first crop the im-
ages containing BBi and BBj to contain only the bound-
ing boxes themselves, while also ensuring that we include
some amount of the surrounding image context. The net-
work then accepts the raw content within each bounding
box and passes it through its layers until it finally produces
a 500-dimensional feature vector for each of the two inputs.
Let φi and φj thus be the final hidden activations ex-
tracted by our network for bounding boxes BBi and BBj .
In order to compute the similarity, we then simply con-
catenate the two vectors to get a 1000-dimensional vector
φ = φi ||φj , and pass this as input to a final fully-connected
layer. We lastly apply a Softmax classifier, which outputs
the probabilities for the positive and negative classes, where

Figure 2. Our appearance model
positive indicates that the inputs match, and negative indi-
cates otherwise.
The actual network structure we use for this challenge
consists of the 16-layer VGG net, which won the ImageNet
2014 localization challenge. In our case, we begin with
the pre-trained weights of this network, but remove the last
fully-connected layer so that the network now outputs a
500-dimensional vector.
We then fine-tune this network by training the overall
network on positive and negative samples extracted from
our training sequences. For positive pairs, we use instances
of the same target that occur in different frames. For neg-
ative examples, we use pairs of different targets that may
span across all frames.
We trained this model on MOT3D dataset which con-
tains 2 scenes with more than 950 frames that contain more
than 5500 objects. We extracted more than 100k of pos-
itive and negative samples. We did the training on one
scene and validated on the other scene. The result was
84 percent accuracy on the binary classification problem of
positive/negative pairs. We used CUHK03 dataset [13] as
the sanity check for our prediction. This dataset contains
13164 images of 1360 pedestrians and contains 150k pairs.
FPNN method which got rank 1 of identification MAP rate
were able to achieve 19.89 percent accuracy. Our method
achieves 18.61 percent of accuracy and outperforms several
other methods such as LDM, KISSME, SDALF.
3.2. Motion
The second component of our overall framework is the
inclusion of an independent motion prior for each target.
The intuition is that the previous movements for a particular
target can strongly influence what position a target is likely
to be at during a future time frame.
Additionally, a nuanced motion prior can help our model
when tracking objects that are occluded or lost, since it pro-
vides a heuristic as to where these objects might generally
be located. Thus, formulating a sophisticated model for the
motion prior of a target will be valuable in achieving robust
performance during tracking.
We can therefore use this information to aid us in the
3
task of data association, in which we can match members
of Tt and Dt+1 based on which detections are closest to the
motion prior’s next predicted location for each target.
To thus incorporate this information, we construct
a Long Short-Term Memory (LSTM) network over the
3D velocities of each target. More concretely, let
(xi0 , y0i , z0i ), (xi1 , y1i , z1i ), . . . (xit , yti , zti ) represent the 3d tra-
jectory of the i-th target from the timestep 0 through
timestep t. Assuming a point (xit+1 , yt+1 i i
, zt+1 ), we want to
see whether this point belongs to the trajectory of i-th target.
Let use define the velocity of target i at the j-th timestep i to
be vji = (vxij , vyji , vzji ) = (xij −xij−1 , yji −yj−1 i
, zji −zj−1
i
).
This can be done by assigning an score to this point and see-
ing whether it is large enough or not. For this purpose, we
train our LSTM to accept as inputs the velocities of a single
target for timesteps 1, . . . , t and produces H-dimensional
outputs. We also pass the t + 1 velocity vector (which
we wish to determine whether it corresponds to a true tra-
jectory or not) from a fully-connector layer that brings it
to H-dimensional vector space. The last LSTM output is
then concatenated with this vector and the result is passed
to another fully connector layer which brings the 2H di-
mensional vector to the space of k features. Finally, another
fully connector layer, reduces the dimension to 2 which will
be used as the 0/1 classification problem during the train-
ing.
Figure 3. Our 3D motion prior model
Note that training occurs from scratch, and weights are
shared across all targets. Once we train the network, then
given a query target i at timestep t0 , the LSTM will output
a predicted velocity vti0 +1 . We can then simply add the ve-
locity to the query target’s position at t0 in order to compute
the motion prior’s predicted position for frame t0 + 1. That
is,
(xit0 +1 , yti0 +1 , zti0 +1 ) = (xit0 +vxit0 +1 , yti0 +vyti0 +1 , zti0 +zti0 +1 )
We therefore obtain the predicted position from the motion
prior, and can use this to filter out candidate detections that
are not sufficiently close to the prior.
For training this model, we used MOT3D dataset, which
only consists of true trajectories. We considered trajecto-
ries of length t + 1 = 7 and we assumed H = 128. For
each true trajectory, we changed the last element of it by a
randomly chosen object among all other objects that exist at
the same frame. By doing this we were able to reach to the
same number of invalid trajectories as the valid trajectories
(it is not good to have unbalanced distributions for train-
ing). After training this model, we were able to achieve the
accuracy of 95 percent for the 0/1 classification problem.
3.3. Integration
Given these three components of our framework for
Multi-object Tracking, we now describe the method by
which we integrate these parts into a coherent system. To
recall, we have identified appearance cues, motion priors,
and interactive forces as critical parts of the MOT problem.
We believe a sophisticated framework should merge these
pieces together in an elegant way. You can find the graphi-
cal model of our approach in figure 4. Each human has an
appearance edge and motion edge, and between every pairs
of humans there is an interaction edge.
A . . . ]
Appearance Edges
Human Node
h1 Interaction Edges
Motion Edges
M
A hn
h2 M For training this model, we used the pretrained compo-
M model end to end using MOT3D dataset.
Figure 4. The graphical model of our approach
Our overarching model is a Long Short-Term Memory
network which we construct over the already pre-trained
appearance, motion, and interaction modules. This LSTM
is trained to perform the task of data association: once
again, suppose we are at timestep t and wish to deter-
mine whether target i is matched to a detection d found
in timestep t + 1. We then train the LSTM to output the
probabilities of whether the t and d correspond to the same
object.
The inputs to the LSTM are feature vectors that we ex-
tract from our individual models. Let φA represent the hid-
den activations extracted from our appearance model before
the final fully connected layer of the network, where we in-
put the bounding boxes surrounding target i and detection
d. Let φM j be the hidden state of the Motion Prior LSTM
extracted at timestep j, and likewise let φIj be the hidden
feature vector of the Interaction model extracted at timestep
j. Then, the input to our integrator is given by
φj = φA ||φM j ||φIj
where we thus concatenate the individual feature vectors
output by the modules. Therefore, when we set up the
model we use these features as inputs to the LSTM and
train it to output either a positive or negative label for each
timestep (indicating whether there is a valid match) using a
standard Softmax classifier and cross-entropy loss.
An important point to note is that we train this LSTM
without fine-tuning the weights of the individual compo-
nents of the framework, which are each in fact trained sep-
arately. The overall model, composed by the previous com-
ponents is illustrated by figure 5 and the output of the model
is a similarity score which is used as a weight for the edges
of matching graph for matching the detections between time
frames.
Similarity Score
H Occluded
0.23
[
0.95
A M I
Motion Interaction
0.12
0.23
A
Appearance
Feature Feature Feature
False alarms
Extractor Extractor Extractor T
T+1
Figure 5. Our overall model
nents described in previous sections and fine-tune the whole
4. Experiments
In this section, we now describe our various experiments
and results, and then later peform a qualitative analysis on
model’s performance.
4.1. Baselines
We first discuss the various baselines that we use to es-
tablish a standard for comparison against our more nuanced
model.
4
 • Markov Decision Process Tracker people into the 3D world. It consists of two publicly avail-
able datasets: a crowded town center, and the well known
In [23], authors demonstrated success on 2D Multi-
PETS2009 dataset.
object tracking by formulating the tracking problem as
a Markov Decision Process (MDP). They represented 4.3. Results
every target as being in either an active, tracked,
lost, or inactive states, and learned the appropriate The accuracy and results of each component of our sys-
transition probabilities and rewards based on extracted tem is described at each of the experimental sections. Here
features. In order to evaluate this method on the 3D we see the final results of the tracker in table 6 for results of
challenge, we project the bottom-midpoint of the our tracker compared to other baselines on MOT3D chal-
predicted 2D bounding boxes to the ground plane lenge. The last 3 rows are our cross validation on MOT
(using the provided calibration parameters given in the challenge training set.
data sequences). Tracker MOTA (H) MOTP (H) MT (H) ML (L)
DBN (State of art) - 1st 51.1 61.0 28.7% 17.9%
KalmanSFM (Baseline) - 5th 25.0 53.6 6.7% 14.6%

• MDP Tracker with Linear Motion Prior Yu’s 3D 45.5 61.0 6% 3%

Though the MDP described above can obtain reason-
able results on the problem of multi-object tracking,
we additionally incorporate a simple linear 3D motion
prior into the feature vectors associated with each state
in the MDP. More specifically, we use the normalized
distance between a candidate detection and the motion
prior prediction as a feature in that state.
• MDP Tracker with LSTM Motion Prior
As a final baseline experiment, we realize that incorpo-
rating a simple linear motion prior may be too simplis-
tic of an approach to accurately model the movements
of a target. A more reasonable method is to use an
LSTM similar to our own motion model to output the
predicted 3D coordinates for every target, and then use
these values in the feature vectors as described above.
We report the results using the proposed method on the
3D MOT 2015 Benchmark which includes the PETS09-
S2L21 and the AVG-TownCentre2 sequences. The sensi-
tivity of the method to the omission of single variables is
evaluated on the PETS09-S2L1 dataset (available for train-
ing in the 3D MOT 2015 Benchmark). The corresponding
results of an evaluation in 3D image space (correct detection
requires at least 50% intersection-over-union score with the
reference) and in 3D world coordinates (correct detection
requires at most 1m offset in position) are reported in fol-
lowing section, respectively.
4.2. Datasets
We test our tracking framework on the Multiple Ob-
ject Tracking Benchmark [10] for people tracking. The
MOT Benchmark collects widely used video sequences in
the MOT community and some new challenging sequences.
We evaluate the proposed algorithm on MOT3D challenge
which provides the 3D coordinate of position of the feet of
Appearance only (Cross validation) 38.1 54.1 15% 20%
LSTM only (Cross validation) 28.9 48.3 9% 28%
Appearance and LSTM (Cross validation) 40.3 57.1 16% 19%
Appearance and LSTM (MOT Challenge) 28.3 51.7 29.1% 21.8%
Figure 6. Primary Results on MOT3D challenge
5. Conclusion
This paper proposes a deep neural network designed for
multi object tracking. Quantitative results show that the
tracking performance is superior to the baseline tracking
methods. Our tracker can also be fine-tuned for various
applications by providing more training videos of certain
types of objects. Overall, our real-time neural network
tracker opens up many possibilities for different applica-
tions and extensions, allowing us to learn from several cues
used for representing the individuals in a crowded scene.
We demonstrate that in general, fusing appearance and mo-
tion cues together can enhance the level of performance on
the MOT task. We show our model is able to fuse and use
different data modalities and get a better performance. One
of the main advantage of our tracker to others is the scala-
bility, one can add another cue component (e.g. exclusion
model) to the model and finetune the model to reason jointly
on the new cue and previous cues.
References
[1] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei,
and S. Savarese. Social lstm: Human trajectory prediction in
crowded spaces.
[2] S. Ali and M. Shah. Floor fields for tracking in high density
crowd scenes. In Computer Vision–ECCV 2008, pages 1–14.
Springer, 2008.
[3] M. D. Breitenstein, F. Reichlin, B. Leibe, E. Koller-Meier,
and L. Van Gool. Robust tracking-by-detection using a de-
tector confidence particle filter. In Computer Vision, 2009
IEEE 12th International Conference on, pages 1515–1522.
IEEE, 2009.

5
 [4] W. Choi and S. Savarese. Multiple target tracking in world
coordinate with single, minimally calibrated camera. In
Computer Vision–ECCV 2010, pages 553–567. Springer,
2010.
[5] E. Fontaine, A. H. Barr, and J. W. Burdick. Model-based
tracking of multiple worms and fish. In ICCV Workshop on
Dynamical Vision. Citeseer, 2007.
[6] H. Grabner, M. Grabner, and H. Bischof. Real-time tracking
via on-line boosting. In BMVC, volume 1, page 6, 2006.
[7] D. Held, S. Thrun, and S. Savarese. Learning to track at 100
FPS with deep regression networks. CoRR, abs/1604.01802,
2016.
[8] C. Huang, B. Wu, and R. Nevatia. Robust object tracking by
hierarchical association of detection responses. In Computer
Vision–ECCV 2008, pages 788–801. Springer, 2008.
[9] Z. Khan, T. Balch, and F. Dellaert. An mcmc-based particle
filter for tracking multiple interacting targets. In Computer
Vision-ECCV 2004, pages 279–290. Springer, 2004.
[10] L. Leal-Taixé, A. Milan, I. Reid, S. Roth, and K. Schindler.
MOTChallenge 2015: Towards a benchmark for multi-
target tracking. arXiv:1504.01942 [cs], Apr. 2015. arXiv:
1504.01942.
[11] B. Leibe, K. Schindler, N. Cornelis, and L. Van Gool. Cou-
pled object detection and tracking from static cameras and
moving vehicles. Pattern Analysis and Machine Intelligence,
IEEE Transactions on, 30(10):1683–1698, 2008.
[12] K. Li, E. D. Miller, M. Chen, T. Kanade, L. E. Weiss, and
P. G. Campbell. Cell population tracking and lineage con-
struction with spatiotemporal context. Medical image anal-
ysis, 12(5):546–566, 2008.
[13] W. Li, R. Zhao, T. Xiao, and X. Wang. Deepreid: Deep filter
pairing neural network for person re-identification. In Pro-
ceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 152–159, 2014.
[14] C.-W. Lu, C.-Y. Lin, C.-Y. Hsu, M.-F. Weng, L.-W. Kang,
and H.-Y. M. Liao. Identification and tracking of players
in sport videos. In Proceedings of the Fifth International
Conference on Internet Multimedia Computing and Service,
pages 113–116. ACM, 2013.
[15] W. Luo, T.-K. Kim, B. Stenger, X. Zhao, and R. Cipolla. Bi-
label propagation for generic multiple object tracking. In
Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, pages 1290–1297, 2014.
[16] E. Meijering, O. Dzyubachyk, I. Smal, and W. A. van Cap-
pellen. Tracking in cell and developmental biology. In Semi-
nars in cell & developmental biology, volume 20, pages 894–
902. Elsevier, 2009.
[17] P. Nillius, J. Sullivan, and S. Carlsson. Multi-target tracking-
linking identities using bayesian network inference. In Com-
puter Vision and Pattern Recognition, 2006 IEEE Computer
Society Conference on, volume 2, pages 2187–2194. IEEE,
2006.
[18] K. Okuma, A. Taleghani, N. De Freitas, J. J. Little, and D. G.
Lowe. A boosted particle filter: Multitarget detection and
tracking. In Computer Vision-ECCV 2004, pages 28–39.
Springer, 2004.
6
[19] S. Pellegrini, A. Ess, K. Schindler, and L. Van Gool. You’ll
never walk alone: Modeling social behavior for multi-target
tracking. In Computer Vision, 2009 IEEE 12th International
Conference on, pages 261–268. IEEE, 2009.
[20] C. Spampinato, Y.-H. Chen-Burger, G. Nadarajan, and R. B.
Fisher. Detecting, tracking and counting fish in low quality
unconstrained underwater videos. VISAPP (2), 2008:514–
519, 2008.
[21] C. Spampinato, S. Palazzo, D. Giordano, I. Kavasidis, F.-P.
Lin, and Y.-T. Lin. Covariance based fish tracking in real-
life underwater environment. In VISAPP (2), pages 409–414,
2012.
[22] Z. Wu, A. Thangali, S. Sclaroff, and M. Betke. Coupling
detection and data association for multiple object tracking.
In Computer Vision and Pattern Recognition (CVPR), 2012
IEEE Conference on, pages 1948–1955. IEEE, 2012.
[23] Y. Xiang, A. Alahi, and S. Savarese. Learning to track: On-
line multi-object tracking by decision making. In Interna-
tional Conference on Computer Vision (ICCV), pages 4705–
4713, 2015.
[24] J. Xing, H. Ai, L. Liu, and S. Lao. Multiple player tracking in
sports video: A dual-mode two-way bayesian inference ap-
proach with progressive observation modeling. Image Pro-
cessing, IEEE Transactions on, 20(6):1652–1667, 2011.
[25] K. Yamaguchi, A. C. Berg, L. E. Ortiz, and T. L. Berg. Who
are you with and where are you going? In Computer Vision
and Pattern Recognition (CVPR), 2011 IEEE Conference on,
pages 1345–1352. IEEE, 2011.
[26] B. Yang, C. Huang, and R. Nevatia. Learning affinities and
dependencies for multi-target tracking using a crf model.
In Computer Vision and Pattern Recognition (CVPR), 2011
IEEE Conference on, pages 1233–1240. IEEE, 2011.
[27] B. Yang and R. Nevatia. Multi-target tracking by online
learning of non-linear motion patterns and robust appear-
ance models. In Computer Vision and Pattern Recogni-
tion (CVPR), 2012 IEEE Conference on, pages 1918–1925.
IEEE, 2012.
[28] A. R. Zamir, A. Dehghan, and M. Shah. Gmcp-tracker:
Global multi-object tracking using generalized minimum
clique graphs. In Computer Vision–ECCV 2012, pages 343–
356. Springer, 2012.
 Near-Eye Display Gaze Tracking via Convolutional Neural Networks
Robert Konrad Shikhar Shrestha
rkkonrad@stanford.edu shikhars@stanford.edu
Abstract
Virtual reality and augmented reality systems are cur-
rently entering the market and attempt to mimic as many
naturally occurring stimuli in order to give a sense of im-
mersion. Many aspects such as orientation and positional
tracking have already been reliably implemented, but an im-
portant missing piece is eye tracking. VR and AR systems
equipped with eye tracking could provide a more natural
interface with the virtual environment, as well as opening
the possibility for foveated rendering and gaze-contingent
focus. In this work, we approach an eye tracking solu-
tion specifically designed for near-eye displays via con-
volutional neural networks, that is robust against light-
ing changes and occlusions that might be introduced when
placing a camera inside of a near-eye display. We create
a new dense eye tracking dataset specifically intended for
neural networks to train on. We present the dataset as well
as report initial results using this method.
1. Introduction
Immersive visual and experiential computing systems
are entering the consumer market and have the potential
to profoundly impact our society. Applications of these
systems range from entertainment, education, collaborative
work, simulation and training to telesurgery, phobia treat-
ment, and basic vision research. In every immersive experi-
ence, the primary interface between the user and the digital
world is the near-eye display. Thus, developing near-eye
display systems that provide a high-quality user experience
is of the utmost importance. Many characteristics of near-
eye displays that define the quality of an experience, such as
resolution, refresh rate, contrast, field of view, orientation
and positional tracking, have been significantly improved
over the last years. However, an important missing inter-
face is gaze tracking.
Although the area of gaze tracking has been studied for
decades, they have been studied in the context of tracking
a user’s gaze on a screen placed a distance away from the
eyes. Many such techniques are not suitable for near-eye
displays where the imaging of the eye is restricted to a small
1
Paroma Varma
paroma@stanford.edu
area, and might be subject to occlusions and only partial
views of the pupil and iris. Currently, one company, SMI,
offers to hack a an Oculus DK2 for a large cost (roughly ten
thousand dollars). We seek a more elegant and cost effective
solution for providing the benefits of gaze tracking for near
eye displays.
Gaze tracking not only provides a new intuitive interface
to near-eye displays, but also allows for important tech-
niques increasing immersion and comfort in foveated ren-
dering and gaze contingent focus.
Applications Depth of field, the range of distances
around the plane of focus that still appear sharp, is a cue
that humans use to detect depth of objects in a scene. Given
a user’s gaze position on the screen, a depth of field can be
rendered into a scene, creating an adaptive depth of field
system. As the gaze moves around the scene, different ob-
jects come into focus and other objects fall out of focus, cre-
ating a more realistic experience than an all-in-focus image
that current VR and AR displays present. Such as system
has been shown to reduce image fusion times in stereo sys-
tem in [8], as well as increasing subjective viewing experi-
ences [4, 9]. In a first person environment, rendering a depth
of field effect showed an increased sense of immersion [5].
Because the area of real-time depth of field rendering is well
studied [3, 18, 17], an adaptive depth of focus system could
be immediately created with the implementation of a robust
gaze tracking system suitable for near eye displays.
A second potential use of gaze tracking in near-eye
displays is gaze contingent focus, which improves com-
fort of VR and AR displays by reducing the vergence-
accommodation conflict. When we look at objects our eyes
perform two tasks simultaneously when looking at an ob-
ject: vergence and accommodation. Vergence refers to the
oculumotor cue created by the muscles in our eyes rotat-
ing our eyeballs so that they verge to the point we look at.
Accommodation is the oculomotor cue created by the mus-
cles in our eyes bending the lenses in our eyes such that
the object of interest comes into focus. In normal, real-
world, conditions our eyes verge and accommodate to the
same distance. However, in near-eye stereoscopic display,
the user the user is able to verge at any distance in the
scene but is forced to focus to a fixed distance. This dis-
tance is a function of the distance between the lenses and
the display in the head-mounted display, as well as the fo-
cal length of the lenses themselves. This forces a mismatch
between vergence (able to verge anywhere) and accommo-
dation (only able to accommodate to one distance), known
as the vergence accommodation conflict. When exposed to
such a conflict for extended periods of time, users develop
symptoms of headache, eye strain, and, in extreme cases,
nausea[13].A system capable of determining the distance
to which the user is verged at, through gaze tracking, can
either use focus-tunable optics or an actuated display to re-
duce the vergence accommodation conflict by changing the
distance to which users focus to, as explained in [7].
2. Related Work
A variety of remote eye gaze tracking (REGT) algo-
rithms have been reported in literature over the past couple
of decades. For our purposes, the general body of knowl-
edge can be divided into two categories: ones which assume
a model of the eye and ones which learns an eye tracking
model.
2.1. Model Assumed REGT
Methods assuming a model of the eye generally extract
features from an image of the eye and map them to a point
on the display. This type of work generally either uses in-
tensity images captured from a traditional camera as seen in
[19], or uses illumination from an infrared light source and
captures the eye with an IR camera.
Infrared-Based REGT IR illumination creates two ef-
fects on the eye. Firstly, it creates the so-called bright-eye
effect, similar to red-eye in photography, which results from
the light “lecting off of the retina. The second effect, a glint
on the surface of the eye, is caused by light reflecting off the
corneal surface, creating a small bright point in the image.
This glint is often used as a reference point, because if we
assume that the eye is spherical, it does not move as the eye
rotates in its socket.
After grabbing an image of the eye, the glint and pupil an
be extracted via image processing algorithms described in
[6]. A glint pupil vector can be calculated, and mapped to a
2-D position on the screen via some mapping function. Al-
though many have been proposed, [20, 19], the most com-
monly used function is the 2nd order polynomial defined in
[10], defined as :
sx = a0 + a1 x + a2 y + a3 xy + a4 x2 + a5 y 2
sy = b0 + b1 x + b2 y + b3 xy + b4 x2 + b5 y 2
2
Figure 1. Table of existing gaze tracking datasets, which are
mostly tailored towards model assumed methods.
where (sx , sy ) are the screen coordinates, and (x, y) are
the pupil-glint vector components. A calibration procedure
is performed to estimate the unknown variables a0 , a1 , ...b5
via least squares analysis by asking a user to looking at (at
least) 9 calibration targets. The accuracy of the best IR-
based methods fall somewhere between 1◦ to 1.5◦ accuracy.
View-Based REGT In view-based REGT, only intensity
images from traditional cameras are used without any ad-
ditional hardware. These techniques rely more on image
processing techniques to extract features from the eyes di-
rectly, which can then be mapped to 2D pixel coordinates.
Tan et. al [16] uses an image as a point in high dimensional
space and through an appearance-manifold technique is able
to achieve a reported accuracy of 0.38◦ . Zhu and Yang [19]
proposed a method for feature extraction from intensity im-
ages and using a linear mapping function are able to achieve
a reported accuracy of 1.4◦ .
2.2. Model Learned REGT
Model learned REGT techniques use some sort of ma-
chine learning algorithm to learn an eye tracking model
from training data consisting of input/output pairs, i.e. 2D
coordinates of points on the screen and images of the eyes.
In the work by Baluja and Pomerleau [2], an artificial
neural network (ANN) was trained to model the relation-
ship between 15x40 pixel images of a single eye and their
corresponding 2D coordinates of the observed point on the
screen. In their calibration procedure the user was told to
look at a cursor moving on the screen along a path made of
two thousand positions. They reported a best accuracy of
1.5◦ .
Similar work by Xu et. al. [12] was presented, but in-
stead of using raw image values as inputs, they segmented
out the eye and performed histogram equalization in order
to boost the contrast between eye features. They used three
thousand points for calibration and reported an accuracy of
around 1.5◦ .
2.3. Our Approach
A key motivation behind this work is that gaze tracking
systems lack the required robustness for commercial ap-
Figure 2. Image of setup, sample captures, and path of point on screen. The left image depicts our setup comprising of an LCD monitor,
webcam, and chin rest to keep the head roughly stable. The upper right images show samples images from what the webcam captures. The
bottom right image displays a part of the path that the moving target follows during its trajectory.
plications. High variability in the appearance of the pupil
and lack of a clear view cause accuracy issues with detec-
tor based tracking systems. Deep learning performs well
on problems with high intra-class variability. The intuition
behind this work was to use a CNN model instead of the
parametric calibration since it would probably be more ro-
bust to variations in skin/eye color, HMD fixation etc.
In particular the key contributions of this work are:
• Introduction and implementation of an end-to-end
CNN based approach to gaze tracking
• Creation of a new gaze tracking dataset with a near-eye
camera covering five different subjects and dense po-
sition sampling of the screen based on smooth pursuit
• Performance evaluation of the chosen method, conclu-
sions and suggestions for improvement that can be in-
corporated in future work
3. Dataset
Many eye-gaze tracking datasets exist [14], as seen in
Figure 1, however they are mostly tailored for model as-
sumed REGT systems. The majority of the datasets use
target based calibration with large spacing between targets.
Neural networks are not able to train on such few, and
sparsely, sampled points and learn a good relationship be-
tween image data and pixel coordinates. The few datasets
that used continuous targets also allowed for continuous
head movement, which is not a good representation for a
near-eye display where the display is strapped to the user’s
head (with the camera rigidly fixed inside of it).
3
Instead of using one of the above datasets, we decided
to create our own dataset suitable for neural network train-
ing. Our strongest criteria was to have a large number of
calibration points densely sampling the entire screen, with
corresponding images of the eye. With such training data,
we would expect the CNN to learn the fine differences be-
tween points on the screen.
In our setup, as seen in Figure 2, we placed a user with
his/her head resting on a chin rest 51 cm away from a 1080p
24 inch monitor. A webcam was placed very close to the
chin rest imitating what a camera placed inside of a near-
eye display would see.
Asking a user to fixate on a series of targets is infeasi-
ble for the large number of calibration points we wanted
to collect. Instead we used the fact that humans are able
to track moving objects well, up to some angular velocity.
This is the notion of smooth pursuit. We moved a point
about a screen at 7.5◦ /s in a winding pattern from left to
right, top to bottom, as seen in Figure 2 and were able to col-
lect 7316 calibration points during a single 4 minute sitting.
At this angular velocity the eye is able to smoothly track the
point as it moves about without sacades (which occur when
the eye attempts to ’catch up’ when a point is moving very
quickly). We chose this particular angular velocity based on
[11], which introduced the concept of a pursuit based cali-
bration and found that points moving about between 3.9◦ /s
and 7.6◦ /s resulted in best accuracy.
In order to achieve a smoothly moving point, we dis-
played four points per angular degree, giving us a total of
7316 points. We captured webcam video frames at approxi-
mately 30 fps, which roughly corresponded to one frame per
1/4 angular degree point shift. Because our goal was around 5. CNN Implementation
1◦ of accuracy, we binned the calibration points into 1◦ bins.
For example, points displayed between 0.5◦ and 1.5◦ would 5.1. Caffe
be considered the same as 1◦ . We found that with a 4x re- The CNN implementation was performed in both Caffe
duction in classes, and a corresponding 4x increase in points and Tensorflow for comparison. Caffe has LeNet in the
per class, the CNN was able to better learn. model repository implemented for MNIST digit classifica-
The cropped and downsampled captured dataset can be tion. The .prototxt file was reconfigured to point to the
found at [1] lmdb file for our dataset. The image files and binary with
class labels are converted to the Caffe compatible lmdb file
4. CNN Learning Approach and then training is performed. The figure below shows the
setup workflow for the Caffe implementation.
In this work we explore the use of Convolutional Neural
Networks (CNN) for gaze tracking in an end-to-end fash-
ion. Traditional approaches like the one mentioned in the
previous section rely on hand-engineered feature detectors
and then use a parametric model to track the gaze direction
of a user.
Recently CNNs have outperformed traditional feature
engineering-based computer vision methods in a variety of
tasks. This work explores their use for gaze tracking. The
key benefit is that this approach is fully data driven. We
train the CNN model to take images of the users eye (taken
from a camera very close to their face) as input and estimate
the gaze direction in terms of x and y pixel coordinates on
the screen.

4.1. Description of CNN

CNNs are a class of artificial neural network algorithms
with multiple hidden layers that are built using convolu-
tional layers. In this work we treat the gaze tracking prob-
lem as a multi-class classification problem, where each
class is a specific point on the screen, and use a simple CNN
(i.e. LeNet) to learn the mapping from images to gaze posi-
tion.
LeNet consists of two convolutional layers and two pool-
ing layers followed by a fully connected layer at the end.
For our implementation, we modify the fully connected
(FC) layer to the desired number of output classes for the
gaze tracking problem.
Figure 4. Caffe Model Architecture
As the results were not very promising and training was
suspending abruptly, we later focused on the TensorFlow
implementation to complete the project.
5.2. TensorFlow
Tensorflow also consists of the LeNet model as an ex-
ample trained for MNIST. The example was reconfigured
by changing the number of classes to train on, size of con-
volutional kernels etc. to work with the CAVE and captured
gaze tracking dataset. The figure below shows a visualiza-
tion of the TensorFlow model.
Some of the parameters that were explored were the size
of the convolutional kernels and number of convolutional
layers. Unfortunately, due to limited computing power, it
remained difficult to add too many layers to the network.

6. Data Organization
Figure 3. LeNet Architecture Due to the large size of the data captured and the limited
computational power available, a few steps were taken to

4

Figure 5. TensorFlow Model Architecture
make the data manageable while not compromising the data
available for the CNN.
6.1. Dataset Augmentation
The dataset consisting of cropped images of the eye and
their respective pixel coordinates for the gaze was aug-
mented before training the CNN. The augmentation con-
sisted of adding different types of noise (Poisson, speckle,
Gaussian etc.) to make the model more robust to variations
in the image during inference. The augmentation extends
the dataset which is very useful as many samples are needed
to fully train the CNN model. The augmented dataset is then
spit into training, test and cross-validation sets depending
on the requirements.
6.2. Image Preprocessing
The pipeline to feed both the CAVE and the captured
dataset were as follows:
1. Convert Images to Grayscale
2. Downsample Images to 28x28 (for efficiency)
3. Crop out eye region from each image
5
Figure 6. Augmented Images
4. Save images and respective labels
The size 28x28 matched the dimensions of the MNIST
dataset. More importantly, we found that downsampling to
a slightly large size, like 50x50, did not necessarily lead to a
better classification accuracy. Similar results were found in
[] where a larger image did not mean better neural network
performance.
Cropping out the eye region was done differently for the
CAVE and captured dataset. For the CAVE dataset, a very
basic estimation of the bounding box around both eyes was
formed. The same bounding box was applied to all images
for a particular pose. This was an error-prone methodology
but did better than using eye recognition software, which of-
ten failed on subjects with glasses. For the captured dataset,
we manually selected the region around the eyes per sub-
ject and then applied the same crop region to the rest of the
images for that particular subject. This manual processing
was also not perfect, since subjects moved slightly between
frames.
The labels were created using one-hot vectors - the
length of the vector was equal to the number of classes per
experiment and the index of the class the image belonged
to was a 1 while the rest of the entries were 0. The images
and the label vector pairs were converted to Tensors/lmdb
format for ease of processing.
7. Results
The CNN was initially tested with the CAVE dataset to
ensure the model learned something. Next, the same model
was applied to the captured dataset, where the images were
significantly better (they were focused just on the eyes) but
the number of classes was significantly greater (21 classes
for the CAVE dataset and 1829 classes for the captured
dataset).
 Figure 7. CAVE Dataset Downsampled Images

Figure 9. CAVE Classification Accuracy with 21 Classes

Figure 8. Captured Dataset Downsampled Images

7.1. CAVE Dataset

Our initial gaze tracking implementation was based on
the CAVE (Columbia Gaze Data Set) [15] dataset. The
dataset has 5,880 images collected for 56 subjects for 21
gaze directions for 5 distinct head poses. To simulate the
near-eye display conditions we only use one head pose
i.e. the subject directly facing the screen. Even though
the dataset was not built for fine-grained tracking and the
captured images are highly inconsistent, the results were Figure 10. CAVE Classification Accuracy with 3 Classes
promising.
The problem was framed as a gaze localization problem
where the screen is divided into 21 (7X3) grids which form learn as well as it would be expected to. Out of the 28x28
the 21 output classes for the CNN model. We show results pixels, a very small fraction actually comprises of the eyes
obtained on the CAVE dataset with 3 output classes (one of and the pupils. In the results, we look at training accuracy
the three horizontal bands on the screen) and with 21 out- which can be interpreted as the inverse of the loss functions
put classes. The graphs show how the classification accu- and shows the progress of the model as it learns a better
racy (measured as the fraction of gazes correctly classified) model. The trained model is then tested on a batch from the
changed across iterations of training the CNN. Each train- test set to get an accuracy which represents the percentage
ing batch was 100 images and the CNN was trained for 500 of correctly classified images. For the 3 and 21 class case,
iterations. The figures 9 and 10 show the training accuracy the final test accuracy was 0.893 and 0.236.
increasing across iterations. The red dotted line shows the
7.2. Captured Dataset
baseline classification accuracy - number 1of classes .
While the accuracy for the 21 class case isn’t significant, Since the number of classes with the real dataset was
the model does learn a mapping and performs better than significantly large, the classification accuracy is not a good
a random guess over the output classes. From the CAVE measure of how well the model does. If the true and esti-
dataset images, it is easy to see why the model does not mated classes are next to each other, that is a misclassifi-

6
Figure 11. Correctly and Incorrectly Classified Images from
CAVE Dataset
cation and measured the same as if the true and estimated
classes were on opposite corners of the viewing screen.
Since our goal is to track the user’s gaze as accurately as
possible, we also look at angular error i.e. actual difference
in angle between the inferred gaze and the actual gaze of the
user. Looking at this particular metric, we can see that the
gaze can be localized to a few degrees in accuracy but there
is scope for improvement. Looking at confusion matrix or
classification accuracy is not appropriate for this problem
as it does not capture the degree of misclassification or how
far the inferred gaze is from the actual gaze direction.
The testing classification accuracy was 0.003. Even
though this is ridiculously low, the testing angular error is
only 6.7 degrees. And in the gaze tracking sense, the angu-
lar degree matters much more than the classification accu-
racy.
Figure 12. Captured Classification Accuracy with 1829 Classes
7
Figure 13. Captured Average Angular Error
8. Conclusion and Future Work
We explored an end-to-end deep learning approach for
gaze tracking in a highly constrained environment (camera
very close to the eye) and obtained promising results with a
vanilla implementation. This demonstrates the viability of
this approach and also highlights some of its shortcomings.
CNN needs a lot of data to be properly trained and lack
of publicly available datasets for fine gaze tracking was a
major impediment. We implemented our own data capture
setup and created an appropriate dataset for this purpose but
given the severe time constraints were only able to gather
data for 5 subjects. This dataset will be extended in the
future for more subjects and made publicly available for fu-
ture experiments.
The model used for this work was the simplest instanti-
ation of a CNN and the eye images had to be significantly
downsampled even with the simple model to facilitate quick
training. This is one of the limitations of using CNNs and
prevents quick iterations or cross-validation.
We believe a higher quality input image of the users eye
that also has support features (fixed in the image irrespec-
tive of the subject) will help improve the gaze tracking ac-
curacy. This is also the situation that will be useful when
doing tracking in a near-eye display. Given that the HMD is
fixed to the user’s head, the camera should see a consistent
images across different users. The model can also be more
complex but due to lack of training data, it did not make
sense to increase model complexity in this work.
To conclude, the work shows promise in using a deep
learning approach for gaze tracking and has potential to
outperform the feature based methods based on parametric
models. However, further exploration is needed to achieve
state of the art results.
References [16] K.-H. Tan, D. J. Kriegman, and N. Ahuja. Appearance-based
eye gaze estimation. In Proceedings of the Sixth IEEE Work-
[1] Captured Dataset. https://www.dropbox.com/s/ shop on Applications of Computer Vision, WACV ’02, pages
mxbis1osiedclkd/data.zip?dl=0. 191–, Washington, DC, USA, 2002. IEEE Computer Society.
[2] S. Baluja and D. Pomerleau. Non-intrusive gaze tracking [17] S. Xu, X. Mei, W. Dong, X. Sun, X. Shen, and X. Zhang.
using artificial neural networks. Technical report, Pittsburgh, Depth of field rendering via adaptive recursive filtering. In
PA, USA, 1994. SIGGRAPH Asia 2014 Technical Briefs, SA ’14, pages 16:1–
[3] B. A. Barsky and T. J. Kosloff. Algorithms for rendering 16:4, New York, NY, USA, 2014. ACM.
depth of field effects in computer graphics. pages 999–1010, [18] T. Zhou, J. X. Chen, and M. Pullen. Accurate depth of
2008. field simulation in real time. Computer Graphics Forum,
[4] S. Hillaire, A. Lecuyer, R. Cozot, and G. Casiez. Using an 26(1):15–23, 2007.
eye-tracking system to improve camera motions and depth- [19] J. Zhu and J. Yang. Subpixel eye gaze tracking. pages 124–
of-field blur effects in virtual environments. In Proc. IEEE 129, May 2002.
VR, pages 47–50, 2008. [20] Z. Zhu and Q. Ji. Novel eye gaze tracking techniques under
[5] S. Hillaire, A. Lcuyer, R. Cozot, and G. Casiez. Depth- natural head movement. IEEE Transactions on Biomedical
of-field blur effects for first-person navigation in virtual en- Engineering, 54(12):2246–2260, Dec 2007.
vironments. IEEE Computer Graphics and Applications,
28(6):47–55, Nov 2008.
[6] T. E. Hutchinson, K. P. White, W. N. Martin, K. C. Reichert,
and L. A. Frey. Human-computer interaction using eye-gaze
input. IEEE Transactions on Systems, Man, and Cybernetics,
19(6):1527–1534, Nov 1989.
[7] R. Konrad, E. Cooper, and G. Wetzstein. Novel optical con-
figurations for virtual reality: Evaluating user preference and
performance with focus-tunable and monovision near-eye
displays. Proceedings of the ACM Conference on Human
Factors in Computing Systems (CHI16), 2016.
[8] G. Maiello, M. Chessa, F. Solari, and P. J. Bex. Simulated
disparity and peripheral blur interact during binocular fusion.
Journal of Vision, 14(8), 2014.
[9] M. Mauderer, S. Conte, M. A. Nacenta, and D. Vishwanath.
Depth perception with gaze-contingent depth of field. ACM
SIGCHI, 2014.
[10] C. H. Morimoto and M. R. M. Mimica. Eye gaze tracking
techniques for interactive applications. Comput. Vis. Image
Underst., 98(1):4–24, Apr. 2005.
[11] K. Pfeuffer, M. Vidal, J. Turner, A. Bulling, and
H. Gellersen. Pursuit calibration: Making gaze calibration
less tedious and more flexible. In Proceedings of the 26th
Annual ACM Symposium on User Interface Software and
Technology, UIST ’13, pages 261–270, New York, NY, USA,
2013. ACM.
[12] L. qun Xu, D. Machin, P. Sheppard, M. Heath, and I. I. Re. A
novel approach to real-time non-intrusive gaze finding, 1998.
[13] T. Shibata, T. Kawai, K. Ohta, M. Otsuki, N. Miyake,
Y. Yoshihara, and T. Iwasaki. Stereoscopic 3-D display with
optical correction for the reduction of the discrepancy be-
tween accommodation and convergence. SID, 13(8):665–
671, 2005.
[14] B. Smith, Q. Yin, S. Feiner, and S. Nayar. Gaze Locking:
Passive Eye Contact Detection for HumanObject Interaction.
In ACM Symposium on User Interface Software and Technol-
ogy (UIST), pages 271–280, Oct 2013.
[15] B. Smith, Q. Yin, S. Feiner, and S. Nayar. Gaze Locking:
Passive Eye Contact Detection for HumanObject Interaction.
In ACM Symposium on User Interface Software and Technol-
ogy (UIST), pages 271–280, Oct 2013.

8
 Fine-grained Flower Classification
Leo Lou
qibinlou@stanford.edu
Abstract
In this paper, we tackle the task of fine-grained flower
classification. Based on the earlier work of Maria et al.[6],
we incorporate both traditional features like HOG, SIFT,
HSV with CNN feature to better describe inter-class differ-
ences between flowers species. We extensively experiment
on different ways to utilize these features to generate a high
performance model, such as feature normalizing, feature
scaling, feature fusing and multiple kernel learning.
Here we use the well-known 102 Oxford flowers bench-
mark dataset to do our experiment. Our approach has
beaten the ground truth work by Maria [6] and the another
CNN based work by [7], though the latter one contains an-
other approach which yield better result than ours(but takes
much longer time to do experiments).
1. Introduction
Nowadays image based classification systems are
achieving better and better performance as a result of large
datasets and neural networks. In this paper, instead of focus-
ing on the task of classifying as many as different objects,
we investigate the problem of recognizing a large number
of classes within one category, in our case, flowers. Such
task is called fine-grained classification.
There has been progress in expanding the set of fine-
grained domains we have data for, which now includes e.g.
birds, aircraft, cars, flowers, leaves, and dogs. In this paper,
we focus on the flowers fine-grained classification task. For
human beings, we can use different features of flowers to
distinguish different species;for example, we can use color,
shape, size, and smell information to help us make a better
decision. But for computers, the only information they can
get is from the input image, which requires us to well design
visual features to describe the flowers.
In this paper, We first discuss several key components of
our classification pipeline, including flowers segmentation
in section 3, flowers representing features in section 4, and
experiment details in section 5.
2. Related Work
There are several noticeable work done on fine-grained
flower classification.
In [6], Maria l investigated on using multiple kernel
learning for flower images acquired under fairly uncon-
trolled image situations the images are mainly downloaded
from the web and vary considerably in scale, resolution,
lighting, clutter, quality, etc. They combined different im-
age features(HSV+SIFT+HOG) to describe a flower image
and to their model. Their work is known as 102 Oxford
flowers benchmark. Anelis et al. [3] designed a better seg-
mentation algorithm to identify potential flower body re-
gions and applied feature extraction on that. They also
developed a much larger dataset than Oxford 102 flower
dataset with 578 flower species and 250,000 images, which
contributed to 4% classification performance improvement
compared to Oxford 102 flower benchmark dataset. There
are other ways to describe flower images. For example, [8]
used generic features extracted from a convolutional neu-
ral network previously used to perform general object clas-
sification. They experimented the CNN features on plant
classification task together with an extremely randomized
forest.
3. Segmentation
Most flowers are recognizable only by their body. Per-
forming a segmentation to separate the flower body from the
background can help reduce the noise in feature extraction
process.
Several papers have proposed methods explicitly for the
automatic segmentation of a flower image into flower as
foreground, and the rest as background. Here We use
the segmentation scheme proposed by Nilsback and Zis-
serman [5]. Figure 1 shows several example segmentaions
from this method. The problem with this schema is that
there are some over-segmented images, where there is no
foreground at all. See Figure 2.
4. Classification
There are several key features that we human use to dif-
ferentiate various kinds of flowers. First of all, color is use-

Figure 1. Example segmentations from [5]
Figure 2. Bad segmentation examples
ful when discriminating a sunflower from a rose. But to
differentiate a buttercup from a dandelion, shape would be
much more useful, but color would not. Smell is also useful
in some cases, though totally useless in our flowers images
classification case. In this section, we discuss several fea-
tures to represent a flower image, and then discuss pros and
cons of multi-kernels SVM classifier and neural network
classifier.
4.1. Features
Color: Colour is described by taken the HSV values of
the pixels. The HSV space is chosen because it is less
sensitive to variations in illumination and should be able
to cope better with pictures of flowers taken in different
weather conditions and at different time of the day. The
HSV values for each pixel in an image are clustered using
k-means. Given a set of cluster centres (visual words)
wic , i = 1, 2, ..., Vc , each pixel in the image I is then
assigned to the nearest cluster centre, and the frequency
of assignments recorded in a a Vc dimensional normalized
frequency histogram n(wc |I).
SIFT: SIFT descriptors are computed at points on a
regular grid with spacing M pixels over the foreground
flower region. At each grid point the descriptors are
computed over circular support patches with radii R
pixels. Only the grey value is used (not color), and the
resulting SIFT descriptor is a 128 vector. To cope with
empty patches, all SIFT descriptors with L2 norm below
a threshold (200) are zeroed. Note, we use rotationally
invariant features. The SIFT features describe both the
texture and the local shape of the flower (e.g. fine petal
structures (such as a sunflower) vs spikes (such as a globe
thistle). We obtain n(wf |I) through vector quantization in
the same way as for the color features.
Histogram of Gradients: HOG features, are similar
to SIFT features, except that they use an overlapping local
contrast normalization between cells in a grid. However,
instead of being applied to local regions (of radius R in
the case of SIFT), the HOG is applied here over the entire
flower region (and it is not made rotation invariant). In this
manner it captures the more global spatial distribution of
the flower, such as the overall arrangement of petals. The
segmentation is used to guide the computation of the HOG
features. We find the smallest bounding box enclosing the
foreground segmentation and compute the HOG feature
for the region inside the bounding box. We then obtain
n(wh|I)
CNN Feature: Convolutional neural networks (CNNs)
were proposed in 1989 by LeCun et al. [4]. Recently, it has
shown its power with the availability of large datasets, per-
formance improvement of GPUs and efficient algorithms.
It has been shown that combining CNN features with a
simple classifier such as SVM can outperform classical
approaches for various classification and detection tasks,
which also require hand-crafted features.
4.2. Classifier
Currently we adopt multiple kernels SVM as our clas-
sifier to train our final model. It is reasonable to have a
weighted linear combination of SVM kernels, each corre-
sponding to each feature. The weights vary for different
flower species, some of which might have a high weight on
Color feature while some might have a high weight on SIFT
feature. For individual feature, we simply use one-vs-other
SVM classifier to get their corresponding model.
5. Experiment
5.1. Dataset
Oxford 102 flowers dataset is a well established dataset
for subcategory recognition proposed by Nilsback et al. [6].
The dataset contains 102 species of flowers and a total of
8189 images, each category containing between 40 and 200
images. It has established protocols for training and testing,
which we have adopted in our work too.
Figure 3. The distribution of numbers of images over 102 classes
5.2. Setup
Our experiment is based on Python environment and var-
ious Python packages, numpy, scipy, scikit-learn, etc. The
complete package list can be obtained from the import sec-
tions of our scripts. It’s worth mentioning that we use
OpenCV 2.4.8 [1] for Python package to process our images
and extract different features. Due to some unknown issues,
our OpenCV version doesn’t support to extract SIFT fea-
tures. Instead, we use the open sourced Octave and VLFeat
[9] to extract SIFT features. It’s a little bit troublesome to
run VLFeat library on Octave because it’s still in experi-
ments. The Overfeat network feature extractor can be in-
stalled from Github source [2] or the pre-build version.
5.3. Procedure
In our experiments, we combine the training set and
the validation set in the dataset since we skip fine tuning
the parameters for different features experiments. Instead,
we directly reuse the reported optimal parameters from [6].
Therefore, the optimum numbers of words is 1000 for HSV,
3000 for SIFT, 1500 for HOG. These optimum numbers
might not really be the optimal ones for our experiments
and other reproducing experiments because of libraries and
implementation differences. We train our classifiers on
the combined training data(both the training and validation
sets) and test our classifiers on the testing sets.
For features like HSV, HOG and SIFT, we use KMeans
to cluster the features to get our visual words. We use Mini-
BatchKMeans in scikit-learn toolbox to overcome the bot-
tleneck that some feature sets are too large to fit into the
entire memory. MiniBatchKMeans can cluster the centers
batch by batch on our split feature sets.
5.4. Result
We report our experiments result in Table 1. It can be
seen that combining all the features contributes to a far bet-
ter performance than using single feature. CNN feature out-
performs all the other features. Though we observe that
CNN features extracted from segmented flowers images are
less descriptive than from images with background. There
are probably two reasons for this; one is that our segmenta-
tion algorithm is still not perfect that it filters some impor-
tant parts of the flowers images when segmenting; another
is that background information actually helps to better clas-
sify a flower image.
Features mAP
HSV 42.3%
HOG 49.1%
SIFT 53.0%
CNN w/o segmentation 73.9%
CNN w/ segmentation 54.1%
HSV+HOG+SIFT+CNN 84.0%
Table 1. Classification performance on the test set. mAP refers
to classification performance averaged over all classes(not over all
images)
Table 2 shows the comparison result between our work
and others. Note that in [7], the best performance is
achieved by further augment the training set by adding
 Method mAP [8] N. Sünderhauf, C. McCool, B. Upcroft, and T. Perez. Fine-
Nilsback et al. [6] 76.3% grained plant classification using convolutional neural net-
Anelia et al. [3] 80.66% works for feature extraction. In CLEF (Working Notes), pages
Ours 84.0% 756–762, 2014.
Ali et al. [7] 86.8% [9] A. Vedaldi and B. Fulkerson. VLFeat: An Open and Portable
Library of Computer Vision Algorithms, 2008.

Table 2. Comparison between our work and others

cropped and rotated samples and doing component wise

power transform, which makes the training model invariant
to scale and rotation. We don’t implement this step because
of priorities and time, but we believe it will also benefit our
model and increase our final performance.

6. Summery and future work

So far we have researched and experimented on avail-
able image features and segmentation algorithms for fine-
grained flower classification task. We observe that multi-
ple features empower the classifier to train a better model
and achieve a better classification accurate on test sets. For
the fine-grained flowers classification task, the learning of
different weights for different classes enables us to use an
optimum feature combination for each classification.
We also realize there are flaws in current pipeline, e.g.
computation time cost is not good enough to be in the re-
altime scale. It poses a barrier to build an usable flowers
classification service for users. Future work should include
using current method to train on a larger dataset and fine
tuning the neural network which we use to extract CNN fea-
tures on flowers dataset.

References
[1] OpenCV. https://github.com/itseez/opencv.
[2] Overfeat. https://github.com/sermanet/OverFeat.
[3] A. Angelova, S. Zhu, and Y. Lin. Image segmentation for
large-scale subcategory flower recognition. In Applications
of Computer Vision (WACV), 2013 IEEE Workshop on, pages
39–45. IEEE, 2013.
[4] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E.
Howard, W. Hubbard, and L. D. Jackel. Backpropagation ap-
plied to handwritten zip code recognition. Neural computa-
tion, 1(4):541–551, 1989.
[5] M.-E. Nilsback and A. Zisserman. Delving into the whorl of
flower segmentation. In BMVC, pages 1–10, 2007.
[6] M.-E. Nilsback and A. Zisserman. Automated flower classi-
fication over a large number of classes. In Computer Vision,
Graphics & Image Processing, 2008. ICVGIP’08. Sixth In-
dian Conference on, pages 722–729. IEEE, 2008.
[7] A. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. Cnn
features off-the-shelf: an astounding baseline for recognition.
In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition Workshops, pages 806–813, 2014.
 Gradient-learned Models for Stereo Matching
CS231A Project Final Report

Leonid Keselman
Stanford University
leonidk@cs.stanford.edu

Abstract

In this project, we are exploring the application of ma-

chine learning to solving the classical stereoscopic corre-
spondence problem. We present a re-implementation of sev-
eral state-of-the-art stereo correspondence methods. Addi-
tionally, we present new methods, replacing one of the state-
of-the-art methods for stereo with a proposed technique Figure 1. An example of a stereo left-right pair from the Middle-
based on machine learning methods. These new methods bury 2014 dataset [18]. The motorcycle scene will be consistently
out-perform existing heuristic baselines significantly. used through this paper as a visual example result .

1. Introduction ported there.

Stereoscopic correspondence is a classical problem in
computer vision, stretching back decades. In its simplest
form, one is given a calibrated, rectified image pair where
differences between the pair exist only along the image
width. The task is to return a dense set of corresponding
matches. An example image pair is shown in figure 1.
While this task may seem straightforward, the primary
challenge comes from challenging, photo-inconsistent parts
of the image. Parts of the image will often contain ambigu-
ous regions, a lack-of-texture, and a photo-metric mismatch
for a variety of reasons (specular reflections, angle of view,
etc.).
This field is well studied, and there exist many stan-
dard datasets, such as Middlebury [18] and KITTI [6].
Both of these datasets contain many rectified left-right pairs,
along with corresponding ground truth matches. The former
dataset contains high resolution images from largely indoor
scenes, and comes from using a method of dense structured
light correspondence method. In contrast, the KITTI dataset
contains much lower resolution images, and consists of out-
door data gathered from a vehicle perspective. Additionally,
the KITTI dataset’s annotations come from a LIDAR tech-
nique, and are generally sparse. While KITTI seems more
popular based on the size of their leader-board, the dense
annotations available in Middlebury [18] are useful for the
problem tackled in this paper, and our results are only re-
The interest in this problem has important practical ap-
plication in autonomous vehicles and commercial applica-
tions. For example, the KITTI dataset was formed to test
if low-cost stereoscopic depth cameras to replace high-cost
LIDAR depth sensors for autonomous vehicle research. In
a different field, commercial depth sensors such as the orig-
inal Microsoft Kinect and Intel RealSense R200 use stereo-
scopic correspondence to resolve depth for tracking peo-
ple and indoor reconstruction problems. As such, improved
methods for stereoscopic correspondence have wide appli-
cation and use.
2. Related Work
2.1. Previous Work
In the field of stereo matching, one of the recent innova-
tions in the past few years was the use of convolutional neu-
ral networks in improving the quality of matching results
[21]. At the time of it’s announcement at CVPR 2015, it
was the top performer in both standard datasets. Even at the
present day, all better performing methods on the Middle-
bury leaderboard use the Matching Cost CNN (MC-CNN)
costs as a core building block. Their primary contribution is
to train a convolutional neural network (CNN) to replace the
block-matching step of a stereo algorithm. That is, instead

1
 Region Neighborhood
Matching Aggregation Propagation
State of-the-art MC-CNN Cross- Semi-global
Aggregation matching
Figure 2. The architecture and algorithm flow for state-of-the-art
methods in stereo. They first use the MC-CNN cost function [21],
then combine those results with cross-based aggregation [22], and
share them with neighbors using semi-global matching [7]. Meth-
ods in dashed lines are heuristic methods, while the ones with solid
lines use a machine-learned method.
of using a sum of absolute differences cost such as
XX
Cost(source, target) = |sourceij − targetij |
i j
or a robust non-parametric cost function such as Census
[20]
R(P ) = ⊗ζ(P, Pij )
Cost(source, target) = popcnt(sourceij ⊕ targetij )
the authors of [21] learn a network to compute a
Cost(source, target) metric based on the ground truth
available from stereoscopic correspondence datasets. An
example of their network architecture is shown in figure 3.
However, in order to obtain their final result, they use
a combination of algorithms to select an optimum match.
Namely, they use a combination of their cost metric [21],
cross-based aggregation [22], and semi-global matching
[7]. This flow is shown in figure 2. We hypothesize that
a short-coming of this state-of-the-art method is that two
of the techniques used in the algorithm flow make use of a
heuristic method for completing a certain task. We hope to
build on the success of the MC-CNN method and use gradi-
ent based learning [12] to replace other components of the
stereo algorithm. The value in picking this specific classifi-
cation algorithm is that it has the ability for us to eventually
design an end-to-end gradient-learned system that trains an
MC-CNN along with our proposed system. The goal for
the project is first implement these baseline algorithm meth-
ods, and then begin to to test and design algorithms and
methods to replace one of the two heuristic algorithms in the
traditional stereoscopic pipeline, namely semiglobal match-
ing [7] or cross-based aggregation [22]. In this report, we
only present methods for replacing semiglobal matching,
but not yet cross-based aggregation.
Outlier
Removal
Learned
Confidence
Figure 3. The matching architecture of [21], the current state-of-
the-art in stereo matching.
2.2. Key Contributions
1. A fast, flexible implementation of stereo matching
We present a new, from-scratch implementation of
state-of-the-art methods in stereo matching, including
Census [20], semiglobal matching [7], cost-volume fil-
tering [10]. Along with standard methods for hole fill-
ing, like those used in MC-CNN [21], and many outlier
removal methods [16]. The implementation is cross-
platform, C++, multi-threaded, and uses no libraries
except those for loading and saving images. It is fast,
and produces competitive RMS error results on stan-
dard datasets. See section 3.1.
2. A machine-learned method for correlation selec-
tion We’ve implemented and tested several semiglobal
matching replacement architectures, trained them on
the Middlebury training data, and demonstrated that
they perform significantly better on out-of-bag exam-
ples than semiglobal matching. See section 4.
3. Baseline Implementation
3.1. C++ Baseline
First, we implemented stereo matching baselines using
current, non-machine learned methods. The stereo algo-
rithms described below were implemented from scratch, in
C++, with no external libraries outside of image loading.
The performance of our baseline is later summarized in ta-
ble 1.
An example left-right pair is show in 1. We’re using
quarter-sized training images from the latest Middlebury
dataset [18]. These are roughly 750x500 pixels in resolu-
tion. The results from our algorithm are compared to the
ground truth visually in figure 4 and quantitatively in table
1. An elaboration of the different papers and methods im-
plemented for each section is described below. The code
is all C++11, and compiles on Visual Studio 2013 and gcc,
with no external libraries. Threading is implemented via
OpenMP [15] to parallelize cost computation across all pix-
2
Figure 4. The image on the left is ground truth for the motorcycle scene in 1. The image on the right is the results of our semi-global
matching pipeline with naive hole filling as described in section 3.1. For visual comparison, occluded and missing ground truth pixels from
both images are masked out.
els in a given scanline. This cost accumulation is the pri-
mary computational bottleneck in the system so paralleliz-
ing that component is enough to provide sufficient scaling
across processor cores.
3.1.1 Cost Computation
As a baseline method of cost computation, we’ve imple-
mented both standard sum of absolute differences, and the
robust Census metric [20]. Census was recently tested and
shown to be the best performing stereo cost metric [8]. The
weighted sum of absolute differences and Census was addi-
tionally state-of-the-art for Middlebury until a year or two
ago [13]. The state of the art in this space is MC-CNN
method [21], which implemented a CNN algorithm to re-
place traditional methods. However, since our project fo-
cuses on implementing neural networks in other parts of the
stereo pipeline, re-implementing this cost metric is not a
high priority.
Specifically, we implemented Census with 7x7 windows,
which allows us to exploit a sparse census transform [5],
and fit the result for every pixel into 32-bits. This enables
efficient performance with the use of a single popcnt in-
struction on modern machines.
3.1.2 Region Selection
For our region selection baseline, we’ve implemented both
box correlation windows and weighting with a non-linear
smoothing algorithm such as the bilateral filter [19]. This
was inspired recent unpublished ECCV 16 submissions
on the Middlebury leader-board, which claim to replace
the popular cross-based [22] with a smooth affinity mask
method like a bilaterial filter, as first shown in [10].
3
3.1.3 Propagation
In order to perform propagation across the image, we’ve im-
plemented semi-global matching [7], in full, as described in
the original paper. We chose to perform 5-path propagation
for each pixel, as it represents a row causal filter on the im-
age, using a pixel’s left, right, top, top-left, and top-right
neighbors. This produces an answer that satisfies the cost
function of Hirchmuller [7]
X
E(D) = (C(p, D(P )) (1)
p
X
+ P1 · 1{D(p) − D(q)} = 1
q
X
+ P2 · 1{D(p) − D(q)} > 1
q
Additionally, we added naive hole filling by propagating
pixels from left-to-right, in order to fill occluded regions.
This is a naive metric, but is a large part of the hole filling
used in the state of the art work [21].
4. Learning Propagation
Most papers in the KITTI dataset build on top of the
successful method of semi-global matching [7], which is
an algorithm for propagating successful matches from pix-
els to their neighbors. The goal of this part of the project
was to replacing this function with either a standard neural
network, or recurrent neural network. Depending on one’s
perspective on what operation semiglobal matching is per-
forming, there is a wide array of neural network architec-
tures that may be amenable to replace it. An overview of
the formulations is shown in figure 5.
The first and most straightforward view of what the en-
ergy function, as stated in equation 1, is that it regularizes a

One-dimensional Smoothing Two-dimensional Smoothing
Probability of correct disparity Column-wise probability
(70d) disparity image (750x70)d
Classifier Classifier
Raw costs per disparity (70d) Disparity Image (750x70)d
Raw values (750x1)d
Figure 5. An example of two different ways to formulate semi-
global matching as a classification task. The one on the left is
explored in section 4.1, while the one on the right is explored in
4.2.
single pixel’s correlation curve into a more intelligent one.
This view is fairly simple, doesn’t incorporate any neigh-
borhood information, but in our testing was the most suc-
cessful model. This is elaborated in section 4.1.
A second view of what semiglobal matching does in
practice is that is regularizes an entire scanline at a time,
performing scanline optimization and producing a robust
match for an entire set of correlation curves at once. This
was the view we took when building models in section 4.2.
A third view of what semiglobal matching does is it
serves as a way of remembering good matches, and propa-
gating their information to their neighbors. This is straight-
forward and almost certainly what semiglobal matching
does. This would require a pixel recurrent neural network
such as that in [14]. In our limited time and testing, we
were unable to get any of these architectures to converge
and hence have excluded them from this paper. However,
our primary focus was on building a bidirection RNN with
GRU [3] activations. In practice, small pixel patches didn’t
converge while large patches were not able to fit into the
memory of the machines we had available for training.
For testing and training, we gather a subset of the Mid-
dlebury images [18], and split into into a random training
and testing set with a 80%-20% split. The unseen samples
are then used for evaluation. Middlebury provides 15 im-
ages for training and 15 for evaluation. For the classifiers in
section 4.1, this results in roughly 500,000 annotations per
image (using quarter sized images), and 500,000 tests of the
network. While for the classifiers in section 4.2, this results
in 500,000 annotations computed over about 1,000 runs of
the network (since it computes 500 outputs at the same
time). See below for details of how this is implemented.
4
4.1. 1D Smoothing
One straightforward view of semi-global matching is
simply as regularization function on top of a pixel’s cor-
relation curve. A correlation curve is the set of matching
costs for a single pixel and it’s candidates. If this input is
negated, and fed a softmax activation function, as used to
train many neural networks, it treats the values as unnor-
malized log probabilities, and selects the maximum (which
would be the candidate with lowest matching cost).
efyi
Li = − log( P fi )
je
Our original implementations for this method were all
straightforward multi-layer perceptions (MLP), using a one,
two, or three layer neural network to produce a smarter
minimum selection algorithm. However, no matter the loss
function, shape, dimensions, regularization, or initialization
function, we were unable to get any MLP to converge. That
is, using a 0-layer neural network (the input itself) was bet-
ter than any learned transformation to that shape and size.
Instead, we found success by using a one dimensional
convolutional neural network as shown in figure 6. We sus-
pect a CNN was able to handle this task better, as one bank
of convolutions could learn an identity transform, while oth-
ers could learn feature detectors that incorporated interest-
ing feedback into that identity transform. In comparison, a
randomly initialized fully connected network may struggle
to learn a largely identity transform with minor modifica-
tions. We implemented the neural network on top of Keras
[4] and TensorFlow [1]. We additionally learned several
non-gradient based classifier baselines such as SVMs and
random forests using scikit-learn [17].
4.2. 2D Smoothing
As shown in figure 5, there is an alternative concept of
how semiglobal propagation. This one incorporates pixel
neighborhoods, and seems a more natural fit for the energy
function presented in equation 1. For this formulation of
a neural network, the correlation curves of an entire scan-
line are reshaped into an image in disparity cost space no-
tation is described in figure 7. We then create a model us-
ing a two-dimensional convolutional neural network [12] on
top of these disparity cost space images The top level is a
column-wise softmax classifier of the same size as the input
dimensions. In order to implement this in TensorFlow [1],
we first pass in a single disparity image as a single batch.
We run our convolutional architecture over this model, and
then reshape the output into pixel-many ”batches” for each
of which we have a label. This allows the built-in softmax
and cross-entropy loss formulations to work out-of-the-box
with no hand-made loops.
 Model RMS Error Runtime
Census 28.92 1.2s
SGBM 28.12 3.1s
softmax
SGBM + BF 32.8 5.8s
70d linear projection OpenCV SGBM 38.00 0.9s
MC-CNN (acct) 27.5 150s
7x1 Average Pool Table 1. A summary table of numerical results on the training Mo-
torcycle image. The error metric is root-mean-squared error in
disparity space, and the run-times are on a quad-core i7 desktop.
The first three lines are baseline implementations implemented by
64, 5x1 convolutions, stride 1 us, while the last two are standard algorithms available on the
dataset website [18]. The MC-CNN results were run on a GPU
[21] (which were on an GPU).
32, 9x1 convolutions, stride 5

quality of all pixels predicted by the classifier. A sum-

16, 9x1 convolutions, stride 2 mary table is shown in table 1. We show that our baseline
implementation is on the same order of magnitude as the
SSE-optimized, hand-tuned implementation of semiglobal
8, 9x1 convolutions, stride 1 matching available from OpenCV [2]. We believe that both
the performance and accurate matching is a function of us
Input: 70d correlation curve using the robust and fast ADCensus [13] [20] weighted
cost function. Since the primary focus of this project as
to simply provide a flexible baseline for quickly generating
data for the machine learned methods in section 4, we did
not spend much time micro-optimizing or tuning algorithm
Figure 6. An architectural view of our most successful machine
learned method, a 1D CNN for predicting better minimums in cor-
hyper-parameters. However, if one wished to tune this al-
relation curves. gorithm there are dozens of knobs, including the relative
weighting of absolute differences and Census, the regular-
ization strengths of P1 and P2 from semiglobal matching,
𝐼𝐶𝑜𝑠𝑡 (𝑤, 𝑑) and the weights used in the bilateral filter.
Disparites

Type equation here.

5.2. Learned Propagation methods

Image Width
Figure 7. A brief visual diagram of a Cost Image for single scan-
line of stereo matching. Each pixel contains the cost of matching
for that value, at that image pixel. Across the entire image, there
exists a cost volume across all scan-lines in a stereo pair, our pro-
posed architectures only deal with a known, discrete number of
cost images.
5. Experiments
5.1. Baseline Method
We tested our baseline C++ implementation of modern
stereo matching for both time and quality of output. Specif-
ically, we focused on simply a single Middlebury training
image (the Motorcycle) to validate that our results were
within expectation for a stereo matching baseline. Our two
key metrics were runtime and root-mean-squared error for
all the dense, all-pixel label ground truth. This is just one
of the metrics for Middlebury, but is one that measures the
In the scope of testing the various propagation classifiers,
we adopt two different evaluation metrics. The first is the
traditional training/test split used in machine learning meth-
ods. The other is the RMS error metric used for stereo algo-
rithm evaluation. A result comparing standard methods and
our proposed classifiers on test data is shown in figure 8.
We see that the one-dimensional CNN as presented in
section 4.1 and shown in figure 6 outperforms the cur-
rent standard methods for smoothing matches. That is,
when fed with the ADCensus correlation curves generated
by our matching algorithm, the neural network generates
predictions that are much more accurate than the heuristic
semiglobal matching method used in state-of-the-art papers
such as MC-CNN [21]. This result is even true when we
take the network’s predictions back to the matching algo-
rithm and use it to generate a full correspondence image.
Even though the neural network (currently) lacks the ability
to make subpixel accurate guesses, it generates lower RMS
error than standard methods like Census and semiglobal
matching, which do have subpixel matching built into the
baseline.

5
 Test Accuracy RMS Error
91.2
84.0
74.4
70.4
11.6 9.4 9.3
ADCensus ADCensus+SGM Random Forest Conv1D NN
Figure 8. Numerical results on the out-of-bag testing data across
the a subset of the Middlebury [18] images.
Model Out-of-bag Accuracy
Census 67.7%
SGBM 69.2%
1D CNN Training 76.4%
1D CNN Test 75.8%
2D CNN Training 58.5%
2D CNN Test 55.4%
Table 2. An summary table of numerical results when testing on a
large batch of Middlebury testing images
Additionally, as can be seen in table 2, the 1D CNN
model is not yet exhibiting overfitting on out-of-bag sam-
ples, and might benefit from additional training time. It
can also be seen that our best 2D CNN architecture dras-
tically underperforms even the standard baselines. While
there may be some more optimal 2D CNN architecture than
the one we tried, our poor initial results made us moved to-
wards trying to build an RNN method instead. However,
we did not have enough time to finish designing and train-
ing our RNN models for replacing semiglobal matching.
Another interesting experimental result is the qualitative
performance of the classifier models. As shown in figure
9, the classification-based models sometimes generate com-
pletely erroneous results for parts of the image. While Cen-
sus will fail to generate a result sometimes, and semiglobal
matching learns a smooth transformation. In contrast, while
the classification models have lower error, they sometimes
predict very non-smooth results, as the classifier is run per
pixel. This is suggestive that a classifier, such as an RNN,
that accounts for neighborhood information may perform
even better. Also, while we did not combined semiglobal
matching with our 1D CNN, it is possible to use the nor-
malized probabilities from the neural network together with
semiglobal matching to overcome this lack of smoothness
and achieve perhaps an even better result.
(a) Census and Semiglobal Results
9.1
(b) Random Forest and 1D CNN Results
Figure 9. A qualitative example using the presented classifiers. It
can seen that the original cost method (Census) is able to resolve
certain parts of the scene. On the other hand, semiglobal propaga-
tion is able to in-paint the image and generate a smooth disparity
image. On the other hand, the errors made by the two classifier
models, although having better accuracy and RMS error than the
heuristic methods, sometimes generate what looks like completely
erroneous results.
6. Conclusion
We have presented a new method for taking stereoscopic
correlation costs and smoothing them into a more refined
estimate. This method is gradient-trainable, and outper-
forms the semiglobal matching [7] heuristic technique used
in state-of-the-art methods such as MC-CNN [21]. This
leads support to the hypothesis proposed in the introduc-
tion, which is that continuing to replace components of the
stereo matching pipeline with machine-learned models is a
way to improve their performance. Since the models pre-
sented here were done with ADCensus costs [13] and not
MC-CNN costs [21], and we did not have enough time to
train on the full Middlebury dataset [18], we don’t present a
new state-of-the-art for stereoscopic correspondence. How-
ever, we believe that these results suggest that one may be
possible by simply running the proposed techniques with
MC-CNN on the full dataset.
In addition, we’ve created a new, simply, fast and cross-
platform stereo correspondence implementation. We’ve
shown it to be about as fast as the one in OpenCV, and to
produce results that are notably more accurate. We hope
this can be used as a base for others to experiment with
other stereoscopic correspondence ideas without having to
dive into complicated OpenCV SSE code or deal with slow
MATLAB implementations.
6

Figure 10. An example of a pixelwise RNN from [14], a gradient-
learned method for propagating information across images.
Figure 11. An example of a spatial transformer for region selection
[11], a gradient-learned method for region selection.
7. Next Steps
To continue this theme of research, we wish to explore
additional architectures for stereo correspondence algo-
rithms that are trained with error gradients. While the one-
dimensional CNN presented here works well, it isn’t able
to capture the neighborhood information that semiglobal
matching can. To incorporate neighborhood information,
we’d like to explore recurrent neural network models, which
we began to design but were unable to get running in time
for this project submission. By coupling our 1D-CNN ar-
chitecture with either a spatial transformer networks front-
end [11], or a recurrent neural network backend [3] [9] , we
might produce a new state-of-the-art algorithm for the clas-
sic stereo problem. Examples of these models are shown in
figures 10 and 11.
8. Code
Code is made available at https://github.com/
leonidk/centest. Running the stereo matching algo-
rithm is straightforward and documented in the README,
but running the learning algorithms (found in the learning/
folder) varies depending on the method.
References
[1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen,
C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghe-
mawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia,
R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané,
R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster,
7
J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker,
V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. War-
den, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. Tensor-
Flow: Large-scale machine learning on heterogeneous sys-
tems, 2015. Software available from tensorflow.org.
[2] G. Bradski. Opencv library. Dr. Dobb’s Journal of Software
Tools, 2000.
[3] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau,
F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase
representations using rnn encoder-decoder for statistical ma-
chine translation. arXiv preprint arXiv:1406.1078, 2014.
[4] F. Chollet. Keras. https://github.com/fchollet/
keras, 2015.
[5] W. S. Fife and J. K. Archibald. Improved census transforms
for resource-optimized stereo vision. Circuits and Systems
for Video Technology, IEEE Transactions on, 23(1):60–73,
2013.
[6] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for au-
tonomous driving? the kitti vision benchmark suite. In
Conference on Computer Vision and Pattern Recognition
(CVPR), 2012.
[7] H. Hirschmüller. Accurate and efficient stereo processing by
semi-global matching and mutual information. In Computer
Vision and Pattern Recognition, 2005. CVPR 2005. IEEE
Computer Society Conference on, volume 2, pages 807–814.
IEEE, 2005.
[8] H. Hirschmüller and D. Scharstein. Evaluation of stereo
matching costs on images with radiometric differences. Pat-
tern Analysis and Machine Intelligence, IEEE Transactions
on, 31(9):1582–1599, 2009.
[9] S. Hochreiter and J. Schmidhuber. Long short-term memory.
Neural computation, 9(8):1735–1780, 1997.
[10] A. Hosni, M. Bleyer, C. Rhemann, M. Gelautz, and
C. Rother. Real-time local stereo matching using guided im-
age filtering. In Multimedia and Expo (ICME), 2011 IEEE
International Conference on, pages 1–6. IEEE, 2011.
[11] M. Jaderberg, K. Simonyan, A. Zisserman, and
K. Kavukcuoglu. Spatial transformer networks. CoRR,
abs/1506.02025, 2015.
[12] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-
based learning applied to document recognition. Proceed-
ings of the IEEE, 86(11):2278–2324, 1998.
[13] X. Mei, X. Sun, M. Zhou, S. Jiao, H. Wang, and X. Zhang.
On building an accurate stereo matching system on graph-
ics hardware. In Computer Vision Workshops (ICCV Work-
shops), 2011 IEEE International Conference on, pages 467–
474. IEEE, 2011.
[14] A. V. D. Oord, N. Kalchbrenner, and K. Kavukcuoglu. Pixel
recurrent neural networks. CoRR, abs/1601.06759, 2016.
[15] OpenMP Architecture Review Board. OpenMP application
program interface version 3.0, May 2008.
[16] M.-G. Park and K.-J. Yoon. Leveraging stereo matching with
learning-based confidence measures. In Computer Vision
and Pattern Recognition (CVPR), 2015 IEEE Conference on,
pages 101–109. IEEE, 2015.
[17] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel,
B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss,
 V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau,
M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Ma-
chine learning in Python. Journal of Machine Learning Re-
search, 12:2825–2830, 2011.
[18] D. Scharstein, H. Hirschmüller, Y. Kitajima, G. Krathwohl,
N. Nešić, X. Wang, and P. Westling. High-resolution stereo
datasets with subpixel-accurate ground truth. In Pattern
Recognition, pages 31–42. Springer, 2014.
[19] C. Tomasi and R. Manduchi. Bilateral filtering for gray and
color images. In Computer Vision, 1998. Sixth International
Conference on, pages 839–846. IEEE, 1998.
[20] R. Zabih and J. Woodfill. Non-parametric local transforms
for computing visual correspondence. In Computer Vi-
sionECCV’94, pages 151–158. Springer, 1994.
[21] J. Zbontar and Y. LeCun. Stereo matching by training a con-
volutional neural network to compare image patches. CoRR,
abs/1510.05970, 2015.
[22] K. Zhang, J. Lu, and G. Lafruit. Cross-based local
stereo matching using orthogonal integral images. Circuits
and Systems for Video Technology, IEEE Transactions on,
19(7):1073–1079, 2009.

8
 Human Pose Estimation for Multiple Frames
Marianna Neubauer Hanna Winter
Stanford University Stanford University
mhneub@stanford.edu hannawii@stanford.edu
Abstract
Human pose estimation is a well studied topic in vision.
However, most modern techniques in human pose estima-
tion on multiple, consecutive frames, or motion capture,
require 3D depth data, which is not always readily avail-
able. Prior work using single view 2D data, on the other
hand, has been limited to pose estimation in single frames.
This raises some interesting questions. Can human pose
estimation in multiple frames be effected using 2D single
frame techniques, thereby discarding the expensive reliance
on 3D data? Can these 2D pose estimation models be im-
proved upon by taking advantage of the data similarities
across multiple consecutive images? In this paper, we en-
deavor to answer these questions. We take Yang et al.’s [1]
single frame pose estimation model using flexible mixture
of parts and apply it in a multi-frame context. We demon-
strate that we can achieve improvements on the original
method by taking advantage of the inherent data similar-
ities between consecutive frames. We achieve speed im-
provements by restricting Yang et al.’s to search locally in
intermediate frames and, under certain circumstances, ac-
curacy improvements by running a second, corrective, pass
using SVMs trained for instance recognition.
1. Introduction
Human pose estimation has become an extremely
important problem in computer vision. Quality solutions
to this problem have potential to impact many different
aspects of vision such as activity recognition and motion
capture. Additionally, success in these aspects can be
applied to gaming, human-computer interaction, athletics,
communication, and health-care. Despite huge progress
in motion capture, as exemplified with the Xbox Kinect,
the current solutions used in gaming require extensive
hardware making it impossible for such technology to be
used in daily human-computer interactions [2]. We hope to
improve motion capture to work with simple RGB single
view cameras allowing this technology to exist on everyday
phones and computers.
1
Lili Yang
Stanford University
yangl369@stanford.edu
State of the art models for human pose estimation
that are implemented for single static RGB images, also
have some minimal but noticeable accuracy shortcomings
[1]. Currently, when used on video frame sequences, these
models do not utilize the additional information provided
by surrounding frames. Operating under the assumption
that human poses change minimally between frames,
we improve the accuracy of Yang et al.’s [1] efficient
and flexible model for human detection and human pose
estimation in single static images. We take into account
the sift features of other frames in the same video clip
by training SVMs on these features. We can improve the
output of Yang’s model by testing the SVMs on parts of
the images and adjusting the original body parts to reflect
the scores calculated by the trained SVMs. The result is
a notable increase in accuracy of the imperfect Yang pose
estimation.
After discussing related work and the implications of
our method in Section 2, we further describe our process,
resulting algorithm, and our evaluation process in detail in
Section 3. Finally, we analyze our testing data and experi-
mental results for our various methods and hyperparameters
in Section 4.
2. Background
2.1. Review of Previous Work
Human pose estimation is a well studied subject, both
in video (multiple frames) and in images (single frames).
Currently, most modern techniques for pose estimation in
video rely on 3D depth data. A well known example of
this is the xBox Kinect [2], which uses pose estimation to
determine the gamer’s motion. 3D depth data has many
advantages over 2D image data, not the least of which is
the additional dimension of information. However, 3D data
can only be captured using specialized, and often expen-
sive equipment and is not as nearly ubiquitous as 2D videos.
Recent work in pose estimation on 2D image data
feature a wide range of techniques and approaches, among
them Yang’s [1], Agarwal’s [3], Dantone’s [4], and To-
shev’s [5]. These methodologies are similar in that they
focus on pose estimation on single images. We focus
primarily on Yang’s [1] method of pose estimation using a
flexible mixture-of-parts. Yang’s method has the advantage
of producing relatively good results on full body images
across a variety of poses and background contexts, while
still retaining a significant speed advantage over certain
other approaches, such as Toshev’s [5] pose estimation
using convolutional neural networks. A relatively fast
algorithm is of particular significance when we consider
pose estimation in the multi-frame context.
2.2. Our Method
Previous methods for pose estimation in the multi-frame
realm rely on 3D depth data. Our method uses only RGB
single view image data to accurately locate 26 different
body parts. Additionally, our SVMs are trained specifi-
cally on information from a given video clip resulting in
a more accurate classification of small, specific body parts.
Because deep learning would not be feasible in this con-
text, as neural networks take too long to train and require
an extremely large amount of training data, we believe our
method is the best learning-based technique to improve pose
estimation in the multi-frame context.
3. Technical Details
3.1. Overview of Methodology
Utilizing the available source code, we are improving
Yang et al.’s Image Parse model algorithm [1] on a vari-
ety of image sequences of human motion gathered from
Youtube. As an initial attempt, we implemented a HOG
features search algorithm where we compute HOG features
for each frame and find the location of body parts by search-
ing for similar features to those calculated for the body part
in the frame prior. We found that although this method
dramatically speeds up the process, the results are wors-
ened. Then, we implemented an SVM correction method
where we train an SVM for each body part for each video
clip. We improve the original Yang by testing the SVMs
on parts of the image and adjusting the Yang output based
on the scoring results. Expanding upon this method, we
integrated hard negative mining [6] for computing logical
negatives for each SVM. Additionally, we added a double-
pass with another SVM trained to classify a sub-image as
a body part or background. Finally, in order to measure
the accuracy of our computed bounding boxes we manually
annotate ground truth bounding boxes on the same image
sequences.
2
3.2. Yang Algorithm Speedup
The original implementation of Yang’s mixture of parts
algorithm runs in 30 seconds on a typical clip from our
test set (see section 4.1). Since we are testing on upwards
of 2000 images, this is unacceptably slow. Also, in a
multi-frame video with multiple people the highest scoring
bounding boxes often migrate from person to person. To
remedy these issues we reduced the space in which the
mixture of parts algorithm searched for the bounding boxes.
For the first frame of the video clip we run the full
Yang algorithm. For the second frame, we crop the image
to the box bounding containing the entire person plus
a little extra, the size of a body part bounding box, on
the top, bottom, and sides. We then run the full Yang
algorithm on the cropped image. We store the pyramid
level that is used for the bounding boxes on the second
image. For the third frame and all subsequent frames,
we crop the image using the same method to crop the
second image and we search only within the pyramid lev-
els above, at, and below the previously stored pyramid level.
Cropping the image ensures the bounding boxes do
not migrate to another person and speeds up the search
for the bounding boxes. Reducing the pyramid levels also
results in significant speedup. Instead of 30 seconds, the
algorithms runs in about 0.1-0.4 seconds per frame. This
speedup made our SVM correction method, described in
section 3.4, feasible because it allowed us to run Yang on
all the frames of a given video clip in a reasonable amount
of time. This was necessary to obtain enough training data
for the SVMs.
3.3. Interpolation with HOGs
The HOG interpolation method relies on the assumption
that a person’s pose can change only so much between
consecutive frames. Therefore, given the bounding boxes
for body parts in one frame, we are assured that the
associated bounding boxes in subsequent frame may be
found in the same vicinity and would retain similar features.
Our implementation uses Yang’s model to select bounding
boxes for the first frame of the target sequence. In each
subsequent frame, for each bounding box, we run a sliding
window search in the local vicinity of its location in the
prior frame to select candidate bounding boxes. We then
select the candidate with the closest match in HOG features
to the associated bounding box in the prior frame.
By running Yang’s relatively expensive procedure only on
the first frame, we are able to achieve significant speed
improvements over a full run of Yang’s across all frames.
However, this methodology has two disadvantages. Firstly,
any pose estimation error made by Yang in first frame
are propagated into the subsequent frames. Secondly,
the quality of the interpolation disintegrates the farther
removed we are from the initial frame. The key weakness
of interpolation with HOGs is that it takes into account
the output of Yang’s model for only a single frame. In
subsequent investigations, we focus instead on producing
accuracy improvements using SVMs trained on the output
across all frames.
3.4. SVM Correction
Considering only a single human in each of the image se-
quences, we notice that various features, such as the color of
their skin or clothing, do not change over frames. Using this
observation, we train video clip specific SVMs to improve
the output from Yang’s model [1]. From Yang, there are
26 bounding boxes indicating locations of 26 body parts for
each frame. We split up the frames into sub-images defined
by each bounding box as seen in Figure 1 and treat each
of these sub-images as training data for the SVMs. Addi-
tionally, for each frame, we compute negative examples by
randomly selecting bounding boxes within a certain area of
the human and then discarding those that overlap with any
of the calculated body part bounding boxes. We then re-
peat this process until enough negative examples are found
(Figure 2).
Figure 1: A visualization of segmenting the frames from the
bounding boxes calculated by Yang [1] to create the training
data used to train the 26 different SVMs.
Utilizing the VLFeat library [7], we compute cluster
centers from the combined training data by calculating the
sift features for each training example and using k-means
clustering to find centers for all the sift features. Note that
sift features were computed using the RGB information as
the colors are important features for training. We found that
a larger number of centers, such as 100, produced better
results. For all of the training data, we create Bag of Words
feature vectors. For each pyramid depth, p, we break the
training example into a p × p grid of sub-images and take
the sift feature vector for each section. After finding the
3
Figure 2: The process of finding negative examples in each
frame. The leftmost image shows the boundary around the
person in which random bounding boxes are found. The
center image shows these boxes. Then, all the boxes that
overlap with any of the body parts are filtered out and the
resulting bounding boxes that will become negative exam-
ples are displayed in the rightmost image. This process is
repeated until a sufficient amount of negative examples are
found.
closest cluster center to each sift feature vector, we create a
histogram of this distribution and concatenate all sub-image
histograms together to form our Bag of Words. The Bag
of Words features are then used to train the 26 SVMs. For
a given SVM for body part a, all features for the 25 other
body parts and for the negative sub-images are treated as
negative examples.
In order to improve the original output from Yang’s
model [1], we test the SVM on every 10 frames using a
sliding window. As shown in Figure 3, for a given frame
and a given body part, a, we initialize a score for the
SVMs associated with a on the original calculation from
Yang. Then, we start sliding a window of the same size as
the original computing a score at every position with the
SVM for a. The window position with the maximum score
becomes the corrected bounding box.
Figure 3: The sliding window method. The image to the
left displays the original Yang output for the left hand. The
middle image shows the sliding window starting from the
to left and moving across and down. A score is calculated
for each position. The image on the right indicates the cor-
rected body part which is the position of the sliding window
that resulted in the best score.
3.4.1 Double-Pass SVM To evaluate the performance of our algorithms, we
compute an AP vs. overlap threshold curve (AOC), similar
After our initial results, we noticed that if Yang’s model
to the AP curve described in [8]. A robust algorithm
mistakenly placed enough bounding boxes on parts of the
should generate a curve that maintains high AP for all
background that our SVM would do the same. We improve
overlap thresholds, however some drop off is expected. If
our method by using an additional, background distinguish-
there is a drop off it should occur at high overlap thresholds.
ing SVM. We train this SVM on the same feature set as the
26 body parts SVMs but using as positives all body parts
Different regions of the body have drastically differ-
bounding boxes and as negatives all background bounding
ent performances. In general arms and legs perform more
boxes. During the sliding windows stage, this SVM is used
poorly than head and torso in Yang’s algorithm. Therefore,
to filter candidate bounding boxes. Only bounding boxes
we also look at the average raw IOU for each region of
that are classified as non-background are kept and subse-
the body for each clip to see if the relative performance
quently scored by the corresponding body part SVM.
between different algorithms depends on the body region.
We defined seven regions: head, left torso, left arm, left
3.4.2 Hard Negative Mining leg, right torso, right arm, and right leg.
To further improve our method, we take advantage of the
4. Experiments
hard negative mining method [6]. In this addition to our
SVM Correction technique, we train our original 26 SVMs 4.1. Dataset
without any negative examples aside from other body parts.
Yang’s model [1] is pre-trained on the Image Parse
Then, using these SVMs, we test the on the randomly se-
dataset [9]. For testing, we require a dataset containing
lected negatives collected by our previous method. We do
human full-body footage because the model is trained on
this over a series of iterations where in each iteration we
images containing full-body poses.
collect new negative examples, test these negative examples
on all 26 SVMs, take the maximum score, and then keep a
To capture a variety of poses, we pulled video footage from
maximum of 30 negative examples for each video frame
Youtube containing varied subject matter [10], [11], [12],
that have a positive score. Our iterations stop once we have
[13] such as people walking, dancing, and playing sports.
kept a sufficient amount of negative examples. Using this
We cut these videos such that each clip contains a single
technique, we are able to collect the most ”confusing” neg-
camera view and the full-body of the subject. We prepro-
atives to train our SVMs on. We then recompute the cluster
cess the clips to obtain image sequences of the frames.
centers and Bag of Words features including the negative
Each frame is downsized using bicubic interpolation to be
examples and re-train all 26 SVMs. The correction step us-
about 256x256 pixels while maintaining the original aspect
ing the sliding window technique remains the same.
ratio. The downsizing is done to match the approximate
3.5. Evaluation size of the testing images used in [1].
To evaluate the performance of our algorithm, we The ground truths associated with our dataset were
measure how many body parts are correctly localized by made by manually clicking the points of all 26 different
comparing the pixel positions of the computed bounding body parts for every 10 frames. Each click is the centroid
boxes and the manually annotated ground truth bounding of a bounding box for a given body part. For evaluation, we
boxes. The Image Parse model outputs the four corners of believe comparing every 10 frames with the ground truth
a square bounding box while the manual annotation only values is sufficient to determine accuracy.
stores the centroid of a bounding box. We measure the
intersection over union of the computed bounding box and
the ground truth. We assume the size of the bounding box
for the ground truth is the same as the size of the computed 4.2. Results
bounding boxes. A bounding box is labeled ”correct” if its
4.2.1 HOG interpolation
IOU is above a certain threshold.
The HOG interpolation failed to provide accurate bounding
To aggregate this data for a single video clip, we count the boxes for subsequent frames because of drift. Any pose
number of frames a body part is correctly localized and estimation error made by Yang in first frame are propagated
divide that by the total number of frames. This number is into the subsequent frames and the quality of the interpo-
the average precision (AP) of the algorithm for that body lation disintegrates the farther removed we are from the
part in that video clip. initial frame. Figure 4 shows the decrease in average IOU
with increasing frame number. In general, the average IOU

4
Figure 4: Average IOU over for all video clips. Each thin
solid line represents a clip. There are 12 clips ranging from
51 to 121 frames. The black dotted line is an average IOU
over all clips.
Figure 5: Average IOU over all clips for each body region
of Yang output (blue) and HOGs interpolation (yellow).
overlap with the ground truth over all frames in all clips is
significantly lower than in the original Yang output (Figure
5).
All clips performed worse under HOG interpolation
except for Walking Clip 3 (the diamond in Figure 6). A
histogram of the average IOU for each body region rein-
forces that finding (Figure 7). This is likely not because the
HOGs performed well but instead because the Yang output
performed poorly for that particular clip. Note that the
left arm in Figure 8 is not properly localized by the Yang
output, but the HOGs have some overlap with the ground
truth. Also note that the right arm has better localization in
the HOGs interpolation than the Yang output.
5
Figure 6: AP vs. Overlap Threshold Curve of the original
Yang output (red) and the HOGs interpolation output (blue).
Lines with corresponding symbols indicate corresponding
clips. For example, the triangle symbol is the Yang and
HOG evaluation for Walking Clip 1.
Figure 7: Average IOU in Walking Clip 3 for each body
region of Yang output (blue) and HOGs interpolation (yel-
low).
4.2.2 One Pass SVM with Randomly Selected Nega-
tives
Our single pass SVM has a pyramid depth of 5 and 100
cluster centers because those parameters produced consis-
tently good results. We trained and tested the SVM on 5
clips and found that it improved the performance of two of
the clips, and decreased performance in two of the clips,
and did not change the performance in one of the clips (see
Figure 9). Specifically, the SVM improved Beyonce Clip
1 and MLB Clip 1, it made worse Dog Walking Clip 2
and Walking Clip 1, while Dog Walking Clip 1 remained
the same. The improvement in Beyonce Clip 1 was very
 (a) Yang Output (b) HOGs
(c) Ground Truth
Figure 8: Frame 41 of Walking Clip 3.
Figure 9: AP vs. Overlap Threshold Curve of the original
Yang output (red), the single pass SVM correction (blue),
and the double pass SVM correction (green). Lines with
corresponding symbols indicate corresponding clips. dia-
mond: Beyonce CLip 1, asterisk: Beyonce CLip 6, x: Dog
Walking Clip 1, triangle: Dog Walking Clip 2, square: MLB
Clip 1, carrot: Walking Clip 1.
significant (the asterisk in Figure 9). Figure 10 shows that
the original Yang output placed the bounding boxes too
far right and the SVM correction shifted them back to the
center. The SVM also fixed one of the bounding boxes in
the left (pink) arm.
Averaging the IOU over all of the clips (Figure 11)
reveals that the SVM did slightly worse for all body regions
except for the head, left torso and left arm.
4.2.3 Double Pass SVM with Hard Negatives
The double pass with hard negative mining improves the
performance over the single pass SVM in Beyonce Clip
6
(a) Yang Output (b) Single Pass SVM
(c) Ground Truth
Figure 10: Frame 91 of Beyonce Clip 1
Figure 11: Average IOU over all clips for each body region
of Yang output (blue), single pass SVM correction (green)
and double pass SVM correction (yellow).
6, Dog Walking Clip 2, and Walking Clip 1 (Figure 9).
However, in Dog Walking Clip 2 and Walking Clip 1, it still
performed worse than the original Yang output. The double
pass SVM performed significantly better than the original
Yang output in Beyonce Clip 1 and Beyonce Clip 6. For
MLB Clip 6, the SVM corrections have higher average
precision at lower and middle thresholds while the original
Yang output has a higher average precision at the highest
thresholds. In Dog Walking Clip 1 the performance of all
three methods are similar.
In general, both the single and double pass SVM,
when averaged over all the clips, resulted in more accurate
bounding boxes than the original Yang output (Figure ??).
The Single pass SVM performs the best for the left torso,
and right leg, while the double pass SVM performs the
 (a) MLB Clip 1 (b) Dog Walking Clip 1
Figure 12: Example frames from various different clips displaying the SVM correction using hard negative mining and a
double pass. The top row is the original Yang result and the bottom row is the result after our SVM correction.
best with the head, left arm, left leg, and right torso. This
indicates that the extra background SVM pass and the hard
negatives mining did improve the performance of the SVM
correction overall especially since the arms from incorrect
Yang outputs tend to include background sub-images.
Figure 12 shows example frames of where the double
pass SVM corrects errors in the original Yang output. For
example, the right arm for Walking Clip 1 in the Yang
output bounds the background while in the SVM correction
it bounds the right arm. For MLB Clip 1 and Beyonce Clip
1 the arms move closer to the body in the SVM correction
except for one bounding box. The solitary bounding
box remains far away because the true arm is outside of
the search space defined by our correction algorithm. In
Beyonce Clip 1 the left and right legs alternate probably
because the sift features are very similar between left and
right legs. There is also a right arm bounding box on the
left leg because, please, Beyonce’s legs basically look like
arms anyway.
5. Future Work
If given the time, we could make several modifications
to our SVM. Firstly, we did not tune all of the parameters of
the SVM across all of the clips to find the best overall set of
parameters. We also noticed that while some values worked
well for some clips, they worked less well for others. More
investigation in this area could produce interesting insights.
Secondly, in the implementation of the hard negatives
for the SVM, we arbitrarily set a threshold to decide
whether to include the negative example in our set of
negatives for the final SVM. For some clips this threshold
was too high and it was difficult to collect enough negative
examples in a reasonable time. In the future we could
7
(c) Beyonce Clip 1 (d) Walking Clip 1
vary the threshold and create another AP curve or ROC
curve based on that threshold to determine its effect on the
performance of the SVM.
The current implementation of the SVM is impracti-
cally slow. The most time is spent computing the Bag of
Words feature vectors in various parts of our algorithm
including the hard negative mining loop and the sliding
window correction section. Therefore, we believe this to
be the bottleneck of our method. Thus, parallelizing this
computation such that all frames or even all body parts in
each frame are computed in tandem could have a significant
speed up.
6. Conclusion
It is certainly true that human pose estimation is a
challenging subject with many avenues of research yet to
be explored. We have made a small effort by introducing
a method that utilizes the similarities among video frames
to improve a single image pose estimation model when
used in a multi-frame context. The improvement was
particularly marked on the clips where the original Yang’s
algorithm performed the most poorly - and arguably where
improvement was most necessary.
More importantly, we have highlighted areas where
more research is possible and laid the groundwork for
future avenues of investigation.
References
[1] Y. Yang and D. Ramanan. Articulated pose es-
timation with flexible mixtures-of-parts. In IEEE
Conf. on Computer Vision and Pattern Recognition
 (CVPR), pages 1385–1392, Washington, DC, USA,
2011. IEEE.
[2] B. Bonnechre, Jansen B., P. Salvia, H. Bouzahouene,
Omelina L., J. Cornelis, M. Rooze, and S. Van
Sint Jan. What are the current limits of the kinect sen-
sor? In 9th International Conf. on Disability, Virtual
Reality and Associated Technologies, pages 287–294,
Laval, France, 2012.
[3] A. Agarwal and B Triggs. 3d human pose from silhou-
ettes by relevance vector regression. In IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR),
volume 2, pages II–882–II–888 Vol.2, June 2004.
[4] M. Dantone, J. Gall, C. Leistner, and L. van Gool. Hu-
man pose estimation using body parts dependent joint
regressors. In IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), pages 3041–3048, Port-
land, OR, USA, June 2013. IEEE.
[5] A Toshev and C Szegedy. Deeppose: Human
pose estimation via deep neural networks. CoRR,
abs/1312.4659, 2013.
[6] Andrea Vedaldi. Object category detection practical.
http://www.robots.ox.ac.uk/ vgg/practicals/category-
detection.
[7] A. Vedaldi and B. Fulkerson. VLFeat: An open
and portable library of computer vision algorithms.
http://www.vlfeat.org/, 2008.
[8] M. Everingham, L. Van Gool, C. K. I. Williams,
J. Winn, and A. Zisserman. The pascal visual object
classes (voc) challenge. International Journal of Com-
puter Vision, 88(2):303–338, June 2010.
[9] D. Ramanan. Learning to parse images of articulated
bodies. In Advances in Neural Information Processing
Systems 19, Proceedings of the Twentieth Annual Con-
ference on Neural Information Processing Systems,
Vancouver, British Columbia, Canada, December 4-7,
2006, pages 1129–1136, 2006.
[10] beyonceVEVO. Beyonce - sin-
gle ladies (put a ring on it).
https://www.youtube.com/watch?v=4m1EFMoRFvY.
[11] Barcroft TV. Dog whisperer: Trainer
walks pack of dogs without a leash.
https://www.youtube.com/watch?v=Cbtkoo3zAyI.
[12] Cesar Bess. Mlb top plays april 2015.
https://www.youtube.com/watch?v=mpe9w-CHsoE.
[13] BigDawsVlogs. Walking next to people extras.
https://www.youtube.com/watch?v=776niN4-A58.

8
 Indoor Scene Segmentation using Conditional Random Fields
Colin Wei and Helen Jiang
Stanford University
{colinwei, helennn}@stanford.edu
Abstract
Indoor scene segmentation is a problem that has become
very popular in the field of computer vision with applica-
tions that include robotics, medical imaging, home remodel-
ing, and video surveillance. This problem proves even more
difficult when the scene is cluttered. Our project aims to
explore ways to improve indoor scene segmentation algo-
rithms by examining and evaluating a popular method.
We focus on evaluating the robustness of the algorithm
for indoor scene segmentation described in [16] by Silber-
man et. al which uses SIFT features and conditional ran-
dom fields to produce segmentations. In our project, we re-
implement their method and compare performance by mod-
ifying specific sections.
We find that the neural network used in [16] is not robust
to an increasing number of classes, but the CRF model is
in the sense that as we increase the number of classes, the
CRF becomes increasingly important to produce an accu-
rate segmentation. Furthermore, we also demonstrate that
changing the algorithm we use to generate superpixel seg-
mentations increases classification accuracy of the entire
pipeline.
1. Introduction
In our paper, we explore the semantic segmentation of
indoor scenes, even cluttered ones. The main goal is that
given an RGB or RGB-D image of a cluttered indoor scene,
we output a properly labeled image with each individual
pixel corresponding to an object class, such as a television,
chair, or table. Although semantically segmenting a scene
is an easy task for humans, automatic segmentation using
machines proves to be a challenging problem.
The solution we are implementing is modeled after the
segmentation algorithm described in [16], which utilizes
neural networks trained on SIFT features along with con-
ditional random fields in order to produce a semantic seg-
mentation.
The goal of our paper is to thoroughly evaluate the al-
gorithm described in [16] in order to understand its suc-
1
cess cases and failure cases. We hope that in doing so, we
can gain intuition on directions for future work. We follow
and re-implement the technical details of Silberman et. al’s
method of indoor semantic segmentation. Furthermore, we
test modifications to the method.
2. Related Work and Contributions
2.1. Literature Review
There is a large body of work on using conditional ran-
dom fields (CRF’s) in order to produce semantic segmen-
tations of images. In [11], He et. al present an approach
using multi-scale conditional random fields for image seg-
mentation. They leverage 3 different probabilistic mod-
els: a classifier relying only on local information, a con-
ditional random field relying on hidden regional variables
to model interactions between object classes, and a CRF
that incorporates global label information. Their model
relied on a complicated training loop. Subsequent works
such as [15, 18] use pairwise potentials to model the in-
teractions between neighboring pixels when producing se-
mantic segmentations. In [16, 14], Silberman et. al fol-
low the same CRF framework and also introduce the NYU
dataset, a densely labeled dataset of indoor scenes. Finally,
in [8], Chen et. al introduce a state-of-the-art segmenta-
tion pipeline which utilizes a deep convolutional neural net-
work and fully-connected CRF to produce accurate segmen-
tations.
2.2. Our Contribution
We reimplement the semantic segmentation approach in
[16], and run the following main experiments:
1. We evaluate the ability of the neural network used in
[16] to learn 100 object classes. In [16], 13 object
classes are used. This experiment allows us to mea-
sure the robustness of their neural network.
2. We evaluate the performance of their CRF pipeline
using different superpixel algorithms to create initial
low-level segmentations. We also qualitatively analyze
the performance of the CRF on images in the test set
and note potential areas for improvement.

Figure 1: Example data from the NYU dataset: Left=RGB
Image; Middle=Raw Depth Image; Right=ground truth
class labels created by Amazon Turk
3. We evaluate the performance of their CRF pipeline in
the 100 object class setting, and show that the CRF
model is robust to an increasing number of object
classes.
3. Technical Details
3.1. Dataset
We use the SUN RGB-D dataset [17], which contains the
RGB-D images from the NYU depth dataset [14] which is
the one that Silberman et. al. created and used, the Berkeley
B3D0 dataset [12], and the SUN3D dataset [21].
In our project, we use 1449 images from only the NYU
v2 dataset [14], although we use the labels provided by the
SUN dataset [17]. The NYU dataset includes the raw RGB
images, the raw depth images, and the labeled images, as
shown by example in Figure 1. We chose to use the SUN
RGB-D version of the NYU images because the SUN RGB-
D dataset is around 10000 images total, allowing us to ex-
tend our work to settings with more data in the future. The
SUN RGB-D dataset provides object class labels for each
individual pixel of every image. Although the dataset pro-
vides depth information for each image, we only use RGB
information for all of our implementations.
3.2. Segmentation Pipeline
To produce a semantic segmentation of an image, we
take the following steps, following the method of Silberman
et. al:
1. Extract SIFT features {fij } from a dense grid on the
image using a sliding window. As in [16], we use a
grid with a stride of 10, where each sliding window is
of size 40 by 40. Although Silberman et. al concate-
nate SIFT features from three different scales, we only
use one scale for our project in the interest of saving
time and computational power.
2. Use a neural network to produce class probabilities
P (·|fi ) for each feature location i in the image. As
in [16], our neural network has a single hidden layer
of size 1000.
3. Use a low-level segmentation algorithm to segment the
image into superpixels. We experiment with Felzen-
szwalb segmentation, the approach used in [16], as
well as quickshift and SLIC segmentation. All points i
in a superpixel are assigned class probabilities Pi that
are equal to the average of the class probabilities for
all descriptors corresponding to grid points in a super-
pixel. If no grid points fall inside a superpixel, we as-
sign the superpixel uniform class probabilities.
4. Model pixel labels as a conditional random field. The
energy of the CRF is defined as follows:
X X X
E(y) = φi (yi ) + ψij (yi , yj )
i∈I i∈I j∈N (i)
The summations are taken over all pixels in the image.
The first summation models unary potentials for each
class, while the second summation models pairwise in-
teractions between neighboring pixels.
Silberman et. al model φi as the negative log of the
product of Pi (yi ) and a location prior on the class yi .
We did not implement location priors and instead set
φi (yi ) = − log Pi (yi ).
Finally, we set pairwise potentials
ψij (yi , yj ) = 1(yi 6= yj )ηe−αkIi −Ij k2
2
where Ii and Ij are the i-th and j-th RGB color chan-
nels and η, α are hyperparameters. This potential func-
tion mirrors the one used in [10]. Although Silberman
et. al use a different pairwise potential function, we
find that this potential function is easier to tune.
5. Minimize the energy function of the CRF. In [16], Sil-
berman et. al use the scheme provided by [4]. We
experiment with both Boykov et. al’s α-expansion al-
gorithm in [4] and simulated annealing [2].
3.3. Low-level Segmentation Methods
In our semantic segmentation framework, we utilize
three different unsupervised algorithms for the purpose of
producing low-level segmentations:
3.3.1 Felzenszwalb [9]
This algorithm is used by Silberman et. al and preserves
detail in low-variability regions in the image rather than
high-variability regions. The algorithm proceeds by using
a graph-based representation of the image to find region
boundaries, sequentially combining different regions based
on a similarity score across regions.
3.3.2 Quickshift [20]
Quickshift is a mode-seeking clustering algorithm that
builds off of variants of mean-shift. The algorithm provides
a density estimate for each pixel location; it then constructs Table 1: Statistics for 11 Class Train/Test Set
a tree where images are connected to their nearest neigh- Class Name # Training Descriptors # Test Index
bors with larger pixel values. By cutting branches of the Bed 31791 39161 0
tree whose distance cross a max threshold, we obtain clus- Bookshelf 58853 38289 1
ters for the image. Cabinet 99732 80484 2
Ceiling 14025 11884 3
3.3.3 SLIC (Simple Linear Iterative Clustering) [1] Floor 108333 80595 4
Picture 24932 25235 5
SLIC is a clustering method that essentially applies the k- Sofa 34744 47957 6
means algorithm on a different feature space, which takes Table 29964 28882 7
into account pixel location and color intensity. After con- Television 7126 12905 8
structing this feature space, SLIC uses Lloyd’s algorithm to Wall 373894 311308 9
output cluster assignments. Window 30695 28160 10
3.4. CRF Optimization The total number of training and test examples for each
class for our 11 class model.
Although the minimization of the energy function of a
CRF is an NP-hard problem, the α-expansion algorithm [4]
and simulated annealing provide general methods for per- the probability distribution is near uniform, to low values
forming this optimization [2]. near 0, where the distribution concentrates on labelings that
minimize the energy function.
3.4.1 α-expansion Algorithm
3.5. Our Implementation Details
This algorithm by [4] formulates the problem of minimiz-
ing the energy function of a CRF as a min-cut problem. We implement everything using Python. To extract SIFT
Although this problem is still NP-hard in the worst case, features, we use OpenCV’s Python wrapper [5]. To train
Boykov et. al provide an approximation algorithm that finds the neural network over the dense grid of SIFT features,
a local minimum with respect to α-expansion moves, which we use the Python package scikit-neuralnetwork [7]. We
consists of moves where pixels change their labels to α for use the package scikit-image to run the superpixel seg-
a given label α. Given a labeling y and label α, we can mentation algorithms [19]. Finally, for the graph-cut op-
compute timization algorithm of [4], we use gco-python, Andreas
Mueller’s Python wrappers for the gco optimization pack-
min E(y 0 ) age [4, 13, 3]. We implemented simulated annealing from
y 0 ∈Nα (y)
scratch using Cython to optimize for speed. We did not
where Nα (y) is the set of labels within one α-expansion of use any code from [16], as we implemented all of the pre-
y and E(y 0 ) is the energy of the labeling y 0 , in polynomial processing steps ourselves.
time using a reduction to standard min-cut. The algorithm
proceeds by iteratively performing this local minimization 4. Experimental Setup and Results
procedure. 4.1. Neural Network
We train and test a neural network for classifying image
3.4.2 Simulated Annealing
patches. We partition all images from the NYU dataset into
Simulated annealing is a black-box combinatorial optimiza- the train/test splits provided in [14]. For each training im-
tion problem that is guaranteed to converge to the optimal age, we extract SIFT features from the dense grid described
solution, though it may be very slow in practice [2]. The in Section 3.2. Likewise, we construct a test set by extract-
algorithm proceeds by sampling local changes to a label as- ing SIFT features from each image in the test set using the
signment (i.e. the configuration of a single pixel at location same dense grid. After consolidating training features, we
i) with conditional probability equal to randomly partition the SIFT descriptors into 12 total subsets
so we do not have to load the entire training set into mem-
exp − T1 E(y new )


new ory. One training loop consists of a pass over all 12 training
P (yi |y) = P 1 0


y 0 ∈Ni (y) exp − T E(y )
subsets, starting with a learning rate of 0.1 and multiplying
by a decay rate of 0.75 with each subsequent subset. Every
where Ni (y) is the set of labelings obtained by changing other hyperparameter for our neural network is set to the
only pixel i, and T is a changing temperature parameter. scikit-neuralnetwork default.
The idea is to gradually reduce T from high values, where Because the SUN RGB-D dataset contains 894 different
 Table 2: Train/Test Accuracy on 11 Class Set
% Train Correct % Test Correct
62.06% 49.96%
Accuracy on 11 class training and test sets. Computed for
all grid points for each train/test image.

Table 3: Train/Test Accuracy on 100 Class Set

% Train Correct % Test Correct
25.89% 13.37%
Accuracy on 100 class training and test sets. Computed for
all grid points for each train/test image.
(a) Confusion matrix for training.

object class labels, a number that is too big for a neural net-
work with a single hidden layer to accurately classify, we
instead train on subsets of the classes. In total, we train
2 different neural networks. Following [16], we handpick
11 object classes to train on, shown in Table 1. Silberman
et. al use 13 classes - the classes that we use, in addition
to the “blind” and “background” classes. We do not use
these classes, however, because they do not appear in the
SUN RGB-D labels. We also choose the 100 most common
labels, and we train a neural network to classify between
these classes. For reference, our code includes a pickle file
containing the names of these 100 classes. The 11 hand-
picked classes form a large subset of these 100 most com-
mon classes.
From Table 1, we can see that class distributions between
train and test are pretty similar, but class distributions are
both very skewed. This also holds for the 100 classes set.
Because of this skewed distribution, it is possible to train
a neural network that achieves high test accuracy but only
learns a few classes properly. To remedy this issue, we bal-
ance the training distribution by capping each class at 5000
examples per split in the 11 class case and 1000 examples
per split in the 100 class case. We are unclear on how Sil-
berman et. al work around this problem.
In Table 2 and Figure 2, we show the accuracy results
and confusion matrices for the 11 class dataset. In Table 3
and Figure 3, we show the accuracy results and confusion
matrices for the 100 class dataset. From the discrepancy be-
tween the training and testing accuracy for both datasets, it
is clear that our models overfit, even though we use a sub-
stantial amount of training data given the size of networks
we train. There are two main reasons why this could hap-
pen. First, descriptors from the same image corresponding
to nearby points possess some redundancy, which means the
effective number of training samples is smaller than the ac-
tual number. Second, intra-class discrepancy is very high
between different indoor scenes. Since the train/test splits
in [14] are arranged so that not a single scene is in both train
and test, test images could present variations of a class not
(b) Confusion matrix for testing.
Figure 2: Confusion matrix for 11 class dataset. Indices
correspond to indices in Table 1.
seen during training.
Judging from the confusion matrices in Figure 2, it
seems that the 11 class neural network performs worst on
classes that are both relatively scarce and similar in appear-
ance to other classes. For example, sofas and tables are
commonly classified as floors. These sofa and table classes
are very scarce compared to floors and exhibit similar prop-
erties such as a large flat surface, leading to this incorrect
classification. We have already mitigated many of these in-
correct classifications by trying to balance classes during
training, but we are unsure how to improve this further with-
out switching to a deeper network architecture. Since Sil-
berman et. al do not provide their neural network results
in [16], we cannot perform a direct comparison. However,
their CRF with only unary potentials achieves a 40.9% pixel
accuracy on 13 classes, which implies that our results on 2
fewer classes are on a comparable performance level.
We cannot make any comparison to [16] on the 100 class
case because they only provide results for 13 classes. How-
ever, the confusion matrix in Figure 3 shows that the sin-
gle layer neural network is not robust enough for the 100
class case. Many of the classes that are very incorrectly

(a) Confusion matrix for training.
(b) Confusion matrix for testing.
Figure 3: Confusion matrix for 100 class dataset. Indices
are available in the code.
classified in the test confusion matrix are also very uncom-
mon; for example the class ”backpack” (index 8) appears
only 539 times throughout the entire training set, and the
model rarely outputs that label. An interesting question is
whether the model fails because SIFT descriptors cannot
capture enough information about all 100 classes or because
the network architecture itself is flawed. In future work, we
could investigate this by training the single layer network
on the raw image patches and observing whether this im-
proves results, which would imply SIFT as the problem. If
the results do not improve, a deeper network may be neces-
sary.
Table 4: Superpixel Algorithm and 11 Class CRF Perfor-
mance
Superpixel Alg. Unary Acc. CRF Acc.
Felzenszwalb 48.74% 49.95%
Quickshift 48.20% 51.21%
SLIC 49.93% 51.35%
Accuracy of pixel-level labels for segmentation of test im-
ages on 11 classes. We only consider pixels that fall in one
of the 11 classes. Unary accuracy is computed from the
segmentation given by minimizing the unary terms of the
energy function. CRF accuracy is computed from consider-
ing pairwise terms too.
(a) Original image
(b) Felzenszwalb, quickshift, SLIC superpixels
Figure 4: Example superpixel segmentation.
4.2. Superpixel Algorithms and CRF Performance
For the 11 class case, we analyze the performance of
the entire segmentation pipeline described in Section 3.2,
varying the algorithm we use to create initial superpixel
segmentations. For the scikit-image implementation of the
Felzenszwalb algorithm, we set the scale parameter to 100
based on qualitative evaluation of a few training images.
For quickshift and SLIC, we use the default hyperparame-
ters from the implementation of scikit-image.
Table 4 shows the accuracy results for our different trials
over the entire test set. To produce the results in Table 4, we
use Boykov et. al’s α-expansion algorithm for minimizing
the CRF energy for each trial [4, 13, 3]. We use 5 iterations
for the α-expansion algorithm.
Figure 5: The images on the left show segmentation results Surprisingly, we find that performing Felzenszwalb seg-
for different superpixel initializations in Figure 4. On the mentation to create superpixels, the method that Silberman
right, the truth map is show. Black means the model clas- et. al use in [16], actually results in the worst performance
sified the pixel correct, white means the model classified out of all the superpixel segmentation methods that we try.
incorrect, and gray means the pixel does not belong to any The accuracy rate of the conditional random field, 49.95%,
of the 11 classes. is more or less the same as the test accuracy of the 11 class
set, and the accuracy of the unary model with Felzenszwalb
is worse. We should note that the two test sets are differ-
ent - the test set in Section 4.1 uses only the subset of pixel
locations along the dense grid, while our test set here uses
all pixels. Even so, it seems that first performing the super-
(a) Felzenszwalb unary pixel segmentation with Felzenszwalb and quickshift actu-
ally hurts the performance of the classifier. This is because,
as seen from Figure 5, both Felzenszwalb and quickshift
create tiny clusters that do not contain any grid points (see
the tiny yellow clusters in Figures 5a and 5c) and are thus
assigned uniform superpixel probabilities. SLIC works well
even restricted to unary potentials because it produces larger
(b) Felzenszwalb CRF
clusters, ensuring that a grid point falls in each cluster.
In the CRF setting with pairwise potentials, quickshift
and SLIC both provide better performance than Felzen-
szwalb segmentation. This is because, as seen in 4b,
quickshift and SLIC provide more balanced cluster sizes,
whereas Felzenszwalb segmentation can often produce very
(c) Quickshift unary large superpixels. Since all pixels within a superpixel
share the same class probabilities, if these large superpixels
have the wrong class probabilities, performance will suffer.
Meanwhile, for quickshift and SLIC superpixels which are
smaller, as long as a few superpixels have the correct class
probabilities, the CRF will be able to fix the labels for adja-
cent superpixels too.
(d) Quickshift CRF For all three superpixel algorithms, the CRF is able to
correct some of the mistakes seen in the unary version of
the model. For example, it can fix nearly all of the inaccura-
cies introduced by superpixel clusters that are too small, as
seen in the smoothness of the CRF segmentations in Figure
5. Furthermore, the CRF is able to produce minor improve-
(e) SLIC unary ments in fixing some small wrongly classified regions. The
potential for improvement is limited in three ways:

1. The accuracy of the neural network is too low. If the

neural network is too confident in the wrong labels,
then it is hard for the CRF to change as desired. In fu-
ture work, we can examine whether using a CRF with
(f) SLIC CRF more powerful neural network models results in more
performance gains solely from the CRF.

2. The spatial transition potentials do not provide a good

enough model of the transitions between classes. As
an example, this occurs in Figure 5f. The CRF is able
to correctly label more parts of the floor near the right
side of the bed. As seen in Figure 4b, a large chunk of
the patch of the floor incorrectly classified in the SLIC
unary belongs to the same superpixel. Since the CRF
 Table 5: 100 Class CRF Performance
Superpixel Alg. Unary Acc. CRF Acc.
Felzenszwalb 15.16% 15.78%
SLIC 14.77% 15.67%
Accuracy of pixel-level labels for segmentation of test im-
ages on 100 classes. We only consider pixels that fall in one
of the 100 classes. Unary accuracy is computed from the
segmentation given by minimizing the unary terms of the
energy function. CRF accuracy is computed from consider-
ing pairwise terms too.
is able to fix half of this superpixel, neural network in-
accuracy is not the only problem - therefore, transition
potentials must be an issue too.
3. The CRF can fix labels for pixels only if labels for
nearby pixels in the same object are correct. This is
because the CRF model is based on pairwise interac-
tions between neighboring pixels. In future work, we
could attempt to fix this issue by using a fully con-
nected CRF model as in [8], which allows the model
to account for global feature interactions.
Another way to potentially address the first and second lim-
itations is to fine-tune the neural network probabilities and
learn the pairwise interaction potentials by directly train-
ing a CRF model instead of handcrafting the pairwise po-
tentials. We can do this by modeling the pairwise poten-
tials as the result of some convolution kernel applied to
the local image patch followed by some nonlinearity. We
could formulate a training objective that maximizes the log-
likelihood of the true labels, optimize it using contrastive di-
vergence [6], and perform back-propagation into the unary
and pairwise potentials to learn these functions. We suspect
that this approach could mitigate the first and second limita-
tions by providing an energy function that is optimized for
the desired task, producing correct pixel labels. Because of
lack of computational power, we leave this idea for future
work.
4.3. Varying the Number of Classes
We also explore the robustness of the approach in [16]
to a varying number of classes. To do so, we run the seg-
mentation pipeline in Section 3.2 the test set using the 100
most common class labels. We run our experiments using
Felzenszwalb segmentation and SLIC in order to generate
superpixels; we forgo testing quickshift because it provided
similar performance to SLIC in our 11 class test set and the
segmentation algorithm takes too long to run in conjunction
with performing α-expansion optimization on 100 classes,
which already requires significantly more time than the 11
class case.
Original/SLIC Original/SLIC
SLIC unary SLIC unary
SLIC CRF SLIC CRF
Figure 6: Sample truth maps and segmentations for the 100
class case.
We provide our accuracy results in Table 5. Surprisingly,
whereas the superpixel segmentations hurt our unary poten-
tial performance in the 11 class case, they actually improve
our performance in the 100 class case over train and test ac-
curacy given in Table 3. We are unsure why this is the case.
In addition, Felzenszwalb actually provides better perfor-
mance for unary potentials now. This discrepancy might
be due to the fact that in the 100 class case, larger clusters
might result in significant improvement for some test im-
ages because averaging class probabilities of large clusters
reduces noise, and there is more noise in the 100 class case
as opposed to the 11 class case.
The increase in accuracy percentage between the unary
and CRF models is an interesting result that demonstrates
the robustness of the CRF model. As the number of classes
increases, the CRF actually seems to make a bigger impact
on the final segmentation. Although the actual increase in
accuracy is lower in the 100 class case than the 11 class
case, the increase is higher in proportion because many
fewer pixels get labeled correctly in the 100 class case.
From the examples shown in Figure 6, we can also qual-
itatively observe the increased impact of the CRF on seg-
mentation quality. In Figure 6, we show example segmenta-
tions using the SLIC superpixel algorithm; we choose to an-
alyze SLIC because SLIC and Felzenszwalb exhibit similar
CRF performance on the 100 class set, while SLIC is clearly
better on the 11 class set. Whereas the truth maps in Fig-
ure 5 do not change much between the unary and pairwise
cases, the truth maps shown in Figure 6 exhibit significant
changes between the two cases. Furthermore, the segmenta-
tions obtained using the CRF contain far fewer clusters than
the segmentations obtained only using the unary potentials.
We can explain the increased impact of the CRF as fol-
lows: since there are a larger number of classes the network
is less certain about its classification choices and therefore
assigns more uniform class probabilities. As a result, the
pairwise potential term is larger in magnitude than the unary
 Table 6: Optimization Algorithm Comparison
Optimization Alg. % Acc on Random Sample
Simulated Annealing 51.55%
Boykov et. al 53.01%
Unary 51.56%
Accuracy of pixel-level labels for segmentation on 11
classes using different optimization scheme. In the “Unary”
scheme, we ignore pairwise potentials and take the argmax
class probability for each pixel. For all optimization meth-
ods, we use SLIC to obtain superpixels.

term, which means the decisions resulting from pairwise

potentials are weighted more heavily. Thus, we can con-
clude that while the neural network model is not robust to an
increase in the number of classes, the CRF is quite robust.
Thus, perhaps we can expect further improvements by us-
ing better CRF potentials as described at the end of Section
4.2. One interesting question for future work is whether the
CRF would still make as big of an impact in the case where
we have a better neural network model for classifying the
grid points.
(a) Simulated annealing with SLIC

4.4. Simulated Annealing

In an attempt to determine whether the optimization al-

gorithm used for the CRF is optimal, we implement simu-
lated annealing, a black box algorithm useful for Markov
Random Fields [2]. We anneal from a temperature of 1
to 0.01, using 15 total linearly spaced temperatures. Each
annealing step, we perform a Gibbs sampling sweep over
every single pixel in the image, sampling new class labels
with probability proportional to exp − T1 E(y) , where T

is the current temperature and E(y) is the energy of CRF

on labels y. We initialize our labels using the minimum en-
ergy configuration for unary potentials - although we tried
different initialization methods, we found that this worked
best empirically. Because we sweep every single pixel in
the image every Gibbs sampling step, our implementation
is extremely slow, even when we use Cython. Therefore,
we only run our segmentation pipeline on a random sample
of 69 test images, using only the 11 class model. For fair-
ness, we compare to the α-expansion algorithm in [4] and
unary potentials using the same random sample. We show
results in Table 6. (b) Unary with SLIC
Figure 7: Example segmentation using simulated annealing.
 Unfortunately, simulated annealing does not seem to op-
timize the CRF energy at all. As shown in our example,
which is the same initial image as used in Figure 4 and Fig-
ure 5, our simulated annealing implementation does very
little to change the initial assignment. All examples in our
test set appear similar to this. This suggests that some of
our hyperparameters are not set correctly. For example, our
initial temperature might be too high, resulting in the Gibbs
sampler getting “stuck”. Furthermore, we might not have
run a sufficient number of iterations of simulated annealing.
Given more time, we could try to optimize these hyperpa-
rameters to obtain better results. However, we also conclude
that given the large number of hyperparameters to optimize
and the high computational cost, simulated annealing is not
worth the effort compared to cut-based segmentation.
5. Conclusion and Future Work
In this paper, we implement Silberman et. al’s image
segmentation pipeline in [16] with the final goal of explor-
ing the strengths and weaknesses of their algorithm. We
show that the neural network model based on SIFT features
works sufficiently well for a small number of classes, but
does not work for classification tasks on a larger number
of object classes. We also analyze and explain the perfor-
mance of different superpixel algorithms in place of Felzen-
szwalb’s segmentation algorithm, and we find that SLIC
is optimal in terms of both speed and final segmentation
quality. Furthermore, we experiment with a larger num-
ber of object classes in our test set, and we show that the
CRF framework is very robust to an increasing number of
classes, even if the neural network model is not. Finally,
we compare the performance of Boykov’s segmentation al-
gorithm to simulated annealing, and we find that Boykov’s
algorithm works much better for our problem.
Our hope is that our experimentation provides interesting
directions for future research and we believe that our anal-
ysis of the CRF’s performance does this. In Section 4.2,
we discuss limitations of the CRF and argue that directly
learning some CRF potentials from data is a promising di-
rection for further work. In Section 4.3, we show that the
CRF becomes more important as the number of classes in-
creases. Another interesting direction for future work is to
see if this still holds for an even larger number of classes
and also differing CRF connectivities.
Our code can be found at:
https://github.com/cwein3/
im-seg
References
[1] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and
S. Ssstrunk. Slic superpixels compared to state-of-the-art su-
perpixel methods. 3
[2] D. Bertsimas and J. Tsitsiklis. Simulated annealing. Statist.
Sci., 8(1):10–15, 02 1993. 2, 3, 8
[3] Y. Boykov and V. Kolmogorov. An experimental comparison
of min-cut/max- flow algorithms for energy minimization in
vision. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 26(9):1124–1137, Sept 2004. 3, 5
[4] Y. Boykov, O. Veksler, and R. Zabih. Fast approximate en-
ergy minimization via graph cuts. IEEE Trans. Pattern Anal.
Mach. Intell., 23(11):1222–1239, Nov. 2001. 2, 3, 5, 8
[5] G. Bradski. Opencv. Dr. Dobb’s Journal of Software Tools,
2000. 3
[6] M. A. Carreira-Perpinan and G. E. Hinton. On contrastive
divergence learning. 7
[7] A. Champandard and S. Samothrakis. sknn: Deep neural
networks without the learning cliff. 2015. 3
[8] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L.
Yuille. Semantic image segmentation with deep convolu-
tional nets and fully connected crfs. CoRR, abs/1412.7062,
2014. 1, 7
[9] P. F. Felzenszwalb and D. P. Huttenlocher. Efficient
graph-based image segmentation. Int. J. Comput. Vision,
59(2):167–181, Sept. 2004. 2
[10] L. Grady. Random walks for image segmentation. IEEE
Trans. Pattern Anal. Mach. Intell., 28(11):1768–1783, Nov.
2006. 2
[11] X. He, R. S. Zemel, and M. A. Carreira-Perpinan. Multiscale
conditional random fields for image labeling. In Computer
Vision and Pattern Recognition, 2004. CVPR 2004. Proceed-
ings of the 2004 IEEE Computer Society Conference on, vol-
ume 2, pages II–695–II–702 Vol.2, June 2004. 1
[12] A. Janoch, S. Karayev, Y. Jia, J. T. Barron, M. Fritz,
K. Saenko, and T. Darrell. A category-level 3-d object
dataset: Putting the kinect to work. in iccv workshop on con-
sumer depth cameras for computer vision. 2011. 2
[13] V. Kolmogorov and R. Zabih. What energy functions can
be minimized via graph cuts? In Proceedings of the 7th
European Conference on Computer Vision-Part III, ECCV
’02, pages 65–81, London, UK, UK, 2002. Springer-Verlag.
3, 5
[14] P. K. Nathan Silberman, Derek Hoiem and R. Fergus. Indoor
segmentation and support inference from rgbd images. In
ECCV, 2012. 1, 2, 3, 4
[15] J. Shotton, J. Winn, C. Rother, and A. Criminisi. Com-
puter Vision – ECCV 2006: 9th European Conference on
Computer Vision, Graz, Austria, May 7-13, 2006. Proceed-
ings, Part I, chapter TextonBoost: Joint Appearance, Shape
and Context Modeling for Multi-class Object Recognition
and Segmentation, pages 1–15. Springer Berlin Heidelberg,
Berlin, Heidelberg, 2006. 1
[16] N. Silberman and R. Fergus. Indoor scene segmentation us-
ing a structured light sensor. In Proceedings of the Inter-
national Conference on Computer Vision - Workshop on 3D
Representation and Recognition, 2011. 1, 2, 3, 4, 6, 7, 9
[17] S. Song, S. P. Lichtenberg, and J. Xiao. Sun rgb-d: A rgb-d
scene understanding benchmark suite. In The IEEE Confer-
ence on Computer Vision and Pattern Recognition (CVPR),
June 2015. 2
[18] B. Triggs and J. J. Verbeek. Scene segmentation with
crfs learned from partially labeled images. In J. C. Platt,
D. Koller, Y. Singer, and S. T. Roweis, editors, Advances
in Neural Information Processing Systems 20, pages 1553–
1560. Curran Associates, Inc., 2008. 1
[19] S. van der Walt, J. L. Schönberger, J. Nunez-Iglesias,
F. Boulogne, J. D. Warner, N. Yager, E. Gouillart, T. Yu,
and the scikit-image contributors. scikit-image: image pro-
cessing in Python. PeerJ, 2:e453, 6 2014. 3
[20] A. Vedaldi and S. Soatto. Quick shift and kernel methods for
mode seeking. 2008. 2
[21] J. Xiao, A. Owens, and A. Torralba. Sun3d: A database
of big spaces reconstructed using sfm and object labels. In
ICCV, 2013. 2
 Understanding 3D Space-Semantics

Iro Armeni
Voxelized 3D Sliding
Cube (Input)

x,y,z global
location

[0,0,0,0,...,1] [1,0,0,0,...,0]
null ceiling

Leaky ReLU

Leaky ReLU

Leaky ReLU
Leaky ReLU

Leaky ReLU

Softmax
3DConv

3DConv

3DConv

3DConv
3DConv

3DConv
or [0,0,1,0,...,0]
wall

[0,0,0,1,...,0]
other [0,1,0,0,...,0]
floor
Fully Convolutional Multi-class Voxel labels
Neural Network Classification (Output)

Raw Point Cloud Voxelized 3D Enclosed

Space (Input)

Figure 1: Frameworks for 3D Parsing of Large-Scale Indoor Point Clouds into their Space-Semantics. Exploring two different
network architectures: (1) A fully 3D CNN receives as an input a 3D voxelized sliding cube with binary occupancy and performs a per
voxel multi-class classification into 5 semantic labels. (2) A fully 3D CNN receives as an input a voxelized enclosed space with binary
occupancy and performs a per voxel multi-class classification into 10 semantic labels. The result in both cases is a class prediction per
output voxel.

Abstract they need to have an understanding of it. 3D depth sensors

Point clouds are comprehensive and detailed 3D repre-
sentations for indoor spaces, however they contain no high-
level information of the depicted area. In contrast to previ-
ous methods that have mostly focused on more traditional
pipelines, I propose a Fully 3D Convolutional Neural Net-
work for the semantic parsing of such data. In this pa-
per I explore different network architectures and perform
per voxel multi-class classification of the input into its se-
mantics. Two main inputs are explored: (a) a 3D sliding
cube on the large-scale point cloud and (b) a consistently
aligned and normalized enclosed space of the point cloud
(i.e. rooms, hallways, etc.). Both inputs are voxelized with
values equal to their binary occupancy. The network out-
puts a class prediction per voxel. I provide experiments and
results on the different architectures and used the Stanford
Large-Scale 3D Indoor Dataset for their evaluation.
1. Introduction
We spend 90% of our time indoors [3] and for systems to
operate in this environment (e.g. assist us in daily activities)
are quickly becoming standard practice and have enabled
the representation of our surroundings as whole scenes in
an ever increasing number of point clouds. Although such
data is becoming more and more available, it is not directly
useful since it does not contain high-level information of the
depicted elements, as e.g. space-semantics. Such an under-
standing would be beneficial to many applications related to
augmented reality, robotics, graphics, the construction in-
dustry, etc.
Previous work has focused on extracting this informa-
tion from point cloud data by following semantic parsing
approaches (e.g. segmentation or object detection). How-
ever the great majority of them resorts to more tradi-
tional pipelines [26], where hand-engineered features are
extracted and fed into an off-the-shelf classifier such as Sup-
port Vector Machines (SVMs). The past few years these
methods are gradually being abandoned in favor of Deep
Learning ones that produce superior results by learning fea-
tures and classifiers in a joint manner. Convolutional Neural
Networks (CNNs) in specific have made great progress and
are widely used especially in the case of 2D data. Neverthe-
less, the area of CNNs on 3D data, especially for the task
of parsing, is significantly less explored to its 2D counter-

1
parts [18].
Most work in the context of 2.5D and 3D using
ConvNets is targeting other applications like depth from
RGB [7], camera registration [13], and human action recog-
nition [11]. This has limited the amount of produced knowl-
edge and available implementations, pretrained models and
training data for 3D related tasks. Alternative approaches to
the problem of detecting 3D space semantics in large-scale
indoor point clouds could be formed by posing the problem
as 2D or 2.5D. Although these could benefit from pretrained
models, existing architecture and other 2D or 2.5D datasets
for training (e.g. [27] and [22] respectively), they would
not take advantage of the rich spatial information provided
in 3D point clouds, which can help disambiguate problem-
atic cases. It has been shown that 3D parsing methods can
perform better than their 2.5D counterparts [5].
I propose instead a framework for the task of parsing 3D
point clouds of large scale indoor areas into their space-
semantics using an end-to end 3D CNN approach. In a
higher level, the network receives as an input a voxelized
3D portion of a large-scale point cloud 1 and through a se-
ries of fully 3D convolutional layers it performs multi-class
classification on the voxel level, and outputs the predicted
class for each voxel. The network classifies each input voxel
into 10 semantic labels2 related to structural and building
elements, clutter and empty space.v
2. Related Work
Traditional Approaches: Semantic RGB-D and 3D
segmentation have been the topic of a large number of pa-
pers and have lead to a considerable leap in this area dur-
ing the past few years. For instance [29, 24, 22] propose a
RGB-D segmentation method using a set of heuristics for
leveraging 3D geometric priors. [21] developed a search-
classify based method for segmentation and modeling of in-
door spaces. These are different from the proposed method
as they mostly address the problem in a small scale. A few
methods attempted using multiple depth views [28, 9], but
they remain limited to small scale. Unlike approaches such
as [26], [15], my method learns to extract features and clas-
sify voxels from the raw volumetric data. Vote3D [31] pro-
poses an effective voting scheme to the sliding window ap-
proach on 3D data to address their sparse nature.
2.5D Convolutional Neural Networks: A subsequent
extension to RGB-D data followed the success of 2D CNNs
([17], [30], [4], [10]). However, most work handles the
depth data as an additional channel and hence it does not
make full use of the geometric information inherent in the
3D data. [8] proposes an encoding that makes better use
1 The scale of the point cloud can range from a whole building to a floor,
or any large portion of the former.
2 Due to memory restrictions some of the presented experiments use
either 5 labels. For more details see 4.3.
2
of the 3D information in the depth, but remains 2D-centric.
The presented work differs from these in that I employ a
fully volumetric representation, resulting in a richer and
more discriminative representation of the environment.
3D Convolutional Neural Networks: 3D convolutions
have been successfully used in video analysis ([11], [12])
where time acts as the third dimension. Although on an al-
gorithmic level such work is similar to the proposed one, the
data is of very different nature. In the RGB-D domain, [16]
uses an unsupervised volumetric feature learning approach
as part of a pipeline to detect indoor objects. [32] proposes
a generative 3D convolutional model of shape and apply it
to RGB-D object recognition, among other tasks. VoxNet
[20] presents a 3D CNN architecture that can be applied
to create fast object class detectors for 3D point cloud and
RGB-D data. This work has similarities, however among
the differences: it uses a different input representation, it is
not performing voxel-to-voxel classification and since the
task is detection it uses fully connected layers.
3. Method
The proposed method receives as an input a voxelized
3D portion of the point cloud and through a series of 3D
convolutions results to a class label prediction for each
voxel. I gradually explored 3 different approaches:
• 3D Sliding Window: In this network, the input is a
voxelized 3D cube of constant size with binary oc-
cupancy that is sled over the large-scale point cloud.
When fed to the network, it passes through a series of
3D Fully Convolutional layers which result to a per-
voxel multi-class classification (see Figure 1-Left).
• Adding Context: The previous approach does not
provide any context about the content of the sliding
cube in relation to the rest of the point cloud. How-
ever, context can strongly influence inference. To this
end, I provide the global position of the sliding cube in
the point cloud as a second input to the network, fol-
lowing a similar approach to [6] (see Figure 1-Right).
• Enclosed Spaces: The use of a sliding cube with con-
stant size cannot account for the different sizes that el-
ements in the point cloud appear with. Although for el-
ements that belong to the category of things (e.g. chairs
or tables) one can learn a dictionary of shapes, for el-
ements that can be categorized as stuff (e.g. walls or
ceiling) it is harder to identify repetitive shape and size
patterns. To address this issue I explored an approach
similar to [5] to take advantage of the repetitive layout
configuration that indoor enclosed spaces present (e.g.
elements are placed in a consistent way inside a room
with respect to the entrance location). The semantics
in such spaces remain intact (e.g. the wall, ceiling and
 3.1. Input
n
(Sliding 3D Cube or

x,y,z global
Enclosed Space)

1 location The input to the method is a voxelized cube that repre-

0 1
sents the underlying spatial data with binary occupancy.
Input

1 0 1
0 1
n 1 0 0
1
0
3D Sliding Window: Here the input is a 3D sliding cube
0

0 1
1
0 binary
of size 10x10x10 voxels. The stride of the cube on the point
00 1 1

n value cloud in all 3 dimensions is 10 voxels, which means that

there is no overlap between spatially consecutive cubes 3
Global The size of the input is heuristically defined and takes into
3DConv Information consideration the voxel resolution and space-structure. In
Leaky ReLU specific, I targeted to capture the representation of walls
and room borders in point clouds as an empty space in be-
FC tween two surfaces. To reflect that in the voxelized space
3DConv
we chose a voxel resolution of 5x5x5cm, since the standard
Leaky ReLU FC
(3DConv - Leaky ReLU) x6

minimum wall width is 7-10cm. As a result, a sliding cube

of 10x10x10 voxels corresponds to 50x50x50cm in space,
Fully Convolutional

3DConv FC which can encompass both the minimum wall width and a
gap between rooms larger than the standard wall size (ei-
Leaky ReLU ther due to noise or occlusion of the wall surfaces by e.g. a
bookcase in highly cluttered scenes).
3DConv Unpool3D Adding Context: In this approach a second input to the
Leaky ReLU 3DConv
voxelized sliding cube is fed to the network, which is the
global location of the cube with respect to the whole point
3DConv Leaky ReLU cloud. This is represented by its x, y, z coordinates from a
defined starting point (one of the point cloud’s corners) and
Leaky ReLU Unpool3D forms a vector of size 3.
Enclosed Spaces: As mentioned above the input here
3DConv
3DConv is an enclosed space. One can segment the point cloud
Classification

into such spaces with a variety of different approaches ([5],

Multiclass

[23]). Once these spaces are identified, they are projected to

Softmax
a canonical reference coordinate system. In this reference
[1,0,0,0,...,0] system all spaces are systematically aligned with respect
ceiling to their entrance location and subsequently normalized in
[0,0,0,0,...,1]
a unit cube. Before they are fed to the network, they are
null
[0,0,1,0,...,0] voxelized with binary occupancy values. Due to memory
wall restrictions, the resolution of the voxelization was selected
as 0.2x0.2x0.2, thus forming an input of size 50x50x50
[0,1,0,0,...,0]
[0,0,0,1,...,0] floor voxels.
other
3.2. Fully Convolution 3D Neural Network
Figure 1. 3D Convolutional Neural Network Architecture. Left:
3D Sliding Window or Enclosed Space.Right: Adding Contextual The input is then fed into a 3D Fully Convolutional Neu-
Information. ral Network. The choice of not including any Fully Con-
nected layer is based on the task in hand; since we are
performing a voxel to voxel operation, retaining the spa-
other elements appear in the data as a whole). These tial information throughout the network is considered es-
spaces are voxelized, consistently aligned and normal- sential. The network comprises of the following repetitive
ized in a unit cube before they are fed into a 3D Fully unit: a 3D Convolutional Layer (3D Conv) followed by a
Convolutional Neural network. The task is again that Leaky Rectified Linear Unit (ReLU) [19] (apart from the
of per voxel classification (see Figure 1-Right). last layer). The choice of Leaky ReLU over e.g. ReLU is
3 The size of the stride as well as other network details decisions have

been largely driven by memory limitations. An overlapping stride would

In the remainder of this section I will explain the main net- allow to infer the class label of each voxel not only based on its neighbors
work architecture and offer details about each of the above- in one cube location, but also by taking into account a larger area around
mentioned approaches. it.

3
Table 1. Details of 3D Fully Convolutional Neural Network 4. Experiments
Approach 3D Sliding Cube Enclosed Space
Input 4.1. Dataset
Size: N x10x10x10 N x50x50x50
Number of Channels: 1 1 For the evaluation I used the Stanford Large-Scale 3D
Stride: 10x10x10
-
Indoor Dataset [5] which comprises of six large indoor
3D Conv1 parts in three buildings of mainly educational and office
Number of Filters: 32 use (see Figure 2). The entire point clouds are automati-
Filter Size: 5x5x5 cally generated without any manual intervention as the out-
Stride: 1x1x1
Padding: 2x2x2
put of the Matterport camera ([1]). Each area covers ap-
Output Size: N x32x10x10x10 N x32x50x50x50 proximately 965, 1100, 450, 870, 1700 and 935 square me-
3D Conv2 and 3D Conv3 ters (total of 6020 square meters). Conference rooms, per-
Number of Filters: 64 sonal offices, auditoriums, restrooms, open spaces, lobbies,
Filter Size: 5x5x5
stairways and hallways are commonly found. The areas
Stride: 1x1x1
Padding: 2x2x2 show diverse properties in architectural style and appear-
Output Size: N x64x10x10x10 N x64x50x50x50 ance. The dataset has been annotated for 12 semantic el-
3D Conv4 and 3D Conv5 ements which pertain in the categories of structural build-
Number of Filters: 128 ing elements (ceiling, floor, wall, beam, column, window
Filter Size: 5x5x5
Stride: 1x1x1 and door) and commonly found furniture (table, chair, sofa,
Padding: 2x2x2 bookcase and board). A clutter class exists as well for all
Output Size: N x128x10x10x10 N x128x50x50x50 other elements. The dataset was slpit into training, vali-
3D Conv6 dation and testing as follows: 4 areas for training, one for
Number of Filters: 5 10
Filter Size: 5x5x5
validation and one for testing. In this way I ensure that the
Stride: 1x1x1 network sees areas from different buildings during training
Padding: 2x2x2 and testing. The same data split was used in all approaches.
Output Size: N x5x10x10x10 N x10x50x50x50
Output
Size: N x5x10x10x10 N x10x50x50x50 4.1.1 Preprocessing

3D Sliding Cube: The raw colored point clouds were vox-

made to avoid saturated neurons. Mathematically, we have
y = xi if xi ≥0, else y =xi /ai , where ai is a fixed param-
eter in the range (1, ∞). I followed the original paper’s
configuration and set ai to 100. For this experiment I used
6 layers 4 , which details are tabulated in Table 1 1.
Adding Context: In this approach we use a double input
to the network and follow a similar architecture to the one
proposed in [6]. The voxelized cube passes first through
2 3D Convolutional layers to encode spatial information
and then gets concatenated with the global position. The
concatenated vector passes first through a Fully Connected
layer and subsequently through layers of deconvolution to
acquire again its spatial configuration. For more details see
Figure 1.
3.3. Multi-class Voxel Classification
At the end of the final 3D Convolutional Layer the net-
work performs multi-class Softmax classification and will
P the scores of each class label per voxel: fj (z) =
predict
ezj / k ezk , where z is each voxel, j is the class evaluated
and k represents all classes.
elized in a grid of 5x5x5cm (for more details on the choice
of grid resolution see Section 3.1). After voxelization I as-
signed binary occupancy to all voxels as 0 if the voxel is
empty, 1 if occupied. The ground truth labels were gener-
ated by finding the mode of all point labels per voxel. Due
to memory restrictions, the number of classes was limited
to 5: walls, floor, ceiling, other and empty space.
Enclosed Spaces: Similarly, the aligned point clouds
that correspond to each enclosed space were normalized in
a 0.2x0.2x0.2 grid. The generated voxels was the populated
with binary occupancy values. The ground truth labels were
generated as in the previous case. Due to memory restric-
tions, the number of classes was limited to 10: walls, floor,
ceiling, door, beam, column, chair, table, other, and empty
space.
4.2. Implementation
I performed three different experiments, one for each in-
put approach. The implementation details are:
• I implemented the framework in Python3 using the
deep learning Python library Theano [2].

4 The number of layers and filters in the network was a direct result of • All data preprocessing steps were implemented in
the memory limitations. Python3 as well.

4
Figure 2. Stanford Large-Scale 3D Indoor Dataset [5]: I split the dataset into training (4 areas), validation (1 area) and testing sets (1
area). The raw point clouds are shown in the first row and the voxelized ground truth ones in the second row.
• 3D Sliding Cube: After generating the input, I no-
ticed that the amount of sliding cubes that contained
only empty voxels was greatly larger than the sliding
cubes that contained at least one voxel of the rest of the
classes. To counterpoise the skewness of the distribu-
tion I removed part of the empty sliding cubes.
• I shuffled the data before training since without it
the learning process was getting compromised. The
network was receiving sequentially inputs of similar
classes in the first case due to the sliding nature of the
input and the semantic consistency of the configuration
of spaces and in the second due to spaces with similar
functions.
• I used the Adam [14] adaptive learning rate method,
with parameters: 0.9, 0.99, and 1e − 08.
• The size of the batch per iteration was limited to 500
sliding cubes for the first and second approaches (slid-
ing cube) and 4 for the third (enclosed space) due to
memory restrictions.
• I used as metric the mean accuracy of prediction per
voxel.
4.3. Results
3D Sliding Cube: The initial idea towards this problem
was to use a 3D sliding window approach. The main moti-
vation behind it was the fact that the input size to the net-
work and the size of the voxelization grid could remain con-
stant no matter the size of the point cloud (buildings have
different sizes). Previous experience with this framework in
traditional pipelines has been shown successful. However,
5
although substantial effort was put to tune the hyper param-
eters, the network did not learn. I identify four main factors
as the principal reasons: (a) memory limitations did not al-
low to explore a number of hyper parameters such as using
all available classes in the dataset, or different sliding cube
sizes. In both cases the resulting matrices were too large and
the GPU would fail to handle them; (b) limiting the number
of classes forced to place a great number of elements under
the other class, which as a result created a class with low
discriminative power due to the resulting amorphous shape
and geometry, but highly represented in the dataset due to
the number of voxels falling in this category; (c) using a
generic constant size of the cube did not permit to capture
the geometry of other elements; and (d) there was a lack
of context regarding the content of the voxelized input with
respect to the rest of the point cloud. An example of the
training loss can be seen in Figure 3.
Adding Context: Following the previous failed attempt
to learn space-semantics, my next step was to add global
context as a second input to the network. Following the ar-
chitecture described above, the network continued not to be
able to learn. Once again memory limitations restricted the
number of layers, number of filters, and other network pa-
rameters. The training loss of this network is marked with
green line in Figure 4. Although the results are not as ex-
pected (see Figure 5), it did perform better than the previ-
ous network, which demonstrates that the global informa-
tion was helpful, however not powerful enough to solve the
ill-posed problem the sliding window approach created. I
also experimented with the same architecture as that pro-
posed in [6], however the results did not differ.
Enclosed Spaces: For the enclosed spaces approach I
experimented with 5 and 10 semantic classes. In this case

Figure 3. Training Loss of 3D Sliding Cube Approach.
Figure 4. Training Loss.
Table 2. Mean testing accuracy of different apporaches
Approach Sliding Cube+Context Enclosed Space
Mean Acccuracy 0.05 0.60
the network was able to learn (see Figure 4), lines blue and
red). Due to the smaller amount of training data than in
the other experiments, I reduced the number of iterations to
avoid over-fitting. In both cases the network performance
shows similarities, although the behavior when using 10
classes appears slightly more stable (Figure 5).
The mean accuracy of the of the networks is also pre-
sented in Table 2.
5. Conclusion
During this project I faced a lot of challenges related to
the size of the data and the memory limitations. These fac-
tors hindered the process in a great degree. From the exper-
6
Figure 5. Testing Accuracy.
iments it became obvious that information about the context
enabled the network to learn, from adding the global loca-
tion of the sliding cube to providing whole space semantics.
The final results of the enclosed space approach did boost
the accuracy, however there are aspects of it that require im-
provement. First of all it assumes that one is able to identify
rooms and consistently align them in a canonical reference
coordinate system. Second, by normalizing spaces of dif-
ferent sizes into a unit cube, the objects in the scene un-
dergo distortion. Although this step allows to convert stuff
into things since now their dimensions are more consistent
among different spaces, it compromises objects with consis-
tent dimensions to arbitrary ones, which makes the learning
process more difficult. Third, the grid resolution of the vox-
elization that was used was quite coarse, especially in the
case of larger spaces.
A more sophisticated method needs to be used in or-
der to achieve an end-to-end 3D Fully Convolutional ap-
proach for the problem of understanding space semantics,
by e.g. combining segmentation, object proposal [25] and
object detection approaches. This would allow to address
both amorphous and not objects. Another interesting aspect
would be to incorporate RGB information and observe how
it affects the performance. However, this would essentially
mean adding 3 more channels to each input and thus going
back to the issue of memory limitations.
References
[1] Matterport 3d models of interior spaces. http://
matterport.com/. Accessed: 2016-13-03.
[2] Theano theano 0.7 documentation. http:
//deeplearning.net/software/theano/.
Accessed: 2016-13-03.
[3] U. E. P. Agency and the United States Consumer Product
Safety Commission.
 [4] L. A. Alexandre. 3d object recognition using convolutional
neural networks with transfer learning between input chan-
nels. In Intelligent Autonomous Systems 13, pages 889–898.
Springer, 2016.
[5] I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis,
M. Fischer, and S. Savarese. 3d semantic parsing of large-
scale indoor spaces. In Proceedings of the IEEE Interna-
tional Conference on Computer Vision and Pattern Recogni-
tion, 2016.
[6] A. Dosovitskiy, J. Tobias Springenberg, and T. Brox. Learn-
ing to generate chairs with convolutional neural networks.
In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, pages 1538–1546, 2015.
[7] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction
from a single image using a multi-scale deep network. In
Advances in neural information processing systems, pages
2366–2374, 2014.
[8] S. Gupta, R. Girshick, P. Arbeláez, and J. Malik. Learning
rich features from rgb-d images for object detection and seg-
mentation. In Computer Vision–ECCV 2014, pages 345–360.
Springer, 2014.
[9] A. Hermans, G. Floros, and B. Leibe. Dense 3d semantic
mapping of indoor scenes from rgb-d images. In Robotics
and Automation (ICRA), 2014 IEEE International Confer-
ence on, pages 2631–2638. IEEE, 2014.
[10] N. Höft, H. Schulz, and S. Behnke. Fast semantic segmen-
tation of rgb-d scenes with gpu-accelerated deep neural net-
works. In KI 2014: Advances in Artificial Intelligence, pages
80–85. Springer, 2014.
[11] S. Ji, W. Xu, M. Yang, and K. Yu. 3d convolutional neu-
ral networks for human action recognition. Pattern Analysis
and Machine Intelligence, IEEE Transactions on, 35(1):221–
231, 2013.
[12] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar,
and L. Fei-Fei. Large-scale video classification with convo-
lutional neural networks. In Proceedings of the IEEE con-
ference on Computer Vision and Pattern Recognition, pages
1725–1732, 2014.
[13] A. Kendall and R. Cipolla. Modelling uncertainty in deep
learning for camera relocalization. Proceedings of the In-
ternational Conference on Robotics and Automation (ICRA),
2016.
[14] D. P. Kingma and J. Ba. Adam: A method for stochastic
optimization. CoRR, abs/1412.6980, 2014.
[15] H. S. Koppula, A. Anand, T. Joachims, and A. Saxena. Se-
mantic labeling of 3d point clouds for indoor scenes. In
J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. C. N. Pereira,
and K. Q. Weinberger, editors, NIPS, pages 244–252, 2011.
[16] K. Lai, L. Bo, and D. Fox. Unsupervised feature learning for
3d scene labeling. In Robotics and Automation (ICRA), 2014
IEEE International Conference on, pages 3050–3057. IEEE,
2014.
[17] I. Lenz, H. Lee, and A. Saxena. Deep learning for detecting
robotic grasps. IJRR, 2015.
[18] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional
networks for semantic segmentation. CVPR (to appear),
Nov. 2015.
7
[19] A. L. Maas, A. Y. Hannun, and A. Y. Ng. Rectifier nonlin-
earities improve neural network acoustic models. In Proc.
ICML, volume 30, page 1, 2013.
[20] D. Maturana and S. Scherer. Voxnet: A 3d convolutional
neural network for real-time object recognition. In IEEE/RSJ
International Conference on Intelligent Robots and Systems,
September 2015.
[21] L. Nan, K. Xie, and A. Sharf. A search-classify approach for
cluttered indoor scene understanding. ACM Transactions on
Graphics (TOG), 31(6):137, 2012.
[22] P. K. Nathan Silberman, Derek Hoiem and R. Fergus. Indoor
segmentation and support inference from rgbd images. In
ECCV, 2012.
[23] S. Ochmann, R. Vock, R. Wessel, M. Tamke, and R. Klein.
Automatic generation of structural building descriptions
from 3d point cloud scans. In GRAPP 2014 - International
Conference on Computer Graphics Theory and Applications.
SCITEPRESS, Jan. 2014.
[24] J. Papon, A. Abramov, M. Schoeler, and F. Worgot-
ter. Voxel cloud connectivity segmentation-supervoxels for
point clouds. In Computer Vision and Pattern Recogni-
tion (CVPR), 2013 IEEE Conference on, pages 2027–2034.
IEEE, 2013.
[25] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards
real-time object detection with region proposal networks. In
Advances in Neural Information Processing Systems, pages
91–99, 2015.
[26] X. Ren, L. Bo, and D. Fox. Rgb-(d) scene labeling: Features
and algorithms. In Computer Vision and Pattern Recogni-
tion (CVPR), 2012 IEEE Conference on, pages 2759–2766.
IEEE, 2012.
[27] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual
Recognition Challenge. International Journal of Computer
Vision (IJCV), 115(3):211–252, 2015.
[28] T. Shao, W. Xu, K. Zhou, J. Wang, D. Li, and B. Guo. An
interactive approach to semantic modeling of indoor scenes
with an rgbd camera. ACM Transactions on Graphics (TOG),
31(6):136, 2012.
[29] N. Silberman and R. Fergus. Indoor scene segmentation us-
ing a structured light sensor. In Computer Vision Workshops
(ICCV Workshops), 2011 IEEE International Conference on,
pages 601–608. IEEE, 2011.
[30] R. Socher, B. Huval, B. Bath, C. D. Manning, and A. Y. Ng.
Convolutional-recursive deep learning for 3d object classifi-
cation. In Advances in Neural Information Processing Sys-
tems, pages 665–673, 2012.
[31] D. Z. Wang and I. Posner. Voting for voting in online point
cloud object detection. In Proceedings of Robotics: Science
and Systems, Rome, Italy, July 2015.
[32] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and
J. Xiao. 3d shapenets: A deep representation for volumetric
shapes. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pages 1912–1920, 2015.
 Adding Shot Chart Data to NBA Scenes

Neerav Dixit
Stanford University
450 Serra Mall, Stanford, CA 94305
ndixit@stanford.edu

Abstract

Shot charts have become a popular way to visualize the

shooting efficiency of NBA players and teams from differ-
ent parts of the court. Despite their utility, it is difficult to
present shot charts as an effective visual aid in a TV broad-
cast. Here, a technique is demonstrated to superimpose shot
charts onto NBA broadcast images, enhancing the TV view-
ing experience for NBA games.
Previous work has discussed the processing of basket-
ball broadcast images to identify the court outline and per-
form camera calibration [1]. In this work, court identifica-
tion and camera calibration methods are implemented and
shown to be robust in the presence of most typical court
occlusions. Following camera calibration, the desired shot
chart coloring is projected onto the broadcast image of the
playing court. The proposed method works well across a Figure 1: A shot chart obtained from NBA.com/stats
wide range of courts and performs accurate calibration and
projection. The shot chart coloring is superimposed onto
the court only, improving aesthetic presentation and visual whole.
depth perception by excluding players, referees, and fans
A shot chart (Figure 1) provides an effectively way
blocking the court in the image.
to do this visually. On the chart, the court is divided
into 14 regions. Each of these regions is assigned a
color – either red, yellow, or green – according to the
1. Introduction shooting percentage compared to the NBA average in
The tracking and analysis of data in the National that region. This provides an effective visualization
Basketball Association (NBA) has seen a marked in- of specific locations on the court from which certain
crease in recent years. Data and analytics has al- players excel or struggle shooting the ball. Shot charts
tered teams’ understanding of the sport, resulting in have become widespread enough that they are made
changes to coaching and decision making in the NBA. easily available to fans on NBA.com/stats.
As data changes the way the game is played, it be- While shot charts are useful visual tools, it is dif-
comes more important to be able to accurately and in- ficult to use them to convey meaningful information
tuitively present informative data to fans. during a TV broadcast of an NBA game. Viewers see
One of the more commonly cited statistics in bas- the court from a broadcast camera angle, but a shot
ketball is a player’s or team’s shooting percentage. chart is only displayed as a separate image shown
This gives some idea of offensive efficiency and is from an overhead view of the court, such as in Figure
often used to compare players. However, a much 1. Projecting a shot chart onto the court in a broad-
more insightful view can be gained when looking at cast image could add the context needed to help a
shooting percentages from different parts of the court, shot chart easily convey the desired information to the
rather than looking at a single number to represent the viewer. While adding color to the court may be exces-

1
 (a) Broadcast image of left side of court
Figure 2: Broadcast image (a) with image coordinates (u, v). Court geometry shown in (b), (c) along with court
coordinates (x, y) used in each case
sive during live gameplay, a projected shot chart could
help commentators provide salient insight about a
player’s or team’s shot selection quality during a re-
play.
This work presents an algorithm to project shot
chart coloring onto a broadcast image. Given a broad-
cast image, the pixels corresponding to the different
region in the shot chart must be identified, and pixels
corresponding to players, referees, or fans obstructing
the court should be excluded from this classification.
Then, the court can be colored like the appropriate re-
gion of the shot chart. The key (colored rectangle near
the hoop) is excluded from the shot chart projection
in this work. A broadcast image and shot chart are
assumed to be given; there is no attempt to identify
which player is shooting the ball. The broadcast im-
ages used are typical views used of half-court sets on
the left or right side of the court.
2. Problem Statement
In order to project the desired shot chart onto
the broadcast image, camera calibration must be per-
formed to obtain a mapping between the image coor-
dinates and the position on the court. As shown in
Figure 2a, each pixel in the image is assigned a (u, v)
coordinate pair. Each position on the court has a (x, y)
coordinate pair, with the origin located according to
Figure 2b for the left side of the court or Figure 2c for
the right side of the court.
As illustrated by Hu et al. [1], the camera matrix
to map from homogeneous court coordinates p to the
homogeneous image coordinates p0 can be expressed
as a 3x3 homography matrix H.
2
(b) Left side court geometry (c) Right side court geometry
   
u x
p0 =  v  p= y  (1)
1 1
p0 = Hp (2)
Camera calibration using at least 4 known points
on the court can be used to find the matrix H. Once
H is known, the (x, y) coordinate on the court for
each pixel in the court mask can be obtained using
p = H −1 p0 . Each (x, y) position maps to one of the re-
gions in the shot chart, allowing the appropriate color
to be assigned to each court mask pixel. Finally, an
image of the colored court mask is combined with the
original image to yield the image of the projected shot
chart.
3. Technical Content
The basic algorithm and corresponding sample im-
ages are shown in Figure 3. The steps shown in Figure
3 are detailed in the following sections.
3.1. Court Mask Identification
Previous work has shown that detecting the domi-
nant colors in an image of a basketball court can seg-
ment out the court from the rest of the image[1]. Since
the entire court, excluding the key, is of a relatively ho-
mogeneous color, identifying which pixels contribute
to the most common colors in the image identifies the
pixels on the court fairly selectively. In this work, a
Hough voting scheme was used to identify these pix-
els. The Hough space was 3-dimensional, correspond-
 Figure 3: Flow chart of algorithm including images generated at each stage
ing to the HSV colorspace of the pixels. Each dimen-
sion was divided into a n bins (n = 3 or n = 4) for a
total of n3 bins in the Hough space. Each pixel in the
bottom 75% of the image voted for the bin correspond-
ing to its HSV numbers. The top 25% of the image was
discarded because the court typically does not extend
to that part of the image. Bins corresponding to low
value pixels were discarded since the court is not dark
and black is a common background color that can re-
sult in coherent votes for non-court colors. The high-
est remaining m bins (m = 1 or m = 2) were retained,
and pixels contributing to these bins were identified
as the court mask. The values of n and m were var-
ied depending on which court was shown in the im-
age. Because of some differences in the court designs
of the 30 NBA teams, different values of n and m gave
the best results for the different courts.
3
Using this Hough voting scheme, some isolated
pixels would end up as false positives or false nega-
tives. Since the court is mostly continuous in the im-
age, a median filter was applied to the court mask out-
put from the Hough voting scheme in order to elimi-
nate isolated pixels. The size of the median filter was
varied for images of different courts to give the best
results. After thresholding the output of the median
filter, the final court mask, shown in Figure 3b, was
obtained. This court mask does not include players or
referees, and it successfully separates the court from
most of the occlusions on the court.
3.2. Key Point Identification
Following identification of the court mask, this
mask was used to identify key points on the court to
be used for camera calibration. Without prior knowl-
 fitting the selected points using the Hough transform,
so long as enough of the points fall along the desired
line of interest. These detected lines are superimposed
on the image in Figure 3d. Once the court lines in the
image have been found, their intersection can be used
to locate the key points in the image. These key points
are shown in green in Figure 3d.

3.3. Camera Calibration

Once the key points have been identified, the 3x3
camera matrix H can be identified using camera cali-
bration. If we express H as
 
h1
Figure 4: Key points used for camera calibration for H =  h2  (3)
images of the left (red) and right (blue) sides of the h3
court
then the image coordinates (ui , vi ) for key point i can
be expressed in terms of the homogeneous court coor-
edge of the scene, it cannot be known whether any dinates pi defined in (1) as follows:
particular point on the court is occluded. To make
the identification of key points for camera calibration h1 pi h2 pi
ui = , vi = (4)
more robust to variations between broadcast images, h3 pi h3 pi
key points are selected at the intersection of lines on
the court, as shown in Figure 4. To find these points, From (4), the following steps can be performed to
the lines are identified in the image, and the point obtain a homogeneous system to be solved for the
where they intersect is taken as the key point. As long camera matrix H using the 5 correspondence points.
as enough of the relevant court lines are unobstructed
in the image, key points can be accurately located. The h1 pi − ui h3 pi = 0, h2 pi − vi h3 pi = 0 (5)
 T 
3x3 camera calibration matrix H has 8 independent h1
unknowns, so at least 4 points are required for cam- h ,  hT2  (6)
era calibration. In this work 5 points shown in Figure hT3
4 were used for camera calibration.  T
p1 0T −u1 pT1

The court lines that intersect at the key point lo-  0T pT1 −v1 pT1 
cations in Figure 4 are all along the perimeter of the
 
P ,  ... .. .. (7)
 
court mask shown in Figure 3b. Therefore, potential  . . 

points that lie on these lines can be identified by find-  pT5 0T −u5 pT5 
ing the edges of the court mask. First, the side of the 0T pT5 −v5 pT5
court shown in the image is determined by counting Ph = 0 (8)
whether there are more court mask pixels near the left
or right edge of the image. Then, the extent of the The matrix P defined in (7) is known from the court
court in the image is determined by finding the mini- coordinates pi of the key points in Figure 4, as well
mum and maximum rows and columns in the image as the corresponding image coordinates (ui , vi ) found
that contained a significant number of court mask pix- using the key point identification algorithm. The set
els. Using this information, the appropriate places in of court coordinates pi are referenced using the coor-
the image to search for the 5 lines of interest are esti- dinate system in Figure 2b or 2c for images of the left
mated. or right side of the court, respectively. These coordi-
Figure 3c shows the points identified as possibly nate systems allow the different shot chart regions to
belonging to the lines of interest on the court. Some be treated identically for images of the left and right
of these points clearly do not lie on the desired court of the court following camera calibration. The vector
lines because the edge of the court mask is misshapen h defined in (6) is unknown and can be solved for from
due to the presence of players on the court. However, the homogeneous system in (8) using SVD. Once h is
the proper lines on the court can still be determined by known, it can be rearranged into the camera matrix H.

4
3.4. Shot Chart Projection
Following camera calibration, the court coordinates 20
(x, y) of each pixel in the court mask can be found by
first solving for the homogeneous court coordinates p

Normalized Pixel Error

and then extracting the x and y coordinates. 15

 
u
p = H −1  v  (9) 10
1
p1 p2
x= , y= (10)
p3 p3 5

Using the (x, y) coordinates of each court mask pixel,

the corresponding shot chart region is be identified
by converting the (x, y) court coordinates to polar co-
ordinates centered under the hoop. Figure 3e shows
the coloring of court mask pixels according to the shot
chart in Figure 1.
The final image combination is performed by com-
bining the pixels that are not in the court mask with
a weighted average of the original image pixels in the
court mask and the colored court mask. One of these
final combined images is shown in Figure 3f. The shot
chart is projected onto the court (excluding the key),
while players, referees, fans, and other areas that are
not part of the court maintain their original coloring.
4. Experimental Setup and Results
The entire shot chart projection algorithm was
tested using several combinations of NBA broadcast
images and shot charts. Some of the resulting shot
chart projections are shown in Figure 6. The algo-
rithm appears to perform fairly accurately, although
slight differences between the desired and projected
shot chart boundaries can be observed in some in-
stances. In each case, the court is successfully identi-
fied and largely separated from occlusions. However,
in some images there are variations in the lighting on
the court that result in small patches of the court not
being recognized by the dominant color detection and
therefore being excluded from the court mask.
The error in key point location in the image was
used as a quantitative metric to evaluate the algo-
rithm. Inaccuracies in the inferred key point loca-
tion in the image result in inaccuracies in the camera
matrix H obtained from the camera calibration. This
leads to error in the shot chart regions on the court
when the shot chart is projected onto the image. Ac-
curate pixel coordinates for each key point were se-
lected manually and tested to ensure that they gave
good shot chart projections when used for camera cal-
ibration. The key point image coordinates obtained
0
0 1 2 3 4 5 6
Key Point
Figure 5: Normalized distance between inferred and
actual key point locations for 9 images, with each im-
age represented by a different color and shape. The
key points are numbered according to the labels in
Figure 4. Key point 5 was not visible in all images.
using the algorithm are compared to the manually se-
lected key point coordinates, and the distance in pixels
between the accurate and inferred key points is nor-
malized to a 640x360 image size and used as an error
metric. Nine separate broadcast images of either the
left or right side of the court were tested, and the error
values are shown in Figure 5.
The greatest inaccuracies appear to occur in cases
in which there are significant player occlusions along
the court lines whose intersections are used to locate
the key points for camera calibration. In these cases,
many of the points used for Hough line detection are
located along the player outlines rather than the court
lines, resulting in some error in the court line returned
by the Hough transform method. If the returned line
is inaccurate, the key points taken at the intersection
of the line with other court lines will also have errors.
The results shown in Figure 6 demonstrate that the
method is mostly robust to player occlusions of court
lines, but the algorithm does not perform well in ex-
treme cases when one of the court lines used in the
calibration is almost completely blocked.
5. Conclusions
A method of projecting shot charts onto NBA
broadcast images has been demonstrated. The major
steps of the algorithm – illustrated in Figure 3 – in-
clude dominant color detection to identify which pix-
els belong to the court, identification of lines on the

5
Figure 6: Four results of the algorithm projecting shot charts onto broadcast NBA images. Image (a) corresponds
to blue X in Figure 5; (b) to green X; (c) to red X, (d) to blue O

court to find key points in the image, camera cali-
bration and assignment of pixels on the court to shot
chart regions, and combination of the original im-
age with the image of the projected shot chart. The
method shows reasonable accuracy in separating the
court from other features in the image as well as hav-
ing boundaries between shot chart regions at the ap-
propriate locations in the image.
The method was applied to a variety of different
NBA courts, and the dominant color detection method
that was implemented through a Hough transform
appeared to work very well despite some differing
color schemes for these different courts. Some diffi-
culties were encountered in images of certain playoff
games, in which fans were provided with shirts of the
same color. Rather than the region of fans near the top
and bottom of the image having an arbitrary assort-
ment of colors, in these cases there were many pixels
in the image that would vote for the same incorrect
bin in the HSV Hough space that was used. This issue
was overcome by ignoring the top 25% of the image,
which is nearly always occupied by the crowd, and by
enforcing that the Hough bins corresponding to dark
pixels with low values were ignored when identifying
the dominant colors.
Only images using a broadcast angle and focusing

6
primarily on the left or right side of the court were
considered. However, the camera angle in the broad-
cast is not stable in these situations, so there was some
variability in the images used to test the algorithm.
This variability resulted in some difficulty in the step
for detection of potential points on the court lines of
interest, shown as the transition from Figure 3b to 3c.
Knowledge of the court based on the shape and ex-
tent of the court mask had to be incorporated in order
to accurately identify these candidate points across a
range of camera angles.
Considering images of both the left and right sides
of the court also resulted in a need to differentiate be-
tween these images when identifying points along the
lines of interest. Using different sets of court coordi-
nates, shown in Figures 2b and 2c, for camera calibra-
tion of images showing the left and right sides of the
court allowed these different cases to be treated iden-
tically for the later steps in the algorithm following
camera calibration.
This method excludes the key, the colored rectangle
near the hoop, when projecting the shot chart onto the
court. In order for a good projection to be made onto
the key, its color would have to be changed so that
the projected shot chart would be visible in the final
image. This was not done for this work due to dif-
ficulties in changing the color of the entire key while
ignoring occluding players and maintaining the court
markings that exist in the key. However, implement-
ing this into the algorithm would be a good next step
to extend this work further.
This algorithm could definitely improve the viewer
experience of an NBA broadcast if implemented ap-
propriately. The addition of shot chart data in proper
context to the broadcast would help commentators
make salient points with the help of a great visual
aid. Future work to further increase the accuracy of
the projection and identification of court pixels, along
with the inclusion of the key in the shot chart pro-
jection, would result in a useful tool to enhance NBA
broadcasts.
The Matlab code and images used for
this work can be accessed at the follow-
ing link: https://stanford.box.com/s/
akh4sdyput0ez37yy0he3la43y2ono3s
The shot chart projection is done by running pro-
gram main.m.

6. References
[1] Hu, Min-Chun, Ming-Hsiu Chang, Ja-Ling Wu,
and Lin Chi (2011), ”Robust camera calibration and
player tracking in broadcast basketball video.” IEEE
Transactions on Multimedia, 13(2): 266- 279.

7
 Optical Recognition of Hand-Drawn Chemical Structures
Bradley Emi
Stanford University
Dept. of Computer Science

and inaccurate in many cases. Wide availability of this

Abstract
Optical chemical structure recognition is the task of
converting a graphical image of a chemical molecule into
its standard structural representation. Specifically, the
chemical structure recognition algorithm should correctly
identify the graph structure with correct atomic/group
labels for each node, and the correct type of bond label for
each vertex. We introduce a novel method to improve upon
state-of-the-art methods with an eye towards solving the
problem in the face of the additional difficulties when
molecules are hand-drawn. We employ basic text
recognition and corner detection methods to first label the
atoms and groups that form the nodes of the chemical
structure graph, and conclude that our approach to corner
detection outperforms the line vectorization algorithms
typically used in other systems. A Hough transform is used
to recognize the presence of bonds between the nodes. The
major difference in our approach is to use a new
technique that classifies bonds according to various
feature descriptors of sliding-window cross-sections of
bonds using a supervised machine learning approach. In
addition to the baseline method of using the Hough
transform to also classify bonds, we use local maxima
detectors on single-pixel slices of bond cross-sections and
histogram of oriented gradients (HOG) features of wider
bond cross-sections coupled with support vector machine
(SVM), logistic regression, decision tree, and neural
network classifiers. We compare the results of these
feature descriptors, analyzing our pipeline on a hand-
drawn dataset of 360 simple molecules and conclude that
these new bond recognition technique leads to major
improvements in recognition performance over the
baseline.
kind of data from scientific patents, journal articles,
textbooks and other printed sources would lead to major
progress in not only chemistry, but also pharmaceuticals
[1], chemical biology [2], medicine [3], and several other
fields of scientific research. A tool for chemical structure
recognition would also open up new possibilities for
modern artificial intelligence and data mining applied to
already existing datasets that currently only exist in image
format. Furthermore, virtually no work has been done on
the generalization of optical structure recognition to hand-
drawn chemical molecules. Current research approaches
focus on optimizing computer-generated algorithms to
improve accuracy on extremely large molecules typically
presented in scientific patents. (Fig. 1) Little work has
been done on smaller, more common hand-drawn
molecules. (Fig. 2) While computer generated molecules
are difficult enough to recognize, smaller hand-drawn
molecules present different and unique challenges from a
computer vision perspective.
Fig. 1: A molecule from U.S. Patent Class #435; most
current optical structure recognition research focuses on
similar molecules.

1 Introduction

The standard presentation of organic chemical data in a

wide variety of fields, such as biology, chemistry, and
medicine, remains the structural diagram, which contains
all the chemical information of a given molecule, but is
unsuitable for computational analysis. The problem of Fig. 2: A molecule from our dataset. Its overall structure is
optical structure recognition, the conversion of these much simpler, but its bonds and labels require different
images of structures into the usable, machine-readable treatment.
labeled graph data formats, remains highly inconvenient

1

There are several additional advantages of being able to
recognize handwritten molecules in addition to computer-
generated molecules. For example, the computer
generation of molecular structures is currently quite
tedious, and an application to perform real-time
recognition of small components of hand-drawn structures
does not yet exist.
2 Review of Previous Work
2.1 Summary of Previous Work
Previous work on the optical structure recognition
problem has to date focused exclusively on computer-
generated structures. Early research began in the 1990s,
with IBM receiving a patent for recognition of chemical
graphics among other printed material on a page as well as
basic line tracing techniques (Fig. 3) to recognize
structures. [4] A similar approach was developed by
University of Leeds researchers and called CLiDE in the
same year. [5]
Fig. 3: IBM’s line-tracing algorithm.
More modern approaches, such as ChemReader from the
University of Michigan [6], and the National Cancer
Institute’s open-source OSRA [7] have both employed
more sophisticated text recognition (OCR) and line
detection algorithms; ChemReader uses a generalized
Hough Transform and OSRA uses the Potrace library.
State-of-the-art approaches, such as the approach of
MLOCSR, developed by Italian researchers Frasconi,
Gabbrielli, Lippi, and Marinai, generally recognize
molecules in two stages. The first stage is a low-level
processing module, which detects edges, corners, and text;
and a high-level reasoning engine, which uses Markov
logic networks, utilizing prior chemical and graphical
knowledge to correct errors in the low-level module. [8]
Modern approaches, such as a more recent iteration of
CLiDE, also use a specialized artificial neural network to
classify text labels.
2
2.2 Improvements to Existing Approaches
The focus of this paper is the novel approach to the correct
identification of hand-drawn bonds in the low-level
module, the correct identification of atoms and edges
without the use of high-level correction using chemical
and graphical knowledge. Previous attempts at optical
structure recognition, even state-of-the-art approaches, are
heavily dependent on the correct identification of fine
lines (the individual thin lines constituting double, triple,
and dashed bones), which fails in the case of imperfect
hand-drawn bonds. Frasconi et. al.’s algorithm, MLOCSR,
uses the Douglas-Peucker algorithm [9] to approximate
the contour of the molecule with a polygon which fits the
least-vertex polygon to the contour within a certain
precision. We hypothesize that line detection-only
vectorization algorithms such as the Douglas-Peucker
algorithm may fail in cases where bonds are not straight
(Fig. 4), assigning too many vertices to the molecule.
Furthermore, classification algorithms can fail when
dashes follow an irregular pattern and/or touch (Fig. 5).
Fig. 4: Bonds are not Fig. 5: Dashes
straight, a difficulty for touch and are
line-only vectorization irregular
Several algorithms use the Hough transform to detect lines
and line segments. However, Fig. 6 displays the difficulty
of using the Hough transform on a hand-drawn image.
Even when the threshold for the required number of votes
for the Hough transform to detect a line is optimized, both
false positives and false negatives still occur for reasons
specific to hand-drawn images.
The solution to these difficulties is by using the Hough
transform to only recognize bonds. Bonds are then
classified according to the features of their horizontal
cross-sections. In this paper, we experiment with a number
of features and classifiers to optimize the accuracy of bond
type recognition. These experiments constitute a majority
of the current paper, with enhancements to text
recognition and further experimentation on the higher-
level module incorporating chemical knowledge
representations forthcoming in future work.

Fig 6: Difficulty of using the Hough transform on
a hand-drawn molecule. A false positive is
detected on the bottom left because of the bend in
the single bond. The double bond on the right,
meanwhile, is undetected as a false negative.
3 Methods
3.1 Summary
The abstract general structure of the pipeline we use in
order to recognize the chemical structure of our molecules
is as follows:
• Recognize text labels using scale-invariant
template matching
• Removal of text from image
• Bond and corner detection
• Bond detection
• Bond classification
• Association of corners to atoms and groups
Due to the lack of a large amount of hand-drawn chemical
data and the small number of labels, a simpler scale-
invariant template-matching approach was used to detect
text in images with reasonable accuracy. Other
approaches, such as Google Tesseract [9] and other
supervised learning classifiers with histogram of oriented
gradients (HOG) [10] features were attempted. Tesseract
was exceedingly difficult to configure due to its more
general use parameters, which use a number of language
models which make a large number of assumptions in
order to more accurately recognize more structured
sources of text. However, these assumptions, which do
not apply to our purposes, were difficult to remove
programmatically and debug. Future work may
incorporate a state-of-the-art OCR engine; however, this
3
approach was ultimately unsuccessful. HOG classifiers
were found to suffer from a large number of false positives
due to a lack of negative training examples. With more
data, this approach could also prove to be more successful,
but was inapplicable with our limited amount of training
data.
Bond and corner detection was used to identify various
points of interest of the coarse line features; intersections
of lines that represent a carbon, or lines which end in a
connection to a text box, representing a different atom or
group. A coarse Gaussian filter was applied to the image
before applying a Hough transform to find the lines and
points of interest.
Finally, these points of interest were assembled to locate
the atoms and groups of the molecule as well as the bonds,
leaving bond classification as the final task. Cross-sections
of the bonds were analyzed using various feature
descriptors and classified according to several machine-
learning classifiers trained on 45 hand-drawn molecules.
The success of our approach with a small training set size
is again reason to believe this approach will become more
accurate as more data is acquired. More details on the
bond classification algorithm is described in the methods
section.
3.2 Technical Solution
3.2.1 Data
Our data comprises 360 images of 9 different simple hand-
drawn molecules (40 images of each molecule) drawn on
standard white printer paper in fine point black Sharpie
marker. Each image was taken with an iPhone 6 camera at
3264 x 2448 resolution with three color channels (no alpha
channel) downsampled to 400 x 300 grayscale using
bilinear interpolation. All the images were taken in
identical lighting, and were drawn by three different
people, so our model would not overfit to a particular
person’s drawing style. Each image was preprocessed with
binarization using a 40% threshold. No other
preprocessing stages were applied.
45 of the images (5 per molecule) were set aside as a
training set.
3.2.2 Text Recognition
For text recognition, we tried a number of approaches,
including scale-invariant template matching using 5
images of each of 6 templates (“O,” “H”, “OR”, “RO”,
“N”, and “OH”), and a number of supervised learning
classifiers using 4 templates (“O,” “H,” “R,” and “N”). By
visual of the dataset, we estimated a minimum and
maximum scale for the images at 20x20 pixels and 60x60
pixels respectively and implemented a spatial pyramid
sliding window with the length of each square window
increasing from 20 to 60 pixels in steps of 5 pixels. This
was a very conservative estimate, and for
reimplementation on different image sizes, we recommend
scaling from 0.3% of the total area of the image to 3%.
For the supervised learning classifiers, to collect negative
training examples, we randomly selected 1200 of these
windows that were verified by hand to have no text from
the training set. We then collected 5 of each of the 4
templates from the training set. To augment the number of
positive examples, we additionally used 55 images for
each template from the open source Chars74K handwritten
dataset [11]. We cropped each image to eliminate
whitespace and extracted histogram of oriented gradients
features from each using 64 bins. We then compared the
performance of a logistic regression classifier, a linear
SVM classifier, and a neural network with one hidden
layer with 30 nodes. Results are presented in section 4.
For the scale-invariant template-matching, we applied a
Gaussian filter with size equal to half the width of the
measured strokes to all training templates and the image
for matching, and then used the spatial pyramid sliding
window described above to match the images. We then
chose the tolerance level, 0.77, for which the F1 score was
maximized. Non-maximal suppression is used to remove
overlapping bounding boxes. A sample of the output of
this stage is presented in Fig. 7. We used the results of this
algorithm for the next stages of the pipeline. More details
are presented in section 4.
Fig. 7: Sample results of template matching OCR.
3.2.3 Corner Detection Overview
We reimplement the Douglas-Peucker algorithm on our
dataset for comparison with MLOCSR, and we also
implement a corner detection algorithm based on a broad
Harris corner detector.
4
For clarity, we use the terminology of MLOCSR, defining
a C-point to be a corner corresponding to the intersection
of the main bonds of a carbon, a D-point to be the
endpoint of a line segment not connected to the main bond
to represent a double or triple bond, and a T-point to be the
end of a line segment drawn to a text box to indicate a
bond to a non-carbon.
3.2.4 Best-Fit Polygon Reimplementation
As in MLOCSR, we use the Douglas-Peucker algorithm to
detect the vicinity of C-points and T-points, and look for
D-points later once the main corners are located. This
algorithm for each contour iteratively tries to fit n-vertex
polygons to the contour, increasing n until no point on a
contour is further than a threshold distance away from the
polygon. The algorithm then returns the vertices of the
polygon.
We search for clusters of all points of polygons that fit the
opposite contours of the image after a Canny edge detector
is applied in order to accomplish this goal. We use the
threshold of √2 times the edge length as prescribed in
MLOSCR. Fig. 8 shows the results of this stage.
Fig. 8: Result of the Douglas-Peucker algorithm to fit a
polygon to the contours of the Canny edges of the
molecule image.
We then use a basic agglomerative clustering algorithm,
setting a maximum distance between clusters to 50 pixels..
If a polygon vertex is less than 50 pixels away from an
existing cluster center, we assign it to that cluster,
updating the center point of that cluster. Otherwise, we
initialize a new cluster. The results of this stage applied
correctly to a molecule image are shown below in Fig. 9,
with the blue points representing final cluster centers.
Testing results are shown in section 4.

Fig. 9: Results of agglomerative clustering stage.
3.2.5 Harris Corner Detector
The goal of the Harris detector [12] in this context is the
same; to identify the C- and T-points but not necessarily
the finer D-points that distinguish double and triple bonds.
The Harris corner detector looks for a high variation in the
gradient of an image in two directions.
We first apply a coarse Gaussian filter to the image with
the size of the estimated stroke width. We then run the
Harris corner detector, once again requiring corners to be a
threshold distance apart.
A sample result on the same molecule after filtering is
presented in Fig. 10.
Fig. 10: Results of Harris corner detector.
A comparison of the results is presented in detail in Sec. 4,
but we find that the Harris corner detector (89% accuracy)
outperforms the polygon reconstruction method (75%
accuracy).
5
3.2.6 Bond Detection
In contrast to methods based on line vectorization, which
comprises not only MLOCSR but also a majority of the
existing methods in the literature, we use the Hough
transform only to detect bonds, rather than to classify
them. Line vectorization methods make several errors and
in the case of hand-drawn molecules, are not precise
enough to detect the D-points that can be detected by the
polygon reconstruction method when molecules are
perfectly straight. As the state-of-the-art method in
MLOCSR only recognizes under 80% of the C- and T-
points, there is very little hope for such an algorithm to be
able to detect the finer D-points given the large amount of
variability in hand-drawn bonds.
Since a carbon can only have four bonds, for each of the
nodes detected in the previous stage, we look at the four
closest nodes to see if there is a bond between them. While
further molecules are not strictly forbidden from being
connected to a carbon, it is extremely uncommon, and this
case does not occur in any of the molecules in our dataset.
For more general molecules, more nodes can be examined
and spurious matches can be removed using a Markov
logic network similar to what is implemented in
MLOCSR, but we do not implement that here for
simplicity.
The other heuristic we use is that if three nodes are
collinear, there is not a bond between the two outer nodes.
This situation only occurs when there are two bonds at a
180-degree bond angle, so the outer nodes cannot have a
bond between them.
These heuristics leave us with a number of candidate
bonds, a list of possible node-node pairs that could contain
a bond between them. To refine this list, we create a
bounding box of the edge between the two nodes at a fixed
width (40 pixels) and split the bounding box into windows
of fixed size. On each window, we then apply the Hough
transform with a very low threshold to look for lines in the
window. We only accept lines that are within 1 degree of
the expected direction of the bond. We then require that all
of the windows in the bounding box contain a line
detected by the Hough transform. We assume that if a
node-node pair does not contain a bond, at least one of its
windows will not have a matching line in the orientation
of the node-node pair. This approach leads to several false
negatives, but these can be rectified by a Markov logic
network in later steps, because most of the false negatives
are simply a missing bond in a ring or another predictable
structure. This approach does lead to a relatively low false
positive rate, and the false positives are generally simply
spurious triangular closings, which are very rare in organic
molecules and would also be removed by a Markov logic window gets a “vote” for the overall type of bond. A
network. Results are presented in Sec. 4. The process is sample result is shown below in Fig. 12, and overall
visualized in Fig. 11. results are presented in Sec. 4.

Fig. 12: Final result of bond classification for a benzene

ring.

4 Results

4.1 Text Recognition Results

Fig. 11: Top left, top right: Hough line detections (blue)
for the node-node pairs with a bond for a given window
(red) in the bounding box. Bottom: A window between
two opposite nodes that will not have a Hough line
detection, so the algorithm will not assign it a bond, even
though there is contamination elsewhere in the bounding
box.
4.1.1 Scale-Invariant Template Matching
The results of text recognition are presented here. In order
to optimize the tolerance of the scale-invariant template
matching, we measured the precision and recall on the test
set. The results are presented in Fig. 13. We chose the
tolerance that maximizes the F1 score, 0.77.

3.2.7 Bond Classification

The final part of the pipeline is bond classification, before

higher-level modules use chemical knowledge to correct
errors (future work).

We use a sliding window moving along the cross-sections

of bonds extracted from our training set via screenshots,
using a window size of 10 pixels down the length of the
bond and 40 pixels across. Typically this will result in
around 3 to 10 windows per bond. Our training set
contains 62 single bonds, 33 double bonds, 10 wedge
bonds, 10 dashed bonds, and 5 triple bonds.

We then use HOG features on each of the sliding windows

and use these features to train a supervised learning
classifier. We experiment with a multiclass logistic
regression classifier, a linear support vector machine, and
a decision tree. (We avoided using a neural network since
it is prone to over-fitting in the case of a small training
set).
Then, for each bond in the test set, we extract the same
sliding window HOG features and use the classifier to
predict the type of bond that appears in each of the
windows. Then we employ a voting system where each
Fig. 13: OCR precision and recall on test set. The optimal
value was found to be 0.77.
We then apply the template matching to the test set. By
molecule and in total, the results are presented below in
Table 1. Accuracy is by molecule image, so a molecule
has to have all of its text completely recognized with no
false positives for it to count positively towards the
accuracy metric. Diagrams of the molecules associated
with each molecule ID can be found in the Appendix.

6
 Molecule ID Precision Recall Accuracy For future work we would like to expand the size of the
1 1.0 1.0 1.0 training set to improve the accuracy; but for now we use
2 1.0 1.0 1.0
3 0.54 1.0 0.50
template matching.
4 1.0 1.0 1.0
5 0.95 0.95 0.90 4.2 Corner Detection Results
6 1.0 1.0 1.0
7 0.96 0.79 0.40 As shown in Table 3, we conclude that the Harris corner
8 0.79 0.65 0.53
detector outperforms the baseline method of the MLOCSR
9 0.98 0.90 0.58
Total 0.91 0.92 0.77 polygon reconstruction method quite significantly, by
approximately 15% on the molecule level (node level
Table 1: Results of scale-invariant template matching on refers to the number of correctly detected nodes over the
test set. total number of nodes, molecule level refers to the number
of correctly detected molecules with no false positives
While an accuracy of 77% is far from ideal, it is divided by the total number of molecules. There are
surprisingly effective considering we only used 5 training several reasons that this result is the case. First, the
images to build the templates. With more examples, this polygon reconstruction method performs very poorly in
method could perform even better in future work. We use the case of dashed bonds; whereas the Gaussian smoothing
the images where text was accurately identified from this applied to the image before applying Harris corner
stage in the further stages of the pipeline. detection “blends” dashed bonds into an edge before
finding the corners. This kind of preprocessing is not
4.1.2 Supervised Classifiers feasible for the polygon reconstruction method since it
relies on the narrow opposite contours that form the edges
Parameters Train Cross- #Iter- Avg. of the thick lines. The performance on a dashed molecule
Set Validation ations Acc. is demonstrated in Fig. []. While neither method performs
Logistic Regularization 1330 10-fold 100 0.97 particularly well on dashed bonds (polygon method
Regression coeff. = 1.0, L2 examples
with HOG norm
performs at 15% on these dashed bonds, while Harris
features method performs at 45% on dashed bonds), which when
Linear Regularization 1330 10-fold 100 0.96 blended are very wide, making corner detection difficult
Support coeff. = 1.0, L2 examples especially when they a dashed bond is near other corners.
Vector norm
Machine Molecule Overall Overall
with HOG accuracy Precision Recall
features
One-Layer One hidden 1330 10-fold 100 0.99
Polygon 0.748 0.960 0.970
Neural layer with 30 examples
Method
Network nodes
with HOG Harris Method 0.896 0.987 0.989
features
Table 2: Results of supervised classifiers on the OCR
training set.

The performance of the supervised learning classifiers

using HOG features with 64 bins was evaluated using
cross-validation, training a classifier with 90% of the 1100 Table 3: Comparison of corner detection methods.
negative test images and 60 positive test images per
character, and testing on the remaining 10%. We conclude
our training set is not large enough to provide accurate
detections on the test set, recording less than 1% overall
accuracy. This is because although the performance of the
supervised learning classifiers on the cross-validation set
is relatively good, perfect matching on the test set
requires a correct match on each of the 1000+ sliding
windows used for detection, so even the neural network,
with 99% accuracy on the cross-validation set, is unable
to perform well in the natural setting, with nearly 0%
accuracy and several false positives.
Fig. 14

7
Fig. 14: Harris corner detection on a molecule with dashed Molecule ID Molecule Overall Overall
bonds (left) and polygon reconstruction method with Accuracy Precision Recall
initial corners in blue and clustered corners in red. (right). 1 1.0 1.0 1.0
A wide Gaussian filter helps “blend” dashed bonds 2 0.31 0.94 0.95
together. Since each dash of the dashed bond is a contour, 3 0.0 1.0 0.78
many spurious corners in blue are detected with the 4 0.26 1.0 0.79
polygon reconstruction method and make the 5 0.70 1.0 0.88
agglomerative clustering inaccurate. 6 0.82 1.0 0.94
7 0.22 0.86 0.97
As we hypothesized, the Harris method also outperforms
8 0.70 1.0 0.90
the polygon reconstruction method when bonds are not
9 0.10 0.90 0.83
perfectly straight. This was particularly evident in the
benzene rings, which the Harris corner detector (95% Total 0.55 0.96 0.91
accuracy on benzene rings) was able to substantially Table 4: Bond detection results.
outperform the polygon reconstruction method (50%
accuracy on benzene rings). This effect is shown in Fig. Typical errors included a missing bond, as shown in Fig.
15. 16, and false triangular closures with bonds at very wide
angles (bonds that are nearly, but not quite collinear), as
shown in Fig. 17.

Fig. 15: The polygon reconstruction method detects

several incorrect corners due to curved lines, as shown in
blue. The Harris method does not suffer from this
Fig. 16: A typical bond detection error, a missing bond,
drawback.
likely due to slight inaccuracy of corner detection. These
“missing bonds” can be detected in later stages as long as
4.3 Bond Detection Results
most of the structure is correct.
The bond detection is the worst performing stage of the
pipeline, but fortunately it is the most correctable by a
higher-level Markov model as described in MLOCSR.
Still, large improvements can still be made to the
algorithm by applying more heuristics about how the
various atoms bond. We did not take chemical knowledge
about the topological structure of molecules into account
when looking for bonds, but there exist further constraints
that can reduce our false positive rate, which would let us
adjust the tolerance thresholds on the Hough detector and
the angle acceptance to reduce the false negative rate. The
results are presented below in Table 4 and errors are
characterized further in Figs. 16 and 17.

Fig. 17: Another common bond detection error, triangular

closure of wide bonds, due to there being too much
contamination in the bounding box. These can also be
corrected later if most of the molecule is correct.

8
4.4 Bond Classification Results with more training data we will be able to obtain nearly
100% accuracy with this method in the future.
4.4.1 Comparison of Classifiers

We split our training bonds randomly according to a 90-10

split and run cross-validation 10 times.

Classifier Cross-Validation Cross-Validation

Accuracy
Logistic Regression 10-fold 0.85
Linear Support 10-fold 0.97
Vector Machine
Decision Tree 10-fold 0.88

Table 5: Cross-Validation Results on a 90-10 training set Fig. 18: Two examples of correctly recognized molecules
split of known bond labels. after completion of the full pipeline. These can be easily
converted to a standard chemical data format.
4.4.2 Performance on Test Set
5 Conclusion
Based on the results on the cross-validation set, we use the
SVM for classification on the full set of molecules. We Although our overall accuracy is low, we believe that the
find that there is no great disparity in confusing one type work presented in this paper will lay the foundation for
for another; despite the only 5 training examples of triple hand-drawn structure recognition in the future.
bonds, we find that double bonds are no more often
mistaken for single bonds as triple bonds, for example. Much of the low accuracy can simply be attributed to a
lack of training data. State-of-the-art OCR methods, for
Molecule ID Accuracy By Bond Accuracy By example, would boost the accuracy of text recognition
Molecule
1 0.98 0.90 from 77% to near perfect. We also believe that more
2 0.97 0.81 training data will ultimately allow us to use a
3 0.57 0.0 convolutional neural network for the bond classification
4 0.80 0.30 stage rather than an SVM, and more data will be able to
5 1.0 1.0 significantly improve the accuracy of bond classification
6 0.83 0.50
7 1.0 1.0
as well.
8 0.98 0.93
9 1.0 1.0 Additionally, as mentioned previously, the focus of this
Total 0.94 0.75 project was on the low-level recognition of atoms and
Table 6: Test results by molecule using an SVM classifier. bonds; or the nodes and vertices that make up the overall
graph. There are additional heuristics that can be applied
4.5 Overall Results in higher-level modules, such as bonding patterns like
valence rules that we did not take into account, which will
When the overall pipeline is run on the entire set of significantly improve the performance of bond detection.
molecules, 94 out of the original 360 molecules are
correctly recognized in their entirety. While this accuracy Accuracy may also be a misleading metric for certain
may seem low, it is still higher than the performance of the applications of hand-drawn structure recognition as well,
“out of the box” existing optical structure recognition in cases where more information is obtained. For example,
algorithms, the most well-known being OSRA, which in an electronic tablet drawing application, in a similar
when used on handwritten data have nearly 0% accuracy. way to how Chinese and Japanese characters are
We also find that even the approach of MLOCSR applied recognized by OCR software, information about how the
to the data, which relies on the Douglas-Peucker polygon user is drawing the structure is also available. This can
fitting algorithm, does not even detect C- and T- points as improve the localization of corners (using information
successfully as our algorithm on our hand-written dataset. about when the user picks up and puts down the pen) and
We also find that our supervised learning bond identify bonds with much greater accuracy (based on
classification algorithm performs extremely well given the speed of stroke, etc.). Additionally, if there is a limited
very small training data set, which was extracted from subset of molecules that the engine is required to
only 5 images of each molecule. We are optimistic that recognize, various molecule similarity algorithms can be
used to compare the molecule against the database of

9
possible molecules and return the one with the greatest
similarity. This is often the case for simple molecules and
could be very useful in chemistry education.
We conclude that handwritten structure recognition and
analysis is a difficult problem, one that cannot be treated
in the same way that computer-generated structure
recognition is treated. More flexibility must be applied in
accounting for the greater degree of variability in hand-
drawn images, and we have accounted for that in this work
with modern corner and line detection techniques. The key
insight of this project was analyzing small cross-sections
of bonds so the algorithm can gain a consensus from many
cross-sections instead of trying to analyze bonds as a
whole, as previous algorithms have done. Overall, there
are many parts of this pipeline that can be improved as
mentioned, but much progress has been made towards
being able to apply these methods to a public application.
6 References
[1] Gaulton, A.; Overington, J. P. Role of open chemical
data in aiding drug discovery and design. Future Med.
Chem. 2010, 2, 903−7.
[2] Kind, T.; Scholz, M.; Fiehn, O. How large is the
metabolome? A critical analysis of data exchange
practices in chemistry. PLoS One 2009, 4, e5440.
[3] G.R. Rosania, G. Crippen, P. Woolf, D. States, and K.
Shedden, R. A cheminformatic toolkit for mining
biomedical knowledge. Pharmaceutical Research, vol. 24,
(no. 10), pp. 1791-1802, Oct 2007.
[4] Casey, R. et. al. Optical Recognition of Chemical
Graphics. Document Analysis and Recognition:
Proceedings of the 2nd International Conference on
Document Analysis and Recognition. 1993.
[5] Ibison, P. et. al. Chemical Literature Data Extraction:
The CLiDE project. Journal of Chemical Informatics and
Computer Science, 33, pp. 338-344. 1993.
[6] Park, J. et. al. Image-to-Structure Task by
ChemReader. Text Retrieval Conference, 2011.
10
[7] Filippov, I. and Marc Nicklaus. Optical Structure
Recognition Software to Recover Chemical Information:
OSRA, An Open Source Solution. J. Chem. Inf. Model., 49
(3), pp. 740-743, 2009.
[8] Frasconi, P. et. al. Markov Logic Networks for Optical
Chemical Structure Recognition. Journal of Chemical
Information and Modeling. 54, pp. 2380-2390, 2014.
[9] Douglas, D.; Peucker, T. Algorithms for the reduction
of the number of points required for represent a digitized
line or its caricature. Can. Cartogr. 1973, 10, 112−122.
[10] Navneet Dalal, Bill Triggs. Histograms of Oriented
Gradients for Human Detection. Cordelia Schmid and
Stefano Soatto and Carlo Tomasi. International
Conference on Computer Vision & Pattern Recognition
(CVPR ’05), Jun 2005, San Diego, United States. IEEE
Computer Society, 1, pp.886–893, 2005.
[11] T.E. de Campos, B.R. Babu and M. Varma.
Character Recognition in natural images. In Proceedings
of the International Conference on Computer Vision
Theory and Applications, Lisbon, Portugal, February
2009.
[12] Harris, C. and M. Stephens, A Combined Corner and
Edge Detector. Plessey Research, 1988.
7 Code Access
The code is available open-source. The repository is
located at https://github.com/bradleyemi/chemtype2.
Instructions for downloading data and usage are located on
GitHub.
8 Acknowledgements
This project was implemented in Python 2.7 with
additional use of the Anaconda distribution, Scikit-Learn
(for machine learning classifiers), and OpenCV for CS
231A at Stanford University.
A Appendix: Molecule Table

2
 Pedestrian Detection and Tracking in Images and Videos
Azar Fazel
Stanford University
azarf@stanford.edu
Abstract
The increase in population density and accessibil-
ity to cars over the past decade has led to extensive
computer vision research in recognition and detection
to promote a safer environment. Primarily, much of
the research focuses on detecting pedestrians in or-
der to reduce the chance of collision, and to improve
traffic control. The need for increased surveillance at
the work place and at home also promotes research
in this area. We implemented a pedestrian detection
and tracking algorithm by using histogram of oriented
gradients (HOG) features and a linear support vec-
tor machine (SVM) and Random Forest classifier. Our
goal was to analyze and generate different bounding
boxes for people within static images, and finally ap-
ply this strategy to localize and track people within
videos. We benchmarked different HOG parameters to
find the best model, and furthered our experimentation
by comparing the effectiveness of SVM versus Random
Forests. Our implementation was able to achieve 80%
accuracy in static images, and was able to track pedes-
trians in videos if the detected pedestrians’ poses do
not vary significantly.
1. INTRODUCTION
Just in the United States, 5,000 of the 35,000 annual
traffic crash fatalities involve pedestrians [1]. Com-
puter vision research in the area of pedestrian detection
is becoming increasingly crucial as more intelligent
motor vehicles are introduced into the streets. How-
ever, pedestrian tracking and detection is inherently a
hard problem to solve due to high intra-class variabil-
ity and partial occlusions. Our goal was to benchmark
different feature parameters from HOG, and compare
1
Viet Vo
Stanford University
vtvo@stanford.edu
the success of SVM versus Random Forests for pedes-
trian detection. We implemented our algorithms on
Python and utilized several computer vision pack-
ages from OpenCV, machine learning packages from
sklearn, and imaging processing packages from scikit-
image. In order to train our model, we used a com-
bination of 5,400 positive 64 x 128 images from the
Inria’s Person dataset [2], PETA dataset [3], and the
MIT database [4]. Our negative images were also from
these databases and images from the Daimler Mono
dataset [5], containing a total of 2,100 images. For
each non-pedestrian image, 10 random windows of 64
x 128 pixels were extracted for training, giving a to-
tal of 21,000 negative images. This trained model was
then used to test the detection accuracy on images, and
track pedestrians in videos.
2. PREVIOUS WORK
Many techniques are being used today for pedes-
trian detection. One such technique that is similar
to HOG is Scale Invariant Feature Transform (SIFT).
This technique generates features by using Difference-
of-Gaussians (DoG) in an image’s scale-space pyra-
mid to find interesting local keypoints. Each keypoint
will have a orientation vector, and is invariant to scale
and rotation. Due to the high dimensionality of SIFT
features, principal component analysis is often used
in conjunction with SIFT [10]. Although SIFT can
be effective for detecting human features, Dalal and
Triggs explained in their paper that locally normalized
HOG descriptors were more effective [6]. They ex-
perimentally showed that the dense grid of uniformly
spaced cells and overlapping local contrast normaliza-
tion improved the detection performance as compared
to SIFT. In fact, HOG features have been shown to
have 1 to 2 orders of magnitude less false positives
then other approaches.
The Deformable Parts Model (DPM) is another
technique for object detection that performs well at
classifying highly variable object classes. In this tech-
nique, for each image, a HOG feature pyramid is
formed by varying the scale of the image, and defin-
ing a root and parts filter. The root filter is coarse and
is used to capture the general shape of the object, while
higher resolution part filters are used to capture small
parts of the object. Objects are then detected by com-
puting the overall score for each root location based on
the best possible placement of the parts [7].
The other technique in pedestrian detection is Con-
volutional Neural Networks (CNN). This technique
shows outstanding power in addressing the pedestrian
detection problem, especially in the context of au-
tonomous driving. In CNN, it learns which convo-
lution parameters can produce better features to eas-
ily predict an optimal output. Then it uses these
features by extracting them from the last fully con-
nected layers to train an SVM model for pedestrian
detection[11][12].
3. TECHNICAL APPROACH
Using our dataset of positive and negative images,
we extracted features using the histogram of oriented
gradients technique described by Dalal and Triggs.
This technique divides the image into dense equal
sized overlapping blocks. Each of these blocks are
further divided into cells which will be used to find
a 1-D histogram of gradient edge orientations over the
pixels of the cell. For this project, we experimented
with block sizes that were 2x2 and 4x4, and cells sizes
that were 8x8 and 16x16 in order to find the best com-
bination. For our histograms, we used 9 orientation
bins across all experiments. Histograms for each block
are combined and finally normalized to have better in-
variance to illumination and shadowing [6]. Figure
1 shows an example of extracted HOG features for a
pedestrian.
In order to train the model, the feature vectors for
the images were fed into a linear SVM classifier. This
model was then used to classify pedestrians from non-
pedestrians. We implemented a sliding window ap-
proach to exhaustively search static images for win-
dows with the scores greater than 0.2. The scores for
each window was calculated using the weight and bias
2
Figure 1: Example of HOG features, the right picture is the
original image and the left one is the extracted HOG
features.
found from our SVM model. Since our sliding window
was kept at a constant size of 64x128 pixels, we imple-
mented an image pyramid approach for our detection.
In this approach, for each image, we scaled down the
image by 15% of its original size for several iterations
until the size is below a threshold of 64 pixels for width
and 128 pixels for height. For each iteration, our de-
tector window searched the entire scaled image and
calculated scores using our SVM model. Once this al-
gorithm was finished, scaled bounding boxes was dis-
played on the original image and non-maximal sup-
pression applied to eliminate redundant boxes.
In order to reduce false positive rates, we mined
for hard negative examples using our negative train-
ing data. We extracted all false positive objects found
within negative images and included these examples
into our training data for retraining the classifier.
Due to the exhaustive search performed during
HOG feature extraction, the time complexity for
object detection is very high. This poses a problem for
pedestrian tracking in videos because detection rates
would be too slow. In order to remedy this problem,
objects that are moving will be extracted from each
frame using background subtraction [8]. Using this
method, we detected motion by segmenting moving
objects from the background and passing these smaller
images into our model instead of passing the whole
frame for detection. The nth frame can be represented
as In which is its intensity value. In−1 will correspond
to the previous frame. Doing a pixelwise subtraction,
we get the equation
 (
In (i, j) ∆(i, j) ≥ Tthreshold
Mn =
0 ∆(i, j) < Tthreshold
where i and j are pixel positions and Mn is the mo-
tion image. By finding the motion image, we can dra-
matically reduce the complexity of our computation
[9]. Using these motion images, we were be able to
run our model on video frames much faster than when
we did not have any motion detection.
Figure 2 provides a summary of the steps we have in
our detection algorithm.
4. EXPERIMENTS AND RESULTS
To obtain the model with the highest accuracy, we
tried two different classifiers: SVM and Random For-
est. To evaluate our models, we tested them on vali-
dation set including 1,000 pedestrian images. For the
SVM classifier, we investigated different regulariza-
tion parameters(C) to get the highest accuracy. The
regularization parameter tells the SVM optimization
how much we want to avoid misclassifying each train-
ing example. For large values of C, the optimization
will choose a smaller-margin hyperplane if that hy-
perplane does a better job of getting all the training
points classified correctly. Conversely, a very small
value of C will cause the optimizer to look for a larger-
margin separating hyperplane, even if that hyperplane
misclassifies more points. We got the highest accuracy
for the SVM model when the regularization parameter
has the value of 0.001.
For Random Forest, we examined different number
of trees in training the model. Random Forest uses
bagging (picking a sample of observations rather than
all of them) and random subspace method (picking a
sample of features rather than all of them) to grow a
tree. If the number of observations is large, but the
number of trees is too small, then some observations
will be predicted only once or even not at all. If the
number of predictors is large but the number of trees
is too small, then some features can be missed in all
subspaces used. Both cases results in the decrease of
random forest predictive power. But the last is a rather
extreme case, since the selection of subspace is per-
formed at each node. In general, the more trees we
use the better get the results. However, the improve-
ment decreases as the number of trees increases, i.e.
at a certain point the benefit in prediction performance
3
from learning more trees will be lower than the cost in
computation time for learning these additional trees.
For our dataset, Random Forest provided the best ac-
curacy with 1,000 trees.
Furthermore, in order to tune the hyper parameters
for HOG features, we extracted them using different
block sizes and cell sizes. Table 1 shows the results
of these experiments. As seen in this table, the block
size and the cell size have a significant affect on the
accuracy of our models. In the other words, the effec-
tiveness of the models strongly depends on the HOG
feature parameters. Also from the table, we can see
that the Random Forest outperforms SVM in all the
cases except when the block size is 2x2 and the cell
size is 4x4 which the accuracy of the SVM model is
higher than the Random Forest. According to these re-
sults, we can conclude that there is no optimal config-
uration for HOG features and it depends on the dataset
we are using. In order to reduce false positive rates in
our models, we exhaustively searched all 2,100 nega-
tive images and extracted 5,800 windows with the size
of 64x128 pixels as false positive objects and then re-
train our model with the new augmented set. Using
1,000 new negative images for validation, the original
model had a false positive rate of 0.005% while the
new model with hard negative mining had a 0% false
positive rate. Most of the false positives came from ob-
jects that are erect and skinny such as poles and trees.
However, our hard negative mined model eliminated
many of these false positives. An example of this im-
provement is seen in Figure 3.
As mentioned in the section 3, since we found mul-
tiple bounding boxes for each object, we used non-
maximal suppression to remove the redundant bound-
ing boxes. Figure 4 shows an example of using non-
maximal suppression for two images.
For the purpose of background subtraction, we
calculated a reference image using a Gaussian
Mixture-based background/foreground segmentation
algorithm. Then, we subtracted each new frame from
this image to compute a foreground mask. The result
is a binary segmentation of the image which highlights
regions of non-stationary objects. This way we were
be able to get the segmentation of moving regions in
image sequences in Real-time. Figure 5 shows an ex-
ample of the background subtraction for one frame of
a video.
 Figure 2: Flow Chart of Major Steps for Pedestrian Detection and Tracking.

Block size Cell Size SVM Accuracy Random Forest Accuracy

2 8 80.8% 67.7%
2 16 69.7% 81.1%
4 8 66.7% 69.1%
4 16 0% 80.4%
Table 1: The accuracy of SVM and Random Forest models using different HOG parameters

Figure 3: Reduction in false positive rates using hard

negative mining. Poles and other erect patterns were
eliminated in the new SVM model trained with the
augmented negative dataset.

To track the pedestrian in videos, after applying

the background subtraction and getting the foreground
mask, we found the contours for each frame and then
computed the bounding boxes for each contour of that
frame. Since all the training images have the size of 64
Figure 4: Redundant bounding boxes were eliminated
x 128, we re-sized the contours whenever their heights using non-maximal suppression on these images.
and widths were smaller than our training image sizes.
This was accomplished by adding some padding to the
heights of widths of the contours. Afterward, we ap-
plied our classifier on that contour to see if the contour one frame of a video. As seen in this figure, there are
is a pedestrian or not. We then used the non-maximal six pedestrian in the frame and the classifier detected 3
suppression technique to remove multiple bounding of them. The others either are occluded or they are in
boxes for each object. Figure 6 shows the results for a pose that the classifier can not detect. To observe if

4
Figure 5: Example of applying background subtraction on
one frame of a video.
Figure 6: Pedestrian detecting and tracking in one frame of
a sample video.
there are any confusion for the classifier when there are
moving objects other than the pedestrians, we tested it
on the videos that have different type of moving ob-
jects. Figure 7 is an example of a frame that contains
pedestrians, motorcycles and a truck. As shown in the
figure, the classifier only detected pedestrians and ex-
cluded the truck and motorcycles.
Furthermore, to evaluate our classifier, we tried 5
different videos with different duration. The total
number of the pedestrians in these videos was 40 and
our classifier detected 24 of them which means it has
60% accuracy. As a side note, we should mention
that finding videos that have both moving pedestrians
and moving non-pedestrian objects was difficult since
since the cameras of most videos were not fixed and so
the detection was not possible. We instead evaluated
our classifier on a few number of videos. To watch
a complete demo of the performance of our classifier,
please refer to the Youtube link that we have provided
5
Figure 7: Pedestrian detecting and tracking in a frame with
different type of moving objects.
in section 6.
5. CONCLUSION AND FUTURE WORK
We have demonstrated that HOG feature descriptors
combined with SVM or Random forest and negative
hard mining provides an effective strategy for pedes-
trian detection and tracking. The main draw backs
of this approach is that our model is unable to track
a large variety of human poses, and can only track
pedestrians after some delay is added to the video due
to the high complexity of HOG extraction. For many
videos, occlusion is often present while pedestrians are
moving in the scene, causing difficulties in detection.
In the future, other techniques for pedestrian tracking
can be added to our system such as optical flow and
Kalman filtering. Tracking humans is inherently a dif-
ficult problem in the computer vision society, but solv-
ing this problem can greatly reduce the number of an-
nual motor vehicle casualties, and reduce crime rates
through improved surveillance systems at home and at
work.
6. GITHUB AND YOUTUBE LINKS
Our GitHub Code:
https://github.com/afazel/CS231A_
Project
Our Youtube Video on Pedestrian Tracking:
https://www.youtube.com/watch?v=
0lEJIh6dWAE
References
[1] 2014 Motor Vehicle Crashes: Overview. U.S.
Department of Transportation, March 2016.
 [2] http://pascal.inrialpes.fr/data/human/

[3] http://mmlab.ie.cuhk.edu.hk/projects/PETA.html

[4] http://cbcl.mit.edu/software-
datasets/PedestrianData.html

[5] S. Munder and D. M. Gavrila. An Experimental

Study on Pedestrian Classification. IEEE Trans-
actions on Pattern Analysis and Machine Intelli-
gence, vol. 28, no. 11, pp.1863-1868, November
2006

[6] Dalal Navneet, Triggs Bill Histograms of Ori-

ented Gradients for Human Detection. Interna-
tional Conference on Computer Vision & Pattern
Recognition - June 2005.

[7] Felzenszwalb Pedro, Girshick Ross, McAllester

David and Ramanan Deva. Object Detection with
Discriminatively Trained Part Based Models.

[8] Nan Lu, Jihong Wang, Q.H. Wu and Li Yang

An Improved Motion Detection Method for
Real-Time Surveillance . Annalen der Physik,
322(10):891921, 1905.

[9] Nan Lu, Jihong Wang, Q.H. Wu and Li Yang His-

tograms of Oriented Gradients for Human Detec-
tion. IAENG International Journal of Computer
Science, 35:1.

[10] Zickler Stefan, and Efors Alexei Detection of

Multiple Deformable Objects using PCA-SIFT.
Carnegie Mellon University. 2007.

[11] Canyameres Masip, Sergi, and Antonio Manuel

Lpez Pea On the use of Convolutional Neural
Networks for Pedestrian Detection. 2015.

[12] Szarvas, M., Yoshizawa, A., Yamamoto, M.,

and Ogata, J. Pedestrian detection with convolu-
tional neural networks. Intelligent Vehicles Sym-
posium, 2005.

6
 Reconstructing Roller Coasters
Tyler J. Sellmayer
Stanford University
tsellmay@stanford.edu
Abstract
In this paper, we describe a method of three-dimensional
reconstruction whose input is a series of two-dimensional
images recorded by a passenger on a roller coaster (a
”first-person ride video”), and whose output is a three-
dimensional model which approximates the path of the
roller coaster’s track. We also describe a method for de-
termining the approximate color of the roller coaster track
from the same video. We conclude that our methods are ir-
revocably flawed and cannot be used to achieve our goal of
printing scale 3D models of roller coasters.
This builds on previous work in the structure from motion
problem, wherein an entire three-dimensional scene is re-
constructed from a series of images. We modify this problem
by attempting to only reconstruct one element of the scene,
the roller coaster’s track. We also implement a method of
choosing a subset of input frames from a video. Previous
SFM work has largely focused on either individually-taken
photographs [6, 12] or complete video sequences [2], tak-
ing every frame as input.
Our approach relies on a fundamental assumption about
first-person ride videos: the camera’s path through the
world is an approximation of the roller coaster track. In
other words, the rider keeps all hands and feet, and his
camera, inside the ride at all times. This allows us to use
the camera pose location as an approximation of the track
location.
1. Introduction
The motivation of this paper is to enable the author to
create 3D-printable models of roller coasters automatically.
The problem is a modification of the structure-from-motion
(SFM) problem as described in [6], simplified because we
only need to recover the camera poses for our final output.
We do not attempt to recover the 3D world points compris-
ing the roller coaster track itself. To accomplish the cam-
era pose estimation, we rely on MATLAB’s structure-from-
motion toolkit as described in [12].
We begin by breaking apart a video into individual
1
frames at 30 frames per second of video, using the avconv
tool [13]. This gives us a collection of frames stored as in-
dividual PNG images. Our MATLAB code then reads these
images from disk.
We author our own MATLAB code for automated
calculation of the color of the track. This color,
track color centroid, is later used to paint the fi-
nal rendering of points, and in the future could be used to
choose an appropriate color for a 3D-printed model of the
track.
We extend MATLAB’s tutorial code [12] to make use of
the high framerate of our input video [3]. Instead of tak-
ing each and every frame of video as input, we choose a
start point s and desired number of frames n, and intro-
duce a frameskip parameter which controls how many
video frames we ignore between SFM input frames. We call
these ignored frames inbetweens. The unignored frames are
called keyframes.
The SFM pipeline takes these keyframes as input. For
each keyframe, it corrects for image distortion (using a
manually-tuned set of camera parameters), detects SURF
features [1], finds feature correspondences with the previ-
ous frame, and uses these correspondences to estimate the
fundamental matrix [6, p. 284] between the previous frame
and current frame.
This estimated fundamental matrix is then used to com-
pute the relative camera pose adjustment between frames,
which is used to update a list of world-coordinate camera
poses. After each new frame is successfully processed in
this manner, we perform bundle adjustment [15] to increase
the quality of our camera poses. Once we have processed n
frames, we report the list of camera pose locations in world
coordinates and plot them. The plot points are colored with
the RGB value from track color centroid and dis-
played.
To enhance our results, we enable our algorithm to sub-
stitute a keyframe with a nearby inbetween frame when this
fundamental matrix estimation fails for any reason, trying
up to frameskip inbetweens.
In our experimental results, we report plot of camera
poses generated under a variety of SURF parameters and

Figure 1. Frame number 3070 from a first-person ride video [3],
unedited.
Figure 2. Frame number 2784 from a first-person ride video [3],
unedited. Here, the rider is inside a dark tunnel.
frameskip values. We then draw conclusions from these
results.
2. Problem Statement
Our problem has two independent pieces: estimating
the roller coaster track’s color, and estimating a three-
dimensional model of the roller coaster track’s path. We
examine these problems separately.
2.1. Estimating Track Color
This problem can be concisely stated as ”Given a first-
person ride video of a roller coaster, return the RGB value
which most closely approximates the paint color of the
roller coaster’s track.”
First-person ride videos have the property that the image
of the track always touches the bottom of the frame near
the center, as shown in 1. This property is only untrue in
cases where the camera is not pointing forward along the
track (we found no examples of this) or when the track is
not fully visible. For example, the track is not visible at
the bottom of the frame when the camera’s automatic white
balance adjustment causes it to be blacked out (or whited
out) in response to changing environment light, as seen in
figure 2.
Using this mostly-true property, we can conclude that
ideally, the track color will be approximately that color
which appears most often in the bottom-center area of our
2
video frames. But our images are noisy, and the light-
ing changes throughout the video, so this ideal scenario
doesn’t quite work if we use the pixel colors directly from
the recorded images. Instead we bucket these pixel colors
into a color palette using nearest-neighbor search [11], then
find the palette color whose member pixels occur most of-
ten in the bottom-center of the frame. This color we call our
track color. The full explanation of this algorithm is in the
Technical Content section 3.
2.2. Estimating Track Structure
As stated above, we use the locations in world space of
our camera as an approximation of the track position. This
lets us use the camera pose estimation stage of structure-
from-motion [12] as the basis of our algorithm. We mark
every frameskipth frame as a keyframe with the frames
between them called inbetweens.
Before processing any frame, we first undistort it using
manually-tuned camera parameters for correcting fisheye
distortion and a calculated intrinsic camera matrix K. To
obtain the approximate focal length of our camera, we com-
pute the average width-at-widest-point of the roller coaster
track’s image in pixels across a random subset of frames,
and use this average width to obtain a ratio between a width
in world-space (the real width of the track, which we as-
sume to be 48 inches) and a width in the image plane in
pixels. We use this ratio to convert the known focal length
of our camera (14mm, according to [5]) to pixels. We as-
sume square pixels and zero skew, so this focal length is all
we need to compute our intrinsic camera matrix K.
For each keyframe, we attempt to compute the camera
pose relative to that of the previous frame. MATLAB’s pose
estimation toolkit assumes that the camera poses are exactly
1 unit distance apart, an assumption which is corrected by
performing bundle adjustment [15] after each frame’s pose
is added.
During the computation of the relative camera pose,
MATLAB’s helperEstimateRelativePose func-
tion [12] attempts to estimate the fundamental matrix F [6,
p. 284]. This estimation can throw an exception when there
are not enough matching correspondence points to complete
the eight-point algorithm, or when there is no fundamen-
tal matrix found which creates enough epipolar inliers to
meet the requirement set by our MetricThreshold pa-
rameter. When an exception occurs, we do not want to
simply stop calculating. Instead, we make use of the in-
between frames, retrying the relative camera pose computa-
tion with each inbetween frame after our second keyframe
until we find one that succeeds, or until we run into the
next keyframe, whichever comes first. If we run out of in-
betweens without successfully computing a relative cam-
era pose, we terminate our SFM computations immediately
and return a partial result. In our full results (see section 6)
we report the mean and maximum numbers of unsuccessful
fundamental matrix computations per keyframe for each of
our experimental runs.
SFM requires feature correspondences for the funda-
mental matrix calculation [6], which requires features. We
choose to use SURF [1] as our feature detection algorithm
because it provides scale- and rotation-invariant features.
This is necessary because roller coasters often rotate the
rider relative to the environment (which rotates the pro-
jected images of features in our scene between frames), and
because the camera moves forward along the z-axis between
frames which changes the scale of the projected images of
features in our scene. In our results we report experiments
with controlling the SURF parameters NumOctaves,
NumScaleLevels, and MetricThreshold as defined
in [7].
To avoid needless reimplementation of past work, we use
MATLAB’s built-in toolkit [12] for computing correspon-
dences between features, estimating camera poses, tracking
views, triangulating 3D world points, and performing bun-
dle adjustment [15]. Together, these produce a final set of
camera poses, including camera location and orientation in
world-space. We then plot the camera locations and color
our plot using the RGB value of the calculated track color.
3. Technical Content
3.1. Splitting Video Into Individual Frames
Our video is downloaded from YouTube [3]. We run the
following command to split it into its individual PNG for-
mat images at a rate of 30 frames per second of video [13]:
$ avconv -i video.mp4 -r 30 -f image2 \
output_dir/%05d.png
We manually select the range of frames [f1 , . . . , fe ] from
the video which comprises the first-person ride video, ex-
cluding the copyright notice at the beginning and the credits
at the end.
3.2. Calculating Track Color
3.2.1 Determining The Color Palette
We first decide on a palette size. For our experiments, we
use palette size 10, meaning we will calculate 10 centroids
in the RGB space.
Our code examines a random subset of t
frames [s1 , . . . , st ] ⊂ [f1 , . . . , fe ], and takes
a random subset of q pixels in each frame
[p1,1 , . . . , pq,1 , p1,2 , . . . , pq,2 , . . . , pq,t ],
pixel pi,j is represented as a triplet of values between 0 and
255, indicating the red, green, and blue values comprising
the color of that pixel, respectively. This is a standard
representation of colors in RGB space.
Figure 3. Ten color centroids calculated from a random subset of
pixels in [3]. Notice that these colors are similar to those found in
figure 1.
We take this set of several thousand pixels, and run k-
means clustering [10] on it. This gives us k centroids in
RGB space, and we use the colors those centroids represent
as our color palette. Because of the randomness in this al-
gorithm, we do not get the exact same palette every time.
One example of a k = 10 color palette is seen in seen in
figure 3.
3.2.2 Finding The Track
Once we have established our color palette, we need to de-
termine which of the colors in the palette most closely ap-
proximates the color of our track. To accomplish this, we
must rely on our knowledge that in first-person ride videos,
the roller coaster track usually touches the bottom of the im-
age frame, near the center, and almost never touches the left
or right sides of the frame.
We first select a new random subset of t frames
[r1 , . . . , rt ] ⊂ [f1 , . . . , fe ]. In each frame, we examine only
the bottom 10 rows of pixels. We split this 10-pixel-high
strip horizontally into g 10-pixel-high segments. For an im-
age of width W , this gives us g regions [γ1 , . . . , γg ] each
of size 10 × W g . Our ultimate goal with these regions is to
find which palette color is least often present in the left- and
right-most regions.
Rather than just counting every pixel, we choose to count
only those pixels which lie on either side of an edge. This
increases the number of pixels we count that represent track
(which is made of hard-edged steel parts, in focus, and rel-
atively large in the frame, giving it more sharp edges) com-
pared to the number of pixels we count in noisy background
regions (which tend to be out of focus, motion-blurred, or
so far away that their edges are not distinguishable at the
camera’s resolution, giving them few sharp edges). We
count the pixels (ex − 1, ey ), (ex + 1, ey ) which lie on ei-
ther side of the edge, rather than the pixel (ex , ey ) which
lies directly on the edge, because we want to capture the
colors inside the regions more than we want to capture the
colors of the edges themselves. We call the set of points
where each (ei,x − 1, ei,y ), (ei,x + 1, ei,y ) for all edge pixels ei our set
of half-edge pixels.
Furthermore, because we care about hue more than satu-
ration or value when determining which pixels to count, we
perform the edge-finding computation on the ’hue’ layer of
3
 the column [0, 5, 124, 26, 0]T satisfies this condition, so our
track color is the one corresponding to that column. In this
example, that color happens to be the reddish-orange palette
color which covers the bulk of the track in figure 4.

3.3. Calculating Track Width

Now that we know the color palette and the track color,
we can examine yet another random subset of frames and
compute the average width (in pixels) of our image of the
track. This will allow us to scale our model appropriately
by adjusting the focal length parameter in our camera matrix
K. We again examine only the bottom 10 rows of pixels of
each frame. In each row of pixels, we find the leftmost and
rightmost pixel whose color lands in the track-color bucket,
and consider the distance between these two pixels to be the
track width at that row. We simply take the mean of these
track widths for each of the 10 bottom rows in each image
as our average track width.
Figure 4. Top: Frame number 3854 from a first-person ride video 3.4. Camera Pose Estimation
[3], unedited. Bottom: the same image with each pixel’s color
replaced by its nearest RGB-space neighbor from the color palette The camera pose estimation task is the first part of the
in figure 3. Structure-From-Motion problem as described in [12]. We
operate on grayscale version of our frames, color is not im-
Table 1. A histogram of half-edge pixel counts over 5 regions and portant for this part. We use a manually-tuned parameter for
10 colors.
correcting the radial distortion caused by the GoPro’s fish-
1612 163 15 14 244 117 72 0 59 818 eye lens effect, giving us less distorted frames, like the one
471 47 276 18 253 112 41 5 0 433 seen in figure 5.
683 0 70 55 32 285 142 124 0 251 We use SURF [1] to detect features as seen in
182 0 168 1 281 104 0 26 45 441 figure 6. We modify the parameters NumOctaves,
312 141 74 0 94 183 18 0 14 663 NumScaleLevels, and MetricThreshold as defined
in [7] in our experiments. SURF provides rotation- and
scale-invariant features, which is useful to us because we
want to find feature correspondences between frames in
our images after converting them to HSV (hue-saturation- an environment where our camera is rotating and moving
value) format. We use MATLAB’s edge function to ac- through space (because it’s on a roller coaster!). As de-
complish this [8]. scribed in the Problem Statement 2 section, we use MAT-
We use a nearest-neighbors search [11] to bucket our LAB’s toolkit to accomplish fundamental matrix estima-
half-edge pixels’ RGB values into our color palette. To il- tion, triangulation, and bundle adjustment. We extend the
lustrate this concept, we provide figure 4 where every pixel tutorial code [12] to make use of the relatively large num-
in the image has been replaced with its bucketed color. ber of frames available to us.
We then count the number of half-edge pixels in each We begin our SFM camera pose update loop with
color palette bucket, in each segment [γ1 , . . . , γg ]. This the intention of operating only on every frameskipth
gives us a two-dimensional histogram where each cell rep- frame [f1 , f1+f rameskip , f1+2∗f rameskip , ...], called our
resents the total number of half-edge pixels of one particu- keyframes. At each step in the loop, we first find feature
lar color ci in one particular region γj across all the frames correspondences between the next keyframe and the cur-
in [r1 , . . . , rt ]. Table 1 gives an example of one such his- rent last-used frame. Figure 7 shows one such set of cor-
togram. From this histogram we can find a color column respondences. When these feature correspondences pro-
which most closely matches the pattern we expect for our vide enough high-quality points for us to estimate the fun-
track color. As stated above, we expect the track color to damental matrix and meet our threshold for epipolar in-
appear almost-only in the center region(s), and almost-never lier count, the calculation succeeds and our loop contin-
in the far left and far right regions. We get good results sim- ues on to the next keyframe. When the feature corre-
ply choosing the color whose far-left region and far-right spondences do not meet this requirement, or if MATLAB’s
region counts have the minimum sum. In table 1 we see that estimateFundamentalMatrix function [9] fails for

4

Figure 5. Above: Frame number 839 in [3], unedited. Below: The
undistorted frame.
Figure 6. SURF features in frame 91.
any other reason, we re-attempt the calculation with an
inbetween instead of the failed keyframe. We try subse-
quent inbetween frames in the same order they appear in
the video.
For example when frameskip = 20 the algorithm
will first attempt to compute the fundamental matrix be-
tween frames 1 and 21. If this computation fails, we try
the computation again between frames 1 and 22, then 1 and
23, and so on, until the computation succeeds, or until we
reach frame 41 (which is the next keyframe) and terminate.
Examples of feature correspondences which led to a failed
fundamental matrix calculation can be seen in figure 8.
5
Figure 7. SURF feature correspondences between frames 192 (red)
and 212 (blue).
Figure 8. Top: SURF feature correspondences between frames 212
(red) and 232 (blue). Bottom: SURF feature correspondences
between frames 212 (red) and 241 (blue). Both of these sets
of features lead to a failed fundamental matrix estimate calcu-
lation. Notice the yellow lines which are much longer than the
distance between the image centers - these are incorrect corre-
spondences. Having so many of these creates the situation where
estimateFundamentalMatrix finds only fundamental ma-
trices that do not meet the threshold for number of epipolar inliers.
3.5. Output
Our output is a set of points, plotted in a 3D view, col-
ored to match our calculated track color. Each point repre-
sents the estimated camera pose location from one frame of
video. An example plot is given in figure 9.

Figure 9. Plot of 53 estimated camera pose locations,
generated with frameskip = 16, NumOctaves =
4, NumScaleLevels = 3, MetricThreshold = 1100.0.
4. Experimental Setup and Results
4.1. Implementation And Setup
For all our experiments, we operate on the same set of
images from [3]. This data set is roughly 14GB, and we’re
running our code on Stanford’s corn servers, so this uses
99% of our filesystem quota. We did not have a computer
with enough available storage to hold another data set.
All our code is implemented in MATLAB. We imple-
ment the following functions:
• main.m – This is the main function of our code. Re-
sponsible for generating figures 4, 5 and all of the cam-
era pose plots, as well as running all other functions
described below. Controls which image files are used
as input to the other functions. Defines the experiment
parameter TEST FRAMESKIP, the desired number of
inbetween frames between keyframes (also referred to
as frameskip above.)
• random subset images.m – Used by main.m to
select random subsets of image paths, which are then
passed to cluster colors, track color, and
average track width.
• cluster colors.m – Determines the color palette
of a set of images using k-means [10], as described
in section 3.2.1. Defines the experiment parameter
NUM COLORS, the number of color centroids in the
palette. Returns a NUM COLORS × 3 matrix of RGB
values, with each row representing one palette color.
• track color.m – Determines which color centroid
from the output of cluster colors best approxi-
mates the color of the roller coaster track, as described
6
in section 3.2.2. Defines the experiment parameter
BOTTOM STRIP SEGMENTS.
• average track width.m – Determines the width
of the track, as described in section 3.3.
• sfm.m – Computes camera poses using the
SFM pipeline as described in section 3.4. De-
fines the experiment parameters NUM OCTAVES,
NUM SCALE LEVELS, and METRIC THRESHOLD.
For our color experiments, we run our main MAT-
LAB function once per experiment, altering either the
NUM COLORS parameter in cluster colors.m
or the BOTTOM STRIP SEGMENTS parameter in
track color.m for each experiment. We record
the output of main, which includes the list of color cen-
troids, the histogram of pixels-per-centroid-per-segment,
and the final track color.
For our SFM experiments, we run the same MAT-
LAB function (main) once per experiment, changing
one parameter out of NumOctaves, NumScaleLevels,
MetricThreshold, and TEST FRAMESKIP each time.
We conduct testing in an exploratory manner, altering only
one parameter at a time, searching for a configuration which
gives us the longest possible series of camera poses before
running into the failure condition of being unable to cal-
culate a fundamental matrix for a keyframe or any of its
subsequent inbetweens.
4.2. Results
4.2.1 Track Color Estimation
We experimented with computing the track color for various
values of NUM COLORS and BOTTOM STRIP SEGMENTS.
These results are so uninteresting - the track color always
comes out as a shade of orange (as seen in figures 1, 3, and
4) or nearly white (the color of the sky in those same figures)
- that it is wholly unnecessary to present more than one full
example. We present this example in tables 2 and 3.
We also present a table of track color centroids as cal-
culated under different parameters in table 4. Remember
that these color coordinates are on a scale of 0-255 with
(0, 0, 0) representing pure black and (255, 255, 255) repre-
senting pure white.
4.2.2 Camera Pose Plots
We present a representative subset of our camera pose esti-
mation results. Figures 9, 10, 11, 12, and 13 show a variety
of camera pose location plots. Note that the long, straight
path of points in each plot corresponds to the long, straight
lift hill of the roller coaster. The jumbles of points are
badly-reconstructed camera locations on and after the peak
of the lift hill, or during the initial turn out of the ride shelter
 Table 2. A histogram over 5 segments and 10 colors.

902 1 153 87 4 191 775 3 467 428

454 4 45 206 7 146 344 98 293 721
429 4 20 114 6 492 172 7 203 1084
375 6 60 233 0 333 0 0 103 1131
918 0 148 338 0 334 0 0 97 730
Table 3. The color centroids for each of the 10 colors in 2.
Red Green Blue
81.1008 76.3207 57.9355
187.2241 128.2615 83.6437
72.8899 45.4787 24.6535
244.1506 242.3728 241.9210
171.5434 90.5595 36.1833
127.8087 105.3556 84.7148
36.8842 28.1582 19.3491
177.7524 181.0321 182.5577
Figure 10. Plot of 62 estimated camera pose locations
184.3315 152.2799 126.4728
(between frames 91 and 249) in [3], generated with
129.8242 55.6536 23.2184 frameskip = 4, NumOctaves = 4, NumScaleLevels =
Table 4. Estimated track colors.
3, MetricThreshold = 900.0.
#colors #segments Red Green Blue
3 3 161.7516 117.9050 85.0151
5 3 243.3168 241.6485 241.0941
7 3 243.2237 242.1020 242.0921
14 3 244.6650 243.3350 243.2677
3 5 161.7516 117.9050 85.0151
5 5 243.3168 241.6485 241.0941
7 5 142.3456 67.5242 28.2738
14 5 244.6650 243.3350 243.2677

area. Only figure 13 represents our coaster somewhat accu-

rately - watch the video in [14] and observe the presence
of the roller coaster’s second hill. We were unable to find
any configuration of our code which could reconstruct past
frame 2689 of the video [3] without hitting the failure con-
dition of being unable to find an inbetween that allows for
successful fundamental matrix estimation. For full results,
see the link in section 6.
5. Conclusions
5.1. Color Extraction
Our relatively low level of success in determining track
color (see table 4, often the track color is determined to be
white, which is actually the color of the sky in our video)
suggests that our goal was not achieved, and our method
was too reliant on manual tuning of the NUM COLORS and
BOTTOM STRIP SEGMENTS parameters. We suggest that
future work on this topic should ignore our results and use
better, existing image segmentation algorithms like the ones
found in [4]. Should anyone choose to use our method, we
offer some suggestions and conclusions about our results.
In section 2.1, we introduced a method for extracting the
Figure 11. Plot of 19 estimated camera pose locations
(between frames 91 and 153) in [3], generated with
frameskip = 8, NumOctaves = 4, NumScaleLevels =
3, MetricThreshold = 900.0.
color of a roller coaster track from a first-person ride video.
This method is extensible to other color extraction prob-
lems, where the general region that contains an object is
known across multiple views. For instance, one might ex-
tract a person’s eye color by overlaying a grid on the image,
then creating a 2D histogram similar to table 1 with each
row referring to a single cell in the grid. An existing face-
detection algorithm would be used to determine the general
location of the eye in each frame. After doing the nearest-
neighbors color palette bucketing and counting half-edge
pixels to create the histogram, whichever color was present
near the center of the eye, but not present at all in the bound-

7

Figure 12. Plot of 52 estimated camera pose locations
(between frames 91 and 833) in [3], generated with
frameskip = 16, NumOctaves = 4, NumScaleLevels =
3, MetricThreshold = 900.0.
Figure 13. Plot of 165 estimated camera pose locations
(between frames 91 and 2641) in [3], generated with
frameskip = 16, NumOctaves = 4, NumScaleLevels =
3, MetricThreshold = 2000.0. A video of these points is
available in [14].
ary regions around the eye, would most likely represent the
closest approximation of the person’s eye color. Further re-
finements could be made by detecting circles to more accu-
rately locate the iris and pupil.
For our specific roller coaster problem, our goal was to
find a close approximation to our track color. Here we de-
fine ”close approximation” in the context of our original
goal, which was to create 3D-printed models of roller coast-
ers, so we only need to approximate up to the color res-
olution of available 3D-printing filaments. In general this
means we only need to be able to accurately distinguish or-
8
ange from red or brown or any other primary or secondary
color, and we have achieved this level of accuracy in this
paper, but only by manually tuning the NUM COLORS and
BOTTOM STRIP SEGMENTS parameters until we got the
desired result. This is less useful than simply picking the
color manually.
5.2. Camera Pose Estimation
In section 3.4 we describe our approach to reconstruct-
ing camera poses for a series of frames taken from a video.
Our experimentation with modifying the frameskip pa-
rameter reveals that frameskip = 18 gives the longest
reconstructable sequences, but the resulting plot looks far
less smooth than with frameskip = 16. For our ultimate
goal of creating 3D-printed models of the track, we want a
smooth-looking plot, so we chose frameskip = 16 and
explored the behavior when other parameters are modified.
Increasing SURF’s NumOctaves parameter allows
SURF to find larger blob features [7]. This is useful
when our roller coaster moves past objects that are large
in the frame, such as the tree seen in figure 8. Mod-
ifying this did not affect our results very much. Run-
ning with NumOctaves = 5, NumScaleLevels =
3, MetricThreshold = 2000.0, frameskip = 16
allowed our algorithm to reconstruct 168 total frames,
whereas the same configuration with NumOctaves ∈
[3, 4, 6] only allowed our algorithm to reconstruct 165 to-
tal frames. This is only a 1.8% increase.
Increasing SURF’s NumScaleLevels parameter al-
lows SURF to find a greater quantity of small blobs [7].
It cannot be less than 3, but increasing it above 3 did
not improve our results. Running with NumOctaves =
5, NumScaleLevels = 4, MetricThreshold =
2000.0, frameskip = 16 allowed our algorithm to recon-
struct 40 total frames, and running with NumOctaves =
5, NumScaleLevels = 5, MetricThreshold =
2000.0, frameskip = 16 allowed our algorithm to re-
construct 167 total frames.
Increasing SURF’s MetricThreshold parameter in-
creases the minimum threshold for feature ’strength’ [7].
This gives us a greater quantity of high-quality fea-
tures which are more likely to find strong correspon-
dences in subsequent keyframes. This also makes us
less likely to find correspondences in general, because
it may be that the same feature has ’strength’ higher
than MetricThreshold in one image but not in the
other. The higher we set this threshold the more likely
it becomes that we will fail to find the same feature
in two consecutive images, thus the less likely we are
to find a correspondence. When the threshold is much
higher than 2000 (we tested with MetricThreshold =
4000), we will reach a failure state earlier, because
we will be unable to calculate a fundamental matrix
due to too few features. When the threshold is much
lower than 2000 (we tested with MetricThreshold ∈
[800, 850, 900, 1000, 1100]) we will reach a failure state
earlier. With low thresholds we obtain so many erroneous
feature correspondences that they will cause MATLAB’s
estimateFundamentalMatrix function to fail with
an exception because there are never enough epipolar in-
liers for any of the sampled fundamental matrices [9]. Un-
fortunately, we cannot provide a specific suggestion for a
good MetricThreshold parameter, because the effects
of this value are entirely dependent on the quality and struc-
ture of the input images. We can suggest that future work
start by doubling MetricThreshold until the quality of Figure 14. Frame number 2814 from a first-person ride video [3],
their output degrades, then doing binary search to find a unedited.
good-enough MetricThreshold between the two best
values.
6. Code And Full Results
Choosing a high MetricThreshold also increases
MATLAB code and the full experimental results of this
the (totally subjective) smoothness of our point plot. This is
paper are available at https://github.com/tsell/
because the high quality features are less likely to be incor-
reconstructing-roller-coasters.
rectly corresponded with the wrong feature in an adjacent
frame. This is especially important because the scene in and
around a roller coaster is full of repetitive elements, like the References
repeating structure of the track, the similar pieces of support [1] H. Bay, A. Ess, T. Tuytelaars, and L. V. Gool. Surf: Speeded
steel, and repeating patterns in the nearby rides and build- up robust features. Computer Vision and Image Understand-
ings. These elements are often incorrectly matched as corre- ing, 110(3):346–359, 2008.
spondences, as seen in figure 8. Modifying NumOctaves [2] P. Beardsley, P. Torr, and A. Zisserman. 3d model acquisition
and NumScaleLevels also helps with this by narrow- from extended image sequences. Computer Vision, pages
ing the range of feature scales we detect, reducing the oc- 683–695, 1996.
currence where a nearby feature in one frame is incorrectly [3] FrontSeatCoasters. Six flags magic mountain goliath pov hd
corresponded with a far-away feature in another frame. roller coaster on ride front seat gopro steel 2013. Web, 2014.
https://www.youtube.com/watch?v=N uV0Q2UH98.
[4] M. Frucci and G. S. di Baja. From segmentation to binariza-
tion of gray-level images. Journal of Pattern Recognition
5.3. Final Word Research, 1:1–13, 2008.
[5] GoPro. Hero3 field of view (fov) information. Web,
Overall, we consider these experiments a failure. Our 2016. https://gopro.com/support/articles/hero3-field-of-
camera pose estimation is not robust enough to create view-fov-information.
smooth models of the entire track. There are large portions [6] R. Hartley and A. Zisserman. Multiple View Geometry in
Computer Vision. Cambridge University Press, 2003.
of first-person ride videos which are totally inscrutable to
our methods, including frames like the ones seen in figure [7] MathWorks. detectsurffeatures: Detect surf fea-
tures and return surfpoints object. Web, 2016.
2 and 14 which have been nearly destroyed by the cam-
http://www.mathworks.com/help/vision/ref/detectsurffeatures.html.
era’s auto white-balance feature. We were unable to find a
[8] MathWorks. edge: Find edges
configuration of SURF parameters and frameskip value
in intensity image. Web, 2016.
which reduced the reconstruction error sufficiently to make http://www.mathworks.com/help/images/ref/edge.html.
a smooth-looking track model, so none of our results are [9] MathWorks. estimatefundamentalmatrix: Es-
worthy of being 3D printed. Also, the processing takes so timate fundamental matrix from correspond-
long (on the order of 1 hour per 150 frames successfully ing points in stereo images. Web, 2016.
processed, though we did not take explicit notes of our tim- http://www.mathworks.com/help/vision/ref/estimatefundamentalmatrix.html.
ing), and needs to be manually re-calibrated for each video [10] MathWorks. kmeans: K-means clustering. Web, 2016.
(because the scale and quality of features in different videos http://www.mathworks.com/help/stats/kmeans.html.
varies widely depending on video quality and camera reso- [11] MathWorks. knnsearch: Find k-nearest
lution), that this is not faster or better than simply construct- neighbors using data. Web, 2016.
ing the model manually in some 3D modeling software. http://www.mathworks.com/help/stats/knnsearch.html.

9
[12] MathWorks. Structure from motion
from multiple views. Web, 2016.
http://www.mathworks.com/help/vision/examples/structure-
from-motion-from-multiple-views.html.
[13] J. Nielsen. How to extract images from a video with av-
conv on linux. Web, 2015. http://www.dototot.com/how-to-
extract-images-from-a-video-with-avconv-on-linux/.
[14] T. Sellmayer. Rotate camera points fig. 13. Web, 2016.
https://www.youtube.com/watch?v=N uV0Q2UH98.
[15] B. Triggs, P. McLauchlan, R. Hartley, and A. Fitzgibbon.
Bundle adjustment: A modern synthesis. Proceedings of the
International Workshop on Vision Algorithms, pages 298–
372, 1999.

10
 Recovery and Reconstruction of Blackboard or 
Whiteboard images with occlusions 
 
Vijayaraj Gopinath 
vgopinat@stanford.edu 
Computer Vision: From 3D reconstruction to recognition 
CS231A 
Stanford University 
 
Abstract  2 Proposed solutions 
We  have  all  taken  the  pictures  of  In  this  work,  the  multiple  pictures 
blackboard  for  reference  purpose,  but  the  of  the  blackboard can be in any orientation 
problem is most  of the time those pictures comes  and  we  assume  the  first  picture  dictate our 
with  occlusion  such  as  professor  standing  in  required  orientation  for  result  purpose  and 
between  or  other  students.  Even  if  you  can  take 
all  other  pictures   would   be  orientated 
multiple image,  there will one or other occlusion 
towards  our  first  picture.  Also  the 
obstructing  the  blackboard  information.  By  the 
time  one  student  or  professor  has  moved  out  proposed  algorithm  will  find  out  whether  
another  will  often  have  moved  in,  making  it  the  given  pictures  would  able  to  recover 
difficult  and  time­consuming  to  get   the  perfect  the  blackboard  (or  whiteboard)  or  not 
blackboard  picture.  Also  we  can’t  wait  for  the  otherwise  report  an   error.  Since  we  have a 
right   moment,  since  the  blackboard  can   be  specific scenario here, we would tap on the 
erased  multiple  time  which  makes  this  problem  properties  of  this  scenario  (blackboard 
even  more  difficult.  In  this  project  we  will  make  only)  to  segment  the  occlusion  with  a 
use   of  available  multiple   images  and  better  techniques  such  as   background 
automatically  recover  the  blackboard  by 
subtraction  and  image  morphology  
removing all the occlusions. 
techniques  which  provides  better  results 
 
than  without  worrying  about  other   more 
1 Review of previous work 
complex segmentation techniques. 
We  can  find  many  works  related  to 
 
generic  scene  reconstruction  with  occlusion 
3 Summary of the technical solution 
using  multiple  images.  Especially  the   work 
To  solve  this  problem,  we  have 
[1]  from  Microsoft  research  which  tries  to 
remove  any  occlusion  from   landmarks  or  proposed the following solution to this. 
monuments  from  multiple  pictures.  Generic  1,  Since  we  are  dealing  with  multiple 
scene construction has lot of assumptions and  image,  find  homography  for  first  image 
work  around  with  that,  but  here  in this work,  and  to  all  other  image   and  using the found 
we  are  only  concerned  about  a  specific  homography  warp  all  other  images  to  the 
scenario,  reconstructing  the  blackboard  first  image.  2,  Detect,  segment  and  label 
all  the  occlusions  in  all  the  images.  Use 
images  from  occlusions  such  as  person 
image subtraction and other morphology 
(professors or students) or any other objects. 
 
techniques  to  detect  and  segment.  3,  Once  In  figure  1,  you  can  see  that  lot  of  features 
labelled,  we  need   to  find  out,  which  (more outliers than inliers) are found around 
occlusions  comes  from  which  original  occlusion  since  those  regions  have  more 
image  and  decide  whether  the  scene  can  be  intensity  changes.  Also,  we  will  not  get 
recovered  or  not.  4,  Using  the  label  and  enough  features  on  the corners always since 
identification  of  occlusion  from  the original  the  color  of  the  whiteboards   sometimes 
image,  recover  the  scene  by  copying  the  closely  match  the  background  of  the  scene 
pixels  from  non  occluded  region  to  the  itself.  For  all  these  reason,  we   need  user 
occluded  region.  5,  Finally  blend   the  image  feedback  to  select  four  corner  of  the 
to complete the reconstruction.   blackboard  for  us.  We  use  these  four points 
  to  calculate  our  initial  homography  and  we 
4.1 Finding Homography  remove  all  the   outliers   using  RANSAC  or 
It  is  important  to  have  an  accurate  other such methods.  
homography  since  we  later  use  image   
subtraction  to  identify  occlusions.  It  is 
common  nowadays  we  have  more 
whiteboards  than  blackboards  which  makes 
finding  homography  more  challenging. 
With  our  experiments,   we  found   that 
detecting  rich   features  proven  very  difficult 
around  whiteboards  because  of 
homographic  surface  nature  of  the  scene. 
Also  most  of  the  found  features  are  also 
around  the  occlusion  since  we  often  see  a   
steep  change  in  intensity  levels  near  the  Figure 2 ­ Scene after the removing all 
occlusions.  outlier features.  
   
Only  using  the  inliers  set  which  we  found 
using  the  last  step,  we  recalculate  the 
homography  which  becomes  our  final 
homography.  We  need  these  two  step 
approach  to  get  our  homography  since  we 
need  accurate  homography  and  also  we 
assume  that  the  user selection of  four points 
will be not accurate enough for homography 
calculation  but  accurate  enough  to  remove 
outliers.  We  have  used  SURF  features here. 
  Using  the  final homography we warp all the 
Figure 1 ­ with outlier features.  images to match our base first image. 
   
 
4.2  Detection  Segmenting  and  Labeling  accurate  homography,  results  after  image 
occlusion.  subtraction  will  have noises around it  which 
The  first  attempt  we  made  in  we  will  deal  later.  After  image  subtraction, 
detecting  the  occlusion   was  face  detection.  we  can  use  global  image  threshold  using 
Since  most  of  the  time,  our  occlusion  will   Otsu’s  method  [2],  to  compute  the   global  
be  a  person  occluding  the   blackboard  it  threshold  level which is normalized to [0 1]. 
would  be  very  interesting  to   take  this  Using  the  found  threshold  level,  we  can 
approach.  We can find the bounding box for  convert  the  intensity  image  to  a  binary 
face  after  detecting  it  and  We  can  use  face  image. 
to  body  ratio  to  define  a  segment  box  to   
cover entire occlusion. 
 

  Figure 4 ­ After image subtraction 
Figure 3 ­ Face detection   
 
This  idea  has  clearly  many  issues,  We  will 
get  little  noise  in  terms  of  detecting  other 
small  or  non  faces  in  the  images  also  the 
occlusion  need  not  to  be   a  person  here  or 
even  if  the  occlusions  are  assumed  to  be 
only person, finding bounding box would be 
wrong  to  do  since  person  can  take  other 
forms.  After  experimenting   with  lot  other 
methods,  we  found  out  that  the 
morphological  techniques  would  suit  this  Figure 5 ­ Binary image with noise 
scenario more perfectly.   
  To  remove  noises  in  the  binary  image  we 
Once  we  have  accurate  homography  can  use  morphological  techniques.  Use 
and  good  warping,  we  can  use  image  structural  element  of ‘Disk’ with radius of 3 
subtraction  between  base  image  and  all  and  erode  the  picture.  The  above  step  helps 
other  image  to  remove  everything  except  to  remove  the  noise  which  we   got  after 
the occlusion. Though we found out   image subtraction and thresholding. 
 
The  binary  image  we   found  from  the  last 
step  will  have  lot  of  discontinuity.  In  order 
to  get  the  complete  segment,   we  need  to 
dilate  the  binary  image  to   get  supersize  of 
the  occlusion.  We  can  use  the  structural 
element  of  ‘Disk’  with  radius  of  25  to  get 
the  supersize  of  the  occlusion.  Figure  6 
shows the binary image after dilation.   
 
 
Figure 7 ­ After filling the holes 
 

Figure 6 ­ Image dilation, Disk radius 25  
 
Depends  upon  the  intensity  level  of 
occlusion  in  comparison  to  the  blackboard  Figure 8 ­ Erode to get original size 
or  whiteboard,  the  found  occlusion    
potentially  can  have  holes  inside  it.  Which  After  finding  the  binary  image  of  the 
can  be  noticed  from   the  figure  6.  Here  occlusions,  label  all  the  independent 
occlusion  originally  had  intensity  level  connected  components  to  get  the  accurate 
similar  to  the  background  so  after  image  boundary for every occlusions in the scene.  
subtraction  and thresholding will have holes   
in  it.  We  can  find   holes  by  finding   the 
connected  components  in  the  binary  image 
and  finding  the  missing  pixel  inside it. Find 
such  missing  pixel  and  fill  it.  Figure  7 
shows  the  binary  image  after  filling  up  the 
hole.  Once  finding  the  supersize  occlusion, 
erode  the  binary  image  with  SE of  radius of 
22  to  get  back  the  original   size  of  the 
occlusion.  Figure  8  ­  shows  the  final binary 
image with the occlusion having same size. 
Figure 9 ­ Color map of the found label. 
 
4.3  Mapping  occlusions  to  the  original  We  address  this  in  the  future  works section. 
image region.  Blackboard  reconstruction  by  removing 
After  the  segmenting and labeling, it  occlusion  is  very  interesting  project and  has 
is  important  to  map  each  found   occlusions  an  important  application  in  the  education 
to  the  original  image.  We  assume  that  the  domain. This can be pretty  much a cool app 
object  which  is  occluding  has  totally  which  students can install in the smartphone  
different  average  intensity  value  in  and  take  multiple  pictures  and  app  can 
comparison  to  the blackboard itself. We can  automatically  recover  the  entire  blackboard 
build  a  model  based  on  the  intensity  level  scene in a self contained manner. 
around  the  occluded  object  and  occluded   
region  in  the  original  images,  which  can  be 6 Future works 
used to map the objects.  1,  As  previously  discussed 
  sometimes  we  won't  get  enough  features  to 
find  homography,  and  so  need  user 
selecting  conors  for  us.  We  can  work  on 
sophisticated  rectangle  detection  to  avoid 
user  input.  2,  Fine  tuning  segmentation 
using  other  techniques  such  as  Fast 
marching  method, 3, For mapping occlusion 
to  the  original  image,  currently  we  build 
model  based  on  intensity  value,  this  can   be 
improved to other sophisticated model using 
 
Figure 10 ­ Recovered image  features.  4,After  the  scene  has  been 
reconstructed,  we  can  try  recognizing  the 
4.4 Recovering and Blending  text in it and create a doc. 
Once  we  identified,  labeled  and 
mapped  the  occlusions  we  can  recover  the           REFERENCES 
scene  by  copying  the  pixels  from  non 
occluded  portion  of  the  images  to  the  1. http://research.microsoft.com/pubs/69386/peop
occluded  portion  of  the  images.  Since,  we  lemover.pdf 
copy  pixels  come  from  different  image  to  2. http://ijarcet.org/wp­content/uploads/IJARCET­
recover  the   scene,  the  boundary  of  the  VOL­2­ISSUE­2­387­389.pdf 
3. http://www.eiti.uottawa.ca/school/research/viva
recovered  region  in  the  final  image  will  be 
/papers/homographie.pdf 
visible,  so  we  need  to  blend  the  image  to 
4. http://www.cescg.org/CESCG­2016/papers/Jari
complete  our  reconstruction.  Figure  10 
abka­Generation_of_lecture_notes_as_images
shows  the  final  recovered  image  by  _from_recorded_whiteboard_and_blackboard_
removing all the occlusions.  based_presentations.pdf 
  5. http://visual.cs.ucl.ac.uk/pubs/learningOcclusio
5 Conclusion  n/CVPR_2011_learning­occl.pdf 
Dealing  with  similar  background  in    
computer  vision  is  very  challenging, human 
eyes has evolved to detect this seamlessly.  
 
 
 Real-Time Semi-Global Matching Using CUDA Implementation
Robert Mahieu
Stanford University
Department of Electrical Engineering
rmahieu@stanford.edu
Abstract—With recent rise of technology such as
augmented reality and autonomous vehicles, there
comes a necessity for speedy and accurate depth
estimation to allow these products to effectively interact
with their environments. Previous work using local
methods to produce depth maps has generally been fast
but inaccurate and work using global methods has been
accurate but too slow. A new technique referred to as
semi-global matching combines local and global
methodologies to balance speed and accuracy,
producing particularly useful results. This project
focuses on the implementation of slight variations on the
original algorithm set forth by Hirschmuller [2] to
increase accuracy and using CUDA to accelerate the
runtime. Results show sufficiently low error, though
runtime was found to be imperfectly optimized.
1. Introduction
Acquiring depth information from sets of images is
incredibly important in many emerging fields such as
augmented reality, robotics, autonomous vehicles, etc.
However, these applications rely on the produced depth
information to be both accurate and generated in a short
amount of time—ideally close to real-time—to ensure
safety of the system and users, as well as for reliable pose
tracking.
Many algorithms for computing this information have
been previously explored, either looking at localized
information or global information throughout the entire
image. Local techniques such as Winner-Takes-All (WTA)
or scanline optimization (SO) [7] compute results for pixels
independently and require minimal computation, but due to
lack of consideration for global trends typically result in
inaccurate conclusions. The dynamic programming [5]
approach is also computationally efficient, but because the
algorithm only looks at a single row per iteration, it also
lacks consideration for global trends and commonly causes
streaking patterns to show up in the output. On the other
hand, while global techniques such as a Graph Cuts [1] and
Belief Propagation [9] produce more accurate results and
better avoid the errors encountered in the local methods,
Michael Lowney
Stanford University
Department of Electrical Engineering
mlowney@stanford.edu
these techniques are significantly more memory intensive
and end up being much slower.
To obtain both reasonable accuracy as well as real-time
performance, we instead move to what can be referred to as
a semi-global matching technique [2], which makes some
use of both local and global methods. Additionally,
offloading data calculation onto the GPU, which is ideal for
handling SIMD (single instruction multiple data)
computation, allows us to exploit the parallelizable nature
of our image-based calculations and significantly reduce
computation time for estimating the optimal disparity map.
2. Problem Statement
In order to tackle this problem, we make the assumption
that the input images are a rectified stereo pair. This is
inherently the case when two cameras are orthogonal to
the baseline, and point in the same direction. Mpopular
stereo vision datasets, such as the Middlebury dataset,
which is used in this paper [3][6][8], provide stereo pairs
that have been rectified. The benefit to having rectified
images is that the epipolar lines are horizontal and
corresponding lines are at the same height in each image.
This simplifies the problem because corresponding points
will lie on the same epipolar line, and so we only have to
search in horizontal directions.
Local methods are more prone to noise in their disparity
maps due to the fact that there may be several local
minima in their cost function. Because of this the semi
global approach uses a model which penalizes changes in
disparity values in local neighborhoods. This causes the
resulting disparity map to be smoother by attenuating high
frequency noise, which provides a clearer estimate of the
true relative depth of the objects in the scene.
3. Technical Content
The implementation of the semi-global matching method
comes down to minimizing an energy function describing
the quality of a potential disparity image. This is
represented by the expression below:

𝐸(𝐷) = ∑ (𝐶(𝑝, 𝐷𝑝 ) + ∑ 𝑃1 𝟏{|𝐷𝑝 − 𝐷𝑞 | = 1}
𝑝 𝑞∈𝑁𝑝
+ ∑ 𝑃2 𝟏{|𝐷𝑝 − 𝐷𝑞 | > 1})
𝑞∈𝑁𝑝
Where 𝐷 is the disparity map, 𝑝 is a pixel location on the
map, 𝐷𝑝 is the disparity value at pixel 𝑝, 𝑁𝑝 is the
neighborhood of pixels around 𝑝, 𝟏{? } is an indicator
function that is equal to one if the argument within the
braces is true and zero if false, and 𝑃1 and 𝑃2 represent
penalty value given to various changes in disparity within
the local neighborhood. 𝐶(𝑝, 𝑑) represents an initial cost
function which is based on the absolute difference between
gray levels at a pixel 𝑝 in the base reference image and pixel
𝑝 shifted by 𝑑 along the epipolar line in the matching image
(assumed to be the right image in the stereo pair):
𝐶(𝑝, 𝑑) = |𝐼𝑏 (𝑝𝑥 , 𝑝𝑦 ) − 𝐼𝑚 (𝑝𝑥 − 𝑑, 𝑝𝑦 )|
Note also as stated in section 2 that the images are
assumed to be rectified. The energy function thus penalizes
heavily (with 𝑃2 ) large jumps in the disparity map and less
heavily (with 𝑃1 ) small changes that may represent sloped
surfaces. Note that 𝑃1 < 𝑃2 . This allows us to reduce high
frequency noise in the resulting disparity image.
Optimizing this energy function using global
minimization in 2-dimensions is NP-complete, and requires
too much computation to solve for many practical
applications. On the other side of the spectrum, minimizing
in 1-dimension over image rows, such as in the dynamic
programming approach, is light on computation, but suffers
from accuracy issues as discussed above. To handle this
problem, semi-global matching leverages several
1-dimensional minimization functions to more efficiently
construct an adequate estimate of the solution.
The first step in the actual implementation of the
algorithm is to calculate initial costs for all pixels in the
image pair at disparities ranging from 0 to some selected
𝑑𝑚𝑎𝑥 . We found that using a 𝑑𝑚𝑎𝑥 value of 64 was more
than enough for all images we tested. This computation can
be efficiently carried out on the GPU with a kernel that
gives each thread a computation for one pixel and one
disparity value. The system takes an RGB stereo pair as an
input and then converts the image to grayscale. The cost
function uses the difference in grayscale intensity values as
a metric to determine how good or bad a potential match is.
Once the images are converted to grayscale they are store
in texture memory on the GPU to increase speed. Texture
memory is cached on the chip and allows for much faster
reads than global memory.
To carry out the next step denoted “cost aggregation”, we
iterate and compute the energy function locally over 8
directions (two horizontal, two vertical, two for each
diagonal). An example for the recursive expression is
shown below for the horizontal direction, going from left to
right across the image:
𝐸(𝑝𝑥 , 𝑝𝑦 , 𝑑) = 𝐶(𝑝, 𝑑)
+ min(𝐸(𝑝𝑥 − 1, 𝑝𝑦 , 𝑑),
𝐸(𝑝𝑥 − 1, 𝑝𝑦 , 𝑑 − 1) + 𝑃1 ,
𝐸(𝑝𝑥 − 1, 𝑝𝑦 , 𝑑 + 1) + 𝑃1 ,
𝑚𝑖𝑛 (𝐸(𝑝𝑥 − 1, 𝑝𝑦 , 𝑖) + 𝑃2 ))
𝑖
This step can be parallelized by having a different block
for each direction, and within each block having each thread
handle one iteration for one disparity. Note that after each
step along the direction, the threads must be synchronized.
This can be done by utilizing the CUDA command
__syncthreads().
Once all paths have been traversed, results are compiled
into a single value:
𝑆(𝑝, 𝑑) = ∑ 𝑤𝑟 ∗ 𝐸𝑟 (𝑝, 𝑑)
𝑟
Where 𝑟 represents a given direction and 𝑤𝑟 represents a
weight value for that particular direction. By introducing
this weighting term, we account for the fact that results
obtained from a certain path orientation may still lead to
better estimates than the results from the other orientations.
Therefore, we are able to weight each direction accordingly
based on the scene type.
The final step is to, for each pixel, iterate over the
calculated energy values and determine which disparity
value corresponds to the minimum energy. This is
represented by the following function:
𝐷(𝑝) = argmin 𝑆(𝑝, 𝑑)
𝑑
This can be parallelized on the GPU by having each
thread handle all disparities for a given pixel. The
disparities returned from this step represent the optimal
disparity map that minimizes the energy function.
Once we have a reasonable disparity map, the next step
 Figure 1- First row shows the left input image of the stereo pair, second row is the depth
map using semi-global matching, third row is ground truth
is to determine areas of occlusion in the image, meaning
areas that are visible in the base image, but blocked in the
matching image. This can be done by running a slightly
modified version of the algorithm as defined above again to
generate a disparity map for the match image. The only
modification is in the initial cost function which becomes:
𝐶(𝑝, 𝑑) = |𝐼𝑚 (𝑝𝑥 , 𝑝𝑦 ) − 𝐼𝑏 (𝑝𝑥 + 𝑑, 𝑝𝑦 )|
Once we have disparity maps for both the base and match
image we can compare the results to identify occluded
regions. For each pixel in the base disparity map we sample
the disparity value. We then compare this to the value in the
match disparity map at the same pixel location shifted by
the base disparity value we just sampled. If these two values
are the same (within some small tolerance), we judge them
to be true correspondences, otherwise we make them as
occluded pixel and set them to zero in the base disparity
map. This technique outputs a refined base disparity map.
Finally, to eliminate residual noise in the output we filter
the disparity map using a median filter. Good results were
observed while using a small kernel of 3x3. This allows us
to keep the major edges and details in the map while
removing the unwanted high frequency components.
A consideration worth noting is the memory
requirements of this algorithm. The amount of memory
used scales like 𝑂(𝑚𝑛𝑑𝑚𝑎𝑥 ), where 𝑚 it the number of
rows in the image, 𝑛 is the number of columns in the image,
and 𝑑𝑚𝑎𝑥 is the number is the maximum number of
disparity values. Storing the initial cost matrix and the
matrices for the 8 search directions may exceed the total
amount of memory on the GPU. For this paper the
algorithm was run on a laptop with a NVIDIA GeForce
940M GPU with 2GB of memory. In order to stay below
this 2GB threshold the input images must be down sampled.
Unless otherwise specified the images will be down
sampled so that the number of columns will be 450, and the
number of rows will be scaled accordingly.
4. Results
The quality of our algorithm was tested using the
Middleburry dataset. Figure 1 shows the results of our
algorithm compared to the ground truth of the depth
map for various image pairs. For these trials we used
the values suggested in [4] for 𝑃1 , 𝑃2 . All 𝑤𝑟 were set
to the same value to ensure equal weighting. Table 1
shows the values of 𝑃1 and 𝑃2 used for our results.
Table 1: Penalty Values
↔ ↕ ↖↘ ↙↗
𝑃1 22.02 17.75 14.93 10.67
𝑃2 82.79 80.87 23.30 28.80
 Qualitatively the results appear to be quite a close Unfortunately, due to time constraints on the project, we
match to the ground truth. The regions towards the left were unable to spend much time optimizing the CUDA
border of the image are consistently unlabeled. This is implementation, so tests on runtimes have returned sub-
due to the fact that they represent pixels that are only optimal results. In the current somewhat naïve
seen in the base image. Only pixels that are in the field implementation, we are still able to get runtime down to
of view of both cameras will result in accurate disparity around one frame/sec for images with sizes below about
values. 250x217 (54250 pixels). The relationship between runtime
and input image size is illustrated in Figure 2. As shown in
Table 2: MSE and runtime for images in Midlleburry the figure, the Semi-Global Matching algorithm has a
dataset runtime that scales linearly with the number of pixels in the
input images. It is also worth noting that the graphics cards
IMAGE PAIR MSE Runtime
used in modern stereo research are much more powerful
aloe 0.0296 3715ms than the ones used in this paper (consumer grade laptop
GPUs). This difference in hardware is a main contributor to
books 0.0687 3394ms
the longer runtimes found in this project.
dolls 0.0712 3385ms
It is important to note that one of the most significant
laundry 0.0935 3517ms
factors in the runtime, however, is actually the resizing of
pots 0.1053 3607ms images that occurs at the start of the program after being
baby 0.0385 3759ms read in by the CPU. This is necessary to ensure that we do
not overload the GPU memory, however the time cost is
bowling 0.1083 3692ms very high. When images do not need to be resized on-the-
art 0.0949 3385ms fly, total speeds are greatly increased (about 2x speedup).
In future implementations, intelligent use of shared
cones 0.0409 3152ms memory and memory access within warps should be able to
wood 0.0671 3403ms dramatically increase performance. Some of these
techniques are outlined by Michael et al. [4].

Table 2 shows the mean-squared error (MSE) between For posterity, to demonstrate the robustness of our
our experimental depth maps and their respective ground algorithm we also selected an arbitrary stereo image pair
truths. Note that the error values are generally quite low,
never reaching any higher than around 10% for any of the
images we tested. Although differences in scaling of depth
to grayscale may be present between the experimental
results and the ground truth, this appears to be minimal and
therefore the MSE should still provide a good metric for
analyzing the success of the algorithm.

While the performance of the algorithm seems to

decrease in highly cluttered scenes such as the Art image
pair or scenes with many strong gradients such as the Pots
image pair, in general, variation in the scene structure
appears to affect the quality of the produced depth map very
little, both qualitatively and quantitatively.

These results also appear to indicate that our decision to

weight the results from all path directions equally, as well
as our decision to keep constant the penalty values
throughout all tests, had relatively small impact on the
quality of our depth maps. The results from Michael et al.
[4] report that training these values for specific scenes does
increase quality noticeably, however our results seem to
show that it is still possible to get reasonably good results
without worrying about changing these parameters for
every scene.
from reddit (https://www.reddit.com/r/crossview) of a
Figure 2- Runtime vs Image Size
Batman figurine. The results are displayed in Figure 3.
While some poorly defined occlusion areas create a good
bit of distracting noise in the depth map image, the large
non-occluded areas actually show quite a high amount of
detail. Looking closely, we can even make out definition at
the level of the muscles on the figurine.
Figure 3- Batman robustness test
5. Conclusions
In this paper we demonstrate the effectiveness of the
Semi-Global Matching technique for depth estimation from
a set of stereo images. Leveraging a combination of global
and local techniques our results exhibited low error when
compared to ground truths. With more powerful hardware
and optimized CUDA implementation, our initial results
imply that this would be possible to run in real time. This
has been shown to be the case in [2][4].
The error could be further reduced in the future by
changing how we handle occlusions. Currently we are able
to detect occlusions by comparing the depth map for both
the base and match image. Once occlusions are found they
are set to zero, which can cause an increase in the measured
MSE. By adapting an interpolation technique as suggest by
[2] we can increase clarity of the depth map and increase
the accuracy.
Implementing a mutual information based initial cost
function can also further increase the accuracy of the depth
maps. We experimented with a basic mutual information
approach, but were unable to produce reasonable outputs.
We believe this is due to the need of multiple iterations for
the depth map to converge. Future work would also
explorer further the concept of mutual information, and
finding a way to incorporate it into stereo matching with as
little iterations as possible.
The results from this project further indicate that Semi-
Global Matching is a robust approach to estimating depth
information from a scene. Semi-Global Matching well
balances the tradeoffs of both speed and accuracy, making
it a strong contender for use in new technologies.
Link to code:
https://github.com/rmahieu/SemiGlobalMatching
References
[1] Y. Boykov, O. Veksler, and R. Zabih. Efficient
approximate energy minimization via graph cuts.
IEEE Transactions on Pattern Analysis and Machine
Intelligence, 23(11):1222–1239, 2001.
[2] H. Hirschmuller, "Accurate and efficient stereo
processing by semi-global matching and mutual
information," 2005 IEEE Computer Society
Conference on Computer Vision and Pattern
Recognition (CVPR'05), 2005, pp. 807-814 vol. 2.
[3] H. Hirschmüller and D. Scharstein. Evaluation of cost
functions for stereo matching. In IEEE Computer
Society Conference on Computer Vision and Pattern
Recognition (CVPR 2007), Minneapolis, MN, June
2007.
[4] Michael, M., Salmen, J., Stallkamp, J., & Schlipsing,
M. (2013, June). Real-time stereo vision: Optimizing
semi-global matching. In Intelligent Vehicles
Symposium (IV), 2013 IEEE (pp. 1197-1202). IEEE.
[5] G. Van Meerbergen, M. Vergauwen, M. Pollefeys, and
L. Van Gool. A hierarchical symmetric stereo
algorithm using dynamic programming. International
Journal of Computer Vision, 47(1/2/3):275–285,
April-June 2002.
[6] D. Scharstein and R. Szeliski. High-accuracy stereo
depth maps using structured light. In IEEE Computer
Society Conference on Computer Vision and Pattern
Recognition (CVPR 2003), volume 1, pages 195-202,
Madison, WI, June 2003.
[7] D. Scharstein and R. Szeliski, “A taxonomy and
evaluation of dense two-frame stereo correspondence
algorithms,” International Journal of Computer
Vision, vol. 47, pp. 7–42, 2002.
[8] D. Scharstein and C. Pal. Learning conditional random
fields for stereo. In IEEE Computer Society
Conference on Computer Vision and Pattern
Recognition (CVPR 2007), Minneapolis, MN, June
2007.
[9] J. Sun, H. Y. Shum, and N. N. Zheng. Stereo matching
using belief propagation. IEEE Transactions on
Pattern Analysis and Machine Intelligence,
25(7):787–800, July 2003
Appendix:
CS231A FINAL PROJECT, JUNE 2016 1

Solving Large Jigsaw Puzzles

L. Dery and C. Fufa

Abstract—This project attempts to reproduce the genetic algorithm in a paper entitled ”A Genetic Algorithm-Based
Solver for Very Large Puzzles” by D. Sholomon, O. David, and N. Netanyahu. [3] There are two main challenges in
solving jigsaw puzzles. The first is finding the right fitness function to judge the compatibility of two pieces. This has
thoroughly been studied and as a result, there are many fitness functions available. This paper explores the second part
that is crucial to solving jigsaw puzzles: finding an efficient and accurate way to place the pieces. The genetic algorithm
attempts to do just that. The crucial part of the algorithm is in generating a new ordering of pieces called ’child’ from
two possible orderings of pieces, called ’parents’. Each generation learns from good traits in the parents. After going
through a hundred generations, the ordering will reflect the original image to a high accuracy. This paper also makes
use of CNN to start with reasonable orderings of ’parents’. This cuts down on the number of generations required to
reach the correct ordering of the pieces.

Index Terms—CS231A, Jigsaw Puzzles Algorithms

1 I NTRODUCTION DNA/RNA modeling, image base CAPTCHA

T HE problem of automating the solving of
jigsaw puzzles is one that has been around
since at least the 1950s. Jigsaw puzzles are im-
age reconstruction problems where the image
provided has been cut into non overlapping
boxes and shuffled around. The problem is
then to reconstruct the original image from the
shuffled pieces. For the problem to be tractable,
the puzzle pieces are assumed to be of identical
dimensions and that no piece has been rotated.
The
problem has multiple applications, both in and
outside of image reconstruction. Puzzle solu-
tion techniques can be applied to broken tiles
to simulate the reconstruction of archaeological
artifacts. In fall, 2011, DARPA held a compe-
tition, with a fifty thousand dollar prize, to
automatically reconstruct a collection of shred-
ded documents.Other applications include the
molecular docking problem for drug design,
construction, and speech descrambling.
2 R EVIEW OF PREVIOUS WORK
2.1 Previous Work
In 1964, Freeman and Garder[1] proposed a
solution for a 9 piece problem. The shapes were
allowed to be of different dimensions. After
Freeman and Garder, most of the work has
been based on color-based solvers, with the
assumption that all pieces are rectangles of the
same dimension. Recently, probabilistic puzzle
solvers have been developed[2]. These algo-
rithms were solving 432 pieces. These solvers
however require apriori knowledge of the puz-
zle. There are also particle filter-based solvers
which are improvements over the probabilistic
puzzle solvers.
In 2013 Sholomon et Al[3] introduced a ge-
netic algorithm based technique for solving
large jigsaw puzzles. It is our goal in this paper
to replicate the results of this paper and also
suggest areas where it could be improved.
2.2 This paper’s contribution
This paper uses the genetic algorithm as a
strategy for piece placement. It uses a stan-
dard estimation function. While this is not the
CS231A FINAL PROJECT, JUNE 2016
first time that the genetic algorithm has been
used to solve the jigsaw puzzle problem, it has
only been used to solve puzzles of a limited
size. This paper attempts to solve puzzles with
larger pieces. In addition to the genetic algo-
rithm, this paper also attempts to use CNN to
arrive at the correct reconstruction of the image
in less iterations.
3 T ECHNICAL D ETAILS
3.1 Genetic Algorithm
The genetic algorithm as implemented for solv-
ing the jigsaw puzzle problems starts out with
a thousand different ways to order the pieces.
Each way of ordering a piece is called a chro-
mosome. The entire set of a thousand chro-
mosomes is called a population. At each stage
of the process, called a generation, we have a
population of a thousand chromosomes. Now,
the goal is that with each passing generation,
i.e. with the next thousand chromosomes or
a population, the orderings of the pieces will
begin to look more and more like the original
or correct image. During each population, the
best chromosome will be determined by the
estimation function.
Above is the higher level pseudo code for the
Genetic Algorithm framework. The four best
scorers according to the estimation or fitness
function will automatically be placed into the The fitness function for a given chromosome
next generation. The rest of the chromosomes will compute the sum of the score for every
for the next generation are going to be hy- edge in the chromosome. Below are examples
brids of chromosomes from the current one. of functions which compute the compatibility
2
Two chromosomes from the current population
are selected, and a function called crossover
generates a child chromosome that learns from
the parents, and has a better reordering of the
pieces, and hence, a better fitness score. It is
via this mechanism that each generation gets
a better fitness score than the previous genera-
tion. The selection process of which parents to
choose to give birth to a new child chromosome
discriminates towards parents with a better
fitness score. The selection process is called
a roulette selection. The likelihood of being
selected is directly proportional to how good
the fitness score is. This way, the algorithm
makes sure that selected parent chromosomes
have good traits (as evidenced by their fitness
scores) to be passed on to the children.
Fitness Function
The estimation function utilizes the fact that
adjacent pieces in the original image will most
likely share similar colors along their edges.
Hence, computing the sum of the squared color
differences along pixels that are adjacent to
each other (between two different pieces) will
give us an indication of whether the two pieces
belong adjacently in the direction they shared
the pixels. Hence, the less the specific sum is ,
the more likely they are to be adjacent to each
other. From the image below for example, we
can expect the fitness function to give us a high
score for piece 5 and 6 as the color difference in
the edges seem to be high, while piece 8 and
9 will have a very low fitness score. We can
further assume that piece 5 and 8 will have a
high fitness score while 6 and 9 will have a
lower one in comparison.
CS231A FINAL PROJECT, JUNE 2016 3

score of two pieces in a left-right adjacency are similar to that of the parents. The process
relationship, and a function which computes of growing the kernel will go on until all the
the fitness score of a given chromosome (i.e. pieces have been used.
computes the score for all edges and direc- The final absolute location of a given piece
tions.) is only determined after all the pieces have
been used. This is because as recommended
earlier, the kernel growing process must allow
for independence or flexibility in the placement
as the algorithm plays out. To begin, crossover
K is the number of pixels in each piece in the selects a random piece from either parent and
vertical direction. places it in the kernel. After that, it keeps
track of all the available boundaries for a new
piece to be added to the kernel. An available
boundary can be thought of as a piece and the
This way, it covers all the available edges direction in which a new piece can be placed
in a chromosome. Note that D is the fitness adjacent to it. There are three main phases
score for the compatibility of piece xj to the involved in crossover.
direction (left, right, down, or up) of xi. In
the selection process of the algorithm (roulette Phase One
selection) make sure that a lower fitness score It goes through the boundary pieces in the
is treated as more likely to be chosen. kernel. Let’s say that piece xi in the direction
d, for example is selected. Phase one checks to
Crossover see if both parents have the same piece xj in
the direction of d of xi . If it so, xi is added
Crossover can be considered the heart of the to the kernel. If x has been already added,
i
algorithm. Crossover receives two parent chro- it will of course be skipped. The only pieces
mosomes and creates a child chromosome. It under consideration should be unused pieces
allows ”good traits” to be transmitted from (pieces not in the kernel). This phase keeps on
the parents to the child. The goal is to have going until there is no boundary on which both
the child with a better fitness score than both parents agree.
parents. The fitness function does a good job
of discriminating between adjacent pieces, but Phase Two
does not give any indication of whether the
pieces are placed at the correct absolute po- Assume (xi ,R) is available on the kernel. Check
sition in the image. The implementation of if one of the parents contains a piece xj in
crossover then must allow for independence spatial relation R of xj , which is also a best-
in the placement of pieces. (It should be a buddy of xi in that relation. Two pieces xi and
dynamic process. Just because a piece was at xj are considered best-buddies if D(xi ,xj ,R) is
some point assigned to say, (2,3) of the the the lowest fitness score they can achieve.I.e.
image, it must not remain there, it should be there is no better piece xk , that will give a
able to transition into a different place based lower fitness score D(xk ,xj ,R) as well as no xk
on how the pieces build up around it.) available, that can give D(xi ,xk , R) lower than
The implementation of crossover suggested D(xi ,xj ,R). The piece considered xj must be
starts out with a single piece and then grad- adjacent to xi in one of the parents.
ually joins other pieces at available bound- If such a piece is found, go back to phase
aries.The image is always contiguous since one, if not, proceed to three
new pieces are only added adjacent to existing
ones. Keeping track of the pieces used and Phase Three
the dimensions of the child being formed is Pick random (xi,R) from the kernel and assign
important so that the dimensions of the child it xj from available pieces such that D(xi ,xj ,R)
CS231A FINAL PROJECT, JUNE 2016
is lowest. Go to Phase One. The three phases
keep on going until all the pieces are used
up. Mutation is introduced in phases one and
three. With a 5 percent probability, a random
available piece is assigned as opposed to one
that both parents agree on in phase one and
the most compatible on e in phase three.
3.2 Convolutional Neural Network Augmen-
tation
The Algorithm proposed as is, always re-
quires 1000 randomly initiating chromosomes.
As an extension to the Algorithm proposed by
Sholomon et al[1], we decided to try to influ-
ence the starting state of the Genetic Algorithm.
The rationale is that if the the genetic algorithm
starts out with chromosomes that are already
quite good, then convergence would be faster
and the number of generations required for
the algorithm could be reduced. We decided to
train a convolutional Neural Network to solve
the Jigsaw task and then use its output as input
to the Genetic Algorithm.
4
Problem Formulation
The input to the neural network is an image
whose color channel are made up the jigsaw
pieces stacked side by side. The task of the
network was to predict the order in which the
pieces were stacked together, thus assigning
each piece its right position in the original
image. To clarify, say we have a 3x3 puzzle. The
pieces are numbered 1 to 9 according to their
position in the original image. They are then
stacked in a random configuration along the
color channel. It is then the task of the network
to predict the configuration in which they were
stacked. We cast the problem as a classification
problem. Since the configurations space is re-
ally large, 9 pieces produce 9! = 362990 possi-
bilities, it would be near impossible, given the
computing resources at our disposal, to have
9! classes. As such, we decided to reduce the
problem to the following. We keep track of 100
classes representing 100 randomly generated
configurations. The objective of the network is
now to predict the configuration (1 - 100) that
has the closest Hamming Distance to the actual
configuration of the given image. This work
around made the solution space of the problem
tractable.
Network Architecture and Implementation
Details
The diagram below shows the network
structure. Our implementation of the
network was done in TensorFlow. We used
a softmax cross entropy loss function. A
hyper parameter search lead us to Adam
Optimizer with a learning rate of 10−3 and
a batch size of 128. We had to normalize
the image channels to 1 by dividing by 255.
CS231A FINAL PROJECT, JUNE 2016 5

4.2 Solving Jigsaws via Genetic Algorithm

3.3 Evaluation Metric
There are two major metrics used to evaluate a
jigsaw reconstruction. There is the direct com-
parison which measures the fraction of pieces
located in their correct absolute location, and
there is neighbor comparison, which measures
the fraction of correct neighbors. The direct
method has been shown [1] to be less accurate
and less meaningful since it cannot handle
slightly shifted cases. We thus used the neigh-
bor comparison as our evaluation metric.
4 E XPERIMENTS
4.1 Convolutional Neural Network
The table below shows the run time and ac-
curacy results for difference puzzle piece sizes
averaged over 10 runs. As can be seen from the
table, we were able to achieve results compa-
rable to those of the original paper in terms of
the accuracy of rearrangement of jigsaw pieces
and the fitness score of the result returned.
Our reconstructions, though not always as ac-
curate as the original by the neighbor accuracy
metric described, produced a reconstructions
whose fitness score were equal to the fitness
score of the original image.This suggests that
the algorithm sometimes gets stuck in a local
minimum whose fitness score is the same as
the un-jumbled image’s score.
Figure 1. Algorithm Performances on Different
Piece Numbers

Given that we cast our solution space for the

network into 100 classes, we set our baseline to
be a validation accuracy of 0.01 corresponding
to random guessing out of the 100 classes.
Our convolutional neural network, was
able to achieve validation accuracy of 0.022
which is more than twice our baseline. This
performance is quite impressive considering
that what the network is actually trying to
achieve is predicting a configuration space of
9! that is being represented by 100 classes. The
more puzzles provided to the network to solve,
the better it got as the plot below suggests.
CS231A FINAL PROJECT, JUNE 2016
In the domain of run time however, we were
unable to match the original paper’s results.
The paper describes solving a 432 piece puzzle
in 43.63 seconds, however our implementation
takes around 2 hours to run on a puzzle of
the same size. We believe this huge difference
is due to differences in the specifics of the
implementation of the crossover function.
Figure 2. 96 PIECES:GENERATION 1
Left: Best Reconstruction so far. Right: Sec- run time. Though the difference is 1 second
ond Best
Figure 3. 96 PIECES:GENERATION 100
Left: Actual Image. Right: Reconstructed Im-
age.
4.3 Genetic Algorithm + Convolutional
Neural Network (CNN)
As an augmentation to the original algorithm,
we fed the reconstruction output of the CNN
as the starting population of the Genetic
Algorithm. We were mainly interested in two
effects
1.Did the run time of the algorithm improve?
2.How did the accuracy of the reconstruction
change?
6
The table above contrasts the performance of
the CNN augmentation in different regimes
of training data size with the pure Genetic
Algorithm performance on the 3x3 Jigsaw. In
general, the augmented algorithm has a better
for this regime of 3x3 puzzles, it is easy to
see how this gain could be more significant
as the puzzle size increases. In general, the
reconstructed puzzle from the augmented
model are not as accurate as the pure Genetic
Algorithm. However, the reconstructions
always had the same minimal fitness score
as the original, meaning that it is finding a
good minimum, even if this is not the original
reconstruction.
5 C ONCLUSION
The Jigsaw puzzle problem is an interesting
problem with applications in many domains.
Looking forward, one extension we plan to
explore is to solve the jigsaw problem using
only a neural network. We envision embed-
ding convolutional layers in a Long Short Term
Memory or Recurrent Neural Network which
would directly predict the right configurations
instead of using our current trick of having 100
representative configurations. We would also
like to investigate more avenues for improving
the run time of our current model.
R EFERENCES
[1] H. Freeman and L. Garder. Apictorial jigsaw puzzles:
The computer solution of a problem in pattern recog-
nition. IEEE Transactions on Electronic Computers, EC-
13(2):118127, 196
CS231A FINAL PROJECT, JUNE 2016 7

[2] T. Cho, S. Avidan, and W. Freeman. A probabilistic image

jigsaw puzzle solver. In IEEE Conference on Computer Vi-
sion and Pattern Recognition, pages 183190, 2010.
[3] D. Sholomon, O. E. David, and N. S. Netanyahu. A genetic
algorithm-based solver for very large jigsaw puzzles. In
IEEE Conference on Computer Vision and Pattern Recog-
nition, pages 17671774, 2013.
[4] D. Pomeranz, M. Shemesh, and O. Ben-Shahar. A fully
automated greedy square jigsaw puzzle solver. In IEEE
Con- ference on Computer Vision and Pattern Recognition,
pages 916, 2011
 Video Stabilization and Face Saliency-based Retargeting
Yinglan Ma1 , Qian Lin2 , Hongyu Xiong2
1
Department of Computer Science, Stanford University 2 Department of Applied Physics, Stanford University
Abstract
Technology revolution has brought great convenience of
daily life recording using cellphones and wearable devices
nowadays. However, hand shake and human body move-
ment is likely to happen during the capture period, which
significantly degrades the video quality. In this work, we
study and implement an algorithm that automatically stabi-
lizes the shaky videos. We first calculate the video motion
path using feature matching and then smooth out high fre-
quency undesired jitters with L1 optimization. The method
ensures that the smoothed paths only compose of constant,
linear and parabolic segments, mimicking the camera mo-
tions employed by professional cinematographers. Since
the human face are of broad interest and appear in large
amount of videos, we further incorporated face feature de-
tection module for video retargeting purposes. The detected
faces in the video also enables many potential applications,
and we add decoration features in this work, e.g., glasses
and hats on the faces.
1. Introduction
Nowadays nearly 2 billion people own smartphones
worldwide, and an increasing number of videos are cap-
tured by mobile devices. However, videos captured by hand
handled devices are always shaky and undirected due to the
lack of stabilization equipment on the handhold devices.
Even though there are commercial hardware components
that could stabilize the device when we record, they are rel-
atively redundant and not handy for daily use. Moreover,
most hardware stabilization systems only removes high fre-
quencies jitters but are unable to remove low frequency mo-
tions arise from panning shots or walking movements. Such
slow motion is particular problematic in shots that intend to
track prominent foreground object or person.
To overcome the above difficulties, we implement a
post-processing video stabilization pipeline aiming to re-
move undesirable high and low frequency motions from
casually captured videos. Similar to most post-processing
video stabilization algorithms, our implementation involves
three main steps: (1) estimate original shaky camera path
1
from feature tracking in the video; (2) calculate a smoothed
path, which is cast as an constraint optimization problem;
(3) Synthesizing the stabilized video using the calculated
smooth camera path. To reduce high frequency noise, we
use the L1 path optimization method described in [1] to
produce purely constant, linear or parabolic segments of
smoothed motion, which follows cinematographic rules. To
reduce low frequency swanning in videos containing a per-
son as the central object, we apply further restraint to the
motion of the facial features. In order to make the solution
approachable, our method uses automatic feature detection
and do not require user interaction.
Our video stabilization method is a purely software ap-
proach, and can be applied to videos from any camera de-
vices and sources. Another popular class of mobile video
stabilization methods use the phone’s build-in gyroscope to
measure the camera path. Our method has the advantage of
being applicable to any video from any sources, for example
online video, without any prior knowledge of the capturing
device or other physical parameters of the scene. Our ap-
proach also enables facial retargeting, which can be extent
to other kinds of salient features.
2. Previous Work
2.1. Literatures
Video stabilization methods can be categorized into three
major directions: 2D method, 3D method and motion esti-
mation method.
2D methods estimate frame-to-frame 2D transforma-
tions, and smooth the transformations to create a more sta-
ble camera path. Early work by Matsushita et al. [5] applied
low-path filters to smooth the camera trajectories. Gleicher
and Liu [4] proposed to create a smooth camera path by
inserting linearly interpolated frames. Liu et al.[6] later in-
corporated subspace constraints in smoothing camera tra-
jectories, but it required longer feature tracks.
3D methods rely on feature tracking to stabilize shaky
videos. Beuhler et al. [8] utilized projective 3D reconstruc-
tion to stabilize videos from uncalibrated cameras. Liu et
al. [9] were the first to introduce content-preserving warp-
ing in video stabilization. However, 3D reconstruction is
difficult and unrobust. Liu et al. [6] reduced the problem
to smoothing long feature trajectories, and achieved com-
parable results to 3D reconstruction based methods. Gold-
stein and Fattal[10] proposed an epipolar transfer method
to avoid direct 3D reconstruction. Obtaining long feature
tracks is often fragile in consumer videos due to occlusion,
rapid camera motion and motion blur. Lee et al. [11] incor-
porated feature pruning to select more robust feature trajec-
tories to resolve the occlusion issue.
Motion estimation methods calculate transitions between
consecutive frames with view-overlap. To reduce the align-
ment error due to parallax, Shum and Szeliski[12] im-
posed local alignment, and Gao et al.[7] introduced a dual-
homography model. Liu et al[13] proposed a mesh-based,
spatially-variant homography model to represent the motion
between video frames, but the smoothing strategy did not
follow cinematographic rules.
Our implementation, based on [1], apply L1 -norm op-
timization to generate a camera path that consists of only
constant, linear and parabolic segments, which follow cine-
matographic principles in producing professional videos.
2.2. Our Contribution
In this work, we re-implement the L1 -norm optimization
algorithm [1] to automatically stabilize the videos captured,
with a smoothed feature path containing only constant, lin-
ear and parabolic segments. Additionally, in order to en-
able the video to retarget on human faces, we use the facial
landmark detection algorithm from OpenFace toolkit [3] to
set facial saliency constraints for the path smoothing; the
strength of the constraint could be tuned from 0 (no facial
retargeting) to 1 (video fixing on facial features), and in this
way we are able to combine both video path smoothing and
facial retargeting according to specific user needs.
Beyond that, in order to make our work more fun, we
also manage to attach interesting decorations such as hat,
glasses, and tie above, on, or below the human faces de-
tected, and their transformations are based on the movement
of human face in the video.
3. Proposed Method
3.1. L1 -Norm Optimized Video Stablization
In this section, we describe the method of video stabliza-
tion in this work.
3.1.1 Norms of smoothing
When applying path smoothing algorithm, we should al-
ways be careful to choose which regularization method we
use, since different regularization methods works differ-
ently for different error distribution. [2]
For error distributions with sharply defined edge or ex-
tremes (typified by the uniform distribution) one should
2
use Tchebycheff (L1 ) smoothing. For error distributions
at the other end of the spectrum, which is with long tails,
one should use L1 smoothing. In between these extremes,
which are short-tail spectra such as normal distribution,
least squares or L2 smoothing appears to be best.
3.1.2 L1 -Norm Optimization
In the perspective of a single feature point, the video mo-
tion can be viewed as a path of its coordinates (x, y) move-
ment with respect to the frame number. Since it is diffi-
cult to avoid jitters with hand-held devices, we will observe
that the path the is wiggling. Video stablization is to ob-
tain the new coordinates (x, y) at each frame and thus a
new path with enhanced smoothness. In the perspective of
the frames, the task is to smooth the transformations be-
tween frames so that the feature points movement would
be minimal. The frame transformation is generalized as
affine transform, including translational and rotational mo-
tion, and scaling caused by object/camera distance change.
We estimate the camera path by first matching features
between consecutive frames Ct and Ct+1 , and then cal-
culate the affine transformation Ft+1 based on the match-
ing. That is, the process can be formatted as Ct+1 =
Ft+1 Ct . Then we estimate the affine transformation Ft+1
using these two set of feature coordinates, Ct and Ct+1 . In
this work, we extract features of each frame (opencv func-
tion cv::goodFeaturesToTrack), and find the matching in the
next frame using iterative Lucas-Kanade method with pyra-
mids (cv::calcOpticalFlowPyrLK).
We denote the smoothed features as Pt , then we have a
correlation between the original features in frame t and the
smoothed ones, as Pt = Bt Ct , where Bt is is the stabiliza-
tion/retargeting matrix, transforming the original features to
the smoothed ones. Since we only want the smoothed path
to contain constant, linear, and parabolic segments, we min-
imize the first, second, and third derivatives of the smoothed
path with weights c = (c1 , c2 , c3 )T :
O(P ) = c1 |D(P )|1 + c2 |D2 (P )|1 + c3 |D3 (P )|1 , (1)
where
X X
|D(P )|1 = |Pt+1 Pt | 1 = |Rt |1 ,
t t
X
|D2 (P )|1 = |Rt+1 Rt | 1 , (2)
t
X
|D3 (P )|1 = |Rt+2 2Rt+1 + Rt |1 .
t
Here the residual is Rt = Bt+1 Ft+1 Bt .
For each affine transform:

b b t
Bt = 11 12 x (3)
b21 b22 ty
in 6 DOF, we vectorize it as pt = We use Constrained Local Neural Fields (CLNF) for fa-
(b11 , b12 , b21 , b22 , tx , ty )T , which is the parametriza-
tion of Bt ; correspondingly
|Rt (p)|1 = |pTt+1 M (Ft+1 ) pt | 1 . (4)
We make use of Linear Programming (LP) technique
to solve this L1 -norm optimization problem. To minimize
|Rt (p)|1 in LP, we introduce slack variables e1 0, so
that e1  Rt (p)  e1 ; similarly there are e2 and e3 for
|Rt+1 (p) Rt (p)|1 and |Rt+2 (p) 2Rt+1 (p) + Rt (p)|1 ,
respectively. For e = (e1 , e2 , e3 )T , the objective function
of the problem is to minimize cT e.
In addition, we want to limit how much Bt (or pt ) could
deviate from the original path, i.e. the actual shift should
within the cropping window. Thus, we can add constraints
on the parameters in LP, such as: lb  U pt  ub, where U
is the linear combination coefficient of pt . The complete L1
minimization LP for smoothed video path with constraints
is summarized below:
Algorithm 1 Summarized LP for the smoothed video path
Input: Frame pair transform Ft , t = 1, 2, ..., n
Output: Update transform Bt
. Bt could be transformed to pt
Minimize: cT e
w.r.t p = (p1 , p2 , ..., pn )
where e = (e1 , e2 , e3 )T , ei = (ei1 , ei2 , ..., ein ), c =
(c1 , c2 , c3 )T
subject to:
1. e1t  Rt (p)  e1t
2. e2t  Rt+1 (p) Rt (p)  e2t
3. e3t  Rt+2 (p) 2Rt+1 (p) + Rt (p)  e3t
4. eit 0
constraints:
lb  U pt  ub
We use lpsolve library for modeling and solving our LP
system.
3.2. Facial Features Detection and Retargeting
In many videos, a particular subject, usually a person, is
featured. In this case it is not only important to remove fast,
jittering camera motions, but also unintended slow panning
or swanning that momentarily move the subject off-center
and lead to distraction for the viewer. This can be posed
as a constraint on the path optimization as requiring that
salient features of the subject to be closed to the center re-
gion throughout the video.
The first step towards salient-point-preserving video sta-
bilization is salient feature detection and tracking. In par-
ticular, it is desirable to have the algorithm automatically
recognize and detect these salient features without user in-
put. There are many face detectors available for such task.
3
cial landmark detection available on OpenFace. Detail of
the algorithm can be found in [3]. The CLNF algorithm
works robustly under varied illumination and are stabilized
for video. It outputs a fixed number of facial landmarks,
including the face silhouette, the lips, nose tip and eyes, as
shown in Fig. 2c. These multiple landmarks allow a more
stable and accurate estimate of the facial position. In con-
trary, other face detector, for example the opencv built-in
ones, were observed to produce inaccurate bounding box
and are not stable over video frames during our experiment.
The detailed facial landmarks from CLNF also enable us to
perform other post-processing on the video, for example the
face decoration described in Section 3.4.
After detecting the facial landmarks in each frame t, we
estimate the center of face Cf,t by averaging all the land-
marks. Let C0 be the desired position of the center of face,
for example the center of frame. Let Pt and St be the orig-
inal and smoothed camera trajectory, then the saliency con-
straint can be posed as a additional term to the loss function
Lt = (1 ws )(St P̄t )2 + ws (St Pt + Cf,t C0 )2 (5)
where P̄t is average over a window of frames, and ws is a
parameter to adjust how much weight the saliecy constraint
have on the optimization. Minimizing Lt then produce the
desired smoothed trajectory St .
3.3. Metrics & Characterization
3.3.1 Evaluation of Smoothed Path
For the stabilizing problem we are concerning about, it
would be inappropriate to simply regard the undesired shak-
ing as short-tail normal distribution, so using the L1 norm
between each frame pair during minimization is more suit-
able. In addition, L1 optimization has the property that the
resulting solution is sparse, i.e. the computed path there-
fore has derivatives which are exactly zero for most seg-
ments. On the other hand, L2 minimization (in a least-
squared sense), tend to result in small but non-zero gradi-
ents. Qualitatively, the L2 optimized camera path always
has some small non-zero motion (most likely in the direc-
tion of the camera shake), while the L1 optimized we used
(|D(P )|1 , |D2 (P )|1 , and|D3 (P )|1 ) will create path is only
composed of segments resembling a static camera, (uni-
form) linear motion, and constant acceleration [1].
Therefore, we will compare the L1 norm |D(P )|1 be-
tween the original video feature path and the smoothed one,
and use this comparison as metrics of our experiments de-
scribed below. Specifically, we will calculate the average
absolute shift between adjacent points on the video feature
path, with respect to both x and y directions, and average
absolute rotation angle increment. The same calculations
will be done to the smoothed path
3.3.2 Evaluation of Facial Retargeting
As for the part of facial retargeting, in addition to the com-
parison between the L1 norm |D(P )|1 of the original video
feature path and the new one, where we can extract the in-
formation about smoothing, we are also interested to see
how the facial features are targeted. So we will calculate
the average position of the face features with respect to the
center of frame, and simultaneously calculate the average
absolute position deviation.
3.4. Face Decoration
With per frame face features detected, we can add fun
face decorations to our videos, such as glasses, hat and mus-
tache. By incorporating feature locations, we are able to
translate, scale and rotate the decorations to place them ap-
propriately onto human faces. Since our videos are stabi-
lized and focus on faces, the transitions of the decorations
are smoother. Here is an example of how we utilize the fea-
ture points in adding decorations.
Adding glasses: we extract left eye, right eye, left brow
and right brow feature points to calculate a horizontal eye
axis, and use it to estimate the orientation of the glasses.
Scale is approximated from eye distance, and translation de-
pends on the locations of the eye points.
Since face silhouette feature points are usually less sta-
ble, we avoid using those points in adding face decorations.
Screenshots of adding hat and glasses are shown in Fig-
ure 4.
4. Experiments
Table 1 lists the algorithm run time on our laptop. The
second column lists time for path smoothing without facial
feature, and the third column lists time for path smooth-
ing with facial feature as salient constraint. In the lat-
ter case, the CNFL facial landmark detection takes up the
biggest chunk of time (⇠ 45ms per frame). [1] reported 20
fps on low-resolution video, and 10 fps with un-optimized
saliency.
Table 1. Timing per frame of the algorithm. Video resolution
640 ⇥ 360.
w.o. face w. face
motion estimation (ms) 12.1 59.1
optimize camera path (µs) 0.15 0.40
render final result (µs) 2.7 2.7
face decoration (ms) - 5.7
total (ms) 15 68
speed (fps) 67 15
4
4.1. Video Stabilization
We apply our path smoothing algorithm to shaky videos
and observe significant reduction of jittering. An example
output can be found on Youtube.
To visualize the effect of stabilization, we plot the esti-
mated camera trajectory before and after our algorithm in
Fig. 1. We also provide a quantitative measurement of the
L1 norm |D(P )|1 before and after smoothing in Table. 1.
As we can see the L1 norm decreases a lot, which means
the abrupt jitters are significantly decreased.
Figure 1. Path before and after (Left column) L2 -norm smooth-
ing (Right column) L1 -norm smoothing. (Top)x-direction.
(Middle)y-direction. (Bottom)rotational angle.
Table 2. L1 norm |D(P )|1 between the original video feature path
and the smoothed one, in both x and y directions and the rotational
angle.
path < | xt | > < | yt | > < | at | >
original 1569 857 1.12
smoothed 705 234 0.44
4.2. Facial Retargeting
Our experiment with video stabilization using facial fea-
tures are shown in Fig. 2. Fig. 2(a) is the original video,
which contains slow swanning motion of both the camera
and the subject person. Fig. 2(b) is the stabilized output us-
ing only camera path smoothing. The slow motion of the
subject is still prominent. Fig. 2(c) is the stabilized output
using camera path smoothing with a constraint of the mo-
tion of facial features. It leads to stabilization of the subject
at the center over frames. Both result videos can be found
on Youtube link 1 and link 2.
As expected, stabilization comes at a price of reduced
resolution. The original image are cropped by 20% in
Fig. 2(b) and (c) to remove black margins due to warpping.
There are still residue margins in Fig. 2(c).
 We also quantify the smoothing effect and the facial tar-
geting, as we can see from Table. 2. With the increase of the
facial saliency constraint ratio !, both L1 norm and the ab-
solute position shift drops, which means, the larger ! is, the
smoothier the video gets, and the more centered the human
face is. The result is expected from our algorithm.
4.3. Comparison with State-of-the-art Systems
Due to no publicly available implementation of previous
works, we obtain the original and output videos reported in
Grundmann’s paper [1], and calculate the evaluation metrics
described in Section 3.3 on their output video and present
alongside with our results. As we can see from the compari-
son below, our implemented algorithm is comparable to the
state-of-the-art system.
4.4. Face Decoration
With per frame face features detected, we can add fun
face decorations to our videos, such as glasses, hat and mus-
tache. By incorporating feature locations, we are able to
translate, scale and rotate the decorations to place them ap-
propriately onto human faces. Since our videos are stabi-
lized and focus on faces, the transitions of the decorations
are smoother. Screenshots of adding hat and glasses are
shown in Fig. 4.
5. Conclusion & Perspectives
All in all, video feature path is significantly smoothed
using the L1 optimization stabilization algorithm; the L1
norm |D(P )|1 , which signifies the moving between frames,
greatly drops after applying the stabilization.
If the facial retargeting method is included, the video
would be more focused on human faces; the larger the
saliency constraint ratio ! is, the more centered the human
faces are with respect to the cropped video frame.
Decoration addition such as glasses, hat, or tie could also
be attached to the faces in the video, with the same orien-
tation as the faces. More fun stuffs will be applied to make
this work fancier in the future.
Reference
[1] Matthias Grundmann, Vivek Kwatra, Irfan Essa.
Auto-Directed Video Stabilization with Robust L1 Optimal
Camera Paths. CVPR, 2011.
[2] JR Rice, JS White. Norms for smoothing and estima-
tion. SIAM review, 1964.
[3] Tadas Baltruaitis, Peter Robinson, and Louis-
Philippe Morency. Constrained Local Neural Fields for ro-
bust facial landmark detection in the wild. ICCVW, 2013.
[4] Michael L. Gleicher and Feng Liu. 2007. Re-
cinematography: improving the camera dynamics of casual
5
video. In Proceedings of the 15th ACM international con-
ference on Multimedia (MM ’07). ACM, New York, NY,
USA, 27-36.
[5] Yasuyuki Matsushita, Eyal Ofek, Weina Ge, Xiaoou
Tang, and Heung-Yeung Shum. 2006. Full-Frame Video
Stabilization with Motion Inpainting. IEEE Trans. Pattern
Anal. Mach. Intell. 28, 7 (July 2006), 1150-1163.
[6] F. Liu, M. Gleicher, J. Wang, H. Jin, and A. Agar-
wala. Subspace video stabilization. In ACM Transactions
on Graphics, volume 30, 2011.
[7] Junhong Gao , Seon Joo Kim , M. S. Brown, Con-
structing image panoramas using dual-homography warp-
ing, Proceedings of the 2011 IEEE Conference on Com-
puter Vision and Pattern Recognition, p.49-56, June 20-25,
2011
[8] Buehler, C., Bosse, M., and McMillan, L. 2001. Non-
metric image-based rendering for video stabilization. In
Proc. CVPR.
[9] Feng Liu , Michael Gleicher , Hailin Jin , Aseem
Agarwala, Content-preserving warps for 3D video stabiliza-
tion, ACM Transactions on Graphics (TOG), v.28 n.3, Au-
gust 2009
[10] Amit Goldstein , Raanan Fattal, Video stabilization
using epipolar geometry, ACM Transactions on Graphics
(TOG), v.31 n.5, p.1-10, August 2012
[11] Chen, B.-Y., Lee, K.-Y., Huang, W.-T., and Lin, J.-
S. 2008. Capturing intention-based full-frame video stabi-
lization. Computer Graphics Forum 27, 7, 1805–1814.
[12] Heung-Yeung Shum , Richard Szeliski, Systems
and Experiment Paper: Construction of Panoramic Image
Mosaics with Global and Local Alignment, International
Journal of Computer Vision, v.36 n.2, p.101-130, Feb. 2000
[13] Shuaicheng Liu, Lu Yuan, Ping Tan, and Jian Sun.
2013. Bundled camera paths for video stabilization. ACM
Trans. Graph. 32, 4, Article 78 (July 2013), 10 pages.
Figure 2. Demonstration of facial retargeting in video stabilization. The green dot indicates the center of frame. Green lines show boarder
of frame. Red dots in (c) indicated detected facial landmarks from OpenFace [3]. They are intended as a guide to the eye. Both videos can
be found on Youtube (b) and (c).

Table 3. L1 norm |D(P )|1 between the original video feature path and the smoothed one, in both x and y directions and the rotational
angle.

! < | xt | > < | yt | > < |x xcenter | > < |y ycenter | >
original 1392 496 32805 5882
0.2 1139 254 32583 4902
0.5 792 234 21568 3433
0.95 221 247 2695 1954

6
Figure 3. Path smoothing before and after with facial saliency con-
straints. (Left column) x-direction. (Right column) y-direction.
From top to bottom, the facial constraint ratios ! are 0.2, 0.5, and
0.95, respectively.

Table 4. Comparison between our algorithm and the state-of-the-

art one from [1]

method < | xt | > < | yt | > < | at | >

state-of-the-art [1] 273 296 0.53693
our algorithm 705 234 0.44387

Figure 4. Face decoration with glasses and hat