IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO.
11, NOVEMBER 2016
Ranking Highlights in Personal Videos by
Analyzing Edited Videos
Min Sun, Ali Farhadi, Tseng-Hung Chen, and Steve Seitz, Fellow, IEEE
Abstract We present a fully automatic system for ranking
domain-specific highlights in unconstrained personal videos by
analyzing online edited videos. A novel latent linear ranking
model is proposed to handle noisy training data harvested
online. Specifically, given a targeted domain such as surfing,
our system mines the YouTube database to find pairs of raw and
their corresponding edited videos. Leveraging the assumption
that an edited video is more likely to contain highlights than
the trimmed parts of the raw video, we obtain pair-wise ranking
constraints to train our model. The learning task is challenging
due to the amount of noise and variation in the mined data.
Hence, a latent loss function is incorporated to mitigate the
issues caused by the noise. We efficiently learn the latent model
on a large number of videos (about 870 min in total) using a
novel EM-like procedure. Our latent ranking model outperforms
its classification counterpart and is fairly competitive compared
with a fully supervised ranking system that requires labels from
Amazon Mechanical Turk. We further show that a state-of-the-
art audio feature mel-frequency cepstral coefficients is inferior to
a state-of-the-art visual feature. By combining both audio-visual
features, we obtain the best performance in dog activity, surfing,
skating, and viral video domains. Finally, we show that impressive
highlights can be detected without additional human supervision
for seven domains (i.e., skating, surfing, skiing, gymnastics,
parkour, dog activity, and viral video) in unconstrained personal
videos.
Index Terms Video summarization, video highlight detection.
I. I NTRODUCTION
T HANKS to the wide usage of mobile devices, we increas-
ingly share a large amounts of videos, which are typically
captured in a casual and unplanned way. This trend is likely to
accelerate with more wearable devices like GoPro, HTC Re,
etc. becoming available on the consumer market. On YouTube
alone, 400 hours of video are uploaded every minute. More
impressively, one of a recent trendy video streaming service,
Periscope, claims that 40 years of videos are watched per
day [3]. However, among these incredible amount of videos,
most of them are not fun to watch; the best videos have usually
Manuscript received September 21, 2015; revised April 27, 2016 and
August 7, 2016; accepted August 8, 2016. Date of publication August 17,
2016; date of current version September 13, 2016. This work was supported
in part by MOST under Grant 103-2218-E-007-025, in part by Microsoft, in
part by Google, in part by Intel, in part by NSF under Grant IIS-1218683,
in part by ONR under Grant N00014-13-1-0720,and in part by ONR MURI
under Grant N00014-10-1-0934. The associate editor coordinating the review
of this manuscript and approving it for publication was Prof. Janusz Konrad.
M. Sun and T.-H. Chen are with National Tsing Hua University,
Hsinchu 300, Taiwan (e-mail: sunmin@ee.nthu.edu.tw).
A. Farhadi and S. Seitz are with the University of Washington, Seattle,
WA USA.
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TIP.2016.2601147
1057-7149 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
5145
been carefully and manually edited to feature the highlights
and trim out the boring segments. As a result, most unedited
videos are only watched once and become the digital dark
matters [4] on the web.
One way to solve this problem is to develop computer
algorithms to do the editing for us. I.e., wed provide raw video
footage, and out would pop a high quality edited video. Indeed,
both Google and Facebook recently released products that
seek to achieve similar goals. Googles Auto-Awesome movie
feature generates a video summary from all the footage of an
event (complete with filters and background music!). However,
Auto-Awesome works best when the event videos are short and
contain only highlights. It is not clear how it can handle raw
personal videos typically a few minutes long. Facebooks new
Look Back feature provides similar functionality, but focused
on photos and your most popular posts instead of videos.
These applications motivate the importance of research in
automatic video editing.
As a step towards this goal, we address the prob-
lem of detecting domain-specific highlights in raw videos
(see Fig. 1(a)). Many prior researches [5][12] have
explored the highlight selection problem in limited domains
(e.g., baseball game, cricket game, etc.). Recently, many
approaches [13][15] have been proposed to handle uncon-
strained personal videos. However, most methods require large
amounts of human-crafted training data. Since the definition
of a highlight is highly dependent on the domain of interest
(e.g., blowing out the candles on a birthday cake, a ski jump,
raising glasses in a toast), its not clear that these techniques
are scalable to handle video contents from all domains.
Instead, we ask a crucial question: can we learn to detect
highlights by analyzing how users (with domain knowledge)
edit videos? There is a wealth of edited video content on
YouTube, along with the raw source material. This content
captures highlights spanning a vast range of different activities
and actions. Furthermore, we show that its possible to identify
the mapping of raw source material to edited highlights,
leading to a wealth of training data. In this work, we intro-
duce (1) a novel system to automatically harvest domain-
specific information from YouTube, and (2) a novel latent
ranking model which is trained with the harvested noisy data
(see Fig. 1(a)). Leveraging the assumption that edited video is
more likely to contain highlights than the trimmed parts of the
raw video, we formulate the highlight selection problem as a
pair-wise ranking problem between short video clips in the raw
video. (see rank constraints in Fig. 1(b)). We introduce latent
variables into the ranking model to accommodate variation of
highlight selection across different users. For instance, user
5146
Fig. 1. System overview: Given surfing videos, we train a latent linear ranking SVM to predict the h-factors fully automatically (Panel (a)). Our system
automatically harvests training data online by mining raw and edited videos in YouTube related to surfing. The raw and edited pair of videos give us
pair-wise rank constraints as shown in panel (b). Note that the harvested data is noisy due to the variation of highlights selected by the users on YouTube
(Panel (c)).
A might select a very long duration clip as a highlight,
whereas user B prefers shorter clips (see Fig. 1(c)). We use
a novel EM-like procedure to learn the latent ranking model.
Our approach has several advantages. First, our latent rank-
ing model consistently outperforms its classification counter-
part (see Fig. 4). Second, it can be efficiently trained on a large
number of videos by taking advantage of a newly developed
solver [16] for linear ranking SVM. Third, our latent model
nicely takes care of the noise in our automatically harvested
training data in six out of seven domains (see Fig. 5). Finally,
we demonstrate that results using automatically harvested
YouTube data are fairly competitive as compared to those
obtained from a fully supervised ranking approach trained
on specially constructed training sets commissioned on Ama-
zon Mechanical Turk (see Fig. 6). Hence, we demonstrate
state-of-the-art weakly-supervised performance, while achiev-
ing much greater scalability (avoiding the need to manually
construct new annotated datasets for training).
A preliminary version of the paper is described in [17].
In this letter, we update the performance by including
second-order information in the fisher vector representation.
As a result, the visual feature dimension increases to 85200.
Using this more powerful visual feature, our latent ranking
model achieves a better accuracy than the one reported in [17].
In addition to the six domains reported in [17], a new
viral video domain is added. The new viral video domain
contributes to about 38% of the total number of videos in
our dataset. We also conduct additional sensitivity analysis
with respect to the hyper-parameter for classification and
ranking SVM. Finally, we include additional audio cues in
our system. Our experiments show that visual feature alone
is powerful and confirm that audio feature alone achieves
significantly worse results. By combining both audio-visual
features, we obtain the best performance in dog activity,
surfing, skating, and viral video domains.
II. R ELATED W ORK
Our work is highly related to video summarization. There
is a large literature on video summarization (see review [18]),
including techniques based on keyframes [19][22]. In the
following, we focus on subjects most relevant to our work.
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO. 11, NOVEMBER 2016
A. Video Summarization
With the unprecedented growth of online videos, many
researchers recently have focused on summarizing casu-
ally captured personal videos. Gygli et al. [13] propose a
supervised training approach to detect interesting moment
in a personal video. Potapov et al. [15] also propose a
supervised training approach for detecting highlight in each
specific-domain. Gygli et al. [14] further propose to optimize
multiple objectives for finding both interesting and repre-
sentative moments. However, all methods require collecting
videos with manually labeled interesting moments for training.
In contrast, our method can be trained on a large amount
of automatically harvested personal videos. Li et al. [23]
also propose to optimize multiple objectives, but focuses on
transfer learning from text. There are also a few unsupervised
approaches. Zhao and Xing [24] propose a fully unsupervised
approach using sparse coding to summarize a video in
quasi-real-time. Yang et al. [25] propose a recurrent auto-
encoder to unsupervisedly extract video highlights and achieve
impressive results. Finally, there are a few works with addi-
tional assumptions. Chu et al. [26] propose a co-summarization
approach which leverages a large number of edited video in
the same domain as reference videos. Sun et al. [27] propose
to summarize unconstrained personal videos into a rank list of
image composites. However, the proposed method relies on a
robust human detector to summarize only human activities.
Song et al. [28] propose to utilize freely available video
titles to guide the video summarization process. Similarly,
Liu et al. [29] propose to select video thumbnail according to
both visual and side information (e.g., video title). However,
these approaches require videos to be associated with high
quality video titles.
Many methods have been recently proposed for summa-
rizing a video with a known type of content. Given an
ego-centric video, we know hands, objects, and faces are
important cues. References [30], [31] propose to summarize
a video according to discovered interesting objects and faces.
Kopf et al. [32] focus on removing camera shakes in ego-
centric videos and propose to convert a long ego-centric video
into a short and significantly stabilized time-lapse video called
SUN et al.: RANKING HIGHLIGHTS IN PERSONAL VIDEOS BY ANALYZING EDITED VIDEOS
hyper-lapse. Arev et al. [33] propose a method to summa-
rize multiple videos capturing the same event by analyzing
camera viewing directions. Xu et al. [34] recently propose
to utilize gaze tracking information (such as fixation and
saccade) to summarize ego-centric videos. Similarly, given
a video uploaded to e-commerce websites for selling cars
and trucks, Khosla et al. [35] propose to use web-image
priors (i.e., canonical viewpoints of cars and trucks online) to
select frames to summarize the video. Our method is another
step in this direction. However, our proposed system is not
restricted to handling only ego-centric videos, where cues
from hands and objects are easier to extract, nor does it rely
on discovered canonical viewpoints which are shown to have
generalization issues in other domains such as cooking [35].
Whereas, our method is feature independent and we directly
harvest information about how users select domain-specific
highlights to create their own edited videos.
B. Sports Video Analysis
Highlight detection in broadcast sport videos has attracted
several researchers [5][12] due to the popularity of such
videos. Compared to other video types, such as personal
videos, broadcast sports videos have well defined structure
and rules. A long sports game often can be divided into parts
and only a few of these parts contain certain well defined
highlights. For example, common highlights are the score
event in soccer games, the hit event in baseball games, and the
bucket event in basketball games. Due to the well defined
structure, specifically designed mid-level and high-level audio-
visual features, such as player trajectories, crowds, audience
cheering, goal or score events, etc., are used in many methods.
One exception is [12] which uses easy-to-extract low-level
visual features. Most of the methods treat highlight detection
as a binary classification task, where each part of a training
video is labeled as true highlight or not. We argue that for
unconstrained personal videos, it is ambiguous for humans
to label highlights as binary labels. This also imposes extra
burden for annotators. Hence, it is important for a method
to scale-up by naturally harvesting crowd-sourced information
online.
C. Crowd-Sourced Highlight Discovery
Recently, many researchers have demonstrated the ability to
discover highlights from crowd-sourced data such as Twitter.
Olsen et al. show that user interaction data from an inter-
active TV application can be mined to detect events [36].
Hannon et al. use Twitter data in PASSEV [37], combining
summaries of tweet frequency and user-specified search for
terms in tweets to generate a highlight reel. The main differ-
ence between these approaches and others in computer vision
is that crowd-sourced data (users annotations but not the the
video data) are always required as an input. Hence, these
methods cannot work well on videos with no or very few
associated crowd-sourced data. On the contrary, our method
harvests both crowd-sourced and video data from YouTube for
training our latent ranking model. After training, our model
can be applied to rank highlights in any video.
5147
D. Learning to Rank
Learning to rank is an important learning technique in recent
years, because of its applications to search engines and online
advertisement. In computer vision, researchers have adopted
the ranking approach mainly for image retrieval task [38], [39].
Recently, Parikh and Grauman have introduced the concept of
relative attributes [40] which opens up new opportunities to
learn human-nameable visual attributes using a ranking func-
tion. Wang et al. [41] propose a novel deep ranking model that
employs deep learning techniques to learn similarity metric
directly from images. In this work, we propose one of the first
ranking models for domain-specific video highlight detection.
Our domains include popular actions such as skating, surfing,
etc. Our proposed latent ranking method can also be consid-
ered as a Multiple Instance Learning (MIL) problem, where
the automatically harvested edited video segment corresponds
to an unknown set of positive highlight moments. Similar
to [38], we found that formulating the domain-specific video
highlight detection as a MIL ranking task outperforms a MIL
classification task.
III. V IDEO H IGHLIGHTS
A video highlight is a moment (a very short video clip)
of major or special interest in a video. More formally,
a highlightness measure h (later referred to as h-factor) for
every moment in a video can be defined such that moments
with high h are typically selected as highlights. Hence, the goal
of detecting highlights is equivalent to learning a function
f (x) to predict the h-factor (i.e., h = f (x)) given features
x extracted from a moment in the video (see Fig. 1(a)).
One straightforward way to learn f (x) is to treat it as a
supervised regression problem. However, labeling the h-factor
consistently across many videos is hard for humans. On the
contrary, it is much easier for humans to rank pairs of moments
based on their highlightness. Hence, we propose to formulate
the video highlights problem as a pair-wise ranking problem
described next.
A. Pair-Wise Ranking
To establish notation, we use i as a unique index for all
moments in a set of videos. Each moment is associated to a
tuple (y, q, x), where y R is the relative h-factor, q Q
is the index of the video containing the moment, Q is a set
of videos, and x is the features extracted from the moment.
The set of ranking constraints corresponding to the q t h video
is defined as,
Pq {(i, j )|qi = q j = q, yi > y j }, (1)
where qi = q j = q ensures that only moments from the q t h
video are compared to each other according to the relative
h-factor y. Note that y only needs to be a relative value to
express the order within each video. The full set of pair-wise
ranking constraints in the dataset is defined as,
P qQ Pq = {(i, j )|qi = q j , yi > y j }, (2)
where qi = q j ensures that only moments from the same video
are enforced with the ranking constraints.
5148 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO. 11, NOVEMBER 2016
Our goal is to learn a function f (x) such that
f (x i ) > f (x j ), (i, j ) P, (3)
which means none of the pair-wise ranking constraints is
violated. We adopt the L2 regularization and L2 loss linear
ranking SVM formulation to learn f (x; w) = w T x as follows,
 (see Fig 1(c)). Especially for the loose highlight, constraints
1
min w T w + max(0, 1 w T (x i x j ))2 , (4) expressed in Eq. 5 might not always be true. In order to address
w 2
(i, j )P
where w is the linear model parameter and > 0 is a regular-
ization parameter. Ideally, the optimal model parameters can
be learned since the optimization problem is convex. How-
ever, the large number of pair-wise constraints (|P|) becomes
the main difficulty in training the ranking SVM efficiently.
We solve the problem efficiently by taking advantage of a
newly proposed fast truncated newton solver which employs
order-statistic trees to avoid evaluating all the pairs [16].
Note that deep learning based non-linear ranking methods
(e.g., Zhao et al. [42]) also could be used to train on
a large-scale dataset such as our videos dataset (about
870 minutes in total).
B. Ranking From Edited Videos
Asking humans to order moments within each video is
feasible but not scalable. Since the definition of a highlight is
highly dependent on the domain of interest, human annotators
without domain knowledge might find the task ambiguous
and tedious. We argue that we can naturally harvest such
information from edited videos. Assuming that we have raw
videos which are used to create the edited videos, and users
make a (somewhat) informed decision to select the moments
in the raw video to be included in the edited video. We now
get the following ranking constraints,
yi > y j , i E q , j R q \ E q , (5)
where q is the index of the raw video, E q is the set of moments
in the raw video which are included in the edited video,
and Rq is the set of moments in the raw video. Eq. 5 states
that the relative h-factors of moments included in the edited
video (E q ) is higher than the rest of the moments in the raw
video (Rq \ E q ) (see Fig. 1(b)).
Given many pairs of raw videos and their edited versions,
the linear ranking SVM becomes
1 
min w2 + L(q; w), (6)
w 2
qQ
 
L(q; w) = max(0, 1 w T (x i x j ))2 , (7)
iE q j Rq \E q
where Q is a set of raw videos, and L(q; w) is the loss
of violating ranking constraints in the q th raw video. As a
result, we can even extract ranking constraints for training data
where users edit their videos by simply trimming it. The main
advantage of this approach is that a wealth of such data exists
in the digital world. We describe in Sec. IV how we can harvest
such data online automatically.
C. Handling Noisy Data
Not all the users online are experts. The start and end of
a specific highlight can vary significantly depending on the
users. For example, user A might select a loose highlight
with a few minutes included in its edited version. In contrast,
user B might select a tight highlight with a few seconds
this problem, we propose a latent loss as follows,

L zq (q; w) = max(0, 1 w T (x zq x j ))2 , (8)
j Rq \E q
z q = arg max f (x i ; w), (9)
iE q
where z q is the best highlighted moment in E q . Note that the
latent loss relaxes the constraints in Eq. 5 to
y z q > y j , j Rq \ E q , (10)
where only the the best highlighted moment z q in E q must
have higher relative h-factor than the rest of the moment
(Rq \ E q ) in the q th raw video.
1) Latent Linear Ranking SVM: A latent linear ranking
SVM incorporating the latent loss is defined as,
 
1
min w2 + L(q; w)+ L zq (q; w), (11)
w,Z 2
qQ T qQ L
where Q T is a set of raw videos with tightly selected
highlights, Q L is a set of raw videos with loosely selected
highlights, {Q T , Q L } is a partition of Q (i.e., Q T Q L = ,
Q T Q L = Q), and Z = {z q }qQ L is a set of latent best
highlighted moments in Q L .
2) EM-Like Approach: Given Q T and Q L , we solve the
model parameters w and the set of latent best highlighted
moments Z iteratively using a EM-like approach.
1) We set Q = Q T and obtain our initial w by solving
Eq. 6.
2) Given the initial w, we estimate our initial Z by solving
Eq. 9 (E-step).
3) Then, we obtain w by solving Eq. 11 while fixing Z
(M-step).
4) Given w, we estimate Z by solving Eq. 9 (E-step).
5) Go to step 3 if the estimated latent variables Z have
changed; otherwise, stop the procedure.
In our experiments, the EM-like approach stops typically
within five iterations.
D. Self-Paced Model Selection
One simple way to define {Q T , Q L } is to select videos
with highlights less than a fixed length (see Sec. V-A4).
However, such simply partition might not be the best partition.
We propose a self-paced model selection procedure to evaluate
K partitions of Q guided by pair-wise accuracy of every video
A = {aq }q as follows.
1) We set Q T = Q and obtain w by solving Eq. 11.
2) Given the model w, we evaluate the pair-wise accu-
racy A on the training videos.
SUN et al.: RANKING HIGHLIGHTS IN PERSONAL VIDEOS BY ANALYZING EDITED VIDEOS
3) Given A, we order the videos from low to high pair-wise
accuracy, and evenly split the videos into K mutually
exclusive sets.
4) Starting from the set with the lowest accuracy,
we remove one set at a time from Q T to Q L .
5) For each partition of Q T and Q L , we solve Eq. 11 to
obtain a new model w and new pair-wise accuracy A
on the training videos.
6) Finally, we select the model with the highest mean pair-
wise accuracy mean(A).
The pair-wise accuracy aq for the q th video is defined as,
 learned using our EM-like (Sec.III-C2) procedure. In Fig. 4,
(i, j )Pq 1(w x i > w x j )
T T
aq = , (12)
|Pq |
where 1() is a indicator function which is one if a pair-wise
constraint is satisfied in our learned model (w T x), Pq (defined
in Eq. 1) is the set of pair-wise constraints for the q t h video,
and |Pq | is the number of pairs. The pair-wise accuracy is a
normalized value between 0 and 1. Hence, it is comparable
for different pairs of Q T and Q L .
E. Comparison to Binary Highlight Classification
Recall that previous highlight selection methods [5][12]
formulate their problem as a binary highlight classification
problem. Although we argue computing highlights is intrinsi-
cally a ranking problem, we show that our problem can also
be considered as a binary classification problem by setting
{yi = +1; i E q , q Q} and
{y j = 1; j Rq \ E q , q Q}, (13)
where y {1, +1} becomes a binary label. Next we discuss
the advantages and disadvantages of a binary classification
problem.
The binary classification problem is highly related to a
pair-wise ranking problem which ignores the video index and
expresses the following constraints,
P {(i, j ); yi > y j }. (14)
In this case, moments are also compared to each other across
videos. However, training a classification model is much more
efficient than training a ranking model. This is because the
huge number (i.e., quadratic to the number of moments) of
pair-wise violation losses (Eq. 7) are replaced by losses with
respect to a separation hyperplane as defined below,
 to be raw and edited pairs of videos . Hence, we use the
L C (q; w) = max(0, 1 yi (w T x i )). (15)
i;qi =q
The number of losses in Eq. 15 is linear to the number of
moments. The advantages of a classification model is (1) it
implicitly incorporates more pair-wise constraints in Eq. 14
than constraints in Eq. 2, and (2) it can be solved more effi-
ciently using many existing methods. Nevertheless, the newly
added pair-wise constraints which compare moments from two
different videos (i.e., {(i, j ); qi = q j , yi > y j }) could be
harmful. For example, the highlight in video q1 might be less
interesting than many non-highlighted moments in video q2 .
5149
We can also define a latent linear classification SVM to
handle the noisy data by replacing L(q; w) with L C (q; w)
and L zq (q; w) with L Czq (q; w) in Eq. 11, where L Czq (q; w) is
defined as,
L Czq (q; w) = max(0, 1 yzq (w T x zq ))2

+ max(0, 1 y j (w T x j ))2 ,
j Rq \E q
z q = arg max f (x i ; w). (16)
iE q
The latent linear binary classification SVM model can also be
we demonstrate that our proposed latent ranking model is
consistently better than the latent classification model. This
proves that the additional constraints in the latent classification
model are indeed harmful.
IV. H ARVESTING YouTube V IDEOS
Nowadays, many videos are shared online through websites
such as YouTube. We propose a novel approach to harvest a
large amount of raw (i.e.,unedited) videos and their edited
highlights from YouTube. To find the corresponding raw
videos and edited highlights, we can use [43] to efficiently
identify duplicated frames between a pair of videos. Once
a set of consecutively duplicated frames are matched for a
pair of videos (q1 , q2 ), we define these matched frames as
the automatically harvested highlight E. Then, we identify
the longer video among q1 and q2 as the raw video R. Note
that [43] is robust, but it can miss matches when the video
includes after effects. Hence, our dataset is built with high
precision.
Theoretically, our proposed method is feasible even for a
large scale database such as YouTube. In particular, Google is
already using a similar procedure to find videos with copyright
violations. However, this is a daunting task for individual
researchers like us, since we need to retrieve a large number of
videos to find enough matched pairs of edited and raw videos.
Therefore, we explore the following two approaches to retrieve
a smaller set of videos that is likely to contain raw video and
highlight pairs.
YouTube video editor. YouTube has an online editor
called YouTube video editor. If a video has been used
as a content to create other edited videos, the attribution
of the video keeps track of which videos used this video
as content (Fig. 2-Left). This suggests that they are likely
YouTube API to query videos generated by YouTube
video editor to retrieve a smaller set of candidate raw
and edited pairs of videos.
YouTube Channels. There exists many YouTube channels
curating highlight compilations. The compilation in these
channels typical credits the original videos in the descrip-
tion of the video (see original links in Fig. 2-Right). The
original links typically point to other YouTube videos
corresponding to the raw video of the highlights.
Given these candidate pairs of raw and highlight videos,
we use [43] to efficiently identify the duplicated frames as
5150
Fig. 2. The left panel shows the raw (top link) and edited highlight
(bottom link) pair that we retrieved using the property of YouTube video
editor.The right panel shows curated highlight compilation (top) and a list of
original links (indicated by a red box) pointing to the raw videos.
Fig. 3. The screenshot of our web-based highlight annotation interface. Given
the knowledge of a domain (e.g., surfing), an user on Amazon Mechanical
Turk (AMT) is asked to select the beginning of the highlight and the duration
of the highlight.
the highlights. Inevitably, our current data is limited by the
availability of videos generated by YouTube video editor
and curated by YouTube channels including original link
information. For instance, we were not able to retrieve enough
videos for domains such as cooking, fixing bike, etc., since
for those common domains users have less incentive to curate
these videos or edited the video collaboratively using YouTube
video editor. Nevertheless, our current system works well for
common action and animal related domains such as skating,
dog activity, parkour, viral video, etc. We describe in
Sec. V-C about the selected domains and the data statistics.
A. Ground Truth Highlights
For evaluating our ranking results, ideally we would like
to obtain a h-factor order of all moments in every video.
However, it is time consuming and ambiguous to ask gen-
eral people to order all the moments since often it is hard
to compare the h-factors of two random moments. Hence,
we found that it is more effective to ask multiple people
to select a single highlight (i.e., a video segment) within
each video. We use Amazon Mechanical Turk to collect these
ground truth annotations. For each video, we collect a no more
than five seconds highlight from fix turkers. The web-based
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO. 11, NOVEMBER 2016
highlight annotation interface is shown in Fig. 3. Given the
knowledge of a domain (e.g., surfing), an user on Amazon
Mechanical Turk (referred to as turker) is asked to select the
beginning of the highlight and the duration of the highlight.
Since we are crowd-sourcing these annotations from a large
number of turkers, our annotations will inevitably be noisy.
Hence, we consider a video segment as ground truth positive
or negative highlights, if it is selected more than 2 times or
have not been selected, respectively. Please see Sec. V-A3 for
more details.
V. E XPERIMENTS
We conduct experiments on a newly created YouTube
highlight dataset harvested by our system automatically. In the
following sections, we first give details of our implementation
such as feature representation, parameter setting, etc. Then,
we describe details about the dataset and our highlight detec-
tion evaluation. Finally, we report quantitative and qualitative
results on our novel YouTube highlight dataset.
A. Implementation Details
We describe our representation and training parameters in
detail.
1) Moment Definition: We first define each moment as
a 100 frames clip evenly sampled across each raw video.
The start and end frames of each moment is then aligned to
the nearest shot boundary within 50 frames away. We use the
3rd party code1 to detect shot boundaries according to the
pixel-wise RGB difference between two consecutive frames.
2) Feature Representation: Given these moments,
we extract the state-of-the-art dense trajectory motion
feature [2] for action classification (other scene, objects,
and audio features can also be included in our framework).
Then, the dimension of the dense trajectory features is reduced
by half using PCA. The dense trajectory features within each
moment are mapped to a learned Gaussian mixture codebook
to generate a fisher vector including both first and second
moment statistics with a fix dimension (85200). The Gaussian
mixture model with 200 mixture components is learned using
the training raw videos of all domains.
3) Highlight Moment Definition: Recall that the automati-
cally harvested highlights are obtained at frame-level. Hence,
we define a moment as a highlight moment (in E) if at
least 70% of its frames are covered by the frame-based
highlights. A moment is not a highlight (in R \ E) if at
most 30% of its frames are covered by the frame-based
highlights. All the remaining moments are not included in
training. For the manually annotated highlights, we need to
accumulate the annotation from five turkers to each moment.
We define a soft-highlightness score for each moment. For a
moment covered 70% by a manually annotated highlight,
it will accumulate 0.7 soft-highlightness score. After accumu-
lating all the soft-highlightness scores, we define the moments
with equal or more than 2 scores as ground truth positive
highlights and the moments have not been selected as negative
ground truth for evaluation.
1 http://johmathe.name/shotdetect.html
SUN et al.: RANKING HIGHLIGHTS IN PERSONAL VIDEOS BY ANALYZING EDITED VIDEOS
Fig. 4. Left-panel: performance comparison between our latent ranking model and its classification counterpart. Latent ranking model (blue-bar) is consistently
better than latent classification model (red-bar) with an average improvement of 28.6% in mAP. Right-panel: sensitivity analysis of the ranking and classification
models. We test = {0.001, 0.01, 0.1, 1, 10}.
Fig. 5. Performance comparison among non-latent ranking (red bar), latent ranking (blue bar), and self-paced latent ranking (green bar) models. Our latent
ranking models outperform non-latent models except for Parkour.
4) Tight Highlight Q T Definition: By default, we define a
video highlight to be tight, if the training highlight consists
of no more than 2 moments. All the rest of the videos are
considered to have loose highlight. Note that the self-paced
model selection procedure will modify Q T and Q L as descried
in Sec. III-D.
5) Model Training: For all the experiments, several regu-
larization parameters = {0.001, 0.01, 0.1, 1, 10} have been
considered. We report the average mean average precision
across these five values for a robust and fair comparison.
We use the liblinear package [44] to train our models in their
primal form. For self-paced model selection, we set K = 5.
B. Highlight Detection Evaluation
Within each video, the best method should first detect the
ground truth highlighted moments rather than other moments.
We calculate the average precision of highlight detection for
each testing video and report the mean average precision
(mAP) summarizing the performance of all videos. Note that
unlike object detection which accumulates all the detections
from images to calculate the average precision, highlight
detection treats each video separately since a highlighted
moment in one video is not necessary more interesting than a
non-highlighted moment in another video.
C. YouTube Highlight Dataset
By harvesting freely available data from YouTube as
described in Sec. IV, we have collected data for dog
activity, gymnastics, parkour, skiing, surfing, skat-
ing, and viral. The first six domains are selected because
about 20% of the retrieved raw videos in these domains are
5151
publicly downloadable. The viral domain is collected from
curated Jukinvideo YouTube channel.2
Our new dataset is very challenging since (1) it contains
a variety of videos captured by portable devices, (2) the
start and end of a specific highlight can vary significantly
depending on the behavior of the users, and (3) there are
irrelevant videos included in the dataset. Irrelevant videos
typically include videos of interviews, slideshows of images,
and products commercials. For instance, this commercial3
is in the skiing domain. We split our raw dataset into half
training and the other half testing. Then, we manually label
the irrelevant videos in both training and testing so that the
fully supervised explicitly crowd-sourcing ranking baseline
(see Sec. V-D.4) is not trained with irrelevant videos. However,
our fully automatic system is trained agnostic to the label of
irrelevant videos.
For testing, none of the irrelevant videos are evalu-
ated. Moreover, as mentioned in Sec. V-B, we evaluate
on videos where turkers reach consensus on highlighted
moments. The statistics of the training and the evalu-
ated testing data for each domain is shown in Table. I.
For training data, we further show the number of relevant
videos and videos with tightly selected highlights (Q T ),
in each domain in Table. I. The dataset and codes are avail-
able at https://github.com/tsenghungchen/ranking-highlights-
in-personal-videos. Our dataset in t otal has 764 videos with
the total accumulated time of 1044 minutes (62681 seconds),
which is at the similar scale as the state-of-the-art large scale
action recognition dataset [45] (1600 minutes).
2 https://www.youtube.com/user/JukinVideoDotCom
3 https://www.youtube.com/watch?v=4jdtNuVezzE
5152 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO. 11, NOVEMBER 2016

TABLE I
S TATISTICS OF O UR D ATA H ARVESTED F ROM YouTube

Fig. 6. Performance comparison between a fully supervised ranking model trained with turkers annotations (referred to as turk), our latent ranking trained
on relevant videos (referred to as latent-rank-sub), and our latent ranking trained with all harvested data (referred to as latent-rank). Our models are fairly
competitive compared to turk, and suprisingly outperforms turk significantly on parkour.

Fig. 7. Performance comparison between our model with audio feature only, visual feature only, audio-visaul-standardize (mfcc-dense trajectory) features,
and audio-visaul-non-standardize. We show that audio feature only performs much worse than visual feature. By combining both features, we achieve the
best average mAP, and best mAP in dog activity, surfing, skating, and viral domains. Finally, we show that standardization is important when combining
hetegoenous features.

D. Highlight Detection Results
Our fully automatic latent ranking method performs
well on the novel YouTube highlight dataset with a
mean average precision of 56.3% across seven domains.
We compare our system with other sophisticated methods
below.
1) Latent Ranking v.s. Classification: In Fig. 4-Left,
we compare the mean average precision for every domain
between our latent ranking model (Sec. III-C) and a latent clas-
classification model (27.7%4 mAP) in all domains with an
average improvement of 28.6% in mAP.
As mentioned in the implementation details, we show
performance of both ranking and classification models with
different regularization parameter values in Fig. 4-Right.
The performance for = {0.01, 0.1, 1, 10} are similar, which
shows that our latent ranking method is not sensitive to
change of . In order to check statistical significance, we
4 Note that the latent classification model using 26000 feature dimension

sification model (Sec. III-E), where both models are trained achieves 46% mAP in [17], which is still 18.3% less accurate than our latent
ranking model. We believe that this is due to the fact that more powerful
using noisy data harvested from YouTube. Our latent ranking features overfit a wrong classification model on the training set of a ranking
model (56.3% mAP) consistently outperforms the the latent problem and cause even worse generalization accuracy on the testing set.
SUN et al.: RANKING HIGHLIGHTS IN PERSONAL VIDEOS BY ANALYZING EDITED VIDEOS 5153

Fig. 8. Examples of moments ranking from high (left) to low (right) according to our predicted h-factors for surfing, skating, parkour, gymnastics,
dog activity, skiing, and viral videos. We show one sampled frame from each moment to represent it.

compare the AP pairs of the latent classification v.s. the latent and the latent ranking are not distinguishable) is 2e6 . Hence,
ranking methods for each testing video. Given the AP pairs, the null hypothesis is strongly rejected.
we perform a sign test (i.e., only testing which AP is higher). 2) Latent v.s. Non-Latent Ranking: We mentioned in
The p-value of the null hypothesis (i.e., latent classification Sec. III-C that latent model can mitigate issues caused by
5154 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO. 11, NOVEMBER 2016

Fig. 9. Visualization of our predicted h-factors (red lines) and raw turkers annotations (blue bars), overlaid with manually sampled raw frames. For gymnastics,
dog activity, and skiing, the main characters in some frames are too small; hence, we show the manually cropped zoom-in version of the frames. Click to
watch videos on YouTube: surfing link, skating link, parkour link, skiing link, Gymnastics link, dog link, viral video link.

noisy labels in our harvested data. We show in Fig. 5 However, the improvement is less significant as compared to
that our latent ranking outperforms non-latent tanking in our preliminary results in [17]. In average across domains,
dog activity, gymnastics, skiing, surfing, skating, and viral. our latent ranking model is 2% better than the non-latent
SUN et al.: RANKING HIGHLIGHTS IN PERSONAL VIDEOS BY ANALYZING EDITED VIDEOS
Fig. 10. Failure examples. Visualization of our predicted h-factors (red lines) and raw turkers annotations (blue bars), overlaid with manually sampled raw
frames. These examples show that our predictions are not too far off from the turkers annotations.
ranking model in mean AP. We believe this is because our
new feature has increased the performance such that our
latent model has less room for improvement over a non-latent
ranking model. In order to check statistical significance,
we compare the AP pairs of the non-latent v.s. the latent
ranking methods for each testing video.On surfing, skating,
and viral domains, the p-value (probability) of the null hypoth-
esis (i.e., not distinguishable) is 0.064. Hence, the probability
that null hypothesis is true is very low. However, on the whole
dataset, the p-value is 0.643. Note that this is possible even
when the mAP of laten ranking is significantly higher (e.g.,
in gymnastic and skiing), since the sign test do not consider
the difference in AP, but mAP do.
3) Latent v.s. Self-Paced Latent Ranking: In Fig. 5, we also
show the performance of our self-paced model selection pro-
cedure. In average, our self-paced model selection procedure is
slightly better the latent ranking method ( 1% improvement).
In particular, it outperforms both non-latent and latent tanking
methods in skiing, and skating. However, in the sign test,
the p-value is 0.853. Hence, statistically, our self-paced latent
ranking is on-par with the latent ranking.
4) Explicitly Crowd-Sourcing v.s. Naturally Harvesting
Data: In order to analyze the effect of our model trained using
noisy data harvested online, we also trained a fully supervised
ranking model (Sec. III-A) with soft-highlight scores accumu-
lated from turkers. This is a time consuming and ambiguous
task, since labeling highlights in raw personal videos is not
a well defined task for turkers unrelated to the events in the
videos. Moreover, we assume that the human annotators will
discover the irrelevant videos. Hence, the irrelevant videos
in the training set are not used to train the fully supervised
ranking model. However, this means that the fully supervised
ranking model is trained on about 69% of the full training data.
In order to have a fair comparison with the fully supervised
ranking model, we also train our latent ranking method on 69%
of the full training data (referred to as latent-rank-sub).
The comparison between the fully supervised ranking model
trained with multiple turkers annotations and our proposed
latent ranking models (latent-rank-sub and latent-rank) are
shown in Fig. 6. Intuitively, the fully supervised ranking model
5155
should perform much better. However, in parkour, our full
latent ranking model (latent-rank) is even superior to the fully
supervised ranking model trained with turkers annotations.
This proves that our fully automatic system is very competitive
compared to the approach requiring annotations from a crowd-
sourcing platform. Moreover, our latent-rank-sub" indeed
achieves better accuracy than our latent-rank which trained
with both relevant and irrelevant videos. This proves that
training with irrelevant videos in general will hurt the accuracy.
5) Visual Features v.s. Audio Features: In addition to visual
feature including motion and appearance features, personal
videos also includes audio information which is not utilized in
the previous experiments. Intuitively, audio feature should be
a reliable cues, since highlights could come along with loud
audio from the scene and object being captured or from the
cameramen themselves. However, we also notice some of our
videos contain no sound or background music. We conduct the
following experiments to test if audio feature is really reliable
by itself. For audio feature, we extract state-of-the-art Mel-
frequency cepstral coefficients (MFCCs) [1] for each moment.
This corresponds to a 11x13 matrix containing 13 MFCCs
(first 12 DCT coeff. and the log energy) for each moment;
then, we convert it into a 143-dimension feature vector. When
combing both visual and audio features, we standardize every
entry of the features; then, directly concatenate them. In Fig. 7,
we show audio feature alone performs worse than visual
feature. Given the AP pairs, we perform a sign test. Since
the p-value of the null hypothesis is 2e6 , the null hypothesis
is strongly rejected. Moreover, by combining both features,
we achieve the best mAP in dog activity, surfing, skating,
and viral domain. We believe audio feature in gymnastic and
skiing are less informative, since their videos are typically
captured a distance away from the event locations. We also
show that standardization before direct concatenation is very
critical. However, in the sign test, the p-value is 0.298. Hence,
statistically, combining both features is on-par with visual
feature.
6) Qualitative Results: We show a set of moments ranking
from high to low according to our predicted h-factors
for different domains in Fig. 8. Note that our data is
5156
challenging and realistic, since it includes videos captured by
widely used cameras such as GoPro and cellphone cameras.
Detailed visualization of our predicted h-factors and raw
turkers annotations, overlaid with manually sampled raw
frames for each domain is shown in Fig. 9. A few failure
examples are shown in Fig. 10. These examples show that our
predictions are not too far off from the turkers annotations.
VI. C ONCLUSION
As a step towards automatic video editing, we introduce a
fully automatic system for ranking video highlights in uncon-
strained personal videos by analyzing online edited videos.
Our system and the proposed novel latent ranking model are
shown to be superior to its binary classification counterpart
and a non-latent ranking model without handling noise in the
harvested data. Our system is also fairly competitive compared
to a fully supervised ranking system requiring annotations
from Amazon Mechanical Turk. We believe our system paves
the way toward learning a large number of domain-specific
highlights since more and more users behavioral data can be
harvested online. Moreover, we show that our results are not
sensitive to the values of the hyper-parameters for training the
ranking SVM, and audio features alone are inferior than visual
features. Finally, in dog, surfing, and skating domains, audio
features are complementary to visual features.
In the future, we plan to focus on two main directions.
First, we would like to use a bank of domain-specific highlight
rankers to describe every moment in a raw video. Given this
high-level understanding of moments, we hope to reliably
analyze how to fully automatically generate an edited video.
Second, we would like to take advantage of deep learning
techniques to learn a joint audio-visual feature representa-
tion, whereas combing two state-of-the-art hand crafe features
seems to be less effective in our experiment. We believe that
this could fundamentally improve the accuracy of the video
summarization task.
R EFERENCES
[1] S. B. Davis and P. Mermelstein, Comparison of parametric repre-
sentations for monosyllabic word recognition in continuously spoken
sentences, IEEE Trans. Acoust., Speech, Signal Process., vol. 28, no. 4,
pp. 357366, Aug. 1980.
[2] H. Wang and C. Schmid, Action recognition with improved trajecto-
ries, in Proc. ICCV, Dec. 2013, pp. 35513558. [Online]. Available:
https://hal.inria.fr/hal-00873267
[3] TechCrunch. (2015). Periscope Has 10m Registered Users Watching
40 Years of Video Per Day. [Online]. Available: http://techcrunch.
com/2015/08/12/periscope-has-10m-registered-users-watching-40-years-
of-video-per-day/
[4] The Online Economy. (2012). Internets Digital Dark Matter. [Online].
Available: http://www.onlineeconomy.org/internets-digital-dark-matter/
[5] D. Yow, B. Yeo, M. Yeung, and B. Liu, Analysis and presenta-
tion of soccer highlights from digital video, in Proc. ACCV, 1995,
pp. 499503.
[6] Y. Rui, A. Gupta, and A. Acero, Automatically extracting highlights for
TV baseball programs, in Proc. ACM Multimedia, 2000, pp. 105115.
[7] S. Nepal, U. Srinivasan, and G. Reynolds, AAutomatic detection of
Goal segments in basketball videos, in Proc. ACM Multimedia, 2001,
pp. 261269.
[8] J. Wang, C. Xu, E. Chng, and Q. Tian, Sports highlight detection from
keyword sequences using HMM, in Proc. ICME, 2004, pp. 599602.
[9] Z. Xiong, R. Radhakrishnan, A. Divakaran, and T. S. Huang, Highlights
extraction from sports video based on an audio-visual marker detection
framework, in Proc. ICME, 2005, p. 4.
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO. 11, NOVEMBER 2016
[10] M. H. Kolekar and S. Sengupta, Event-importance based customized
and automatic cricket highlight generation, in Proc. ICME, 2006,
pp. 16171620.
[11] A. Hanjalic, Adaptive extraction of highlights from a sport video
based on excitement modeling, IEEE Trans. Multimedia, vol. 7, no. 6,
pp. 11141122, Dec. 2005.
[12] H. Tang, V. Kwatra, M. Sargin, and U. Gargi, Detecting highlights in
sports videos: Cricket as a test case, in Proc. ICME, 2011, pp. 16.
[13] M. Gygli, H. Grabner, H. Riemenschneider, and L. Van Gool, Creating
summaries from user videos, in Proc. ECCV, 2014, pp. 505520.
[14] M. Gygli, H. Grabner, and L. Van Gool, Video summarization by
learning submodular mixtures of objectives, in Proc. CVPR, 2015,
pp. 30903098.
[15] D. Potapov, M. Douze, Z. Harchaoui, and C. Schmid, Category-specific
video summarization, in Proc. ECCV, Jun. 2014, pp. 540555.
[16] C.-P. Lee and C.-J. Lin, Large-scale linear rankSVM, Neural Comput.,
vol. 26, no. 4, pp. 781817, 2013.
[17] M. Sun, A. Farhadi, and S. Seitz, Ranking domain-specific highlights
by analyzing edited videos, in Proc. ECCV, 2014, pp. 787802.
[18] R. Borgo et al., A survey on video-based graphics and video visual-
ization, in Proc. EUROGRAPHICS, 2011, pp. 123.
[19] D. Liu, G. Hua, and T. Chen, A hierarchical visual model for video
object summarization, IEEE Trans. Pattern Anal. Mach. Intell., vol. 32,
no. 12, pp. 21782190, Dec. 2010.
[20] Y. Gong and X. Liu, Video summarization using singular value decom-
position, in Proc. CVPR, 2000, pp. 174180.
[21] C.-W. Ngo, Y.-F. Ma, and H.-J. Zhan, Video summarization and scene
detection by graph modeling, IEEE Trans. Circuits Syst. Video Technol.,
vol. 15, no. 2, pp. 296305, Feb. 2005.
[22] E. Mendi, H. B. Clemente, and C. Bayrak, Sports video summariza-
tion based on motion analysis, Comput. Elect. Eng., vol. 39, no. 3,
pp. 790796, 2013.
[23] L. Li, K. Zhou, G.-R. Xue, H. Zha, and Y. Yu, Video summarization
via transferrable structured learning, in Proc. World Wide Web, 2011,
pp. 287296.
[24] B. Zhao and E. P. Xing, Quasi real-time summarization for consumer
videos, in Proc. CVPR, 2014, pp. 25132520.
[25] H. Yang, B. Wang, S. Lin, D. Wipf, M. Guo, and B. Guo, Unsupervised
extraction of video highlights via robust recurrent auto-encoders, in
Proc. ICCV, 2015, pp. 46334641.
[26] W.-S. Chu, Y. Song, and A. Jaimes, Video co-summarization: Video
summarization by visual co-occurrence, in Proc. CVPR, Jun. 2015,
pp. 35843592.
[27] M. Sun, A. Farhadi, B. Taskar, and S. Seitz, Salient montages from
unconstrained videos, in Proc. ECCV, 2014, pp. 472488.
[28] Y. Song, J. Vallmitjana, A. Stent, and A. Jaimes, TVSum: Summarizing
Web videos using titles, in Proc. CVPR, 2015, pp. 51795187.
[29] W. Liu, T. Mei, Y. Zhang, C. Che, and J. Luo, Multi-task deep visual-
semantic embedding for video thumbnail selection, in Proc. CVPR,
2015, pp. 37073715.
[30] Z. Lu and K. Grauman, Story-driven summarization for egocentric
video, in Proc. CVPR, 2013, pp. 27142721.
[31] Y. J. Lee, J. Ghosh, and K. Grauman, Discovering important people
and objects for egocentric video summarization, in Proc. CVPR, 2012,
pp. 13461353.
[32] J. Kopf, M. F. Cohen, and R. Szeliski, First-person hyper-lapse videos,
in Proc. SIGGRAPH, 2014, Art. no. 78.
[33] I. Arev, H. S. Park, Y. Sheikh, J. Hodgins, and A. Shamir, Automatic
editing of footage from multiple social cameras, in Proc. SIGGRAPH,
2014, Art. no. 81.
[34] J. Xu, L. Mukherjee, Y. Li, J. Warner, J. M. Rehg, and V. Singh, Gaze-
enabled egocentric video summarization via constrained submodular
maximization, in Proc. CVPR, 2015, pp. 22352244.
[35] A. Khosla, R. Hamid, C.-J. Lin, and N. Sundaresan, Large-scale
video summarization using Web-image priors, in Proc. CVPR, 2013,
pp. 26982705.
[36] D. R. Olsen and B. Moon, Video summarization based on user
interaction, in Proc. EuroITV, 2011, pp. 115122.
[37] J. Hannon, K. McCarthy, J. Lynch, and B. Smyth, Personalized and
automatic social summarization of events in video, in Proc. IUI, 2011,
pp. 335338.
[38] Y. Hu, M. Li, and N. Yu, Multiple-instance ranking: Learning to rank
images for image retrieval, in Proc. CVPR, 2008, pp. 18.
[39] B. Siddiquie, R. Feris, and L. Davis, Image ranking and retrieval based
on multi-attribute queries, in Proc. CVPR, 2011, pp. 801808.
[40] D. Parikh and K. Grauman, Relative attributes, in Proc. ICCV, 2011,
pp. 503510.
SUN et al.: RANKING HIGHLIGHTS IN PERSONAL VIDEOS BY ANALYZING EDITED VIDEOS
[41] J. Wang et al., Learning fine-grained image similarity with deep
ranking, in Proc. CVPR, 2014, pp. 13861393.
[42] F. Zhao, Y. Huang, L. Wang, and T. Tan, Deep semantic ranking
based hashing for multi-label image retrieval, in Proc. CVPR, 2015,
pp. 15561564.
[43] C. E. Jacobs, A. Finkelstein, and D. H. Salesin, Fast multiresolution
image querying, in Proc. SIGGRAPH, 1995, pp. 277286.
[44] R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin,
LIBLINEAR: A library for large linear classification, J. Mach. Learn.
Res., vol. 9, pp. 18711874, Jun. 2008.
[45] K. Soomro, A. R. Zamir, and M. Shah, UCF101: A dataset of 101
human actions classes from videos in the wild, in Proc. CRCV-TR,
2013.
Min Sun received the Ph.D. degree from the
University of Michigan, Ann Arbor. He is cur-
rently an Assistant Professor with the National
Tsing Hua University. He is also a Post-Doctoral
Researcher with the University of Washington.
His research interests include object recognition,
image/video understanding, human pose estimation,
machine learning, and deep learning. He received
the best paper awards in 3DRR 2007, CVGIP 2015,
and CVGIP 2016.
Ali Farhadi received the Ph.D. degree from
the CS Department, University of Illinois at
Urbana-Champaign under the supervision of
Prof. D. Forsyth. He is currently an Assistant
Professor with the CSE Department, University of
Washington. He spent a year as a Post-Doctoral
Fellow with the Robotics Institute, Carnegie Mellon
University, under the supervision of Prof. M. Hebert
and Prof. A. Efors. He received the Best Student
Paper Award in CVPR 2011.
5157
Tseng-Hung Chen is currently pursuing the degree
with the Department of Electrical Engineering,
Vision Science Laboratory, National Tsing Hua Uni-
versity under the supervision of Prof. M. Sun.
His research interests are in the areas of computer
vision, natural language processing and deep learn-
ing. His projects focus on natural language genera-
tion and visual semantic understanding.
Steve Seitz received the Ph.D. degree in
computer sciences from the University of
Wisconsin in 1997. He is currently a Professor
with the Department of Computer Science and
Engineering, University of Washington. He also
directs an Imaging Group, Googles Seattle office.
He spent one year visiting the Vision Technology
Group with Microsoft Research and the subsequent
two years as an Assistant Professor with the
Robotics Institute, Carnegie Mellon University. He
joined the University of Washington in 2000 as a
Faculty Member. He received the David Marr Prize for the best paper at the
International Conference of Computer Vision, and he has received an NSF
Career Award, and ONR Young Investigator Award, and an Alfred P. Sloan
Fellowship. He is interested in problems in computer vision and computer
graphics. His current research focuses on 3-D modeling and visualization
from large photo collections.