SEMANTIC PARAMETRIC BODY SHAPE

ESTIMATION FROM NOISY DEPTH

SEQUENCES

Alexandru E. Ichim Federico Tombari

 SOME OF MY OTHER WORK 1

Static Dynamic
8 MP, ~100 images HD, ~2 min
Online Submission ID: 0417

Dynamic 3D Avatar Creation from Hand-held Video Input

Dynamic 3D Avatar Creation From Hand-held Video Input 2

 SOME OF MY OTHER WORK 1

Semantic Parametric Body Shape Estimation from Noisy Depth Sequences 3

 SOME OF MY OTHER WORK 2

Semantic Parametric Body Shape Estimation from Noisy Depth Sequences 4

 SOME OF MY OTHER WORK 2

Semantic Parametric Body Shape Estimation from Noisy Depth Sequences 5

 SOME OF MY OTHER WORK 2

Semantic Parametric Body Shape Estimation from Noisy Depth Sequences 6

 MOTIVATION

Generic framework for tracking and modeling articulated bodies

in our experiments:

human bodies

RGB-D input

can be applied to:

hand tracking
animal tracking

Semantic Parametric Body Shape Estimation from Noisy Depth Sequences 7

 BODY REPRESENTATION

Pose variations - skeleton + linear blend skinning

Rest pose variations - blendshapes + bone lengths

Semantic Parametric Body Shape Estimation from Noisy Depth Sequences 8
 REGISTRATION ENERGY

Semantic Parametric Body Shape Estimation from Noisy Depth Sequences 9

 REGISTRATION ENERGY

X X 2
pose 2 T
Ef eatures = tj j
Esurf ace = ni (xi vi ) 2
2
j i

Features Point-to-plane dense closeness

NiTE landmarks - replaceable ICP iterations

Semantic Parametric Body Shape Estimation from Noisy Depth Sequences 10

 REGISTRATION ENERGY

¯ ¯
(rx I contour , ry I contour , 0) T
n=
(rx I¯contour , ry I¯contour , 0)

Contour

extracted from the 2D depth image

can be extended to RGB-segmentation

Semantic Parametric Body Shape Estimation from Noisy Depth Sequences 11

 REGISTRATION ENERGY

Eprior = 5 Eprior proj + 6 Eprior dev

2
T
Eprior proj = ( µ) MM ( µ)
2
distance to projection
2
Eprior dev = ⌃ 1/2
M ( T
µ) std-dev-weighting
2
of the projection
Statistical Pose Prior

Semantic Parametric Body Shape Estimation from Noisy Depth Sequences 12

 REGISTRATION ENERGY

2
Esmoothness = (t) (t 1)
2

Smoothness term

Semantic Parametric Body Shape Estimation from Noisy Depth Sequences 13

 MODELING ENERGY

Emodeling = 1 Esurf ace + 2 Econtour + 3 Ebs reg

2
Ebs reg L2 = ||s||2
Regularization terms
Ebs reg L1 = ||s||1

accumulate constraints over windows of frames -

performance trade-off

solve and update the body template shape

Semantic Parametric Body Shape Estimation from Noisy Depth Sequences 14

 MODELING ENERGY
L2 regularization

L1 regularization

word cloud
visualizations

Semantic Parametric Body Shape Estimation from Noisy Depth Sequences 15

 RESULTS

Semantic Parametric Body Shape Estimation from Noisy Depth Sequences 16

 Thank you!

Semantic Parametric Body Shape Estimation from Noisy Depth Sequences 17