Closing the Gaps in Inertial Motion Tracking

Sheng Shen, Mahanth Gowda, Romit Roy Choudhury

Inertial Measurement Unit (IMU)

Accelerometer

Gyroscope

Magnetometer
Inertial Measurement Unit (IMU)

Accelerometer

Gyroscope

Magnetometer
Inertial Measurement Unit (IMU)

Accelerometer

Gyroscope

Magnetometer

Wide applications in motion tracking

Lot of work in inertial motion tracking

IMU 3D
Data Motion Trajectory
Tracking
Algorithm

Open problem in mobile computing

Lot of work in inertial motion tracking

IMU 3D
Data Motion Trajectory
Tracking
Algorithm

Open problem in mobile computing

This paper doesn’t solve it either … however, makes progress
Lot of work in inertial motion tracking

IMU 3D
Data Motion Trajectory
Tracking
Algorithm

Open problem in mobile computing

This paper doesn’t solve it either … however, makes progress
Am struggling to explain our algorithm … so will start from scratch
One Prerequisite Slide: Rotation Matrices

• Rotation is a function
0 90° 0
Y: X:
1 −1 1
𝑅𝑜𝑡 90° = = Y:
0
1
X:
0

• Mathematically, rotation is a matrix

cos 90° − sin 90° 1 0 0 −1
=
sin 90° cos 90° 0 1 1 0

• Same for 3D Rotation

3×3
Rotation =
Matrix
MUSE: Our Goal is 3D Localization

Map
Initial Location
3D
Algorithm
IMU = Accel + Gyro + Mag Trajectory
Let’s Understand the Inputs

𝑥0 𝑦0 𝑧0 𝑇

Initial Location
Algorithm
IMU = Accel + Gyro + Mag

Zoom into IMU data:

𝑎𝑥 𝑔𝑥 𝑚𝑥
Accel. = 𝑎𝑦 Gyro. = 𝑔𝑦 Mag. = 𝑚𝑦
𝑎𝑧 𝑔𝑧 𝑚𝑧
(Linear Acceleration) (Angular Velocity) (Earth’s Magnetic North)
Delta rotation in unit time
Can we solve localization with these inputs?

𝑥0 𝑦0 𝑧0 𝑇

Initial Location ? 𝑥𝑡
Algorithm 𝑦𝑡
IMU = Accel + Gyro + Mag
𝑧𝑡
One possibility is:

𝑥0 𝑦0 𝑧0 𝑇

Initial Location 𝑥0 𝑡 𝑥𝑡
𝑦0 + ඵ 𝐴𝑐𝑐𝑒𝑙. 𝑑𝑡 2 𝑦𝑡
Accel., Gyro., Mag.
IMU 𝑧0 0 𝑧𝑡

𝑎𝑥
But there is one BIG problem: Accel. = 𝑎𝑦 is in local reference frame
𝑎𝑧
 𝑎𝑥
But there is one BIG problem: Accel. = 𝑎𝑦 is in local reference frame
𝑎𝑧
 𝑎𝑥
But there is one BIG problem: Accel. = 𝑎𝑦 is in local reference frame
𝑎𝑧

As an analogy
New Delhi Airport What is Map
- Frontwards
3D
- Rightwards
Accelerating Trajectory
- Upwards
𝑎𝑥 : Frontwards w.r.t. the earth?
𝑎𝑦 : Rightwards
𝑎𝑧 : Upwards
This makes orientation necessary
This makes orientation necessary

• What is 3D orientation? Orientation is the 3D rotation needed to make:

Frontwards Northwards
Plane’s Own Rightwards Eastwards
Upwards Vertical
Front

Up

Right
This makes orientation necessary

• What is 3D orientation? Orientation is the 3D rotation needed to make:

Frontwards Northwards
Plane’s Own Rightwards Eastwards
Upwards Vertical

Up: Vertical

Front: North

Right: East
So, how will orientation solve the problem?
What we need to do is:
Initial Location 𝑥𝑡
Accel. Accel. + 𝑦𝑡
Gyro. 3D IMU ඵ 𝑧𝑡
IMU (Local) 𝑡
Mag. Orient. (Global)

So 3D orientation is the key.

So how to get 3D orientation?

2 Main Opportunities: 1. Gravity 2. Magnetic North

Both measurable by IMU

Key idea: What rotation is needed such that

1) Gravity is exactly in my downward direction
2) North is exactly in my frontward direction
So how to get 3D orientation?

Key idea: What rotation is needed such that

1) Gravity is exactly in my downward direction
2) North is exactly in my frontward direction
 So how to get 3D orientation?

Key idea: What rotation is needed such that

1) Gravity is exactly in my downward direction
2) North is exactly in my frontward direction

3×3 𝑎𝑥 0 Tilt is determined (2 out of 3 DoFs)

Rotation 𝑎𝑦 = 0
Matrix 𝑎𝑧 −9.8

Gravity says a lot about orientation,

but not sufficient

Gravity Downward
 So how to get 3D orientation?

Key idea: What rotation is needed such that

1) Gravity is exactly in my downward direction
2) North is exactly in my frontward direction

3×3 𝑎𝑥 0 Tilt + Heading is determined (all 3 DoFs)

Rotation 𝑎𝑦 = 0
Matrix 𝑎𝑧 −9.8
3×3 𝑎𝑥 𝑚𝑥 0 0
Rotation 𝑎𝑦 𝑚𝑦 = 0 100
Matrix 𝑎𝑧 𝑚𝑧 −9.8 0

My orientation
North Frontward
So is Gravity + North enough to get 3D Orientation?

Only when object is static … but not otherwise. Why?

So is Gravity + North enough to get 3D Orientation?

Only when object is static … but not otherwise. Why?

Because any motion of the object will reflect
in the accelerometer … thereby
polluting the gravity estimate
So how to get 3D orientation? (Another idea)

Another Idea for Orientation: Integrate angular velocity from gyro

Initial 𝑡
New
Orientation + න 𝐺𝑦𝑟𝑜. 𝑑𝑡 = Orientation
(at time t)
0

But gyro drifts, so only useful in short time scales

State of the art today: Sensor Fusion
Static Moving Static Moving Static Moving

Time
Rely mostly on Rely mostly on ……
gravity + North gyro integration

Always know 3D orientation

 Getting back to our goal
Initial 𝑥𝑡
Location
+ 𝑦𝑡
𝑧𝑡
Accel. Location
Static?
Y 3D Accel.
IMU Mag. Orient. IMU ඵ
(Local) 𝑡
N Gyro. 3D (Global)
Orient.
 Getting back to our goal
Initial 𝑥𝑡
Location
+ 𝑦𝑡
𝑧𝑡
Accel. Location
Static?
Y 3D Accel.
IMU Mag. Orient. IMU ඵ
(Local) 𝑡
N Gyro. 3D (Global)
Orient.

Main take away: Gravity is the main anchor for 3D orientation

But what if object is not often static

This paper: MUSE

What does MUSE do?
MUSE = Estimate 3D Orientation even for moving object

Static Moving Static Moving

Time
• Use magnetometer as our main anchor
• Joint estimation of orientation and location
What does MUSE do?
MUSE = Estimate 3D Orientation even for moving object

Static Moving Static Moving

Time
• Use magnetometer as our main anchor
• Joint estimation of orientation and location
Earth’s North Pole

Earth’s South Pole

From: Wikipedia
Earth’s North Pole Magnetic Field Direction
on Earth Surface

Earth’s South Pole

From: Wikipedia
Earth’s Magnetic Field as Main Anchor

Z: Up

Global Reference Frame Y: North

X: East

Magnetic Field Direction

Earth’s Magnetic Field as Main Anchor

This vector is not influenced by motion!

Let’s make it our main anchor

North

Horizontal Plane

Magnetic Field Direction

Earth’s Magnetic Field as Main Anchor

3×3 𝑚𝑥 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑥

Rotation 𝑚𝑦 = 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑦
Matrix 𝑚𝑧 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑧

Main Anchor
Magnetic Field Direction
Earth’s Magnetic Field as Main Anchor

3×3 𝑚𝑥 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑥

Rotation 𝑚𝑦 = 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑦
Matrix 𝑚𝑧 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑧

Three questions:

1) What does it say about my orientation?

2) Magnetometer’s resolution is low …
3) How to know x, y, z of earth magnetic field?
Earth’s Magnetic Field as Main Anchor

3×3 𝑚𝑥 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑥

Rotation 𝑚𝑦 = 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑦
Matrix 𝑚𝑧 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑧

Three questions:

1) What does it say about my orientation?

2) Magnetometer’s resolution is low …
3) How to know x, y, z of earth magnetic field?
 Actual vs. Inferred Measurement

Z: Outward Y: Front Z: Outward Y: Front

[Local] [Local]

X: Right X: Right

𝒗𝑚𝑒𝑎𝑠𝑢𝑟𝑒 𝒗𝑖𝑛𝑓𝑒𝑟

Actual Measurement Inferred Measurement

(from magnetometer) (inferred from orientation estimation)
 Actual vs. Inferred Measurement

Z: Outward Y: Front Z: Outward Y: Front

[Local] [Local]

X: Right X: Right
𝒗𝑚𝑒𝑎𝑠𝑢𝑟𝑒 and 𝒗𝑖𝑛𝑓𝑒𝑟 are equal, if our orientation is estimated perfectly …

𝒗𝑚𝑒𝑎𝑠𝑢𝑟𝑒 𝒗𝑖𝑛𝑓𝑒𝑟

Actual Measurement Inferred Measurement

(from magnetometer) (inferred from orientation estimation)
 Actual vs. Inferred Measurement

Z: Outward Y: Front

[Local]

X: Right

𝒗𝑚𝑒𝑎𝑠𝑢𝑟𝑒
𝒗𝑖𝑛𝑓𝑒𝑟

2-DoF Orientation Drift == How off 𝒗𝑖𝑛𝑓𝑒𝑟 is from 𝒗𝑚𝑒𝑎𝑠𝑢𝑟𝑒

 Actual vs. Inferred Measurement

Z: Outward Y: Front
∆𝑜𝑟𝑖𝑒𝑛𝑡

[Local]

X: Right

𝒗𝑚𝑒𝑎𝑠𝑢𝑟𝑒
𝒗𝑖𝑛𝑓𝑒𝑟

2-DoF Orientation Drift == How off 𝒗𝑖𝑛𝑓𝑒𝑟 is from 𝒗𝑚𝑒𝑎𝑠𝑢𝑟𝑒

 Actual vs. Inferred Measurement

Z: Outward Y: Front
∆𝑜𝑟𝑖𝑒𝑛𝑡

[Local]

X: Right

𝒗𝑚𝑒𝑎𝑠𝑢𝑟𝑒
𝒗𝑖𝑛𝑓𝑒𝑟

New Orientation = Old Orientation + ∆𝑜𝑟𝑖𝑒𝑛𝑡

 Actual vs. Inferred Measurement

New Orientation = Old Orientation + ∆𝑜𝑟𝑖𝑒𝑛𝑡

Gyroscope: Orientation(t) = Orientation(t-1) + ‫𝑜𝑟𝑦𝑔 𝑡∆׬‬. 𝑑𝑡

Magnetometer: Orientation(t) = Orientation(t) + 0.01 ∗ ∆𝑜𝑟𝑖𝑒𝑛𝑡

Earth’s Magnetic Field as Main Anchor

3×3 𝑚𝑥 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑥

Rotation 𝑚𝑦 = 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑦
Matrix 𝑚𝑧 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑧

Three questions:

1) What does it say about my orientation?

2) Magnetometer’s resolution is low …
3) How to know x, y, z of earth magnetic field?
Earth’s Magnetic Field as Main Anchor

How to know x, y, z of earth magnetic field?

• Obtain GPS address and check the database is one option …
• A easier way is to utilize pause opportunities

𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑥 3×3 𝑚𝑥

𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑦 = Rotation 𝑚𝑦
𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑧 Matrix 𝑚𝑧

Known when object is static

Evaluation
 Comparison Baselines

GyroOnly Only uses gyroscope integration, assuming initial orientation is known

ComplemFilter (Option 1) Use low-pass filter on acceleration

A3 (Option 2) Calibrate using “good” opportunities

MUSE Our algorithm

Ground Truth: Periodically bringing the phone to the same orientation, and excluding
those moments from calibration opportunities
Natural Activities
A Sample Trace
OK, we have 3D Orientation now… how to estimate location?
OK, we have 3D Orientation now… how to estimate location?

Initial Location 𝑥𝑡
Accel. Accel. + 𝑦𝑡
Gyro. 3D IMU ඵ 𝑧𝑡
IMU (Local) 𝑡
Mag. Orient. (Global)
OK, we have 3D Orientation now… how to estimate location?

Initial Location 𝑥𝑡
Accel. Accel. + 𝑦𝑡
Gyro. 3D IMU ඵ 𝑧𝑡
IMU (Local) 𝑡
Mag. Orient. (Global)

Localization impossible if
accelerometer has any noise … why?
 Motion models can bound location error

Initial Location 𝑥𝑡
Accel. Accel. + 𝑦𝑡
Gyro. 3D IMU ඵ 𝑧𝑡
IMU (Local) 𝑡
Mag. Orient. (Global)
 Motion models can bound location error

Initial Location 𝑥𝑡
Accel. Accel. + 𝑦𝑡
Gyro. 3D IMU ඵ 𝑧𝑡
IMU (Local) 𝑡
Mag. Orient. (Global)

Motion model as priors

 MUSE uses this motion model to
jointly estimate both { orientation + location }

Initial Location 𝑥𝑡
Accel. Accel. + 𝑦𝑡
Gyro. 3D IMU ඵ 𝑧𝑡
IMU (Local) 𝑡
Mag. Orient. (Global)

Motion model as priors

 MUSE uses this motion model (in a particle filter) to
jointly estimate both { orientation + location }

Initial Location 𝑥𝑡
Accel. Accel. + 𝑦𝑡
Gyro. 3D IMU ඵ 𝑧𝑡
IMU (Local) 𝑡
Mag. Orient. (Global)

Motion model as priors

OK, we have 3D Orientation now… how to estimate location?

Initial Location 𝑥𝑡
Accel. Accel. + 𝑦𝑡
Gyro. 3D IMU ඵ 𝑧𝑡
IMU (Local) 𝑡
Mag. Orient. (Global)

Motion model as priors

MUSE Localization Error

Fits motion
model the best

(cm)
Motion model breaks
 Conclusion
Conventional Systems This paper

Use magnetometer as
Orientation
a new “gravity” sensor

Joint estimation of
orientation and location
Location