Fusing Sensors into Mobile Operating Systems & Innovative Use Cases

May 23, 2012

Tristan Joo (tristanjoo@wca.org)

Board Director & Co-Chair of Mobile SIG Wireless Communications Alliance, Independent Executive Consultant

Sensors in Todays Smartphones & Tablets

How many Sensor are there?

-Low-end Smartphones -High-end Smartphones

Sensors in Todays Smartphones & Tablets

Color /
Biometric

NFC GPS WiFi Bluetooth Cellular

APPLICATION

12 ~ 18 Sensors!

Front & Rear

Inertial Sensors: Gyroscope, Accelerometer, Magnetometer (e-Compass) Optical Sensors: Proximity, Ambient Light, RGB Color, Image Sensors (Front/Rear) Touch Sensors: Multi-Touch, Touchless Hover, Pressure Touch Environmental Sensors: Temperature, Humidity, Barometric Pressure, Gas (CO)* Wireless/RF Sensors: GPS, WiFi, Cellular A-GPS, Bluetooth Low Energy, NFC Other Sensors: MEMS Microphones , Biometric/Fingerprint*, BioSensors*
TristanJoo@wca.org (Apr. 2012)

: MEMS Sensors

* : Future Sensors (~2015)

Mobile Sensors: Use Cases and Applications

Key Mobile Use Cases & Applications

Single Sensor Use Cases
Compass Apps Tilt Sensing Multi-Touch, Touchless Hover Ambient Light/Color or Proximity Sensing Ambient Temperature & Humidity Sensing

Innovative User Experiences enabled by Multiple Sensors Fusion!

Mobile Sensor Fusion Use Cases & Apps (in App Stores)
Gesture UI Control (Motion, Proximity) Sensor Music Player, Gesture authentication, etc. Remote Control App (Motion, Multi-Touch, RF) NeguSoft UControl, Samsung Remote, etc. Augmented Reality (Inertial, GPS, Image) Wikitude, Layar, etc. Exercise Apps: SitUp, PushUp, PullUp, Pedometer, etc. (Motion, Proximity, GPS) Sensors Games: Alien Attack (AR Motion), A.Squadron & ZombieGate (9-axis Motion), etc. Indoor Navigation and Positioning (Inertial, Pressure, WiFi): SenionLab, Meridian, etc. Context-Aware Mobile Services (ALL Sensors!): Address Sense, Smart Assistant, etc.

Sensor Fusion: Key to Context-Aware UX Future

Each Device
Smart Diary /Assistant Crash / Fall Detection Smart LBS Health Monitor

Across Devices

Millions!

Web Developers

Sensor Fusion APIs

Power-efficient Algorithms

Mobile Sensory Data Cloud

Sensory Data Collection Context Data Analytics App Context Provisioning Inertial (MEMS) Environmental Optical Wireless/RF

Enterprises & Service Providers

(Some MEMS)

: Persistent Ambient Sensing

TristanJoo@wca.org (Apr. 2012)

Reality Check: Mobile Apps that use Sensors Today

Most popular Sensor Apps still <0.5% Share of Entire App Stores!
[Sensor Apps by App Store]
1,200+ 1,100+ 1,000+ 900+ 800+ 700+ 600+ 500+ 400+ 300+ 200+ 100+ 0+

GooglePlay (470K+apps) iPhoneAppStore (600K+apps) WindowsMktplace (82K+apps)

: Search terms
* Source: App Store websites (as of 4/11/2012) 7
TristanJoo@wca.org (Apr. 2012)

So, whats holding back the Sensors Apps?

Lack of Education of App Developers on Sensors capabilities
Some sensors are complex beasts for average developers to embrace Need creative juice to figure out how to differentiate UX using sensors

Fragmentation causing UX Inconsistencies

OS versions with varying Sensors support OEM Devices with varying Sensors integration (that Apps developers dont access) Sensors with varying feature definitions (axis conventions, etc.) and performance

Evolution of Native OS Sensors Support (a Moving Target)

New sensor types added New virtual sensors added (e.g., auto-rotation, gravity, linear accel, etc.) Modified APIs or dropped features (Android Orientation -> Rotation Vector, etc.) Lacking OS support for proprietary extensions

Lack of Industry Standard Sensor specifications

Unspecified or vague performance targets Sometimes overly restrictive

Mobile Sensors: Fusing into Mobile Operating Systems

Sensors Comparison across Mobile OS Platforms

Native OS Sensor API Support
Android 4.0
(Oct. 2011) LINEAR_ACCELERATION ORIENTATION ROTATION VECTOR /QUATERNION GRAVITY ACCELEROMETER GYROSCOPE MAGNETIC FIELD PROXIMITY AMBIENT LIGHT PRESSURE AMBIENT TEMPERATURE RELATIVE HUMIDITY MULTI-TOUCH CAMERA (FRONT / REAR) Yes3 Yes1 Yes3 Yes3 Yes Yes Yes Yes Yes Yes2 Yes2 Yes2 Yes Yes

Sensor API

Android 2.3
(Dec. 2010) Yes3 Yes Yes3 Yes Yes Yes Yes Yes Yes Yes2 No No Yes Yes

Android 2.2
(May 2010) No Yes No No Yes No No Yes Yes No No No Yes Yes

WinPhone7.5
(Sept. 2011) Yes Yes Yes3 Yes Yes Yes Yes No No No No No Yes Yes

iOS 5
(Oct. 2011) Yes Yes Yes3 Yes Yes Yes Yes Yes Yes No No No Yes Yes

iOS 4
(June 2010) Yes Yes Yes3 Yes Yes Yes No Yes Yes No No No Yes Yes

1. Deprecated. 2. Newly added. 3. Requires Gyro. 10 TristanJoo@wca.org (Apr. 2012)

* Source: OSV websites & interviews (Apr. 2012)

: Also available as Software Sensors

Sensor Fusion Implementations: Android Case

4 Levels: App Processors, Sensor Hub (Co-MPU), Sensors (w/ MPU), Linux Kernel

Software (Virtual) Sensors 11 Device/Sensor-agnostic Integration Common Sensor fusion libraries Native OS API: Power inefficient Native OS API: Accuracy & Performance poor 3rd Party API: Accuracy & Performance improved

Hardware Sensors + Software Integration Device/Sensor specific Integration (Fragmented) Proprietary Sensor fusion libraries A/P level: Power hungry Sensor level: Low-pwr Sensor Vendor API: Accuracy & Performance best 3rd Party API: Accuracy & Performance optimized

TristanJoo@wca.org (Apr. 2012)

MEMS Motion Sensor Fusion Software Providers

Sensor Silicon-specific (IMU + Software & API)

Sensor Silicon-agnostic (Software & API)

12

Sensor Fusion System Integration Challenges

18+ Sensor types 18+ Mainstream Sensor vendors 30+ Sensor product lines 3+ Mobile OS requirements sets 35+ OEM device makers 5+ Third-party software libraries Algorithm tuning for each sensor Sensor mechanical design and placement considerations Sensor calibration requirements Sensor driver integration Sensor fusion library integration

Fragmentation

Scalable
13

A Way Forward: Standardization

Opportunities for Standardization
Sensor Data sheet contents Sensor Calibration and Fusion Library APIs Sensor Self-test and Reporting methods Sensor Configuration (e.g. axis conventions) Sensors Features & Performance standards for native Mobile OS support

Recommendations
Standardize basic Sensor functions to enable rapid bringup and test
Combat Fragmentation concerns from diversifying Sensor & OEM vendors

Ease addition of proprietary enhancements on top of standard features

Sensors: Support product differentiations in addition to standard requirements Mobile OS: Support extensible common APIs for improving set of functions

Accelerate innovations and future roadmaps with UX & Use Cases in mind

14

MEMS for All of Us Mobile Summary

Sensor Innovations continue to enable smarter Mobile Devices
12 ~ 18 sensors in todays smartphones, 5 fueled by MEMS

Exciting Mobile UX Transformation ahead leveraging Sensor Fusion

Mobile Sensor Fusion innovations started / led by MEMS Motion sensors Developers need education

As native support of HW Sensors advance across Mobile OSes, simpler & better performing extensions of Virtual / Fused Sensor APIs become crucial Mobile Ecosystem (esp. MEMS) needs concerted Standardization efforts to effectively scale!
Highly fragmented across Sensors, OS & Device Implementations Developers need standardized API support

15

Thank You! Q&A

[ Invitation to Connect ]
- Email: tristanjoo@wca.org - Twitter: @tristanjoo - WCA: www.wca.org/sig/mobile
16