Harness the Rise of the Machines: FastCV, Accelerating Mobile Computer Vision

Michael Mangan
Product Manager, Qualcomm Inc.

Computer Vision Use Cases

Augmented Reality Gestures

Face Recognition

Text Recognition

Depth Mapping

CV with active IR camera

Computer Vision Use Cases

Background Subtraction

Panoramic Photography Adult Content Blocking

Computational Photography

2D to 3D Video Conversion

Image Stabilization

FastCV Overview
FastCV is an API & library which is designed to enable Real-Time Computer Vision (CV) applications. FastCV enables mobile devices to run CV applications efficiently.
FastCV Value* Smaller Scope Specifics APIs most widely used APIs most computationally intense APIs suitable for heterogeneous core optimization More granular API Better power/performance

FastCV allows developers to HW accelerate their CV application.

FastCV is comparable to OpenGL ES in the rendering domain. FastCV is a clean modular library.
Optimize for embedded/ mobile

* FastCV Value relative to OpenCV

4

One Development Toolkit Two Implementations

FastCV for ARM
FastCV for Snapdragon Processors

CPU

GPU

CPU

GPU

Neon

DSP

VeNum

DSP

Benefits of FastCV
Function NCC Dot Product 128x4 Convert YUV420 Sobel Median3x3 Gaussian3x3 Gaussian5x5 Threshold Integral Image Harris Corner Dilate Erode Perspective Fit LK Optical Flow OpenCV 1.0x 1.0x 1.0x 1.0x 1.0x 1.0x 1.0x 1.0x 1.0x 1.0x 1.0x 1.0x 1.0x 1.0x FastCV for ARM 9.0x 4.0x 1.4x 1.8x 3.8x 2.6x 1.4x 0.7x 1.1x 2.8x 1.4x 1.3x 21.5x 2.0x FastCV for Snapdragon 23.1x 10.0x 1.3x 7.8x 51.9x 4.1x 2.9x 9.7x 1.3x 8.6x 15.0x 15.0x 37.8x 14.3x

Source: Qualcomm internal testing on MSM8660 device.

Gain Is More Than Time

Measure CPU frequency along with times Utilize single CPU in Linux performance mode

CPU Frequency Long Algorithm Time Short Algorithm Time

Source: Qualcomm internal testing on MSM8660 device.

FastCV Architecture
CV Applications AR Gestures Facial Recognition Other

Augmented Reality APIs

Gestures APIs

Facial Recognition APIs

Defined APIs

Optimized Framework

Augmented Reality

Gesture Processing

Facial Recognition

3rd Party CV Frameworks

Computer Vision APIs

FastCV Technology on Snapdragon Processors

FastCV ARM

Kernel

Display Drivers

Camera Drivers

Hardware

Snapdragon CPU Core(s)

Adreno GPU

Video Core

Hexagon

Connectivity Sensors etc.

FastCV 1.0 Feature Grouping

Math / Vector Operations Commonly used vector & math functions Image processing Image filtering, convolution and scaling operations Image transformation Warp perspective, affine transformations Feature detection Fast corner detection, harris corner detection, canny edge detection Object detection NCC based template matching object detection functions 3D reconstruction Homography, pose evaluation functions Color conversion Commonly used formats supported: e.g., YUV, RGB, YCrCb, etc. Clustering and search K clusters best fitting of a set of input points
9

Industry Computer Vision Solutions

FastCV is a processor-core agnostic acceleration
Khronos is looking to provide a standard CV API Potentially utilizing portions of OpenCV FastCV will evolve as Khronos standard is defined
High-level CV algorithms library Application Media interface

FastCV Hardware Acceleration API Hardware Abstraction Layer

Open source reference FastCV for ARM implementation (Reference implementation)

Hardware vendor HW Specific Implementations implementations

FastCV for Snapdragon Nvidia Intel Others

10

Feedback from our Developers

iOnRoad
We integrated FastCV very easily and quicklyI downloaded the FastCV SDK and went through the Getting Started Guide on the website. I set up the developer environment in Eclipse, installed the SDK, then compiled and ran the app...all I had to change was one line of code to call the hardware-accelerated function, then recompile, and I was ready to test a FastCV-integrated version of iOnRoad on my HTC Sensation 4G, which has a Snapdragon S3 processor. We went from about 22ms with our own YUV420-to-RGB function down to just 16ms for the FastCV function. Thats 30 percent less time for the image conversion, and an overall performance improvement of 10 to 15 percent for the entire algorithm. We proved that FastCV is very simple and powerfuland we plan to keep looking for additional parts of our algorithm to optimize. I expect iOnRoads performance to increase with more hardware acceleration, which will allow us to enrich the app with new CV-related features.

Ram Dayan, Head of Software, iOnRoad

11

Feedback from our Developers

TvTak
TvTak achieves faster image conversion and better performance using Qualcomm's FastCV technology, the first mobile-optimized computer vision (CV) library for ARM-based devices. It took us just a few days for our R&D team to integrate FastCV technology. Qualcomms mobile-optimized CV library makes our video detection work faster, providing a better experience for the consumer. David Amselem, CEO, TvTak

12

FastCV 1.1 Released on QDN Summer 2012

In addition to our FastCV1.0.4 capabilities: 80+ Additional Functions Enchanced OS Support
**iOS version of FastCV SDK expected: Summer 2012**

Enhanced Documentation Enhanced Image Processing Capabilities Enhanced Feature Detection and Tracking Capabilities Enhanced Gesture Recognition Support

https://developer.qualcomm.com/FastCV
13

https://developer.qualcomm.com/FastCV

14

The Tools and Resources Developers Need to Succeed

Snapdragon Developer Program
DEVELOP DISTRIBUTE DOWNLOAD DISCUSS

Snapdragon MDPs and Dragon Board

Early access to leading edge APIs and next generation features Accelerate development time for next-gen content Create and optimize applications

developer.qualcomm.com
Technical optimization support | SDKs | APIs | Sample Code | Guides and Tutorials | Developer Forums | Blogs | Discovery

15

Snapdragon Developer Program

How it Works
IDENTIFY DEVELOP COMMERCIALIZE
How tighter hardwaresoftware integration can improve your app (lower power, better video performance and killer gfx) What advanced mobile capabilities can be used in your app (AllJoyn P2P, Vuforia AR, advanced camera features, advanced audio capabilities, multi-screen experiences) Start at: developer.qualcomm.com and download SDKs, tools & resources needed to build your app Get early access to hardware and developer tech support Build your app and share it with us! Introductions to OEMs and carriers for preload opportunities Showcase your apps with Qualcomm at events and major tradeshows

16

Thank You
QUALCOMM Incorporated 5775 Morehouse Drive San Diego, CA. 92121-1714 U.S.A. Copyright 2012 Qualcomm Incorporated. All rights reserved.

Not to be used, copied, reproduced in whole or in part, nor its contents revealed in any manner to others without the express written permission of Qualcomm.
Qualcomm, Uplinq, Snapdragon, Adreno, and Hexagon are trademarks of Qualcomm Incorporated, registered in the United States and in other countries. FastCV is a trademark of Qualcomm Incorporated. Other products and brand names may be trademarks or registered trademarks of their respective owners.

Nothing in these materials is an offer to sell any of the components or devices referenced herein. Certain components for use in the U.S. are available only through licensed suppliers. Some components are not available for use in the U.S.

17

APPENDIX

18

Computer Vision Offload ARM to Hexagon

App CPU

Augmented Reality Java App finding objects in image using FastCV Feature Detect

Augmented Reality Java Application

Call Feature Detect

Fast CV Call Router

VeNum
ARM/VeNum FastCV Library
Feature Detect Function

App DSP
ARM Only FastCV Library
Feature Detect Function

Hexagon (QDSP6) FastCV Library

Feature Detect Function

Comparison of Feature Detect run on: App CPU (ARM) App DSP (Hexagon)

CPU Utilization (%) 98.0

100 80 60 40 20 0 120 100 80 60 40 20 0

Detection Time (msec) 117.0

800 600

Total Device Power (mW) 676.9 436.4

40.7

42.1

400 200 0

58% Less CPU

2.7x Faster
*Power measured at the battery on 8660

35% Less Power*

19

Source: Qualcomm internal testing on MSM8660 device.

Computer Vision Offload ARM/Neon to Hexagon

App CPU

Augmented Reality Java App finding objects in image using FastCV Feature Detect

Augmented Reality Java Application

Call Feature Detect

Fast CV Call Router

VeNum
ARM/VeNum FastCV Library
Feature Detect Function

App DSP
ARM Only FastCV Library
Feature Detect Function

Hexagon (QDSP6) FastCV Library

Feature Detect Function

Comparison of Feature Detect run on: App CPU (ARM/Neon) App DSP (Hexagon)

CPU Utilization (%)

100 80 60 40 20 0

Detection Time (msec)

120 100 80 60 40 20 0 800 600

Total Device Power (mW) 639.0 436.4

85.5 40.7

45.2

42.1

400 200 0

52% Less CPU

Similar
*Power measured at the battery on 8660

31% Less Power*

20

Source: Qualcomm internal testing on MSM8660 device.