TAUCHI Tampere Unit for Computer-Human Interaction

Implementation of New Interaction Techniques:

Computer Vision
Erno Mkinen (erno.makinen@uta.fi,
Pinni B0113)
Multimodal Interaction Research Group
TAUCHI

TAUCHI Tampere Unit for Computer-Human Interaction

Topics
Lecture 1
Introduction
Computer vision basics
Example applications

Lecture 2
Libraries & implementing software
Summary

TAUCHI Tampere Unit for Computer-Human Interaction

Touchless SDK*
Touchless library has C# API to
Capture frames from webcamera
Detect and track colored markers

The source code package contains two

samples and rather good documentation
InitExample
Video capturing
Adding and removing markers
Access to camera properties
*http://touchless.codeplex.com/

TAUCHI Tampere Unit for Computer-Human Interaction

Java
There are various ways to create computer
vision apps in Java
Use core Java functionality, e.g. Java Media
Framework and Java Advanced Imaging API
(http://java.net/projects/jai)
Use pure 3rd party Java library
ImageJ (http://rsbweb.nih.gov/ij/)
BoofCV (http://boofcv.org/)

Use Java wrappers for non-Java library

JavaCV (http://code.google.com/p/javacv/)
OpenCV Java and Processing library
(http://ubaa.net/shared/processing/opencv/)

TAUCHI Tampere Unit for Computer-Human Interaction

Java Media Framework

Download
http://www.oracle.com/technetwork/java/javase/downl
oad-142937.html

Material
Homepage:
http://www.oracle.com/technetwork/java/javase/tech/
index-jsp-140239.html
JavaWorld: Program multimedia with JMF
Part 1: http://www.javaworld.com/jw-04-2001/jw-0406jmf1.html
Part 2: http://www.javaworld.com/javaworld/jw-052001/jw-0504-jmf2.html
JMF 2 API specification:
http://www.oracle.com/technetwork/java/javase/specdo
wnload-136569.html

TAUCHI Tampere Unit for Computer-Human Interaction

Example: CaptureApp
Functionality
Captures video using a webcam and shows it in a JFrame

JMF classes in use

CaptureDeviceManager
CaptureDeviceInfo
Format
VideoFormat
Manager
Player
ControllerEvent
RealizeCompleteEvent

TAUCHI Tampere Unit for Computer-Human Interaction

C/C++

Various libraries for implementing general

and specific things

OpenCV
OpenFrameworks
ArToolKit
Zxing (http://code.google.com/p/zxing/)
...

TAUCHI Tampere Unit for Computer-Human Interaction

OpenCV
C functions and C++ classes for image
manipulation and computer vision
Includes trained cascaded detectors for
face and facial feature detection
Available for

Windows
Linux
MacOS X
Android

TAUCHI Tampere Unit for Computer-Human Interaction

OpenCV

Download:
http://sourceforge.net/projects/opencvlibrary/

Material:
Wiki:
http://opencv.willowgarage.com/wiki/
API documentation and examples come with the
package

TAUCHI Tampere Unit for Computer-Human Interaction

OpenCV

Example: facedetect.cpp
Comes with the OpenCV
Detects specified objects from video image
Draws circles to the object locations

TAUCHI Tampere Unit for Computer-Human Interaction

OpenFrameworks
Provides (easy) access to various libraries
Computer vision support
32 addons
For example, OpenCV, ofxZxing,
ofxARToolkitPlus

Many examples

easyCamExample
ofxCvHaarFinderExample
textureScreengrabExample
...

TAUCHI Tampere Unit for Computer-Human Interaction

ARToolKit
ARToolKit has portable C APIs to

Video capturing
Marker tracking and calibration
Camera viewpoint calculation
Virtual object drawing

Home page:
http://www.hitl.washington.edu/artoolkit/

Includes various examples

Comes with the ARToolkit
Tracks two patterns and draws object to video
image

TAUCHI Tampere Unit for Computer-Human Interaction

Microsoft Kinect

(from Microsoft Kinect SDK for Developers web page

http://www.microsoft.com/en-us/kinectforwindows)

Stereo camera
Infrared camera for depth perception

TAUCHI Tampere Unit for Computer-Human Interaction

Kinect SDKs
Microsoft SDK

OpenKinect

Official SDK
C++, C# wrapper
More robust skeleton
tracking
Speech Recognition
Tracks at most 2
skeletons
Closed Source

General purpose Natural

Interaction SDK
C++, C# & Java wrappers
Confidence measure
Missing Audio API
Open Source
http://openkinect.org/w
iki/Main_Page

See also: Evoluce, OpenNI, CL NUI

TAUCHI Tampere Unit for Computer-Human Interaction

SPI Group Work

Presented at Machine Tool World Exposition
(Hannover, Germany)
FIMECC Award 2011
Next: Kinect Home Automation
(projectors, sensors,
mobile, speech, audio...)

Contact sebastian.muller@uta.fi
or markku.turunen@uta.fi for
more information

TAUCHI Tampere Unit for Computer-Human Interaction

Summary

A lot of research carried out in computer

vision field but still many unsolved issues
Fortunately many tools exist
Also for mobile phones
Choose the one that best fits your goals and
skills