Alex Karpus Honours Project 2009

AN OPENCV IMPLEMENTATION OF A QR BARCODE READER FOR THE IPHONE AND WINDOWS MOBILE PLATFORMS
H ONOURS P ROJECT
Written By: Alex Karpus Supervised By: Dr. Gerhard Roth Carleton University December 13, 2009
ABSTRACT
With the ever changing technology market, mobile devices are the latest in powerful, compact and capable line of consumer electronics. Many existing applications are finding new uses on these devices. While there are many different areas currently benefiting from the mobility and computational power offered by todays mobile devices, the field of Computer Vision in particular, is seeing resurgence in new innovation targeted for these devices. This paper details the process of the design, development and implementation of a QR Barcode reader. The main objective of this paper is to serve as a guide for developing or porting new and existing computer vision application to the iPhone and Windows Mobile development platforms. As part of this paper, OpenCV, a leading open source computer vision framework has been ported to work on both the iPhone and Windows Mobile powered mobile devices.
2 Alex Karpus, 2009
ACKNOWLEDGEMENTS
I would like to thank Dr. Gerhard Roth for supervising this project, for his continued input along the way and help with this paper. I hope that the results of this project will be useful to him and his future students. I would also like to thank Daniele Pizzoni and Yoshimasa Niwa for their great work in porting OpenCV to the iPhone. Also thank you to Nishi Takao for his work on the libdecodeqr library. As well as Rhonda Software for providing useful and insightful observations and code for making OpenCV usable on the iPhone platform.
3 Alex Karpus, 2009
TABLE OF CONTENTS
Introduction ............................................................................................................................................ 7 Background ............................................................................................................................................ 9 Computer Vision ................................................................................................................................. 9 Barcodes and the QR Code................................................................................................................ 10 OpenCV ............................................................................................................................................ 11 Libdecodeqr ...................................................................................................................................... 12 iPhone SDK ....................................................................................................................................... 13 Windows Mobile SDK ........................................................................................................................ 14 Goals and Objectives ............................................................................................................................ 15 Motivation ........................................................................................................................................ 15 Objectives ......................................................................................................................................... 15 Learning Goals .................................................................................................................................. 16 QR Code Reader Design .........................................................................................................................17 QR Code Structure .............................................................................................................................17 Reader Design ................................................................................................................................... 18 Status Guide.................................................................................................................................. 20 Experiments ...................................................................................................................................... 22 OpenCV on Mobile Devices ................................................................................................................... 24 iPhone............................................................................................................................................... 24 Prerequisites ................................................................................................................................. 25 Compiling ...................................................................................................................................... 25 Using OpenCV ............................................................................................................................... 26 Windows Mobile ............................................................................................................................... 27 Prerequisites ................................................................................................................................. 28 Compiling ...................................................................................................................................... 29 Using OpenCV ............................................................................................................................... 39 QR Code Reader for the iPhone ............................................................................................................ 41 Design ............................................................................................................................................... 42 User Interface.................................................................................................................................... 44 QR Code Reader for Windows Mobile ................................................................................................... 46 4 Alex Karpus, 2009
Design ............................................................................................................................................... 47 User Interface.................................................................................................................................... 51 Results .................................................................................................................................................. 53 QR Code Reader Accuracy ................................................................................................................ 53 QR Code Reader Performance .......................................................................................................... 54 Conclusion ............................................................................................................................................ 56 Contributions to the Field.................................................................................................................. 56 Future Work ...................................................................................................................................... 57 References ............................................................................................................................................ 58 Appendices ........................................................................................................................................... 60 Appendix A CD Contents ................................................................................................................ 60 Appendix B Test Cases ................................................................................................................... 62 QR Code Test Sample #1 .............................................................................................................. 63 QR Code Test Sample #2 .............................................................................................................. 64 QR Code Test Sample #3 .............................................................................................................. 65
5 Alex Karpus, 2009
LIST OF FIGURES
Figure 1 - Basic OpenCV structure (Bradski & Kaehler, 2008) ................................................................ 12 Figure 2 - QR Code structure (Wheeler, 2008) ........................................................................................17 Figure 3 - Structure of QR Code reader application ............................................................................... 18 Figure 4 - Windows version of iQR ........................................................................................................ 20 Figure 5 - QR Code reader adaptive threshold ....................................................................................... 22 Figure 6 - CMake configuration window ................................................................................................ 30 Figure 7 - Unloading OpenCV projects in Visual Studio ......................................................................... 32 Figure 8 - Adding a new solution platform in Visual Studio ................................................................... 32 Figure 9 - Visual Studio project Properties dialog .................................................................................. 33 Figure 10 - QR Code reader for the iPhone ............................................................................................ 41 Figure 11 - iQR design for the iPhone .................................................................................................... 42 Figure 12 - UIImage conversion to and from IplImage(Rhonda Ltd, 2009) ............................................. 43 Figure 13 - iQR main screen for the iPhone ........................................................................................... 44 Figure 14 - iQR taking a picture for the iPhone ...................................................................................... 44 Figure 15 - iQR processing image for Windows Mobile .......................................................................... 45 Figure 16- iQR decoded message for the iPhone ................................................................................... 45 Figure 17 - QR Code reader for Windows Mobile ................................................................................... 46 Figure 18 - iQR design for Windows Mobile ........................................................................................... 48 Figure 19 - Marshalling strings to and from unmanaged code ............................................................... 50 Figure 20 - Memory management between managed and unmanaged code ........................................ 51 Figure 21 - iQR main screen for Windows Mobile .................................................................................. 52 Figure 22 - iQR taking a picture for Windows Mobile............................................................................. 52 Figure 23 - iQR processing image for Windows Mobile .......................................................................... 52 Figure 24 - iQR decoded message for Windows Mobile......................................................................... 52 Figure 25 - QR Code reader performance graph .................................................................................... 54
LIST OF TABLES
Table 1 - Libdecodeqr successful statuses ............................................................................................. 21 Table 2 - Libdecodeqr failed statuses .................................................................................................... 21
6 Alex Karpus, 2009
INTRODUCTION
Mobile devices have reached a ubiquitous status in society. The number of cell phones, personal digital assistants (PDAs) and general handheld mobile devices has long surpassed the number of computers in the world (CIA - The World Factbook - World - Communications, 2009). Most mobile devices today are empowered by full color displays, integrated digital cameras, fast processors and even dedicated 3D graphic chips. This technology has allowed these mobile devices to become capable and sophisticated computing peripherals capable of fulfilling a variety of personal and commercial purposes. One purpose that mobile devices have started to rapidly fulfill is in the area of computer vision. For instance, bar codes can now be read on most mobile devices. Bar codes are simply two-dimensional machine-readable representations of information, like text, a phone number or a uniform resource locator (URL). Thus an opportunity exists to take advantage of camera equipped mobile devices for computer vision development in areas such as barcode reading without the use of lasers as is common in the commercial sector. A particular type of barcode that is of interest is the Quick Response (QR) Code. This is a matrix code which is not only efficient for storing large amount of information, but has also been designed for high-speed decoding. This paper and the associated deliverables are about the development of a QR code reader for the Apple iPhone and Microsoft Windows Mobile Smartphone. As part of the project, a foundation of knowledge was built to allow rapid computer vision development on the both the iPhone and Window Mobile device platforms using the Open Computer Vision Library (OpenCV). The information in this paper will help minimize the overhead encountered in porting existing code to the new mobile device platforms and reduce the development time of new computer vision algorithms for mobile devices.
7 Alex Karpus, 2009
The accompanying QR code reader serves as an example application, highlighting the possibilities of OpenCV on the mobile platform. The next section of this report introduces background information on the technologies and concepts involved in this project.
8 Alex Karpus, 2009
BACKGROUND
COMPUTER VISION
Computer vision is a branch of computer science concerned with analyzing images to extract information about an environment. It may seem that concepts of computer vision are quite easy. After all, how hard can it be to distinguish between a tree, a car and a barcode all present in the same image? However, our initial intuitions can be quite misleading because the task of recognizing objects in our environment is something we take for granted. In fact, something as simple as recognizing an object is performed by complex and as of yet, little understood system and processes in the brain (Chai, 2009). Much like the human visual system, computer vision systems can perform the same visual functions except that what we perceive as colors or patterns, computers see as grids of numbers, which can represent different colors, luminosities, levels of saturation, and other conceptualized values. The simple cameras used in computer vision are primitive with no pattern recognition, no automatic focusing and most importantly, no previous understanding and experience to rely on. Through artificial intelligence techniques and image processing algorithms, computer vision applications can extract information about an image. There are many algorithms that exist today, capable of performing simple tasks like revealing contours in an image, or balancing the light exposure evenly across an image. The overall task is still quite daunting. To go from a grid of numbers, into a conceptualized idea about an image is still a very difficult thing. With the advent of mobile technology, computer vision now has yet another outlet for testing and refining its algorithms and ideas.
9 Alex Karpus, 2009
BARCODES AND THE QR CODE

Originally invented and patented in 1957 by Woodland and Silver, a barcode was envisioned for automatically reading product information during a grocery store check out (Bellis, 2009). However since then many different kinds of barcodes have been developed and put to use in a variety of environments. In 1994 a Japanese company Denso Wave developed a new type of 2D matrix barcode called the Quick Response (QR) code. This new type of code has a high capacity for encoding data, prints smaller than regular barcodes while encoding the same data and due to its built-in error checking design, is quite damage tolerant. It was initially designed for use in auto part manufacturing, allowing for quick part identification in an assembly line process. However due to its advantages over typical barcodes its usage has since expanded into the consumer world and has become particularly wide spread in Japan. QR Codes now appear on books, magazines, CDs, and other merchandise available to regular consumers. With the advancements in recent years of the technology behind mobile devices and their integrated digital cameras, it is now possible to develop consumer-grade mobile device applications which, using the built in camera technology, can decode barcodes on the fly. Since QR Codes can store anything from uniform resource locators (URLs), phone numbers and text, their usage has become widespread. Although the QR Code has been considerably more popular in Japan than in North American its usefulness is not lost on major technologically centered businesses. Recently Google, a large American corporation offering many web services, released their initiative for QR Codes in the United States. Their plan is to provide the 100,000 most popular local businesses with decals featuring a QR Code. The businesses are encouraged to place these decals in their windows so that passing customers can use their mobile phone to find out more information about the local business. The customer can read
10 Alex Karpus, 2009
reviews, see pictures of the venue or even see any available coupons, all from their mobile device (Hayward & Kim, 2009). Because of its wide popularity, and the backing of companies like Google, the QR Code was chosen as the subject for the application development portion of this paper.
OPENCV
Open Computer Vision Library or OpenCV for short, is an open source computer vision library. It is a very strong framework which aims to provide a simple-to-use computer vision infrastructure that allows developers to build sophisticated computer vision applications with little overhead. The library is written in C with a recently rewritten C++ interface. It runs natively under Windows, Linux and Mac OS X (Bradski & Kaehler, 2008). The OpenCV library contains over 500 functions that cover many areas in computer vision, including camera calibration, stereo vision, robotics, user interface, and factory product inspection. By popular demand of machine learning algorithms with computer vision implementations, OpenCV also has a general-purpose machine learning library.
CV
Image processing and Vision Algorithms
ML
Statistical Classifiers and Clustering Tools
HighGUI
GUI, Image and Video I/O
CXCore
Basic structures and algorithms, XML support, drawing functions
Figure 1 - Basic OpenCV structure (Bradski & Kaehler, 2008)
The basic structure of OpenCV (Figure 1) is split into four main components; CV, ML, HighGUI and CXCore. Each component contains a certain set of functionality. The CV component contains functionality for basic image processing and manipulation as well as computer vision algorithms; ML contains all the machine learning functionality which includes clustering and statistical classifiers. HighGUI contains functionality for loading and saving images, video. Finally CXCore is a common component between all the other components because it contains all the core data structures and content commonly used by other components. Due to a plethora of existing work already done with the OpenCV framework, it was chosen for its robustness and ease of use. The native C/C++ interfaces also made it an ideal choice as both the iPhone SDK and the Windows Mobile SDK support similar languages.
LIBDECODEQR
Libdecodeqr is a C/C++ library developed by Nishi Takao for the decoding of QR Codes based on the JIS X 0510 and ISO/IEC18004 standards. The library relies on OpenCV for the majority of its
image processing functionality. It can be easily integrated into various platforms for high speed decoding (Takao, 2007). While it was feasible to implement a QR Code decoder from scratch, a document describing the QR Code implementation would be required. Due to the cost of the document describing the encoding and time required to implement already existing functionality, libdecodeqr was chosen to aid in reducing the development time required for implementing a useful mobile QR Code reader. Its reliance on OpenCV made it an ideal choice. IPHONE SDK The Apple iPhone SDK is a complete software developer kit (SDK) used in the creation of software for the iPhone OS. The SDK has four distinct layers, which are the Core OS, Core Services, Media, and Cocoa Touch (Apple Inc, 2009). The Core OS layer is composed of the XNU kernel, drivers, and basic interfaces of the iPhone OS. All virtual memory system, file system, threading, inter-process communications, and network are managed in this Core OS layer. It also provides functionality for interacting with the iPhones hardware components. The Core Services layer provides core system services and data structures that are shared by all applications. Basic data management, including data types and collections, bundles, string management, and more are available to the developer through this layer. The Media layer houses all of its graphic, audio, and video functionality. This layer allows developers to extend the standard user interface from 2D to 3D rendering with support for animations, audio tracks, and video playback.
The Cocoa Touch layer provides developers with access to user interface frameworks like with support for things like application management, window support, text and web content support, and gestures with motion based events. The SDK supports C, C++ and Objective-C languages and for the purposes of this project, only Objective-C was used.
WINDOWS MOBILE SDK

The Microsoft Windows Mobile SDK is a complete software developer kit used in the creation of software for the Windows Mobile OS. The latest version of the developer kit and the one used in this project is version 6. The SDK is divided into three different types of deployments, Professional, Standard, and Classic. The difference in the types lies in the hardware that an application was developed for. Professional referring to mobile devices that include cellular phone functionality, while classic is used solely by PDAs. All of the work done in this project was developed and tested on a Palm Treo 700wx Smartphone running Windows Mobile 6 which falls under the Professional deployment category type. The SDK is further divided into many distinct components of functionality like CellCore for wireless connection oriented services, ActiveSync for synchronization of data between your computer and mobile device and many others (Microsoft, 2008). The SDK supports application development using both C++ and C# languages. The Windows Mobile platform supports the .NET Compact Framework. This project relies on both of these languages.
GOALS AND OBJECTIVES

MOTIVATION
The rapid growth of the mobile device market in the last couple of years has created new opportunity for innovation. In particular the area of Computer Vision has been experiencing resurgence in interest as established algorithms and applications are finding new life on mobile devices. This project was inspired by the emerging need for simplifying the migration path of existing applications onto mobile devices. Since the mobile device market has multiple competing devices with very different development platforms, a roadmap for simplifying the transition was something that I felt would be useful both for Carleton Universitys future mobile development stream, as well as the open source community involved in OpenCV development. It is my hope that my work will reduce the development time for individuals interested in computer vision related projects on the popular iPhone and Windows Mobile platforms.
OBJECTIVES
The motivation for this project has some associated objectives that were completed and are outlined in this paper. 1. Research existing QR Code systems. Implement a QR Code reader with basic context awareness using the computer vision OpenCV framework. 2. Research and develop a roadmap for the work required in porting existing OpenCV applications to mobile devices. In particular focusing on the iPhone and Windows Mobile device platforms.
3. Successfully port the QR Code reader application developed as per the first objective to the iPhone and Windows Mobile device platforms in order to demonstrate the usability and completeness of the second objective.
LEARNING GOALS
My personal motivation in taking on this project was to learn and accomplish several things. Mainly I have been fascinated by the growth of the mobile device market and thus wanted to learn the skills necessary for developing mobile applications on the popular mobile platforms. Prior to this project I had rudimentary mobile development experience from various personal projects which I developed using Java2ME. However the mobile device market has changed a lot and so have the dominant platforms. As such I wanted to learn about the iPhone SDK and the Objective-C language, which as of yet, I had never worked with in either my personal or professional careers. As well the Windows Mobile platform and the C++/C# languages were already familiar to me but I had never actually gone through the phases of development on this platform. I was also interested in the difference between mobile (embedded) architecture and the conventional architecture of everyday computers when it came to reusing existing code on new devices. This is why I felt that a great way to learn about differences and similarities would be to go through the work required in successfully porting existing code to a new platform.
QR CODE READER DESIGN

QR CODE STRUCTURE
Figure 2 - QR Code structure (Wheeler, 2008)
In order to understand the QR Code reader (iQR) design it helps to understand a little about the structure of the QR Code. The QR Code region is made up for three position tags located in the upper left, bottom left and upper right corners. This is in fact a unique feature which can be used to distinguish QR Codes from other 2D barcodes. These are used for locating the QR Code in an image and help serve as markers. They are also helpful in determining the degree of skew and rotation of the code in the image. There is also an alignment tag, which can be used for further alignment calculations. For redundancy, the version and format information is stored in at least two places around the position tags. This allows for better data recovery should a corner of the code be damaged or obstructed. Because different types of data can have different representations in the QR Code, it is vital that as part of the decoding process, the format information be recoverable. Otherwise the encoded data is left unreadable. The version information can be used for backward capability, in supporting older revisions of the encoding protocol.
The checkered pattern present, both vertically and horizontally between two neighboring positions tags allows for a calibration routine to perform calculations to determine the size of single black and white regions. One of the main strengths of the QR Code is its multiple levels of error correction. It uses a ReedSolomon error correction technique and depending on the level of data corruption can support up to 30% of data restoration. Due to the complexities involved full error correction support is not built into libdecodeqr.
READER DESIGN
iQR
Acquire Image
libdecodeqr
Process Image
OpenCV
Decode Image
Display Results
Figure 3 - Structure of QR Code reader application
The QR Code reader (iQR) is an application which utilizes both libdecodeqr and OpenCV in order to decode an image containing a QR Code. 18 Alex Karpus, 2009
The design was intended for portability between platforms. Once ported to platform, a user interface can be wrapped around the basic design to simplify the ease of use and better deliver decoded results. A more detailed process flow shown in Figure 3 is described below. 1. Acquire image. Depending on the platform, the image can come either from file, camera or image library of the device. The actual implementation of acquiring an image is platform dependent. 2. Handover the acquired image to the Libdecodeqr library for image processing. All of libdecodeqr is implemented using OpenCV functions. Thus due to OpenCVs portability (See the OpenCV on Mobile Devices section on page 24), this part is platform independent. a. Apply adaptive binary thresholding. Image processing technique which reveals image detail hindered by poor or uneven lighting conditions. b. Fix the effects of skew and camera rotation on the QR Code. The three position tags and alignment tag are used to extract the QR Code and fix perspective issues (skew, rotation). c. Decode the resulting image. 3. Display results to the user.
Figure 4 - Windows version of iQR
The Windows version (Figure 4) works by reading in an image from a file and processing it. In the example figure, a generated QR Code containing a URL was fed into the program. The program was able to locate the position tags, find and extract the QR Code from the image and print out the decoded results. See the QR Code Reader for the iPhone section on page 41 and the QR Code Reader for Windows Mobile section on page 46 for the mobile versions of this application.
STATUS GUIDE
Once the QR Code is analyzed and processed, the decoded data (if any) is accompanied by a status code. Depending on the success of the decoding process, the status helps debug where in the process decoding difficulties were encountered.
Following are two tables highlighting the different possible statuses. Successful Status (base 16) 2000 0f00 0100 0800 00f0 0010 0080 000f 0001 0002 0008 Description Image decoded Genera version info error. Version info mismatch. Unrecoverable version info. General format info error. Invalid format level used. Unrecoverable format info. General data error. Not supported data format. Length of data stored not as expected. Unrecoverable data.
Table 1 - Libdecodeqr successful statuses
Failed Status (base 16) 4000 0100 0200 0400 0800
Description Image could not be decoded. Unsupported image format. Position tags could not be found. Code area could not be found. Code area could not be determined.
Table 2 - Libdecodeqr failed statuses
The first table (Table 1) is a collection of statuses returned when a QR Code was found in the image and an attempt at decoding it was made. The second table (Table 2) is a collection of statues returned when a QR Code cannot be located in the image being decoded. In each case, the first row of the table contains the main status to which the entries following it are logically added. For example, a status of 0x2089 would indicate that the image was processed, QR Code was found, however the format info could to be recovered and neither could the data (perhaps because the data format was unsupported).
EXPERIMENTS
As part of testing the QR Code reader design and reliability, several attempts were made at improving the accuracy and performance aspects of the design. One such experiment was in testing the affects of applying adaptive light thresholding and perspective skew adjustment based on the location and positioning of the three corner position tags of the source image. Using OpenCVs implementation of the adaptive thresholding algorithm the source image was first adjusted and then it was allowed for the libdecodeqr code to try and detect the image.
Figure 5 - QR Code reader adaptive threshold
This two step process would allow for the position tags to be found. Using the position tags, it would be possible to estimate the rotation of the QR Code in the image. Measuring the displacement of the top two position tags and the two left position tags would allow for the calculation of the angle or rotation. Also the stretch of the image could be approximated by measuring the distance between the top two position tags and comparing that to the distance between the two left position tags. Scaling the resulting image to the larger of the two would help eliminate simple stretch.
However, having applied these basic techniques, to the image prior to the decoding process led to the discovery of little to no improvement. The result of this was due to the fact that the libdecodeqr already implemented a similar approach for handling lighting and rotational/skew effects on the source image. Figure 5 shows a screenshot of what an original input image looks like compared to an image where adaptive thresholding and minor rotation compensation has been applied.
OPENCV ON MOBILE DEVICES

When this project was first envisioned, information about OpenCV on mobile devices existed but was quite scarce. This was one of the motivating factors for bringing about a solution that would contribute to the field and offer a chance to learn about the emerging mobile market. There are several things to consider when porting existing code to new environments. In order to successfully cross compile code for a new system, it is important to know what architecture is used by the system. The system architecture commonly used in mobile devices currently on the market is the Advanced RISC Machine (ARM) architecture. It was developed by ARM Limited and originally intended for desktop computers. However the success of the x86 IBM PC has pushed ARMs efforts into the mobile and embedded device markets instead (ARM architecture, 2009). Both the iPhone and the Windows Mobile development platforms offer full ARM processor support and thus share similar paths for porting OpenCV. IPHONE Up to now I have referred to the iPhone as a single device. When in fact there have been three different versions of the iPhone as of the time of this paper. All of the development involved in this project was carried out on the iPhone 3G. From now on, all information provided will be specific to the iPhone 3G. The iPhone use a 412MH ARMv6KZ processor from the ARM11 family of architecture. It has 128MB of RAM and 8GB of internal storage. The primary digital camera has a 2MP CMOS sensor capable of producing images of up to 1600x1200 pixels in size, however there no video support (iPhone, 2009).
The sections that follow outline the prerequisites and give instructions for compiling and using OpenCV 2.0 in iPhone applications.
PREREQUISITES
To successfully compile OpenCV for the iPhone you will need: An Intel powered Macintosh computer running Mac OS X 10.5+ (Leopard tested) The iPhone SDK 3.0+ developer environment and tools. (iPhone SDK 3.0 tested) http://developer.apple.com/iphone/
OpenCV 2.0.0 (OpenCV-2.0.0.tar.bz2, 2009-10-01 tested) http://sourceforge.net/projects/opencvlibrary/files/
COMPILING
In order to compile OpenCV into static universal libraries, follow the steps outlined here. 1. Create a staging directory. 2. Extract OpenCV into the staging directory. 3. Place the `build_opencv.sh` iPhone build script into the staging directory (See Appendix A on page 60 for information on the scripts accompanying this document). 4. Open a Terminal window. 5. Navigate to the staging directory. 6. Run the `build_opencv.sh` script. This script will attempt to patch and build OpenCV. The whole build process can take up to 15 minutes. 7. Once everything is completed, the staging directory will contain three new folders. a. òpencv_simulator` - Contains the static OpenCV libraries which can only be used for testing in the iPhone simulator.
b. òpencv_device` - Contains the static OpenCV libraries which can only be used on iPhone hardware. c. òpencv_universal` - Contains the static OpenCV libraries which are a hybrid of the libraries found in the previous two folders. These libraries can be used both in the simulator and on the hardware. The structure of the òpencv_universal` folder is as follows: opencv_universal - libs Contains the universal static libraries. - libcv.a - libcxcore.a - libcvaux.a - libml.a - libhighgui.a - include Contains all the relevant OpenCV header files.
USING OPENCV
The steps outlined in the previous section will have produced five static libraries, which can now be integrated into your development environment. The Mac OS X Developer toolkit comes with an IDE called Xcode. Since Xcode is the most widely used IDE, the steps outlined below are tailored to it. However the libraries generated in the previous section can be used with any build environment. The instructions that follow assume that an Xcode project already exists. If not create it now. 1. Copy the òpencv_universal` folder generated in the previous section, into your Xcode project directory. 2. Launch Xcode and open the project. 3. From the menu, go to Project > Edit Project Settings 4. Select the Build tab 26 Alex Karpus, 2009
5. Add the following flags to the Òther Linker Flags` Linking setting
$SRCROOT/opencv_universal/libs/libcv.a $SRCROOT/opencv_universal/libs/libcxcore.a $SRCROOT/opencv_universal/libs/libcvaux.a $SRCROOT/opencv_universal/libs/libml.a $SRCROOT/opencv_universal/libs/libhighgui.a lz lstdc++
6. Add the following path to the `Header Search Paths` Search Paths setting:
$SRCROOT/opencv_universal/include/opencv
You are now ready to use OpenCV in your project. Dont forget to include the OpenCV header files in your code:
#include cv.h #include cxcore.h #include highgui.h
See the QR Code Reader for the iPhone section on page 41 for a demonstration of the implementation and usage of OpenCV on the iPhone.
WINDOWS MOBILE
Unlike the Apple iPhone and the iPhone SDK, Microsofts Windows Mobile operating system and associated environment give developers one environment in which to make mobile applications that will work on a multitude of hardware devices. This is because of a fundamentally different approach that Microsoft takes to software development, as opposed to Apple. Microsoft strives to make its software work on a vast array of devices which can support the Windows Mobile OS. Apple on the other hand, controls the hardware and the software giving little choice to the end consumer. Although there are downsides to Microsofts approach, just as there are downsides to Apples. However, I believe that the versatility of Windows Mobile is a greater asset because it allows for a single developer environment to create applications for differently 27 Alex Karpus, 2009
configured mobile devices. This is very important, especially for educational institutions, such as Carleton University, or any college or university interested in offering a mobile development program. All the Windows Mobile work for this project was done on a Palm Treo 700wx running Windows Mobile 6. Similarly to the iPhone, the Treo uses a 312MHz PXA270 Intel XScale processor based on the ARMv5TE family of processors. The phone has 128MB of memory, of which 62MB can be used for program and data storage. It has a 1.3 MP digital camera capable of producing images of 1280x1024 pixels in resolution with up to 2x digital zoom (Kairer, 2006). The sections that follow outline the prerequisites and give instructions for compiling and using OpenCV 2.0 in Windows Mobile applications. Unlike for the iPhone not all of OpenCV has been completely ported to Windows Mobile. Only the main components CV, ML, HighGUI and CXCore are ported as per the following sections. Also important to note that only image loading/saving capability of HighGUI was ported over, excluding OpenCV video and camera support.
PREREQUISITES
To follow the instructions outlined here you will need: A computer running Microsoft Windows XP or newer (Windows XP tested) Microsoft Visual Studio 2008 or newer (Visual Studio 2008 tested) Cross Platform Make (CMake) 2.6+ (CMake 2.8 tested) http://www.cmake.org/
Patch GnuWin32 (version 2.5.9 tested) http://gnuwin32.sourceforge.net/packages/patch.htm
Windows CE C Library Extensions (wcelibcex-1.0.zip) http://wcelibcex.sourceforge.net/
The Windows Mobile 6.0+ SDK developer environment and tools. (Windows Mobile SDK 6 tested) http://www.microsoft.com/downloads/details.aspx?familyid=06111A3A-A651-4745-88EF3D48091A390B&displaylang=en
OpenCV 2.0.0 (OpenCV-2.0.0.tar.bz2, 2009-10-01 tested) http://sourceforge.net/projects/opencvlibrary/files/
COMPILING
1. Create a staging directory. 2. Extract OpenCV into the staging directory. 3. Place the òpencv_winmo.patch` patch file into the staging directory (See Appendix A on page 60 for information on the files accompanying this document). 4. Open a Command Prompt window. 5. Navigate to the staging directory and then into the OpenCV directory. 6. Run the following command. It will patch the OpenCV source and make it Windows Mobile friendly. It should be noted that as part of this patch a lot of highgui functionality is removed due to incompatibility.
patch p1 < ../opencv_winmo.patch
7. Start CMake (cmake-gui)
Figure 6 - CMake configuration window
8. Click `Browse Source` to browse for the OpenCV source code. It should be within the staging directory. 9. Click `Browse Build` and set the folder where you want the configuration and compiling to take place. I recommend a folder on the same level as the OpenCV code inside the staging directory. 10. Click `Configure`. 11. The CMake system will prompt to select a generator for the project. Choose the build system you are going to use. This guide assumes you will use Visual Studio 9 (2008). 12. In the same prompt as in the previous step, select the Ùse default native compilers` option. 13. Click `Finish` to proceed. 14. It may take a moment for your build system to be verified. Once the process is complete the CMake window will look similar to Figure 6. 15. Set the following environment variables in the CMake window.
CMAKE_CXX_STANDARD_LIBRARIES =coredll.lib corelibc.lib ole32.lib oleaut32.lib uuid.lib commctrl.lib
CMAKE_C_STANDARD_LIBRARIES =coredll.lib corelibc.lib ole32.lib oleaut32.lib uuid.lib commctrl.lib CMAKE_EXE_LINKER_FLAGS = /STACK:10000000 /machine:thumb CMAKE_MODULE_LINKER_FLAGS = /STACK:10000000 /machine:thumb CMAKE_SHARED_LINKER_FLAGS = /STACK:10000000 /machine:thumb ENABLE_OPENMP =0 BUILD_TESTS =0 BUILD_NEW_PYTHON_SUPPORT =0
16. Click `Configure` in the CMake window. This will enable you to generate your Visual Studio solution. 17. Click `Generate`. The process will take a few moments and after it is done, you can exit CMake. 18. Navigate to your build directory. 19. Extract the Windows CE C Library Extensions archive into the build directory. 20. Build the Windows CE C Library Extensions library with the Windows Mobile 6 Professional SDK. If you are unsure of how this is done, skip this step and return to it once youre ready to compile libpng. This will you give you a chance to complete more of this guide and give you some experience with compiling for Windows Mobile. 21. Open the OpenCV solution file. This will launch Visual Studio. 22. Now in Visual Studio, in the Solution Explorer window, unload all the associated OpenCV projects. Refer to Figure 7.
Figure 7 - Unloading OpenCV projects in Visual Studio
23. Now select and reload only the following projects: cv, cxcore, flann, highgui, libjpeg, libpng, ml, opencv_lapack, zlib. This minimizes the compile time and includes only the essentials. 24. Right-Click on the `HighGui` project and set it as the startup project. 25. Open the Configuration Manager from the Build > Configuration Manager menu. 26. Add a new solution platform for the Windows Mobile 6 Professional SDK (ARMV4I). See Figure 8 for details. 27. Set the active solution configuration to be `Release` and close the configuration manager.
Figure 8 - Adding a new solution platform in Visual Studio
What follows now is a breakdown by component of the changes required to compile it. The list number continues, implying the order should be kept when following the guide. All the sections below deal with individual project properties. In order to better understand the instructions, the terminology is briefly described in the next section.
Individual Project Properties

The project properties dialog can be opened by selecting a project from the Solution Explorer and Right-Clicking on it and selecting `Properties`, or alternatively from the Project > Properties menu. The Properties dialog in Visual Studio 2008 looks like Figure 9.
Figure 9 - Visual Studio project Properties dialog
The dialog has several important areas. On the left hand side is a tree-view list of the different project areas that can be configured. The different areas control everything from the options used for compiling a project, the way it is debugged, to the output directory. 33 Alex Karpus, 2009
The majority of the options will be left untouched. However there are a few critical options that must be set manually. In the later sections of this guide, when it says Set the `Preprocessor Definitions` in the C/C++ Preprocessor section. What it is referring to is the C/C++ section in the left hand side of the dialog. Underneath which is a Preprocessor child, which when clicked on, reveals a slew of properties, one of them being the Preprocessor Definitions. Once youre done making changes, simply click OK and the changes will be saved. Now continue to the next sections for compiling individual parts of OpenCV.
Compiling Zlib
28. Add the following definitions to the `Preprocessor Definitions` in the C/C++ Preprocessor section.
NO_ERRNO_H _WIN32_WCE=$(CEVER) UNICODE _UNICODE
29. Build zlib. The build should complete without errors.
Compiling Opencv_lapack
UNDER_CE _WIN32_WCE=$(CEVER) UNICODE _UNICODE
31. Build opencv_lapack. The build should complete with some warnings but no errors.
Compiling Flann
WINCE _WIN32_WCE=$(CEVER) $(ARCHFAM) $(_ARCHFAM_) UNDER_CE UNICODE _UNICODE
33. Build flann. The build should complete with some warnings but no errors.
Compiling Cxcore
WINCE _WIN32_WCE=$(CEVER) $(ARCHFAM) $(_ARCHFAM_) UNICODE _UNICODE UNDER_CE
35. Set the `SubSystem` in the Linker System section to `WindowsCE`. 36. Add `secchk.lib` to Ìgnore Specific Library` in the Linker Input section. 37. Build cxcore. The build should complete with some warnings but no errors.
Compiling Cv
WINCE _WIN32_WCE=$(CEVER) UNDER_CE UNICODE
_UNICODE $(ARCHFAM) $(_ARCHFAM_)
39. Set the `SubSystem` in the Linker System section to `WindowsCE`. 40. Build cv. The build should complete with some warnings but no errors.
Compiling Libjpeg
NO_GETENV UNICODE _UNICODE WINCE UNDER_CE $(ARCHFAM) $(_ARCHFAM_) _WIN32_WCE=$(CEVER)
42. Build libjpeg. The build should complete with some warnings but no errors.
Compiling Libpng
_WIN32_WCE=$(CEVER) $(ARCHFAM) $(_ARCHFAM_) UNDER_CE WINCE UNICODE _UNICODE
44. Add the path to the Windows CE C Library Extensions source directory to the Àdditional
Include Directories` property in the C/C++ General section.
45. Build libpng. The build should complete with some warnings but no errors.
Compiling Ml
WINCE _WIN32_WCE=$(CEVER) $(ARCHFAM) $(_ARCHFAM_) UNDER_CE UNICODE _UNICODE
47. Set the `SubSystem` in the Linker System section to `WindowsCE`. 48. Build ml. The build should complete with some warnings but no errors.
Compiling Highgui
49. Exclude all source files that start with `cvcap` from the highgui project. 50. Add the following definitions to the `Preprocessor Definitions` in the C/C++ Preprocessor section.
WINCE _WIN32_WCE=$(CEVER) UNDER_CE UNICODE _UNICODE $(ARCHFAM) $(_ARCHFAM_)
51. Remove the following definitions from the `Preprocessor Definitions` in the C/C++ Preprocessor section.
HAVE_TIFF HAVE_JASPER
52. Add the path to the Windows CE C Library Extensions source directory to the Àdditional
Include Directories` property in the C/C++ General section.
53. Set the `SubSystem` in the Linker System section to `WindowsCE`.
54. Remove the following libraries from the property Àdditional Dependencies` in the Linker Input section.
..\..\3rdparty\lib\Release\libtiff.lib ..\..\3rdparty\lib\Release\libjasper.lib comctl32.lib gdi32.lib vfw32.lib
55. Add the path to the Windows CE C Library Extensions library to the property Àdditional Dependencies` in the Linker Input section.
\wcelibcex-1.0\msvc80\Windows Mobile 6 Professional SDK (ARMV4I)\Release\wcelibcex.lib
56. Build highgui. The build should complete with some warnings but no errors.
Creating Directory Structure

The previous steps were intended to build OpenCV for Windows Mobile 6 ARM architecture. Now in order to help use these libraries in future projects, a useful directory structure needs to be created. 57. In the staging directory, create a new folder called òpencv_winmo`. 58. Copy the three dll files found in the build directory under `bin/Release` into the
òpencv_winmo\dll` directory.
59. Copy the three lib files found in the build directory under `lib/Release` into the
òpencv_winmo\lib` directory.
60. Copy the include files found in the OpenCV directory under ìnclude/opencv` into the
òpencv_winmo\include`.
You have now created a distributable directory which contains everything required for using OpenCV on the Windows Mobile 6 platform. You may have noticed that we did not compile two versions of OpenCV like we did with the iPhone. This is because the Windows Mobile emulator is an actual hardware emulator. This means that binaries 38 Alex Karpus, 2009
compiled for devices can be tested in the emulator. This helps eliminate the need for multiple binaries for different testing environments.
USING OPENCV
The steps outlined in the previous sections will have produced three dynamically linked libraries, which can now be integrated into your development environment. As before, the guide to using OpenCV on Windows Mobile devices relies on the Visual Studio development environment. With Visual Studio, tasks like deployment are vastly simplified. However the libraries generated in the previous sections can be used with any build environment. The instructions that follow assume that a Visual Studio project already exists. If not create it now. 1. Copy the òpencv_winmo` folder generated in the previous section, into your Visual Studio project directory. 2. Launch Visual Studio and open your project. 3. Open the Project Properties dialog. (See the Individual Project Properties section on page 33 for information about the Properties dialog.) 4. Add the following to the Àdditional Files` property of the Deployment section. This will ensure that during deployment to the device, the OpenCV libraries are included as part of the transfer. (As part of the deployment of OpenCV libraries it is important to include the `msvcr90` library. OpenCV relies on it and your application will not work without it. For more information, refer to http://msdn.microsoft.com/en-us/library/abx4dbyh.aspx.) The example strings below are from the iQR project. As such you should replace the string ìQR` with the deployment name of your project.
highgui200.dll|$(SolutionDir)\opencv_winmo\dll|%CSIDL_PROGRAM_FILES%\iQR|0 cv200.dll|$(SolutionDir)\opencv_winmo\dll|%CSIDL_PROGRAM_FILES%\iQR|0 cxcore200.dll|$(SolutionDir)\opencv_winmo\dll|%CSIDL_PROGRAM_FILES%\iQR|0 ml200.dll|$(SolutionDir)\opencv_winmo\dll|%CSIDL_PROGRAM_FILES%\iQR|0 msvcr90.dll|$(BINDIR)\$(INSTRUCTIONSET)\|%CSIDL_PROGRAM_FILES%\iQR|0
5. Add the following paths to the property Àdditional Dependencies` in the Linker Input section. This will ensure that the linker is able to load the proper definitions during compile time.
"$(SolutionDir)\opencv_winmo\lib\cv200.lib" "$(SolutionDir)\opencv_winmo\lib\cxcore200.lib" "$(SolutionDir)\opencv_winmo\lib\highgui200.lib" "$(SolutionDir)\opencv_winmo\lib\ml200.lib"
6. Add the path the OpenCV includes to the Àdditional Include Directories` property in the C/C++ General section.
$(SolutionDir)\opencv_winmo\include
You are now ready to use OpenCV in your project. Dont forget to include the OpenCV header files in your code:
#include cv.h #include cxcore.h #include highgui.h
See the QR Code Reader for Windows Mobile section on page 46 for an overview using OpenCV on a Palm Treo 700wx to develop a QR Code reader.
QR CODE READER FOR THE IPHONE

There were no real surprises when developing the iPhone version of the QR Code reader. A lot of guessing about whether or not OpenCV would work on the actual hardware was dispelled early on in the project by successful tests. This left a lot of time for developing.
Figure 10 - QR Code reader for the iPhone
Although, iPhone development is quite different from other platform development, in that it forces the developer to maintain a model-view-controller (MVC) design pattern. While some may argue that this is a great design pattern to abide by, it definitely has a significant learning curve when youre unaware of 41 Alex Karpus, 2009
the pattern in place. Having done a lot Windows development in the past, and now developing on the iPhone for the first time, there were quite a few things that felt counter intuitive in terms of how to build even the most basic application. Part of the large learning curve also comes from the fact that iPhone development requires at least rudimentary knowledge of Apples Objective-C programming language. Objective-C is a reflective and object-oriented programming language which uses a Smalltalk inspired messaging convention.
DESIGN
As mentioned in the previous section, iPhone development is done mostly in the Objective-C language. Since Objective-C is a strict superset of the C programming language means that it is possible to include and successfully compile C code in Objective-C classes (Apple Inc, 2009). This characteristic of Objective-C meant that porting the QR Code reader code developed in Windows (See the QR Code Reader Design section on page 17) was simple.
libdecodeqr
iQR
Functions
OpenCV
Figure 11 - iQR design for the iPhone
In comparison to the Windows Mobile version, the iPhone design of the QR Code reader is quite simple as shown in Figure 11. One thing to notice is that OpenCV is labeled as being part of the iQR binary. This 42 Alex Karpus, 2009
is because OpenCV for the iPhone can be compiled only as a statically linked library. This means that during the linking of the iQR application, all the relevant symbols from OpenCV are included in the final iQR binary. It follows that, unlike the Windows Mobile version, the final binary has no external dependencies. The downside is that increases the final binarys file size although this is of no consequence. One interesting issue that was isolated to the iPhone implementation had to do with loading images. The OpenCV function that is often used to loading an image for OpenCV processing is `cvLoadImage`. Even though this function behaves as expected on both Windows, Windows Mobile and even in the iPhone simulator; it does not work as expected on the iPhone itself. To work around this issue a set of functions was written to convert the iPhone SDKs UIImage class, which represents an image, to the OpenCVs IplImage class and vice versa. These functions are stored in the `Function` component of Figure 11.
- (IplImage *) getCVImageFromCGImage:(CGImageRef)cgImage; - (CGImageRef) getCGImageFromCVImage:(IplImage *)cvImage;
Figure 12 - UIImage conversion to and from IplImage(Rhonda Ltd, 2009)
One caveat that was discovered from testing was that after processing and extracting the QR Code from the original image it always came with a black area not filled with any image data. This turned out to be because libdecodeqr set a region of interest (ROI) on the returned image, but did not actually crop the image. Thus `getCGImageFromCVImage` had to be modified to abide by ROI restrictions.
USER INTERFACE
The user interface was designed based on the flip-view model commonly used by utility applications. Upon startup, the application opens the digital camera window allowing the user to take a photo without having to click unnecessary buttons. After an image is selected it proceeds to attempt to decode it. If the image is able to be decoded, a new view is displayed with the isolated QR Code, the status of the decode process, the time it took to decode and the decoded message. If the decoded message is recognized as a URL, an option appears to navigate to that URL. Below is a collection of the different views.
Figure 13 - iQR main screen for the iPhone
Figure 14 - iQR taking a picture for the iPhone
Figure 15 - iQR processing image for Windows Mobile
Figure 16- iQR decoded message for the iPhone
QR CODE READER FOR WINDOWS MOBILE

Although the QR Code reader was designed with portability in mind (see QR Code Reader Design on page 17) there were some major hurdles that had to be overcome in order to produce a simple solution. One of the major hurdles was porting OpenCV to Windows Mobile. Unlike for the iPhone, there was no information out there on how to compile OpenCV for Windows Mobile 6. In particular at the time of this project, OpenCV version 2.0 was released, which meant that any and all previous information on the subject was obsolete by the major changes introduced in the new release.
Figure 17 - QR Code reader for Windows Mobile
DESIGN
The second hurdle that had to be overcome was how to design the actual application. When developing for the Windows Mobile platform, developers have two choices. Development can be done in native C++ code, or in managed code leveraging the .NET Compact Framework. Native code is used to refer to code that when compiled issues commands directly to the CPU using a CPUs instruction set. When a C/C++ application is compiled, the binary that is produced is made up of machine level instructions. However managed code is quite different. Managed code is a term used to refer to code which when run has to be interpreted by yet another runtime. This runtime is called the interpreter, sometimes also called a just-in-time compiler, will processes the code and produces native instructions which are then executed just as native code would be. This creates a level of abstraction which is required for ensuring that the code that is being executed is type and memory safe and provides the foundation for useful language features like garbage collection. Microsofts version of managed code comes in the form of the .NET Framework which uses the C# programming language. The managed code generated by compiling C# results in byte code which then is interpreted by the Common Language Runtime (CLR) layer and converted into CPU instructions (native code) at runtime. In the case of this project, both OpenCV and libdecodeqr libraries were written in native C/C++. Conceptually, the simplest thing to do was to develop a user interface in native C++ (ATL) and integrate calls to OpenCV and libdecodeqr. However after some investigation, it became clear that user interface development in C++ is quite cumbersome and complicated. Thus the alternative was to turn to the .NET Compact Framework and design the user interface using the simple drag-and-drop approach. However this exposed the challenge of interoperability between managed and unmanaged code. After some research and testing, a simple and elegant solution emerged.
Managed
Unmanaged
Libdecodeqr
iQR DLL
C++ Code
iQR Application (UI)
C# Code
OpenCV .NET Compact Framework

C++ Code
Figure 18 - iQR design for Windows Mobile
One of the main strengths of the common language runtime (CLR) and the .NET Framework is the interoperability support between natively compiled code and code converted to native during runtime using a just-in-time compiler (JIT). Since the .NET Compact Framework is a subset of the .NET Framework, it has very strong support for managed to unmanaged code interoperability (Keserovic, Mortenson, & Nathan, 2003). The solution to the challenge was to use, what is known as, Platform Invoke (P/Invoke). It is a service provided by the CLR interop layer for calls to unmanaged APIs from managed code. The only requirement is that the signature of the function being called, be re-declared in managed code. Applying this to the QR Code reader, as illustrated by Figure 18, the required interaction between managed code (iQR Application) and unmanaged code (iQR Dll, OpenCV) is all handled by the .NET Compact Framework. 48 Alex Karpus, 2009
The managed iQR application is accompanied by an unmanaged dll. This dll exports a single function. Inside this function all the work relevant to the QR Code process is performed (see Figure 3 on page 18). This involves calling libdecodeqr (which is linked in during the building of the iQR DLL) and exported OpenCV functions. Another alternative to creating a dll separate from the main iQR application, was to make direct calls into OpenCV using P/Invoke, however this would require for the re-declaration of every OpenCV function that would be used, thus creating a lot of unnecessary overhead. Instead the solution has a single function re-declaration. Below are code samples of what the declaration of the main function looks like in the dll and in the managed C# application. In iQR DLL (DLL.h):
/** * Contains all the QR Code decoding logic. */ extern "C" DLL_API short processImage( const char * in_file, const char * out_file );
In iQR Application (Utilities.cs):

[DllImport("DLL", EntryPoint="processImage")] private static extern short _ProcessQRCode( byte[] in_file, byte[] out_file );
The process of transferring data through the managed/unmanaged code boundary is known as marshalling. The .NET Framework is able to automatically marshal C# data types like string into character arrays. However, C# and the .NET Framework is completely Unicode. Whereas native applications may or may not support Unicode strings. This was the case with the QR Code reader. The way around this was to marshal filename paths as arrays of null terminated bytes (See the `StringToASCIIByteArray` method in Figure 19).
/** * Convert a Unicode string to an ASCII null terminated * byte array. */ public static byte[] StringToASCIIByteArray(string str) { return Encoding.ASCII.GetBytes(str + "\0"); } /** * Convert a returned P/Invoke pointer to a null * terminated array of bytes (chars) to a * Unicode string. */ public static string StringFromASCIIPtr(IntPtr str) { List<byte> bytes = new List<byte>(); byte b = 0; int c = 0; do { b = Marshal.ReadByte(str, c); bytes.Add(b); c++; } while (b != 0); return Encoding.ASCII.GetString(bytes.ToArray(), 0, bytes.Count); }
Figure 19 - Marshalling strings to and from unmanaged code
This split code model also creates some difficulties with memory management. Another strength of the .NET Framework is that it is garbage collected, meaning that the headaches of memory management are resolved. However, when calling unmanaged code and expecting to receive a pointer to a string (character array), who is in charge of releasing the memory? The logical approach is to have all memory managed be performed in native unmanaged code. This approach is used in the QR Code reader implementation.
iQR Application (UI) Managed C#
iQR DLL Unmanaged C++
String filename; String tempfile; String decodedMessage; ... Utilities.ProcessQRCode(filename, tempfile); ... decodedMessage = Utilities.GetLastDecodedQRMessage(); ... Utilities.FreeDLLMemory();
uchar * last_message
processImage(char*,char*)
uchar * getLastMessage() disposeDLL()
Figure 20 - Memory management between managed and unmanaged code
First there is a call into the unmanaged code which passes in the input image filename and the output image filename. This triggers the decoding process inside the dll which may allocate memory to store the decoded message in a local dll variable `last_message`. Once the call returns from the dll, this signals that decoding has completed and if the status is successful, it is safe to query the decoded message. Lastly we must tell the dll that we have retrieved the entire message and the memory can now be freed, otherwise we risk introducing memory leaks. Overall this three step process, as illustrated in Figure 20, ensures that memory is managed in a safe manner. Ignoring this splitting up of responsibility between managed and unmanaged code, the actual QR Code detection algorithm and implementation is unchanged from its Windows counterpart (See QR Code Reader Design section on page 17).
USER INTERFACE
The user interface allows the user to either take a picture of a QR Code or select an existing image from their library. After an image is selected it proceeds to attempt to decode it. If the image is able to be decoded, a new screen is displayed with the isolated QR Code, the status of the decode
process, the time it took to decode and the decoded message. If the decoded message is recognized as a URL, an option appears to navigate to that URL. Below is a collection of the main screens.
Figure 21 - iQR main screen for Windows Mobile
Figure 22 - iQR taking a picture for Windows Mobile
Figure 23 - iQR processing image for Windows Mobile
Figure 24 - iQR decoded message for Windows Mobile
RESULTS
QR CODE READER ACCURACY
Reliably measuring the accuracy of a QR Code reader is difficult because the accuracy is dependent on many factors such as lighting conditions, camera distance and camera angle. Another important aspect of determining the accuracy of the QR Code reader algorithm depends on the digital camera used to capture the image to be decoded. The major drawback of the digital cameras integrated into mobile devices, is the poor quality of images they produce. Typically they have very poor light sensitivity, no focus technology, they have a fixed focal-length lens and give the user almost no control over the capture parameters like shutter speed. For all these reasons there is no comprehensive test rubric as part of this paper. However, there are some interesting observations that are worth noting. For example, decreasing the resolution used to capture images improves processing time but hinders accuracy because blurriness is harder to correct in lower resolution images. Digital zoom features (like that on the Palm Treo) usually produce worse images than when brining the camera closer. Typically, the further away a typical camera is from the QR Code the higher the chance of a successful decode. This is because images produced by fixed-focal length cameras with no macro image support, produce clearer and sharper images at about a foot away from the subject, anything higher results in loss of information during the decoding process. Both the iPhone and Palm Treo fared well on the test QR Codes found in Appendix B on page 62. The iPhone was able to successfully decode every QR Code listed from the first try. The Palm Treo had some difficulties with the small versions of each QR Code, most notably having difficulties with test sample n0. 3.
QR CODE READER PERFORMANCE

As part of the multi-platform development undertaking of this project, it is important to compare how nearly identical code and implementations fare on different the devices. The hardware used for testing the iPhone code was the Apple iPhone 3G running the iPhone 3.0 OS. The iPhones processor has a clock speed of 412MHz and includes on-chip vector floating point (VFP) support. It also has 128 MB of RAM, however during my testing only 40 MB was actually available to applications. The hardware used for testing the Windows Mobile code was the Palm Treo 700wx running Windows Mobile 6 OS. The Treos processor has a clock speed of 312MHz and does not include on-chip floating point support. Along with a total of available 62 MB of RAM and only about 40 MB was available to the applications. The lack of floating point number support I suspect put the Treo at a disadvantage.
QR Code Reader Performance

50 45 40 35 30 25 20 15 10 5 0 46 43 44 44 Decoding Time (seconds) 42 43 46 43 45
M 1
M 2
M 3
QR Code Test Sample iPhone WinMo iPhone Average Windows Mobile Average
Figure 25 - QR Code reader performance graph
The QR Codes used in producing the graph (Figure 25) can be found in the Appendix B section on page 62. The two mobile devices were used to take photos of the printed versions of these QR Codes at equal height distances and under the same lighting conditions. The x-axis is split by QR Code test sample number and further by the size of the QR Codes (small, medium, large). The y-axis displays the decoding time in seconds as reported by the QR Code application on each device. The first and most obvious thing to notice is that the code running on the Palm Treo was on average 5.5 times slower than its counterpart on the iPhone. This, I think, can be at least partially accounted for by the fact that the iPhone has a 32% more powerful processor as well as Treos lack of hardware support for floating point operations. Since much of libdecodeqrs and OpenCVs code relies on floating point calculations this will have certainty affected the performance. Some other things of interest to notice are that the QR Code size relative to the image resolution showed no significantly measurable difference in the overall time of decoding. The amount of encoded data also had no measurable effect. Overall, the average time to decode a QR Code on the Palm Treo was 44 seconds, while on the iPhone it was a mere 8 seconds.
CONCLUSION
This project has been a great learning experience and has resulted in a useful addition to field of computer vision. All the goals and objectives (See the Goals and Objectives section on page 14) have been successfully achieved. Three variants of the QR Code reader application have been designed, developed and tested. Also the process of porting the popular computer vision framework, OpenCV, to the iPhone and Windows Mobile platforms has been pioneered and documented providing a foundation for future work. This paper marks the first time that OpenCV has been successfully ported to the Windows Mobile 6 platform. This project represents the culmination of a full semesters work. I believe that this project will serve as a great foundation for future work in the field of computer vision.
CONTRIBUTIONS TO THE FIELD

This project makes several contributions. The main one is the important research, development and testing work done in porting the OpenCV computer vision framework to the Windows Mobile platform. This paper is the first to document the process of porting the OpenCV framework to the iPhone and Windows Mobile platforms. Along with the effort of porting OpenCV, a QR Code reader application was developed for the iPhone and Windows Mobile platforms demonstrating the ideas of this paper. This paper also provides an overview of the work required in successfully developing multi-platform applications. I hope that this paper will serve as a guide for future projects based on OpenCV development for mobile devices.
FUTURE WORK
There is definitely room for a lot of interesting future projects. What follows are some interesting directions to go from this project. MonoTouch is an iPhone SDK developed by Novell, making it possible to develop iPhone applications using the C# language and the .NET Framework (Novell, 2009). This means that platform portability has been made even easier. In the case of this project for instance, the iPhone version of the QR Code reader could have been ported over from Windows Mobile, with little changes. Such multi-platform possibility is very promising and definitely deserves future project consideration. Augmented Reality (AR) is an up and coming part of computer vision which is seeing a lot of attention due to the slew of capable mobile devices. Augmented reality is the fusion of a real world video feed being merged with computer generated imagery (Haller, Billinghurst, & Thomas, 2007). Some applications rely on tags, similar to the QR Code, for calculating relative camera properties and inserting generated content. Future work can look into leveraging OpenCVs augmented reality capability and creating mobile applications that innovate how augmented reality is applied on mobile devices Another interesting project that could further the work accomplished in this paper, is to add the ability to recognize and decode QR Codes in video. The video can be a live camera feed in which case highspeed decoding performance would be critical. Also finding and decoding QR Codes in a pre-recorded video file would have interesting applications. Finally, something that has not been covered as part of this project but would definitely be of use is to build a test kit for benchmarking OpenCV on the new platforms that it has been ported to. 57 Alex Karpus, 2009
REFERENCES
Apple Inc. (2009). iPhone Dev Center. Retrieved November 29, 2009, from iPhone Dev Center: http://developer.apple.com/iphone/ Apple Inc. (2009, July 28). The Objective-C Programming Language. Retrieved November 28, 2009, from Objective-C Programming Guide: http://developer.apple.com/mac/library/documentation/Cocoa/Conceptual/ObjectiveC/ObjC.pdf Apple iPhone 3G - Full phone specifications. (2009). Retrieved November 25, 2009, from GSM Arena: http://www.gsmarena.com/apple_iphone_3g-2424.php ARM architecture. (2009, November 24). Retrieved November 25, 2009, from Wikipedia: http://en.wikipedia.org/wiki/ARM_architecture Baijian Yang, P. Z. (2007). Professional Microsoft Smartphone Programming. Wrox Press. Barcode. (2009, November 18). Retrieved November 20, 2009, from Wikipedia: http://en.wikipedia.org/wiki/Barcode Bellis, M. (2009, November 20). Bar Codes - The History of Bar Codes. Retrieved November 20, 2009, from About.com: http://inventors.about.com/library/inventors/blbar_code.htm Bradski, G., & Kaehler, A. (2008). What Is OpenCV? In G. Bradski, & A. Kaehler, Learning OpenCV (p. 1). OReilly Media, Inc.,. Chai, S. (2009). Mobile Challenges for Embedded Computer Vision . In S. Chai, Embedded Computer Vision (pp. 219-235). Springer London. CIA - The World Factbook - World - Communications. (2009, November 15). Retrieved November 15, 2009, from CIA - The World Factbook: https://www.cia.gov/library/publications/the-worldfactbook/geos/xx.html Haller, M., Billinghurst, M., & Thomas, B. (2007). Emerging Technologies of Augmented Reality: Interfaces and Design. IGI Global. Hayward, R., & Kim, D. (2009, December 7). Explore a whole new way to window shop, with Google and your mobile phone. Retrieved December 9, 2009, from Official Google Blog: http://googleblog.blogspot.com/2009/12/explore-whole-new-way-to-window-shop.html iPhone. (2009, November 25). Retrieved November 25, 2009, from Wikipedia: http://en.wikipedia.org/wiki/IPhone Kairer, R. (2006, October 24). Palm Treo 700wx. Retrieved November 27, 2009, from Palm Infocenter: http://www.palminfocenter.com/news/9058/palm-treo-700wx-review/ 58 Alex Karpus, 2009
Keserovic, S., Mortenson, D., & Nathan, A. (2003, October). An Overview of Managed/Unmanaged Code Interoperability. Retrieved November 27, 2009, from Microsoft MSDN: http://msdn.microsoft.com/enus/library/ms973872.aspx Microsoft. (2008, October 16). Windows Mobile 6. Retrieved November 29, 2009, from Windows Mobile Developer Center: http://msdn.microsoft.com/en-us/library/bb158486.aspx Novell. (2009). MonoTouch From Novell. Retrieved November 29, 2009, from MonoTouch From Novell: http://monotouch.net/ Rhonda Ltd. (2009, February 04). OpenCV vs. Apple iPhone. Retrieved November 28, 2009, from Computer Vision Software: http://www.computer-vision-software.com/blog/2009/04/opencv-vs-appleiphone/ Takao, N. (2007). libdecodeqr. Retrieved November 29, 2009, from libdecodeqr: http://trac.kokain.org/libdecodeqr Wheeler, R (Designer). (2008, January 7). QR Code Structure Example. Retrieved November 25, 2009, from Wikipedia: http://en.wikipedia.org/wiki/File:QR_Code_Structure_Example.svg
APPENDICES
APPENDIX A CD CONTENTS
Along with a copy of this paper the accompanying CD contains the notes I kept during the process of cross compiling OpenCV for Windows Mobile. This includes any and all major changes that had to be done to the code in order to make it compile. What follows is the CD file structure with brief explanations of the contents. Libdecodeqr QR Code decoder library compiled for the three main platforms discussed. iPhone 3.0 Windows Windows Mobile 6.0
OpenCV-2.0.0 OpenCV libraries (cv, cxcore, ml, highgui) compiled for the three main platforms discussed. iPhone 3.0 build_opencv.sh Bash script used for compiling OpenCV. See section iPhone on page 24 for more information. Windows Windows Mobile 6.0 opencv_winmo.patch Patch diff file used compiling OpenCV. See section Windows Mobile on page 27 for more information. Miscellaneous Notes.pdf Notes kept during the cross compiling stage of project.
QR Code Reader Contains the iQR applications discussed in this paper and all the associated code, project and configuration files.
iPhone 3.0 Windows Windows Mobile 6.0
Alex Karpus - Honours Project 2009.pdf This paper for reference.
APPENDIX B TEST CASES

The following QR Code test samples were generated using a QR Code Generator provided by the team from the ZXing Project. The QR Code Generator can be found at: http://zxing.appspot.com/generator/ More information about the ZXing project can be found at: http://code.google.com/p/zxing/
QR CODE TEST SAMPLE #1

http://scs.carleton.ca

http://en.wikipedia.org/wiki/QR_Code

an opencv implementation of a qr barcode reader for the iphone and windows mobile platforms

Alex Karpus Honours Project 2009

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Alex Karpus Honours Project 2009

Hochgeladen von

Copyright:

Verfügbare Formate

AN OPENCV IMPLEMENTATION OF A QR BARCODE READER FOR THE IPHONE AND WINDOWS MOBILE PLATFORMS

2 Alex Karpus, 2009

3 Alex Karpus, 2009

5 Alex Karpus, 2009

6 Alex Karpus, 2009

7 Alex Karpus, 2009

8 Alex Karpus, 2009

9 Alex Karpus, 2009

BARCODES AND THE QR CODE

10 Alex Karpus, 2009

11 Alex Karpus, 2009

Figure 1 - Basic OpenCV structure (Bradski & Kaehler, 2008)

12 Alex Karpus, 2009

13 Alex Karpus, 2009

WINDOWS MOBILE SDK

14 Alex Karpus, 2009

GOALS AND OBJECTIVES

15 Alex Karpus, 2009

16 Alex Karpus, 2009

QR CODE READER DESIGN

Figure 2 - QR Code structure (Wheeler, 2008)

17 Alex Karpus, 2009

Figure 3 - Structure of QR Code reader application

19 Alex Karpus, 2009

Figure 4 - Windows version of iQR

20 Alex Karpus, 2009

Failed Status (base 16) 4000 0100 0200 0400 0800

21 Alex Karpus, 2009

Figure 5 - QR Code reader adaptive threshold

22 Alex Karpus, 2009

23 Alex Karpus, 2009

OPENCV ON MOBILE DEVICES

24 Alex Karpus, 2009

OpenCV 2.0.0 (OpenCV-2.0.0.tar.bz2, 2009-10-01 tested) http://sourceforge.net/projects/opencvlibrary/files/

25 Alex Karpus, 2009

Patch GnuWin32 (version 2.5.9 tested) http://gnuwin32.sourceforge.net/packages/patch.htm

Windows CE C Library Extensions (wcelibcex-1.0.zip) http://wcelibcex.sourceforge.net/

28 Alex Karpus, 2009

OpenCV 2.0.0 (OpenCV-2.0.0.tar.bz2, 2009-10-01 tested) http://sourceforge.net/projects/opencvlibrary/files/

7. Start CMake (cmake-gui)

29 Alex Karpus, 2009

Figure 6 - CMake configuration window

30 Alex Karpus, 2009

31 Alex Karpus, 2009

Figure 7 - Unloading OpenCV projects in Visual Studio

Figure 8 - Adding a new solution platform in Visual Studio

32 Alex Karpus, 2009

Individual Project Properties

Figure 9 - Visual Studio project Properties dialog

29. Build zlib. The build should complete without errors.

34 Alex Karpus, 2009

35 Alex Karpus, 2009

_UNICODE $(ARCHFAM) $(_ARCHFAM_)

36 Alex Karpus, 2009

53. Set the `SubSystem` in the Linker System section to `WindowsCE`.

37 Alex Karpus, 2009

Creating Directory Structure

39 Alex Karpus, 2009

40 Alex Karpus, 2009

QR CODE READER FOR THE IPHONE

Figure 10 - QR Code reader for the iPhone

Figure 11 - iQR design for the iPhone

43 Alex Karpus, 2009