Self tracking web-cam for mobile computers

by DEMAINE Beno t-Pierre supervised by Ian Taylor June 2005

A thesis submitted in partial fullment of the requirements of Staordshire Univesity for the degree of Master of Science

School of Engineering and Advanced Technology Staordshire University United Kingdom

Abstract

This paper discusses about how to implement a camera able to track by itself an arbitrary object selected by the user. The camera used is a USB1.1 webcam. Three standard servo motors are used to move the webcam; they are driven by a USB microcontroller. The whole circuit is self supplied by the USB wires ( one for the webcam, one for the servos and the uC). The most important aspects are that the whole circuit can plugged on a traditional laptop, is hotplug, and does not require extra supply; the second aspect is that the picture recognition is made in real time, and can nd out any object selected when the user uses the system (no precomputed tables). The camera system has been constructed, tested, and is fully operational. A modied version of the Hough Transform has been created by the author to allow tracking objects in real time. The paper will explain how to build the hardware platform, and all the mathematical research about how to implement a new kind of Hough Transform which can detect arbitrary shapes that are determined at run time, when most other approaches required to learn the shape to detect before starting the recognition.

Keywords

Hough Transform, USB, serial port, real time system, embedded system, arbitrary shape image recognition, USB GNU/Linux driver, MC68HC908JB8, micro-controller, serial monitor for Motorola uCs.

Notice: This document will be maintained at [16]; you are welcome to mail the author for any comment. Denitions of technical terms are given page.54. Any source code given in this document is frozen code; code is given here just as example to illustrate the thesis. If the reader wants to implement the circuit, he should refer to [16], and download up to date sources and schematics. It is strongly recommended that the reader have a glance to the logbook as referred on the website. The project has been started in June 2003, and the technical part has been completed by December 2004. Copyright c 2003, 2004 by DEMAINE Beno t-Pierre, http://benoit.demaine.info/ benoit@demaine.info You may keep a private copy of this document , but may NOT modify it nor publish it without the explicit agreement of the author, either by paper mail, or electronically signed email. 2

Acknowledgments

We want to thank all people who provided help on IRC, mainly on the server IRC://irc.freenode.net, channels #c #debian and #linuxfr, Oliver Thamm (othamm@hc12web.de) who send a development board1 , Garrett Mace, Jean-Michel Friedt (friedtj@free.fr, http://friedtj.free.fr/, http://mmyotte.free.fr/), Simon Guinot (simon@ sequanux.org) for writing the hc08 driver, and Guillaumme Faussard for giving tips about the Hough Transform. We also want to thank Emilie Kaczmarek for her support.

http://hc08web.de/usb08/

Table of content

Contents
1 Abstract 2 Keywords 3 Acknowledgments 4 Table of content 5 List of gures 6 List of tables 7 Introduction 7.1 Objectives . . . . . . . . . . 7.2 Description of the project . 7.3 Range of applications . . . . 7.4 Literature review . . . . . . 7.5 New technologies developed 8 Global description 8.1 Introduction . . . . . . . . 8.2 Common problems . . . . 8.3 Proposed solutions . . . . 8.3.1 Hardware available 8.4 Softwares installed . . . . 9 Building the boards 9.1 Introduction . . . . . . . 9.2 about the JB8 . . . . . . 9.3 The development board . 9.4 The nal board . . . . . 9.5 Conclusion . . . . . . . . 10 The serial monitor 10.1 Introduction . . . . . 10.2 Content . . . . . . . 10.2.1 Description . 10.2.2 About rs232.c 10.3 Conclusion . . . . . . 11 The 11.1 11.2 11.3 GNU/Linux Introduction . Content . . . Conclusion . . driver . . . . . . . . . . . . 2 2 3 4 6 6 7 7 7 7 8 9 11 11 11 11 12 13 14 14 14 14 18 18 19 19 20 20 20 22 23 23 23 25 26 26 26

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

and . . . . . . . . .

the rmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12 Image recognition (HT) 12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

12.2.1 Getting the picture from the webcam : xvcam.c 12.2.2 The YUV format . . . . . . . . . . . . . . . . . 12.2.3 The non working RGHT . . . . . . . . . . . . . 12.2.4 from RGHT() to EGHT() . . . . . . . . . . . . 12.2.5 from EGHT() to PGHT() . . . . . . . . . . . . 12.2.6 The ne PGHT . . . . . . . . . . . . . . . . . . 12.2.7 About color . . . . . . . . . . . . . . . . . . . . 12.2.8 About noise . . . . . . . . . . . . . . . . . . . . 12.2.9 Screen shots . . . . . . . . . . . . . . . . . . . . 12.2.10 Cases where it does not work . . . . . . . . . . 12.2.11 Detection in the real world . . . . . . . . . . . . 12.3 Exporting YUV . . . . . . . . . . . . . . . . . . . . . . 12.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 12.4.1 Points to be improved . . . . . . . . . . . . . . 13 Conclusions and Further Work 14 List of softwares used 15 Glossary 16 References Appendices

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

26 27 28 28 29 29 38 38 40 40 45 46 48 49 50 53 54 55 57 57 57 80 80 80 81 81 82 82 83

A Full source code of the Hough Transform A.1 xvcam.c . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B Building the project B.1 building the board . . B.2 testing the board . . . B.3 Uploading the rmware B.4 Installing the driver . . B.5 The pivot . . . . . . . B.6 The image recognition B.7 FAQ . . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

List of gures

List of Figures
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Overview of the project . . . . . . . . . . . . . . . . prototype board 01 . . . . . . . . . . . . . . . . . . Photo of the prototype board . . . . . . . . . . . . Pictures of the nal board . . . . . . . . . . . . . . Diagram of the monitor software. . . . . . . . . . . Representation of planar YUV data. . . . . . . . . Building the R-Table : cutting of a potato . . . . . Taking care of the color space in the edge detection Subtracting the thermal noise . . . . . . . . . . . . tracking a candle . . . . . . . . . . . . . . . . . . . tracking a knife . . . . . . . . . . . . . . . . . . . . Inuence of the edge factor in ordinary space . . . . Inuence of the edge factor in ordinary space . . . . Searching a fuzzy image with edge factor = 7 . . . Searching a fuzzy image with edge factor = 10 . . . Hough space of a fuzzy picture . . . . . . . . . . . Real world detection: the authors face . . . . . . . Real world : trinkets on a bed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 16 17 19 21 28 31 37 39 41 42 43 43 44 44 45 46 47

List of tables

List of Tables
1 List of components for the prototype of the devel board . . . . . . . . . 15

7
7.1

Introduction
Objectives

This report deals with a self tracking webcam, that is a webcam plugged into a PC which can move to always show at the center of the picture a chosen object. The project has the following properties: it is all embedded it works with only two USB wires without extra power supply it is designed to be embedded so that it can work only with a simple laptop it can self rotate to track an object the tracked object can be chosen in real time by the user the system can anticipate moving object the project is free, developed under GPL, and works under GNU/Linux

7.2

Description of the project

This paper will explain how to build an embedded cheap webcam with mobile base, so that the webcam can turn and spin around 3 axes (180 each), using only two USB ports to supply and manage a webcam, 3 servos, and a dedicated board to monitor the servos. It will also explain how to compute image recognition so that the camera can track an arbitrary shape in a noisy picture in real time. Fig.1 gives an overview of the project. This paper will not explain all the theoretical details, but will focus on the practical implementation: which attempts failed, which ones succeeded, what are the weak points. two development boards for the 68HC908JB8 have been built from scratch: the rst required 4 days of work, the second has been made within 1 day. several new peaces of software had to be written to monitor all the project. the picture recognition part can work in real time on a laptop bought in 2002. This document is the fruit of work which has been done on actual devices, and can be re-implemented on any newer system.

7.3

Range of applications

This project can be used in any place where a camera shall automatically point at a given object, or where a mobile platform needs to identify the location of any x or mobile objective: security cameras (parking, casino, home), self guided missiles,

Figure 1: Overview of the project drones, recording devices for planes, cameras embedded into security cars (Police, taxi), video capable model making

7.4

Literature review

The rst document which may have an approach similar to this project is [9], in the part called moving object tracking using camera motion corrected Kalman ltering by J. F. Boyce and D L Toulson. This document explains how a camera mounted on a mobile platform can track an object, given the relative position of the object, the camera, and the background. This approach uses Kalman ltering, and more precisely the tracking lter, and assumes the movement of the object can be approximated to a third order equation. This allows to anticipate the relative movement of the object, even when it is temporally hidden behind some foreground. This document sounds good, but does not deal with the real time aspect of tracking, and does not evaluate the complexity of the algorithm, nor the cost of equipment. The conclusion just claims that the approach of interrelation of coordinates allow to calculate the relative coordinates of the camera, background and object, and evaluate the acceleration uncertainty when compensating the speed and acceleration of tracking. It is just feasible, but without any mention of any kind of cost (computing time, autonomy, ducial cost ...). On the other hand, [27] is a bit more detailed about motion recognition and computing time, but the algorithms presented are a bit old, and modern computers allow to use more advanced approaches. In fact, a complete recognition approach is detailed in [26]: image ltering preprocessing polygonal approximations

 signatures boundary segments skeleton Fourier descriptors statistical moments topological descriptors This is at last a complete description of the work which should be done. But it is not detailed enough, and does not say which parts are optional, and which ones should be customized. We still needed a complete kernel. Some core parts are explained in [28] (elementary invariant functionals, geometry of pattern space, invariance and separability), [29] (feature selection and extraction, image segmentation, computer vision), and [25] (use of Hough Transform to characterize and recover linear features of object). When all image recognition algorithms tend to detect invariants like explained in [28], few of them really provide ready to use solutions. The common problem always deals with feature extraction, and most often, that topic is related with skeleton analysis of either the edge space of the object, or the skeleton of the object itself. Amongst all those approaches, the HT seemed to be one of the rare one not to require skeleton analysis, and to be tolerant to partial obfuscation, either by the side, or the middle of the object. The HT allows to recover the pattern even when the connectivity changes. Furthermore, it used to be really time consuming 10 years ago, but is now aordable for real time work. So the Hough Transform seemed to be the obvious way to go through. [15] may be the most interesting one amongst all. It introduces new concepts: Randomized GHT skew tolerant HT scaling tolerant HT deformation ratio complete comparison of cost and eciency of all HT But we discovered later on that in fact, it uses connectedness of object to develop such qualities. We discovered this problem very late, and before we could gure it out, we solved it an other way. See next section. About controlling home made devices through USB port of computers, there is no book available yet. Some books dealing with electronic topics explain how to use microcontrolers on a USB chain, but there is nothing dedicated to the use of servo motors.

7.5

New technologies developed

The most advanced point developed in this project is obviously the PGHT routine. IT is a part of C program that derive the Hough Transform a new way, and make it either 9

faster, or more tolerant to deformations. It could even support partial improvement of each problem. You can choose to: perform recognition very fast at the expense of deformation, recognize skewed object at the expense of time, detect scaled objects, accept small scaling and small skew and decrease a bit the certainty of recognition, but still in real time Since all those factors can be tweaked, it is possible to choose the speed of recognition, and then adapt one or an other quality factor, and customize if skew is more important than scaling. We also developed a new board to control 3 servo motors through a USB device, and all related softwares for Linux: rmware uploader, rmware, and a feedback software between image recognition and the USB driver.

10

8
8.1

Global description
Introduction

Using embedded and limited systems, we will try to keep the computing time constant, and the error ratio to be the same for each computation. The use of random search may increase slightly the speed of the process, but since that can introduce big variations of error ratio, it is not no be used.

8.2

Common problems

Many systems claim to be embedded even when they have little autonomy, or even depend on external resources. serial port is old: the serial port is now more than 25 years old, have many limitations, and especially have bad support of multiplexing. It is dicult to plug two serial devices on the same plug, have limited and low rates. And because modern alternative are available, fewer and fewer computers are tted with that port. We need to look forward, and use new alternatives, batteries do not last: it is possible to buy powerful batteries, but they are still very expensive, need to be recharged, and anyway, after 100 to 200 cycles, their capacity is divided by a non neglectable factor, supplies are heavy, require usually two wires (one from the sector, one to the device), use either heavy transformer or expensive DC to DC adaptor, plug are always the deal of electronic devices: they are expensive, and most failures of devices are due to te wire breaking at the output of the plug, or the plug itself going away from the electronic board. They also rule the design of the device due to their volume, or visible surface, it is dicult to break with old habits like the use of serial port for home automation, most cameras use SUB or Fire-Wire (aka I-link), so that any way, the project will need one of those plugs, live picture recognition requires large computation power.

8.3

Proposed solutions

We try to design an embedded system small and smart, with few connections, or none at all, and must be mobile. To save connections, USB is chosen because it provides signal AND supply in the same wire and plug. A side eect is that each USB wire can only deliver 500mA, whereas the board requires more than 2.3A. Practically, the laptop used can deliver that current pretty well over two wires. Finally, the only connection between the laptop and the board is a pair of USB wires. Tha problem could be solved easily: use a USB hub to multiplex the signal, 11

and really low power servos. Some Low power servo can require less than 100mA each, but cost more than 150 Euro each at the moment ... USB is multiplexed, what means it is possible to put several devices on the same plug, either using dedicated hubs, or using devices which support chaining, USB provide multiplexed signal and supply in the same wire, Linux is cheap, can be downloaded as full version for free, with a very permissive license, Linux is provided with full source: each distribution provide the source of all softwares for free, due to the use of the GPL [8], modern laptops are powerful enough to allow to do in real time what used to take hours just 5 years earlier. Linux communities provide live help on any Linux related topic, either on IRC, or news groups, or web forums ... USB webcams are cheap USB is plug an play, supports hotplug, and allow high rate data transmission, many manufacturer provide dedicated microcontrolers able to manage various buses. 8.3.1 Hardware available

All the project is run on standard laptop; here is the description of the hardware: laptop Gericom [7] Webgine XL AMD 1.4G 256 Mo RAM 30G HDD 15,2 screen French keyboard converted into Dvorak [5] CD writer / DVD reader oppy, serial, parallel, synaptic touch-pad 1 PCMCIA2 2 USB1.1 1 I-link ( 4 pin IEEE 1394) S-VGA + TV out VIA chip-set ( sound + modem + ethernet ) S3 Pro-Savage KN133 ( AGP) IrDA ( only stu not working under GNU/Linux ) 12

webcams 2 Terratec [18] Terracam Home ( CPIA USB chipset) 2 Terratec Terracam PRO ( OV511 chipset ) both work ne under GNU/Linux. USB08 Eval board [22] 2 boards given by Oliver Thamm Home made 68HC908JB8 dev board 2 boards : one made for testing purposes, big, many features, reusable for any 68HC908JB8 project, and one dedicated for the project, 4 times smaller, and only designed to be used for this project. Both are shown in the log book.

8.4

Softwares installed

The laptop runs Linux GNU/Debian SID (GNU/Linux based Debian distribution, unstable version). The list of software installed is given page.53

13

9
9.1

Building the boards

Introduction

The project is based on the MC68HC908JB8 [10]. Before all, we need to build a board. The specications of the board are quite simple: it must be able to control 3 or 4 servo motors depending on an USB input signal. The JB8 is only responsible for decoding the USB signal, and de-multiplexing it into 4 serial signal (one per servo). Of course, the board will be supplied by USB.

9.2

about the JB8

The 68HC908JB is in fact a family of chips. Historically, the old core CPU was called the hc05. The instruction set of this core was extended and became the hc08. Any code written for the hc05 can run as binary le on a hc08: they are byte compatible. But since the instruction set of the hc08 is extended, reverse way is not possible. Then, the *9xx class use embedded FLASH. Most hc08 exist in both (E)EPROM and FLASH. The HC08 series are (E)EPROM tted, whereas HC908 are FLASH tted. The JB8 is the USB capable series. For development purposes, the FLASH version is quiet more convenient: it can be reprogrammed more than 1000 times, and during the test phase, the board has been ashed more than 100 times. An EPROM would have never supported that. The MC68HC908JB8 exists in 4 packages: MC68HC908JB8ADW (SOIC 28W), MC68HC908JB8FB (QFP 44), MC68HC908JB8JDW (SOIC 20W), MC68HC908JB8JP (PDIP 20). The JP version is the only one which can be soldered manually an easy way on the rst board. Manually soldering an ADW took more than 6h (just to cut the strips, and solder each pin).

9.3

The development board

The schematic of the board is based on http://hc08web.de/usb08/images/usb08_schema.gif [22]. The Figure.2 shows the schematic of the rst prototype board; one should read LEDs instead of photo-diodes : Oregano did not have LEDs in component packages in 2003. The components are listed in Table.1. The design is really simple. The MC68HC908JB8JP ( PDIP 20 ) is the easiest one to solder. The plug J1 is the USB plug, we just link D- and D+ to PTE4/D- and PTE3/D+, and some capacitors on the supply. The signal lines have 27R resistors to equilibrate the input impedance of the circuit with the output impedance of tho host; see the USB specications on [21]. R2 is an optional pull-up; if it is not present, then the rmware must activate the internal pull-up. Like for all micro-controller, the quartz must be as close to the chip as possible; the quartz capacitors and resistors should all be less than 2cm away from OSC1-OSC2.

14

Description Motorola MC68HC908JP8JP (dip20 package) MAX232 Chip holder 2x10 ( for MC908 ) Chip holder 2x8 ( for max232 ) R 27R R 330R R 1.5k R 2.2k R 10k R 1M R 10M carbon C 22pF C 100nF, ceramic C 4.7uF, electrolytic, 25V or higher D 1N4148 D 1N4001 DZ 8.2V LED RED LED GREEN 6,000 MHZ crystal 2x5 pin M plug [J2 J3] (2) 2x3 pin M plug [j4] (1) 1x3 pin M plug [SWs 9 10] (4) Double raw of pin headers 10+10 [J2 J3 J4 SWs 9 10] HE10 plug SUBD9 F plug with top cover and screws USB wire 2m A/B Solderable USB socket B type 1 way 9 way Ribbon cable, 2m Small push button PCB mount [SWs 1 12] Mini switch [SWs 2 3 4 11] 4poles switch [SWs 5 6 7 8] Jumpers Strip board

Farnell ref 348-0252 407-150

509-644 356-0661 896-755

704-246

512-151 316-6855 ? ? 152-754 ? 535-916 311-674 ?

quant F 1 1 1 1 2 4 1 1 4 1 1 2 2 6 2 1 1 2 2 1 0 0 0 2 1 1 1 1 1 4 4 1 7

Table 1: List of components for the prototype of the devel board The numbers between [brackets] refer to the component number in the gure 2.

15

Figure 2: Schematic of the rst prototype of development board for MC68HC908JB8JP For this devel board, extra switches will allow more convenience (SW3 and SW11). The Reset and IRQ switches have an anti-bounce capacitor; they both are active low. In fact, they also have internal pull-up, thus they shall not be linked to big capacitors. When a Reset occurs ( whether due to Reset switch, Power On Reset, or soft Reset) the uC pulls down the reset pin for a while (4163 bus cycles) so that other chips of the circuit (if there are any) are informed that the board shall be reseted: the JB8 acts like a master if it is put on a bus with other uCs. So, checking whether the Reset pin is at +5V is a good way to double check whether the CPU is really running or not; if after power on it remains low, then the CPU did not start ( maybe due to a faulty quartz). SW4 SW5 SW6 SW7 and SW8 allow to congure monitor mode. SW4 selects the monitor mode, and other switches dene the bus frequency scaling. See section Monitor ROM (MON) of the chip specications. If for any reason, the programmer wants to use the port A for general purpose, just open all those 5 switches, and also open connector J4. The monitor mode is activated by putting a high voltage on IRQ pin. The high voltage is produced the MAXIM232, then lowered to 8V with a Zenner diode. To forbid monitor mode, either switch o the MAXIM, or preferably open SW4. J4 has been put for tests purposes, but in fact t has never been used for this project.

16

C0/A7 is wired as an optional serial port. The MAXIM part is quite common. Just 4 capacitors to feed the charge pumps. The real trick about this board is D5: the specications of the uC are that PTA0 is an input/output pin for serial communication. This means, all trac, either incoming or outgoing will ow through this pin. This pin must be connected to both Rx and Tx. D5 just avoids short circuiting of the maxim. But the way the board is wired implies something specic: everything the host sends through Tx is sent back by the diode to Rx: any character sent is received back. This is called the hardware echo. There is no other special trick about this board. The PDIP20 is easy to solder. The nal result is shown on Figure 3. Note that this boards only support one servo. Near Q, you can see some dirty resistors. They aim to limit the current which ows through the servos. Each servo needs 550mA RMS, what means at least 2A peaks since it is powered with square peaks, for coupling purposes 2 .

Figure 3: Photo of the prototype board, component side.

a

A: power switch of the 68HC908 B: power witch of the MAX232 C: serial connector D: LCD connector E: inverter connector for C F: inverter connector for D

G: power led of the 68HC908 H: LED connected to D0/1 I: Oscillator J: power LED of the MAX232 K: the MC68HC908JB8JP L: the Max232 M: Reset button

N: IRQ button O P: Monitor mode selector switches Q: servo plug R: High voltage switch S: Monitor mode jumpers

The few troubles about this board were very minor: we forgot to cut 2 strips, and put one LED upside down. This board is reliable, not that big, and can support many projects.
2

for more informations read about how a servo works on websites dedicated to that topic.

17

9.4

The nal board

After all tests went OK, a second board was required. The rst one could have been extended to 4 servos ( and has been later on), but it was not nice for presentations. All extra features have been removed, and we kept the minimum. Everything is hard-coded on the new board. On the second board, the extra serial port have been removed, the mandatory one has been reduced to 5 pins, and only the switch SW4 has been kept. The board has been tted with a MC68HC908JB8ADW provided by Oliver Thamm. Since it is a dead small packaging, the strips had to be manually cut in halves. The servos have been wired to [A4:A7]. Further parts will explain why this uC can not deal with more than 4 servos. The nal result is shown on Figure 4.

9.5

Conclusion

We did not really learn that much while building these boards. They both worked very fast, and we endured no major trouble. What have been really surprising is that when by accident the supply of the uC was short-circuited, when supplied with USB supply, ampermetre was showing 1.3A. This means, the laptop used can provide up to 1.3A short circuit. When plugging the nal board, part of the USB chain is reseted because of the use of a very big capacitor, but after that, the 3 servos consuming 550mA each is not any trouble. Dont forget that the camera requires 300mA to 500mA, plus 1.65A for servos ( in the same USB wire) when they all move at the same time, what is a total of 2.15A. But the laptop is happy with that. And this is what the ampermetre tells in continuous current. We expected peak current to be a bit higher when all servos initialize the default position ( at this moment, they all need maximum power for a long time). So: the boards work, the laptop can provide more than 2A on two ports. Remember that the maximum current per USB wire is 500mA - for a total of 1A in our case. At the moment all servo move fast, we could expect the laptop to detect overconsumption, and reset the whole chain. It does not ( some motherboards do detect that, and really perform reset, or at least close the USB power line). We could also expect the over current consumption to generate parasites: the USB wires remains cheap, but it looks like the shield is really ecient.

18

(a) Solder side

(b) component side

(c) top view

(d) front view

Figure 4: Pictures of the nal board

10
10.1

The serial monitor

Introduction

By the time the project was started, Motorola only used provided softwares for MSWindows. Since all parts of project ought to be GNU/Linux compliant, it was required to write some monitor for GNU/Linux. 19

The aim of the monitor is to upload rmwares in the MC68HC908JB8. To do so, the chip can support two implementations: the use of the serial port, through pin port A[0], and the USB port.

10.2
10.2.1

Content
Description

There is not much to say about this software. It has been written top-down, starting by the command line parser, to the functions which send the bytes through the serial port. The main saves the properties of the serial port and of the terminal, parses the arguments, run the CLI or the script parser, then restores all initial settings and exits. The prompt is managed by prompt.c. It allows to enter any command. At any time, the help command allows to print out all available commands, their syntax, and a short description. The prompt manager also supports limited completion, and history of previous commands. Everything has been written from scratch, and should be rewritten in order to use libncurses or libreadline. If any script is provided, then line script() is called from line.h. Whatever you use scripts or prompt, any command is parsed with line parse() from line.c. This function extracts the command, and executes the corresponding routine. All routines are listed in cmd.h and cmd extra.h. chip.c contains all functions related to the 68hc08, and chip ash.c all the ones specic to the 68hc908 series ( with FLASH memory). All these functions call rs232* from rs232.c, where are stored all low level routines to interact directly with the serial port. The interactions between the main parts are described in Figure 5. 10.2.2 About rs232.c

rs232.c only performs the serial port communication. It can work using two modes: buered, or direct. Using direct mode, the routines return only when a character is practically sent. In buered mode, it returns at once, in hope that the character has been properly transmitted. Using those two modes is necessary because GNU/Linux uses caches to manage serial port ( and maybe many other devices). In practice, when a byte is sent by a program, it is put in a cache, and sent later on by the driver. The driver is only executed once per scheduling cycle. Thus, the byte is sent only 2ms to 18 ms later, depending on the scheduling frequency, on a 2.4 Linux kernel. Things may dier on 2.6 kernel since the scheduling frequency ought to be changed from 100Hz to 1kHz. The driver only puts the byte in the serial hardware buer. But the monitor always expect the hardware echo ( see section 9.3 ), and the software echo. But, the driver itself can only read those two echoes after it has been itself scheduled. This means, the monitor can only read the echoed bytes after two scheduling cycles. All that process takes time. An average of 18ms on 2.4 kernels. This means, the rst version of the

20

Figure 5: Diagram of the monitor software. monitor could only sent 1 byte every 18ms. Then buered routines were introduced, just to send data over the line, and put the expected results in a software buer. An error occurs if the expected bytes do not come back after 100ms. The other trick is about the sleep() calls. In order to count time, the software needs to pick up the current time, and measure the timing of some sequences. But it also needs to perform some sleeps. In the rst version, a sleep() is required after every single transmission of bytes. Anyway, the whole protocol can not send more than 1 byte every 1.1ms ( 9 bits at 9600bps), and if you consider the double echo, that comes up to 3.4ms per byte. So, in order to avoid lling buers too fast, the monitor just performs some sleeps. The problem is that after any call to sleep(), the system performs a scheduling switch, whatever the sleep is called for 0.01ms or for several seconds. This means, the real duration of any call to sleep will be the scheduling time. On Linux 2.4.x with x<22, it is always ( at least ) 10ms ( 100Hz ); with x>22, the root can choose to lower that to 1ms ( 1kHz). With any 2.6 kernels, the scheduling frequency is ( should be ) always 1kHz. So, on our 2.4 system, any call to sleep() implies a switch of context, and the practical execution of the program continues execution 14ms later. This means sending a byte and making a sleep takes 14ms per byte. So nally, all calls to sleep() have been suppressed, and have bene replaced by a new sleep routine. This routine just does nothing, but waits for time to ow. This routine has the granularity we choose. It is about 0.1ms, but can be more precise. Of course on 2.6 systems the granularity of scheduling is better, what means calls to C sleep() may be ecient, but this would probably make the monitor lose an average of 0.5ms per byte ... what is 1/6. So by keeping the home made sleep routine, we save time, and make things faster. The global time saving is signicant. The bad point is that the routine takes 100% CPU. But this should happen only while sending data over the line. If the monitor is executed 21

on a dedicated machine in a factory, this does not matter: speed of production is more important than CPU activity. In most other cases, the user will see his computer busy for the time he sends data over the line, and nothing once transmission is over: on GNU/Linux machines, this does not aect other process, because the kernel is good enough to not let the monitor really take 100%, but keeps some CPU time for other process. So ... it causes no harm.

10.3

Conclusion

Nowadays, [10] provides more softwares. Especially search for AN2295 and AN2398, and download AN2295SW and AN2398SW which should allow to program the chip using free softwares. We know that AN2295SW allows to use GNU/Linux to reprogram the chip using the serial port. About AN2398SW, we did not checked how it works. But by the time we started this project, those softwares were not available. While writing monitor-68hc08, we did not really learnt very much about C. It is also the rst time we wrote such a big project, and upload it on a public server. What we learnt was about how to read Motorolas documentations, and how to write a software which must comply with technical specications even when no draft is available. We dened ourself what the software must do, and implement everything following the 68HC908 specications. The job in itself is quite easy: just read documentations, and write code. What makes the work hard is the fact the documentation is big, and pieces of information are spread over all the PDF which is 286 pages. At anytime, writing any function, one must remember all aspect of everything. As result, the software works ne on our computer, and we hope it works the same way on other computers, especially by those who build USB-IR-Boy [20].

22

11
11.1

The GNU/Linux driver and the rmware

Introduction

We were very lucky that some other people had the same needs ... just 2 months before us. Two French people, Simon Guinot and Friedt Jean-Michel also developed their own board, and wrote a USB driver for GNU/Linux; the driver is available at [19]. When our started this project, they just started their work. Any way ... our rst task was to write the monitor, and by the time it was nished, some beta driver was available. The driver aims to be compatible with both our board and MCTs evaluation board.

11.2

Content

The very rst version of the rmware was just written to make some servos spin. This proved a rmware could perform some kind of sleep, generate square signals, and control one servo. More examples are available in the Log Book. In order to write the nal rmware, we wanted to base our work on an existing program. In fact, the MCT board came with a CD on which one can nd a rmware written in C. The problem is that this rmware is licensed, and the C compiler is neither free, nor working on GNU/Linux. On the other hand, asl[1] is free, and works ne on more than 10 platforms, including GNU/Linux. So, we needed the source of the rmware in ASM. All we did was compiling the rmware, upload it in the chip, get it back, and disassembled it. To disassemble it, we used sim68xx. Of course, the program as is did not do the job. It had been written for 68hc05 core CPUs, and we had to patch it for the 68hc08 cores. Any way ... that allowed to get a non licensed ASM source using reverse engineering. The source has then to be patched to get rid of the special features of the MCT dev board. We just removed some functions, and added mines. This way, we could reuse all the USB management part as is, and just added our own routines. The most relevant part is the isr timOF routine. The initialization part set up the PWM to generate a TIM interrupt every 20ms. Then the TIM routine puts all pins of port A up, and then down one after the other when required. In fact, the TIM is structured like a for loop: it counts cycles; duration of pulses is encoded in cycle units. Once a pin have been up for enough time, it is put down. The code of such routine is given below : ; set up the pins bset4 PTA bset5 PTA bset6 PTA bset7 PTA ; high state sleep : about 1ms lda #$4 _plop_ jsr _isr_tim_delay2 dbnza _plop_ ; transfer global values to local tmp ; these are the duration in cycles of each pulse 23

mov mov mov mov mov mov mov mov lda _isr_tim_loop: dec bne bclr4 _s4_end: dec bne bclr5 _s5_end dec bne bclr6 _s6_end dec bne bclr7 _s7_end

servo_0,s0_tmp servo_1,s1_tmp servo_2,s2_tmp servo_3,s3_tmp servo_4,s4_tmp servo_5,s5_tmp servo_6,s6_tmp servo_7,s7_tmp #$FF ; loop 255 times s4_tmp _s4_end PTA s5_tmp _s5_end PTA s6_tmp _s6_end PTA s7_tmp _s7_end PTA ; 4c ; 3c ; 4c

Note that in fact, the routine is quite slow. Given a bus frequency of 3MHz, if one wants the whole for loop to be executed 255 times in 2.5ms, the chip can only manage 4 servos. This could be increased to 8 servos, but then there would be only 127 steps, thus a precision of 180/127 degrees per servo. The 2.5ms limitation is due to the way servo expect pulses: the central position is held for a pulse of 1.5ms every 20ms. If the pulse is shorter, the servo turns one way, if it is longer, it turns reverse way. But the extreme positions are held for about 1ms and 2.5ms. Thus the TIM can keep doing nothing for the rst 1ms, what make the 255 steps smaller ( in duration). Even this way, the precision of the servo remains bad. A precision of just less than a degree remains very much compared to the precision of the picture acquired by the camera. When a servo turns of one step 3 , the picture moves of about 5 pixels. This means, the precision of the whole project is in fact limited to the precision of this single routine. But since we need 3 servos, the loop needs to manage all of them ... in limited time. It is impossible to get a precision higher than 0.7 deg. On the other hand, the 68HC908JB8 is already one of the very few chips which has a bus ratio of 1/2 throughout the Motorola products. It is impossible to change the chip frequency, otherwise the USB module will not work properly. And we could not expect from a USB dedicated chip to be able to have large RAM, generate 8 pulses of independent frequency, length, with an arbitrary precision. In fact ... all that work not that bad.
the steps are not determined by the servo itself, but by the limitation of the chip to generate pulses: the precision of a pulse is limited by the granularity of the for loop in the TIM routine
3

24

11.3

Conclusion

It was also good luck that some one else write a USB driver for Linux, otherwise that single part could have been an independent project by itself. The precision of the project is limited by the precision and the quality of the pulses generated by the chip. Since the servo can only move of steps of 0.7 deg, and the camera has a cell of 320 pixels for an angle of about 45 deg, any time the program asks to move one step, the picture get moved of 5 to 7 pixels. This means, all the precision is lowered by the ability of the chip to generate pulses.

25

12
12.1

Image recognition (HT)

Introduction

This paper will not discuss about the various HT (Hough Transform); any one can use Google and nd hundred of websites and pdf les giving details about it ... see for example [17] [6] and [14]. This paper will just explain why our work is based on the RGHT [15], and how we ended up with the PGHT() routine. From the moment the Hough Transform seemed to be able to t the requirements of the project, we stopped searching for alternatives. It is highly probable that there are other algorithms more ecient, but the HT ts the requirements, so we stuck there. The [15] approach appeared to be more interesting than other documents about HT, especially page 4 which set up a table comparing various HTs. Furthermore, it has been very easy to email the author of RGHT to get some help. We also wanted in this project to let the user choose the object he wanted to track, so that the project can be used for example in planes, when the driver wants the RADAR to track a particular object. Thus the ordinary HT nor RHT could not t ... and RGHT claimed to be faster than GHT for arbitrary shapes ... so we ran through RGHT. In fact, the rst implementation of RGHT did not work at all. But since the general idea of RGHT seduced me, we did not want to give up. After several tries, we ended up with PGHT, which is theoretically faster than RGHT, and practically ecient enough for this project.

12.2
12.2.1

Content
Getting the picture from the webcam : xvcam.c

First of all, we needed to get the picture from the webcam, and show it to the user. Many free softwares could do the job, such as mplayer [13], camstream [3], xawtv [23], camE [2] ... see the log book. mplayer camstream and xawtv matched two essential requirements: acquiring the picture of the webcam showing the picture on the screen but ... if we choose any of them, we would have to download, read and understand a source code of at least 10 000 lines. That would have been too much work. We were seduced by camE: it is small, nice source, and enough features for this project. But it did not put the picture on the screen. So if we go for camE, we also have to read about GTK or something alike. Nevertheless, we grabbed the code of camE: since we were using Debian [4], we mostly use program included in the Debian, and downloaded the source through Debian using apt-get source came. And there was a big surprise: in the source tree of camE, there was a small application called xvcam. A second look at apt-cache search xvcam shown that this application was denitely not compiled in any Debian package. This program was not either mentioned in the Makele. Nevertheless, the header of xvcam.c mentioned about how to compile it. 26

And the surprise was that this very small application ( less than 450 lines of code) did exactly what was required: acquire the picture from the webcam + show it on the screen. This very small program is not famous, nor powerful: it only works with OV511 webcams. But we do have an OV511 webcam \o/ . So xvcam is exactly what we need. Since xvcam is very small, it is far from perfect: it uses internally the same format of picture as the webcam itself: it does not convert it: it just take the raw picture in the cam, and give it to Xv [24]; and this format is YUV. It was to be faster learning about YUV than forget xvcam and read the source of any larger program. In fact using the YUV has an advantage: in keeps the picture exactly like the shot was taken with exactly the same quality. The quality of a Terracam is already poor, and any conversion would make it worse. So xvcam let us rework the picture keeping the same quality as far as possible. To have start with xvcam, we spend several days trying to understand how the YUV works. To make our software interactive, we needed to add many routines; especially widely modify the main() to add support of key bindings. For that, we needed to read a lot about Xv, and catch all keyboard and mouse events. We also add run2() and init2(), trying not to modify the original routines as far as possible and put all our own stu in independent functions. About the variable i2, we rst used a global variable, but once we discovered that shutting down the software took a lot of time, we thought it was due to the fact that the system had to free memory; so we changed i2 into a pointer, and measured how much time the system needed to free this pointer. In fact freeing the pointer does not take much time; the small lag we had was due to a dirty trick in the hc08 driver: when the user-land program frees the le descriptor of the device le /dev/hc08 0 ( or what ever its name is - it depends on the kernel ...), during the close(fd) call, the driver performs a non schedulable and non interruptable wait(1) ... what hangs the system for a whole second. This point shall be improved in further work, but we do not plane to do it soon (see point 6 p.50). Finally, left i2 as a pointer not to re-edit all the code. 12.2.2 The YUV format

We could not nd any good documentation about YUV on Internet, not due to a lack of information, but because there are dierent YUV implementations. Books from libraries can not help any more since we are dealing with a video camera, and the manufacturer is free to use any encoding, providing he releases a driver to go with. So we had to do many test in order to reverse engineer the format specic to the OV511 cell. It is very funny: if the resolution of the picture is x*y, the picture is stored in an array of x*y*2 bytes. The rst half is used to store the light ( one byte per pixel - byte 0 for x=1,y=1, 1 for x=2,y=1 ... line per line), then there are two quart (x*y/2 bytes per array) for colors: the rst one is for green to blue, and the second for green to red. In the second array, a value of 0x00 is green, and 0xFF for blue; in the third array, 0xFF is for red. In those arrays, each byte describes 4 pixels. This is called planar YUV 4:2:0 or planar YUV 4:1:1 ( we do not know the dierence between those two). See g: 6 Each macro-pixel is composed of 4 pixel; four bytes are used to store light intensity of each pixel, one for hue, one for saturation. This format uses 6 bytes per macro-pixel,

27

Figure 6: Representation of planar YUV data. what is 12 bits per pixel. Thus the resolution of the light is twice higher than the color. But the noise rate is neither double nor half. Experience showed that there is few noise in the light, but a lot of noise in the color space. The color noise does not seem to be due to the codec, but rather to the quality of the cell. This is perhaps due to the way our webcam convert the raw data from the cell into the YUV format. This characteristic may dier from one webcam to an other. 12.2.3 The non working RGHT

Since the documentations about RGHT were goods [15], we wrote our own implementation of the RGHT, following exactly [15], but the results were not good at all: we just got a fuzzy cloud of points around the places where the threshold of edges was not null. There is not much to say about it: it just did not work for us. We wrote several mails to various people to ask about sources of various HT, but none could help us in debugging RGHT(). So we gave up. Later on, we received an email from Michael Fung; it says: Our approach of RGHT was based on both GHT and RHT. The RHT is basically a statistical paradigm so that the sample size determines both the speed and quality of the detection. In our RGHT implementation, there are some heuristics such as junction detection as well as hashing. Attached please found the le detailing our implementation. We have not been allowed to attach their implementation to this report, but we were allowed to study their code, rewrite our own RGHT(), and publish any comment and comparison about both sources. We did not study their code yet, but we expect them to set up a propagation table of cognitive points, so that they only pick out pairs of points which are always element of the same homogenous object. 12.2.4 from RGHT() to EGHT()

The theoretical idea of RGHT is to rst compute the R-Table of an object, and pick out some random points in the target picture, but only pairs of points element of the same object, consider them as being part of the object, and from the R-Table, compute the anchor point of the object. The center of the tracked object should be the statistical 28

peak of anchor points. What we expect to be wrong about this approach is that there is little chance to nd one point element of the tracked object, and in nearly impossible that random access nds in the same shot two points element of the same object 4 . Thus, the statistical cloud of result is spread all over the picture, and the peak is not high/strong enough to be detected by our software. Just reading the paper of the author did not make us understand our mistake. Nevertheless, this algorithm look to have some good qualities. EGHT (Exhaustive GHT) is written to be an alternative: the design is very simple: for each pixel of the picture set up the R-Table of the object. compute local r() does the this work. The problem with EGHT() is that computing the R-Table for each point takes ages: each time the program have to determine the border of the object, and then set up the local R-Table, and compare it with i2.R. It is a very heavy process. When using EGHT with a value of R RES higher than 50, computing time for each picture goes over 1 minute; it was far too much to let it nish even once. Using lower values of R RES , the results are too fuzzy: the cloud of results is not strong enough at the expected point, so that the last lter will miss the peak. 12.2.5 from EGHT() to PGHT()

After two tries failed, it was essential to keep this interesting aspect of RGHT : the fact that it statistically scans all the points of the picture, and make computation for each of them. But it was necessary to reduce the computing time per pixel. What makes EGHT too heavy is the total amount of computing per pixel: setting up a propagation table (360 radius * an average of 50 pixels per radius * cos&sin computation ) what is far too heavy for the laptop. The other good idea in RGHT is that it assumes that every pixel is part of the border of tracked object, but does not really test whether it is true or not. Adding some tests to RGHT() can help: for each pixel which passed the threshold test, pick up the edge and phase, and compare them with all the edge and phase of the R-Table, and if any (or many) match, then take out the relative position of the anchor point, and increment that pixel in an independent array. The result will be a large cloud, with peaks all around the object (at half distance from the border than the diameter of the original object) and a big peak in the middle of the tracked object. In fact there will be a cloud all over the picture, and especially all around the objects which border passed the threshold, only the tracked object shall have a strong peak in the middle. The end is easy: apply a peak detector, and pick out the location of the highest value. It works, it is fast, and it matches the project proposal: nding in real-time an arbitrary object in a noisy picture. 12.2.6 The ne PGHT

It is surprising that after writing many HT routines, PGHT() is the smallest one, the fastest one, and also the most ecient one. EGHT() could give some good results, even if it is slow, but in order to have a computation time reasonable ( at least one
In fact this point of view seems to be wrong; there may be a solution when only considering point element of the same object; this means the detection algorithm needs a function to determine whether two points are member of the same object or not.
4

29

picture every minute), R RES needs to be so low that the results were not signicant anymore. Here how it works. Computing the R-Table First, the source code: (it is assumed that i2 is previously declared as global variable, and have proper content)
int e x t r a c t R ( i n t mouse x , i n t mouse y ) { int r , w; i n t R s [ R RES ] [ R WIDTH ] ; // s t r e n g t h o f e d g e f o r t h e RT a b l e int i , j ; i n t count ; // number o f v a l i d f i e l d s i n t h e RT a b l e float x , y ; int o ; int e x i t ; int plop ; // r a d i u s when h a l f o t p o i n t s have been found i 2 >mouse x=mouse x ; i 2 >mouse y=mouse y ; tmp x=mouse x ; tmp y=mouse y ; f o r ( i =0; i < R RES ; i ++) f o r (w=0;w< R WIDTH ; w++) { i 2 >R v a l i d [ i ] [ w] = 0 ; R s [ i ] [ w] = 0 ; i 2 >R[ i ] [ w] = 0 ; } f o r ( i= H2 ; i <yuv width H2 ; i ++) f o r ( j= H2 ; j <y u v h e i g h t H2 ; j ++) { i f ( i 2 >S [ i ] [ j ] ) i 2 > i ] [ j ]= l e n g t h ( i , j ) ; W[ else i 2 > i ] [ j ] = 0 ; W[ } r =0; count =0; e x i t =0; p l o p= YUV WIDTH ; while ( ! e x i t ) { r ++; // i n c r e a s e c i r c l e f o r ( o =0; o< R RES ; o++) // w a l k a l o n g t h e w h o l e c i r c l e { // compute a b s o l u t e c o o r d i n a t e s or c u r r e n t p o i n t x=mouse x+( f l o a t ) r c o s f ( ( f l o a t ) o ) ; y=mouse y+( f l o a t ) r s i n f ( ( f l o a t ) o ) ; i f ( x<0 | | y<0 | | x> YUV WIDTH | | y >240) continue ; // d e t e r m i n e l o c a t i o n o f f i r s t a v a i l a b l e c e l l f o r (w=0;(w< R WIDTH )&&( i 2 >R v a l i d [ o ] [ w ] ! = 0 ) ; ) w++; i f (w >= R WIDTH ) continue ; i f ( ( ( i 2 > ( i n t ) x ] [ ( i n t ) y ] >( i 2 >e d g e f a c t o r +2))&&( i 2 >R v a l i d [ o ] [ w]<=( r / 4 ) ) ) | | W[ ( ( i 2 > ( i n t ) x ] [ ( i n t ) y]> i 2 >e d g e f a c t o r ) && ( ! i 2 >R v a l i d [ o ] [ w ] ) ) W[ ) // i f ( ( ( i2 >S [ ( i n t ) x ] [ ( i n t ) y]> R s [ o ])&&(! i 2 >R v a l i d [ o ] ) ) | | // ( i2 >S [ ( i n t ) x ] [ ( i n t ) y ] >(( R s [ ( i n t ) o ] ) + 2 ( ri2 >R[ ( i n t ) o ] ) ) ) ) // compare c u r e n t e d g e w i t h t h e one j u s t n e a r e r t h e c e n t e r { i 2 >R[ o ] [ w]= r ; i f ( ! i 2 >R v a l i d [ o ] [ w ] ) count++; i 2 >R v a l i d [ o ] [ w]+=5; R s [ o ] [ w]= i 2 >S [ ( i n t ) x ] [ ( i n t ) y ] ; i 2 >R x [ o ] [ w]=( i n t ) xmouse x ; i 2 >R y [ o ] [ w]=( i n t ) ymouse y ; } i f ( ( i 2 > ( i n t ) x ] [ ( i n t ) y]> i 2 >e d g e f a c t o r ) && ( i 2 >R v a l i d [ o ] [ w ] ) ) W[ i 2 >R v a l i d [ o ] [ w]++;

30

Figure 7: Building the R-Table : cutting of a potato

i 2 >R P [ o ] [ w]= i 2 >P [ ( i n t ) x ] [ ( i n t ) y ] ; W[ i 2 >R W[ o ] [ w]= i 2 > ( i n t ) x ] [ ( i n t ) y ] ; } i f ( ( count >=( R RES R WIDTH 0 . 7 ) ) & & ( ! p l o p ) ) p l o p=r ; i f ( ( count >( R RES R WIDTH 0 . 9 ) ) | | ( r > 1 3 0 ) | | ( r>p l o p 3 ) ) e x i t =1; } i 2 >s l e e p =1; i 2 >s e a r c h =1; i 2 >show =0; return ( 0 ) ; }

This is the second version of extract R(). The rst one used to only pick up one point per radius, but as described in [15] (paragraph 3.1 Fig 3), a good GHT should support a random number of entries per cell. PGHT() has been re-written to support that; extract R() also had to be patched to add support of several points per radius. It would be very expensive to make it support a random number of entries, and it would be quite dicult to determine how many points are necessary per radius; a global constant is now used to dene the number of points: it is R WIDTH . A value of 1 would take us back into the old time before any change is done. A value of 3 looks ne: it still allows to compute pictures in real time, and improve wildly the results. A value greater than 5 does not seem to improve the results. The algorithm is a bit dicult to explain. The best thing to do is to read the code, but non programmer people may expect a smart explanation. In fact the code has been written a bit like speaking about philosophy: it is the literal expression of a concept. The human brain generates a concept, and the hands tend to write it down using words. But it often very dicult to explain with words the thought that leaded to the concept itself. The R-Table is an array of line of cells describing the edge of an object. The array is ordered as growing angle (see Fig.7). The circle 2 is divided into R RES slices: one line per value of . In the code, is o. So we need o to grow from 0 to 2. Each value of alpha describes a line starting at the center O. The coordinates of O are given by clicking at any place in the picture with the mouse. Then along all the lines D described by , the software nds out all points which passed the threshold

31

of the edges detector. The three rst points will be stored in the array with various parameters: value of edge and phase, and coordinates relatively to the center O. Practically, the code does things the inverse way: consider that R RES has a value of 300, and R WIDTH is 3; the R-Table has thus 300 lines. Each line has three cells. This is an amount of 900 points. The rst loop (the while loop) uses rst a growing radius r, and for each value of r, a for loop will walk along the 300 values of o. An extra variable count stores how many cells have been tted. When count reaches 90% of R RES * R WIDTH =900, then the process is stopped. This is in short the use of the two last if. At the end of the process, the average center of the R-Table is computed, and those average coordinates are used as new center for computing a second time the R-Table. This process allows to nd a better usable center than the one selected by the user. It is never very far from the one where the user clicked, but the second R-Table is better tted for further computations. Listing of the Phase and Edge based Hough Transform First the listing:
int PGHT( i n t xx , i n t yy ) { int i , j ; int x , y ; i n t tmp ; int p ; float o p , o w ; int mat [ 7 ] [ 7 ] = { { 0 , 0, 0, { 0 , 1, 1, { 0 , 1, 2 , { 0 , 1, 2 , { 0 , 1, 2 , { 0 , 1, 1, { 0 , 0, 0, // p h a s e s h a p e GHT

0, 1, 2, 4, 2, 1, 0,

0, 1, 2, 2, 2, 1, 0,

0, 1, 1, 1, 1, 1, 0,

0} , 0} , 0} , 0} , 0} , 0} , 0} ,

}; i n t r e s u l t [ YUV WIDTH ] [ YUV HEIGHT ] ; i n t max r , max x , max y ; // l o c a t i o n where t h e f i l t e r w i l l d e t e c t a maxima f o r ( i =0; i <yuv width ; i ++) f o r ( j =0; j <y u v h e i g h t ; j ++) i 2 >r e s u l t [ i ] [ j ] = 0 ; f o r ( i= H2 ; i <yuv width H2 ; i ++) f o r ( j= H2 ; j <y u v h e i g h t H2 ; j ++) { i f ( i 2 >S [ i ] [ j ] ) i 2 > i ] [ j ]= l e n g t h ( i , j ) ; W[ else i 2 > i ] [ j ] = 0 ; W[ } f o r ( i =0; i < YUV WIDTH ; i +=1) f o r ( j =0; j < YUV HEIGHT ; j +=1) { int w; i f ( ! i 2 >S [ i ] [ j ] ) continue ; o p=i 2 >P [ i ] [ j ] ; o w=i 2 > i ] [ j ] ; W[ f o r ( p=0;p< R RES ; p++) f o r (w=0;w< R WIDTH ; w++) { i f ( ! i 2 >R v a l i d [ p ] )

32

continue ; i f p h a s e i n p i c i s c l o s e t o any p h a s e i n RT a b l e i f ( abs ( o pi 2 >R P [ p ] [ w] ) < 5 ) i f ( abs ( o wi 2 >R W[ p ] [ w] ) < 5 ) i f ( ( i i 2 >R x [ p ] [ w]>=0)&&( i i 2 >R x [ p ] [ w]< YUV WIDTH)&& ( j i 2 >R y [ p ] [ w]>=0)&&( j i 2 >R y [ p ] [ w]< YUV HEIGHT ) ) i 2 >r e s u l t [ i i 2 >R x [ p ] [ w ] ] [ j i 2 >R y [ p ] [ w]]+=0 x1 ; } } max r =0; // p o i n t d e t e c t o r f i l t e r // a l o n g t h e p i c t u r e f o r ( i =3; i < YUV WIDTH3; i +=1) f o r ( j =3; j < YUV HEIGHT3; j +=1) { tmp=0; f o r ( x =0;x <7; x++) // a l o n g t h e m a t r i x f o r ( y =0;y <7; y++) // ( t h e Matrix has you ) { tmp+=i 2 >r e s u l t [ i+x 3 ] [ j+y 3]mat [ x ] [ y ] ; } r e s u l t [ i ] [ j ]=tmp ; i f ( tmp>max r ) { max r=tmp ; max x=i ; max y=j ; } } // t e s t xx=max x ; yy=max y ; return ( 0 ) ; }

How does it work ? Once you understood how the standard GHT works [17], and know how to set up the R-Table, the process of PGHT() is quite easy to explain: for each pixel p for each entry of the R-Table rt if abs(edge(p)-edge(rt)) < Threshold value then read the distance of current point rt to its anchor point add this relative distance to the coordinates of current point in picture increment this point fi done done This process will in fact determine if any point has properties close to the ones of any point in the R-Table ( considering edge value, strength, phase ... ), and for those which matches, we will consider that the current point is same distance to the center of of searched object than the current point of the R-Table to its anchor point. This process will produce a high probability right in the middle of the object, and a cloud all around it. But since the cloud will be spread on a surface about 3 to 10 times larger, the cloud will not be dense enough. Thus the center of the object will produce the highest peak in the resulting matrix. A simple peak detector will nd it easily.

33

The limitations of RGHT Since the computation of the location of expected anchor point is done using recorded distances, the RGHT is not designed to support scaled objects5 . Nevertheless, the center of the object is the statistical peak of such anchor points, thus a small scaling does not interfere with detection: even if the object gets further or closer of a distance about 50% of the original distance, the scaling factor will be less than 15%, what will still produce a good peak. To support rotated objects, [15] proposes to perform n research of the R-Table in the picture (whatever you use RGHT or an other algorithm), and shift the table of k ranks, with k varying from 0 to n, and n the number of times you want to divide 2*. For example, if you want to detect rotated object with a precision of 45 deg, you will choose n=8. Then the algorithm become: for each pixel p for k=0 to k=360 deg, with a step of 360/n perform any search, except that instead of using rt taken at rank r ( from 0 to 360 deg) just take r from -k to 360-k done But there is an other problem: for some objects, the detector always fails : we found that when the detector is wrong more than 50% times, it is generally due to the fact the object selector was not able to nd out the real shape of the object. This usually happen if there is noise in the object, or if it has edges internally. But such a problem is in fact duo to extract R(), not to PGHT(). A solution is then to increase the tolerance of the selector ( horizontal arrow keys ). An other point would be to insert a small low pass lter to remove dot noise, or any small noise which produces non cognitive points. An advanced solution would be to detect cognitive objects, are reject the small ones. What is wrong with PGHT()

it can not actually support scaled objects, it is not self adaptive to changes of the object, it does not detect rotations of the object. ... but all these defaults could be corrected at the expense of more computation: as said in [15], it would be enough for example to shift the R-Table a given number of times to detect rotated objects. For example, it could be easy to shift the R-Table 5 and 10 degrees left and right, and see if one of those 4 new tables produce better peak than original one; if yes, then the object is rotated 5 or 10 degrees.
but this approach may require less computing time that the original RGHT, since PGHT does not need to nd out pairs of points with special properties
5

34

Ideas about possible optimizations We wrote some news on Usenet to ask for some ideas on how to optimize the Hough Transform: - precomputing sin/cos tables and switching to single precision computations gives only a little speed-up. x86-processors provide an opcode fsincos for simultaneous computation of sin/cos for a given angle. using it for precomputing sin/cos tables isnt worth the eort implementing it. But it might be valuable if you do without the tables. This topic has been discussed many times in the log book in section 6.5.8 on the 26/08/2004. - try to restrict your detection angles and use ner step-sizes in angle intervals where you assume your lines are. Your statement ... If possible, even replace the HoughTransform ;-) its very slow - or try to speed it up by implementing more knowledge about your data. #define R RES 50 //resolution of the R-Table, how many points per 2*Pi is only well-suited for very thick lines. Angle detection is very rough ... In fact, since we kept many properties of the RGHT, this comment is not applicable any more to PGHT(). - for repairing the fuzzy clouds in the hough-accumulator I used the following (fast!) procedure to obtain better results: use a dilatation-lter to link very close points in the hough-accumulator. use a thresholding to remove small accumulator. values perform a segmentation of the hough-accumulator to locate clusters (e.g. by using a chain-code contour tracker) for each segment, return its center of gravity as result The use dilatation in PGHT() still looks fuzzy to the author ... but maybe applying a threshold to RGHT could be equivalent to that. But computing some clusters might be a very good idea, in a way, it could possible to apply a lter to detect peaks, and reject clouds, it is just required to have to nd a way to accept useful peaks, and reject clouds due to noise ( such as the noise produced by hair of a person ... ); for example it may be possible to detect noisy parts of a picture, get rid of them, and set up an edge around; this way, the computation of R-Table could still nd out the characteristics of those regions, but could be able to really determine their shape. - faster convolutions can be done using smaller and separable lter masks. For example, your lter1 h: {0, {0, {0, {0, {0, 0, 1, 2, 1, 0, 0, -1, -2, -1, 0, 0, 0, 0, 0, 0, 0}, 0}, {1, -1, 0}, |1| 0}, equals {2, -2, 0}, equals |2| (*) |1 1| 0}, {1, -1, 0} |1| 0},

35

computing a vertical convolution of the image columns with {1,2,1} and a hor. conv. of the rows with {1,-1} is signicantly faster than using the original lter1 h - because of the linearity of your convolution lters you may apply them in arbitrary order. Very important for performance: If you need convolve a few times with dierent lters you may merge the dierent lters into only one convolution lter which is the convolution product of the dierent lters by convolving the lters with each other. (e.g. above (1,2,1)T (*) (1,-1) == lter1 h) In a more general way, we could try to merge some operations of run 2() ... - If your images are large and your convolutions are slow, remember you can compute convolutions using the Fast Fourier Transform Algorithm (FFT). The classical FFT is limited to convolution lengths N that are powers of 2: N=2k. Apart from some dierences between discrete convolution and FFT(DFT)-convolution, the FFT is signicantly faster: O(N*log N) for 1D-FFT vs. O(N2) for 1D-discrete convolution. We have no clue if that would really improve the detection. your images are very small and your CPUs caches might even hold a complete image... This means that RAM accesses should not be a problem as previously described in the log book, but all the raws should stay in the cache. Let us have a look at it; the CPU is a mobile AMD Athlon(tm) 4, which is said to have 256 KB of cache. The picture is 320*240 ... times the number of matrixes to be acceded: this is 76k for the light array, and 304k per array of integers. So if this guy is right, it would be a great benet to use short integers if possible, in order to help pre-caching. merging convolutions was just a general remark ... I didnt study your algorithms thoroughly enough - but the meaning of lter1 h is a vertical frequency low pass (blur) and an approximation of the rst derivative (horiz.). JUST applying the 5x5 lter mask on the image causes the performance problem. Applying rst (1,2,1) on the columns and then (1,-1) on the rows is faster than applying the 5x5 mask on rows & columns: (1,2,1)T and (1,-1): for each pixel three multiplications, two additions for (1,2,1) for each pixel two multiplications and one addition (or equivalently: one subtraction) => about width*height*8 computation steps 55 mask: for each pixel 25 multiplications, 24 additions => about width*height*50 computation steps but I agree: since your images are small, running time of convolution is not so crucial. >> >> >> >> >> The really slow part is the PGHT(), which compute a function applied to the whole picture with a 50*3 (or bigger) table (depending on R RES bigger => better ). Remember that both matrixes ( picture and table) are dynamic, and can change at any time. How can I apply FFT to that ?

the tw library (apt-get install tw3) oers high-speed FFTs for arbitrary input sizes

36

(a) Edge space when using only the light array to compute convolution

(b) Edge space when taking care of both color arrays

Figure 8: Taking care of the color space in the edge detection That point will be mentioned in the further work section. the general tip is to stay in a local memory area ;) IIRC your Linux uses 4k memory pages. If have the suspect that your image doesnt completely t into your CPUs cache, you may think about a tiling of your images into sub-images stored at certain 4k memory osets. But that wont help if your algorithm moves around randomly. This is not to be implemented since the order in which pixels are accessed is really random in PGHT(). but heres another wonderful application of the FFT. Its benet is the description of a contour which is invariant of scale, position and rotation of the contour. If implemented cleverly, its faster than any form of Hough Transform. Does this means that FFT could help detecting scaled and rotated objects ? Other rules : - using faster algorithms is superior to code optimization :-) - in-line small functions that are called very often (e.g. convolution of a single point) - if possible, avoid divisions of oating point values ... instead multiply with the reciprocal: divisions are much slower than multiplications. Also the C-librarys pow()-function is rather expensive. Assuming you want to track a CLOSED contour in your image: 1. do a segmentation of the image and manage to describe every object in your scene as a closed contour of equidistant points (e.g. by using chain-codes with 8-neighborhood) Your contours are lists of points: (x1,y1),(x2,y2),...,(xn,yn) 2. rename your axes: x => Real, y => Imaginary. Your contours can now be interpreted as lists of complex numbers 3. Compute a Fourier-Transform of the contour (using a complex-FFT). The result describes your contour as a sum of sin/cos-functions (or complex exp-functions if you like) which is a Fourier-Series of your contour with rapidly fading coecients.

37

12.2.7

About color

As explained earlier, the resolution of the color space in YUV pictures is not the same as the light space. But the noise rate in color arrays in not either the same as in the light array. In fact, the noise rate is higher in the color arrays. In colorful pictures taken from the webcam, one can see many large red and green lines: not pixels but lines, because colorful noise is often 3 to 5 meta-pixels: the frequency space of noise in color diers from the one for light. In order to be able to know exactly about color noise, the software support some key bindings to enable or disable support of color, which make the convolution functions take care or not of the color: the convolution functions conv1() and conv2() reads a global variable; if the switch is of, they only convolve the mask using the light array; if it is on, they also convolve the mask with color arrays. The bindings are keys Q for blue array, and S for red (QWERTY layout). The result can be shown very well on Fig.8(a) and 8(b), in the edge space. As the pictures show, there is less noise after convolution when the switches are both o. 12.2.8 About noise

Preview paragraph tells about the noise due to color. But there is a way to get rid of some other noise: the black noise, or so called thermal noise. All physicians know that when the temperature of any object is higher than 0o K, the atoms move, and so do the electrons. This phenomena implies that any metal which is exposed to traditional temperature (about 20o C ) are subject to surface currents. Those currents are also present in the sensitive cells of any digital camera. Thus, at common temperature, any camera is to give a non perfectly black picture even when it is put in a perfectly black room. This phenomena gives the so called the black noise, or thermal noise of cells. The noise picture depends on each camera, on the temperature, and can vary from day to day even for a given cell. People who take picture of stars for several minutes are used to put their cam in a black bag and take a 10s shot before taking any picture. Then they can use some advanced software to subtract the black noise to the second picture. The same process can be applied on webcams: just put the nger in front on the objective, and take some shots. This procedure has been automated using the argument -b after xvcam. This argument forces xvcam to take a series of shots, compute an average of them, and save the result in a le. The le produced is valid for some hours, but should be renewed from time to time. Fig.9(a) shows a normal picture, whereas Fig.9(b) shows the result of subtracting the black noise. Figs.9(c) and 9(d) show the corresponding edge space, 9(e) and 9(f) the phase space. So, the black noise subtraction changes the hue space of the picture, but it widely improves edge and phase detection. It is not active by default, because in some rare cases the noise get increased.

38

(a) Normal view of normal shot

(b) Normal view of shot after subtracting the black noise

(c) Edge space of normal shot

(d) Edge space of shot after subtracting the black noise

(e) Phase space of normal shot

(f) Phase space of shot after subtracting the black noise

Figure 9: Subtracting the thermal noise

39

12.2.9

Screen shots

Let put the camera on a stable place, in front of some trinkets. First, let see the ideal case : some objects are put in high light exposure. Just click on an object; the selection is highlighted in red; then the detected object is pointed by a blue point in a small red box. As shown on Fig.10, the selected object is automatically put back in the center of the view. Let select the next object : the small knife (see Fig.11). Again, the camera centers automatically the object. The edge factor Now, let see what about the edge factor; for that we select the candle holder. See Fig.12(a). This candle holder is particular because it has noise inside. This means the edge detector nds edges inside the object. To solve this, the user just has to press the right arrow in order to increase the edge factor, so that the object selector is less sensitive to noise. This means, it only accepts threshold on higher value of edge calculations. This way, the shallow edges, and edges of small points (such as noise), are not considered as object edge, but as content of object. To reduce the edge factor, just press left arrow. The more you increase the edge factor, the more noise will be tolerated; thus the more the shape of object will be large. This option only aects the object selector. It helps getting a more signicant R-Table. It has no inuence on further computations such as object research. On Fig.13 you can compare the result of the Hough research using various edge factors on dierent objects. As you can see, every object has a cloud around. This cloud is the probability of presence of the center around the edge of the object. If the selected object has a radius of 30 pixels, then this cloud is present at 30 pixels away from the edge (outside). The second cloud is 30 pixels away, but at the inside. The inner cloud has a very high probability peak at the right center of the object. In fact, all objects in the picture have both clouds, but the researched object has a higher probability at its center than other objects. When increasing the edge factor, the R-Table get more selective. Thus the clouds are lighter. Finally, the critical point is that the cloud around every object have something common with the original selected object. This is why the external cloud is very light around objects which look like the one researched, and even lighter inside. But only the researched object has the best match. 12.2.10 Cases where it does not work

Fig.14 shows that when selecting an object with large entropy, the red square points to arbitrary locations. The object selector did not really understood how to read the picture (the selector rounded a strange cloud), and the HT matched the R-Table to arbitrary places. Sometimes in the heap, sometimes outside it. Increasing the edge factor can correct that. See Fig.15. 40

(a) Selecting

(b) detection 1

(c) detection 2

(d) detection 3

(e) detection 4

(f) detection 5

Figure 10: tracking a candle

41

(a) Opinel 1

(b) Opinel 2

(c) Opinel 3

(d) Opinel 4

(e) Opinel 5

(f) Opinel 6

Figure 11: tracking a knife

42

(a) edge factor = 2

(b) edge factor = 4

(c) edge factor = 5

(d) edge factor = 7

Figure 12: Inuence of the edge factor in ordinary space

(a) holder edge factor = 2

(b) holder edge factor = 7

(c) candle edge factor = 2

(d) candle edge factor = 7

Figure 13: Inuence of the edge factor in ordinary space 43

(a) selection

(b) search

(c) search

(d) search

Figure 14: Searching a fuzzy image with edge factor = 7

(a) selection

(b) search

(c) search

(d) search

Figure 15: Searching a fuzzy image with edge factor = 10 44

Figure 16: Hough space of a fuzzy picture Fig.16 shows very well that the Hough Space of the picture is far too noisy to oer proper solutions. Detecting a face In the real world, you want to detect arbitrary objects. Let turn o the main light. Fig.17(a) shows the authors face in the dark, with only a small light behind. Even though the face is dark, and of same darkness as the wall behind, the edge space still shows nice lines. The selection is pretty good: it nds only a fourth of the face, but since that fourth is characteristic, the search engine has no problem nding it again. The Hough space has very thin clouds; one is on diagonal, parallel to the microphone branch, and the other one in parallel to the vertical border of the face. Both clouds are as far away from the most signicant edge as the center is far from the average edge. The statistical peak is the cross point of both clouds. 12.2.11 Detection in the real world

Fig.18 shows the same items on a bed, with less light. This is luck again it works better than usual. Black noise subtraction is on. Fig.18(a) shows the normal view. Fig.18(b) shows the edge space. Each object have one or several edges. The candle and the holder are properly detected; thus they will be correctly tracked. Since the clock is quite small and has special edges, the rst identication will fail with an edge factor of two, then the search engine will nd the clock at arbitrary places all over the picture; when increasing the edge factor to 5, the R-Table is computed on the good shape: tracking will be correct. On Fig.18(g) and 18(h), when trying to identify the post card, it is far too fuzzy; it even has some common colors with the bed: a small edge factor will identify a small part of the card which will be identied to random places of the bed ( in fact it will be identied to the noise of the picture). When using a larger value of edge factor, the object selector will run out of the card, and will not work. If the object is moved away from its original place, detection does not work at all.

45

(a) ordinary space

(b) Selection in phase space

(c) Edge space

(d) Hough space

Figure 17: Real world detection: the authors face

12.3

Exporting YUV

Exporting the picture is not at all intuitive. The use of UNIX capture tools gave a blue square at the place of the expected picture. This is due to the way xvcam outputs the picture: it uses the Xv standard. This standard allows the user application to write any data directly in the memory of the video card. This means it is just impossible for any other application of the system to re read that memory location. Thus, only xvcam can produce the shots, and save them to a le. Several key bindings had to be added to xvcam. The new function called scrot() just read the cache of the picture which is about to be shown, and converts it into a .png le. But, as said previously, the internal format of picture is the YUV format so that scrot() has rst to export the YUV array into RGB. To export to RGB, scrot() needs to allocate a large array, then convert each YUV pixel into RGB, and call libpng to export that to a le. The u and v components have to be converted into the space [-128,+127]. Then the conversion is done by the equations: r=y+(v)*1.4023; g=y-(u)*0.3437-(v)*0.748; b=y+(u)*1.7734; A threshold is applied to keep the resulting values in the eld [0,255].

46

(a) ordinary space

(b) Edge space

(c) The candle

(d) The holder

(e) clock ef=2

(f) clock ef=5

(g) Post card ef=2

(h) Post Card ef=6

Figure 18: Real world : trinkets on a bed

47

12.4

Conclusion

Whereas the standard Hough Transform requires a limited number of parameters to describe the shape of the object, we have been able to set up a brand new algorithms able to nd an arbitrary shape in real time using a standard computer. Of course the computation of the algorithm is time consuming (more expensive than some other HT algorithms), but it is still possible to go and work using a rate above 5 picture/s using a 2 years old laptop. This mean any one could either: use a more powerful computer in the industry to perform PGHT; then rate of pictures will directly depend on the speed of the CPU; if our AMD1.4G laptop can reach 5 picture/s, an actual 3G Pentium with better cache management could deal with more than 10 picture/s. optimize the algorithms and use it in an embedded system and use it in planes radars, or missile for self driving; then a dedicated system could work only for picture recognition; when a 2 years old laptop can do 5picture/s, we think it would be possible to build a dedicated (such as an ASIC ?) system working at the same rate, but consuming less power: when one study our code, there are very few complex computation ( cos/sin/atan computation is done only once per point in the whole process ), and mainly RAM access, such as random access to 2D arrays; so that we expect it would be possible to reach incredible rates using a computer either with more cache, or with better cache pre-emption, or with a RAM using the same ( or at least half) speed as the CPU. RGHT does not need to detect cognitivity of objects: it is a straightforward approach after the basic ltering. detected objects can be partially obfuscated If you look at the picture recognition as Digital Signal Processing, you can still resume the work in the following steps: apply a low pass lter to get rid of some noise and bad frequencies convert the data in an appropriate format ( such as using ADC, applying FFT or an other space transformation) : in our case it is computing edge, phase and Hough spaces process the signal in the proper space convert back the data in a normal format ( using DAC or inverse of rst space transformation), after an optional down-sampling lter again to get rid of artifacts and standardize the results DSP usually use FFT in order to work in frequency domain ... HT works in the Hough domain. That is the only dierence. Apart from that, the picture recognition remains standard DSP. An other optimization possible: Rotation of the image produces just a phase shift while scaling of the image scales just the amplitudes. Two contours are similar if they have a similar frequency spectrum ... this is just the idea. It was implemented for applications like recognizing damaged gear wheels, which have a periodic contour (the teeth) which can be described by very few Fourier coecients. 48

The method is very powerful and well suited for any kind of contour as long as it is closed and ... but one has to accomplish a heavy chunk of programming :-/ It is overkill for very simple contours like circles or rectangles. Study of the RGHT shows that the need to determine whether the pairs of points are connected, ie member of the same shape; checking such a thing requires some computation. For each pair of point, they need to check connectivity, compute the horizontal and vertical scaling, what require a lot of processing (cot, arcsin ...). From that point of view, eciency of RGHT vs PGHT divided by processing time may vary depending on the architecture (CISC, RISC, ARM ...) 12.4.1 Points to be improved

write a better edge detector (Canny lter is said to be better than Sobel) rewrite the object selector in order to be more tolerant to noise inside the object to be selected nd a way to make object selector to take care of the color: it might be dicult because of the noise, but the object selection process can be long ... as long as the resulting R Table is more appropriate

49

13

Conclusions and Further Work

This project really works better than expected; when we wrote the rst draft of the project proposal, we thought that it would take life for us to cover all topics, and get everything work together. In fact, the longest part (in human days) was writing monitor 68hc08. It was also the least gratifying part since there were (non free) equivalents for MS-Windows, and Motorola released by the mean time a UNIX compliant monitor. But we learnt a lot about GNU/Linux under-niece, and about how to read and understand Motorola technical specications. By the mean time, we spend several minutes to 1h per day on other aspects of the project, such as the HT, even if that is not mentioned in the log book. About the HT, it was a completely new topic for us; we had to learn everything, but it looks like we learnt it quite fast, and acquired very deep understanding of the topic since the program works pretty well on our laptop. In the end, we have developed: a development board for the MC68HC908JB8, a rmware for the board, a complete image recognition software based on the xvcam project, with an improved GHT All softwares are open-source, and the whole project can be compiled on most Linux systems. List of points which can be modied, updated or improved: 1. The monitor support more CPUs targets. 2. The monitor write a ./congure 3. rmware improve the rmware to make it compliant with new versions of the GNU/Linux driver. 4. rmware rewrite a rmware from scratch so that it is possible to publish it under the terms of GPL2 [8]. Such a work could be based on the USB-IR-Boy [20] project (make a search of 68hc908 in http://www.sourceforge.net/). 5. GNU/Linux driver improve the interrupt mode. 6. GNU/Linux driver suppress the blocking system call of 1s when closing the device le. 7. GNU/Linux driver clean up the code so that the user-land program do not have to perform a usleep(50000) call to avoid kernel panics. 8. GNU/Linux driver write to the GNU/Linux development team to ask for a single and proper major/minor device number. 9. GNU/Linux driver rewrite the driver, clean up the code, and set up a new pair of driver/rmware, and query [21] for a proper VID/PID.

50

10. GNU/Linux driver for the 2.6 driver, support applications which just read or just write to the board, and/or use a circular buer like the 2.4 driver: support non equilibrated read/write operations. 11. xvcam.c keep a copy of the R-Table of all the objects found, set up some statistics, and adapt i2.R-Table depending on those statistics; this way it would be possible to make i2.R self adapt in case the object spins, scales or rotate slowly; giving strong importance to those statistics could also help rejecting bad objects. 12. xvcam.c support scaled and rotated objects. 13. xvcam.c use thinning algorithm to reduce thickness of edges. 14. xvcam.c set up a routine to reject bad objects; the rules of this algorithms are not dened yet, because we have no idea on how to do it. 15. xvcam.c improve the edge strength calculation to be more tolerant to noise so that the value stored is more relevant 16. xvcam.c improve the phase computation (use 45 deg and 135 deg edges) 17. xvcam.c add support of Y axis: either detect horizontal lines, and correct this, or after object detection, shift the R-Table, and convolve it again with the picture: if this second computation provides a stronger cloud peak, then the actual rotation is better than original search 18. xvcam.c since the position of the center of the object in known relatively to any point of the shape, it should NOT be necessary to draw a cloud outside each object: if the local gradient is properly computed, the we shall only increment the right of the shape, the one recorded in the R-Table6 . (it is a self adaptive method which allows to detect rotated objects, and turn around Y axis depending on that) 19. xvcam.c:conv *() try to use FFT instead of normal convolution. (see section 12.2.6) 20. xvcam.c:PGHT() change the last lter to a more intelligent one, so that it comes possible to detect peaks after the HT, and reject clouds due to noise. This way, it could be able to detect objects with more precision, and fewer error. 21. xvcam.c:PGHT() allow not to compute statistical tests on all points; for example skip computations if rand()>0.66 to reduce computation by 33%. 22. xvcam.c:compute local r() and run2() set up and use pre-computed tables of cos() sin() and atan(), after determining what in log book.pdf sections 21/08/2004 and 26/08/2004 leaded us to conclude that real time computed cos() is faster than pre-computed tables access ...
6 To detect rotated objects, shift the R-Table of n ranks; short convolve the old and the new table with the picture: instead of researching the tables in the whole picture, just start from anchor point, and for each entries of R-Tables, go to the edge of the object, compare the properties of that point with the properties expected from the table, compute an average table of those dierences: the RTable which produce the best average dierence is closer ( in angle mean ) to the actual object; this short process avoids to recompute too many times complete convolution of R-Table with picture; it just have to do this three time per frame ( one step left, one step right), keep the best result. It may increase just by 5% the total computation time, but allows to track rotations of the object: any slight rotation is detected on the next frame, and corrected at once: no need to shift the whole R-Table hundred times over the whole 2.

51

23. xvcam.c:compute local r() instead of using a growing value of radius ( o variable), try to use a propagation table of points to save time(by detecting cognitive pixels), and if this process is ecient, retry the Exhaustive HT routine. The two weakest points are the convolution mask of the edge detector, and the way the R-Table is generated. Most those points could be performed or implemented in short time. But if you do implement those optimizations, do not forget that everything is licensed under GPL, so that you must mail back your modied version to the author. Any new code will be published on the website, so that every one can gain benet of it.

52

14

List of softwares used

asl : ASM compiler http://john.ccac.rwth-aachen.de:8000/as/ Dia : vectorial schematic drawing http://www.gnome.org/projects/dia/ ICC : compiler and assembler http://www.pemicro.com/ gcc : C compiler http://gcc.gnu.org/ Latex : document syntax http://faq.tug.org/ OpenOce.org : word processor http://www.openoffice.org/ Oregano : Circuit design GUI http://people.imendio.com/richard/oregano/ sim68xx: simulator http://www.geocities.com/TheTropics/Harbor/8707/simulator/sim68xx/ TheGimp : picture editor http://www.gimp.org/ vim : text editor http://www.vim.org/

53

15

Glossary

Hough Transform (HT) an algorithm RGHT Randomized Generalized Hough Transform [15] is with the PGHT one that detects strait lines in pictures. of the very few GHT which can deThe Generalized HT can detect any tect shapes selected after compiling parametric curve, such as face conthe program. tour, or by extension any contour which can be put into equations beRHT Randomized Hough Transform: yet fore writing the code of the program. an other HT. HT See Hough Transform USB Universal Serial Port : a wonderful MC68HC908JB8 is a micro-controller free technology: it is a 4 wire serial from Motorola [12]; it is based on bus which can work at various rate the 68hc08, and is thus assembly obfrom 1.5MB/s to 480MB/s; there is ject compatible with the 6805 CPUs, a ground wire, a +5V/500mA power but has an internal USB controller; line, and two signal lines called D+ it is very easy to use and well docuand D- ( the two lines shall be mented, provided you are not afraid crossed by opposite intensity so that to read several books of 350 pages noise do not propagate) ... see [21]. each Note that this UC is limited YUV The not documented image format. to interrupt mode !!! No other USB This format is used to store pictures. transfer mode is supported. RGB is widely used, but do not reect what the eye see; worse: the PGHT enhanced Phase and edge based sensibility of the eye is not linear, Generalized Hough Transform : what mean the equations computing brand new algorithms developed by about RGB shall not be linear if the the author to detect arbitrary shape programmer wants linear response. objects in real time. It widely disYUV store in separate places light, cussed in section 12. The complete hue, and saturation. It is an other source code can be found at [16] (you 3D representation of the same space. can read it in the logbook, or downIt is used internally in many video load the source by downloading the devices, cameras, PAL TV ... sources of the documents).

54

16

References

References
[1] The asl project. http://john.ccac.rwth-aachen.de:8000/as/. [2] came. http://cvs.sourceforge.net/viewcvs.py/enlightenment/misc/camE/. [3] Camstream. http://www.smcc.demon.nl/camstream/. [4] The debian project. http://www.debian.org/. [5] Dvorak : the perfect keyboard. http://en.wikipedia.org/wiki/Dvorak_Simplified_Keyboard. [6] An edge based hough transform.
http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/MORSE/hough.pdf.

[7] Gericom. http://www.gericom.com/. [8] The gnu website. http://gnu.org/. [9] Iee 4th international conference on image processing. conference publication n0 354, 7-9 April 1992, part Moving object tracking using camera motion corrected Kalman ltering by J. F. Boyce and D L Toulson. [10] The mc68hc908jb8 by motorola. http://www.freescale.com/webapp/sps/ site/prod_summary.jsp?code=68HC908JB8. [11] monitor-68hc08. http://sourceforge.net/projects/monitor-68hc08/. [12] Motorolla. http://www.motorolla.com/. [13] Mplayer. http://www.mplayerhq.hu/. [14] Real time face detection using the hough transform: the gfht (generalized fuzzy ht). http://www.rennes.supelec.fr/ren/rd/etsn/ftp/articles/tech_etsn_04_ 2001.pdf. [15] Rght.
http://www.cse.cuhk.edu.hk/~pffung/publication/RandomizedGeneralizedHoughTransform_ConferenceICPR1996.pdf

[16] Self tracking webcam project. http://stwp.demaine.info/. [17] Simple description of various hough transform. http://www.dur.ac.uk/andrew1.hunter/Vision/CV06Hough.pdf. [18] Terratec. http://www.terratec.net/. [19] The usb driver for linux for the mct board. http://projets.sequanux.org/membres/sim/usb/. [20] Usb-ir-boy. http://sourceforge.net/projects/usbirboy/. [21] The usb website. http://www.usb.org/. [22] Usb08 eval board from. http://hc08web.de/usb08/ http://www.elektronikladen.de/.

55

[23] xawtv. http://linux.bytesex.org/xawtv/. [24] Xv. http://linux.about.com/library/cmd/blcmdl1_xv.htm. [25] Iee 3rd international conference on image processing, July 1989. conference publication n0 307, part Tracking and segmentation of moving objects in a scene by P. D. Picton. [26] Rafael C. Gonzalez and Richard E. Woods. Digital image processing, 2001. ISBN 0-201-18075-8. [27] Vishvjit S.Nalwa. A guided tour of computer vision, 1993. ISBN 1-201-54853-4. [28] Satosi Watanabe. Frontiers of pattern recognition, 1972. University of Hawaii, Honolulu, Hawaii, academic press, New York, London. [29] Tzay Y. Young and King-Sun Fu. Handbook of pattern recognition and image processing, 1986.

56

Appendices
A
A.1

Full source code of the Hough Transform

xvcam.c

I am sorry it is not colorful ... but I really do not know how to include colored code in A L TEX.
/ / / / View s t r e a m s from a webcam w i t h Xv / P a r t s from an app used t o t e s t T r i d e n t d r i v e r s , and camE / $ I d : xvcam . c , v 1 . 2 2001/06/21 1 9 : 3 0 : 0 2 r i c h l o w e Exp $ / g c c xvcam . c o xvcam I / u s r /X11R6/ i n c l u d e L/ u s r /X11R6/ l i b lX11 l X e x t lXv lm /

/ This s o f t w a r e i s p a r t o f t h e E n l i g h t e n m e n t p r o j e c t , and was found i n t h e camE d i r e c t o r y . The camE p r o j e c t i s w r i t t e n under GPL2 . When I found t h i s f i l e , i t was n o t l i c e n s e d , b u t t h e a u t h o r w i s h e s t o p u b l i s h under GPL2 as w e l l . So , a l l m o d i f i c a t i o n s done by me ( Demaine BP) or any o t h e r h a k e r s h a l l be done under GPL2 . For more d e t a i l s a b o u t t h e GPL : h t t p : / /www. gnu . o r g / For more d e t a i l s a b o u t E n l i g h t e n m e n t : g o o g l e > e n l i g h t e n m e n t For more d e t a i l s a b o u t a u t h o r s o f camE and xvcam , s e e t h e AUTHORS f i l e i n t h e CVS t r e e . For more d e t a i l s a b o u t DEMAINE BP : h t t p : / /www. demaine . i n f o / / / Note o f DEMAINE BenoitP i e r r e : my m o d i f d i e d v e r s i o n might r e q u i r e s t o be c o m p i l e d w i t h o p t i o n lm t h e k e y b o a r d o f my l a p t o p i s f r e n c h c o n v e r t e d i n DVORAK t h u s t h e k e y s r e a d w i l l p r o b a b l y be wrong f o r y o u r computer The o r i g i n a l xvcam was a b o u t 400 l i n e s . I t a c q u i r e s t h e p i c t u r e o f t h e webcam i n t h e YUV f o r m a t I d i d n t know a t t h e b e g i n n i n g . Note t h a t t h i s program works f i n e w i t h ov511 cams , such as my T e r r a t e c Terracam PRO, b u t n o t w i t h t h e Terracam Home wich i s CPIA . . . This program was w r i t t e n t o d e t e c t an o b j e c t i n a p i c from a webcam and u p d a t e s e r v o s p o s i t i o n u s i n g t h e hc08 d r i v e r . See t h e SelfTackingWebcam p r o j e c t i n my home page h t t p : / /www. demaine . i n f o / / // // // // // // // // // // // // // // The p s e e d o f t h e p r o c e s s depends on many f a c t o r s . The main ones a r e R RES and R WIDTH ; t h e s p e e d o f p r o c e s s i n g w i l l be l i n e a r l y proportionally inverse of those values . i f you u s e v e r y low v a l u e s , you can have a v e r y h i g h r a t e . . . b u t you cannot go a b o v e t h e numxer o f p i c / s e c t h e cam can g i v e ( USB1) which i s a b o u t 45 p/ s f o r me . u s i n g 3 6 0 : 1 or 1 0 0 : 3 I have 3 . 5 p/ s on my AMD 1G4 . But i t i s u s e l e s s t o t r y t o go h i g h e r than 5p/ s . i t a l l depend on your CPU and your c a c h e . The q u a l i t y depends on ( i n d e c r e a s i n g i m p o r t a n c e ) [ recommended v a l u e /minimum recommended ) MAX EDGE ERROR [ 5 0 / 2 0 ] ACCEPTED RATE [ 2 0 / 0 5 ] ACCEPTED RATE [ 7 0 / 4 0 ] R RES [ 2 0 0 / 5 0 ] R WIDTH [ 3 / 1 ] // s h o u l d be betmeen 1 and 4

#define ASPECT RATIO 2 #define SIZE 8 #define YUV WIDTH 320 #define YUV HEIGHT 240 #define SKIP 25 #define R RES 50 #define R WIDTH 3 #define MAX EDGE ERROR

// r e s o l u t i o n o f t h e RT a b l e hom many p o i n t s p e r 2 Pi // number o f p o i n t s p e r r d i u s // s p e e d f a c t o r 50 // t e l l s how many t i m e s t h e random // num g e n e r a t o r s e a r c h s f o r u s e f u l l p o i n t s

57

#define

ACCEPTED RATE 20

// // // //

this is or when miminum and t h e

due t o b u g s o f v e r y f u z z y p i c s t h e cam i s moving t o o f a s t accuracy r a t i o between l o c a l R t a b l e o b j e c t R Table

#define ACCEPTED RATE 70 #define PRECOS SIZE 1024 #define PI 3 . 1 4 1 5 9 2 #define BLACK FILE /tmp/ b l a c k n o i s e . raw // min and max f o r s e r v o s #define MIN 5 0 x10 #define MAX 5 0xF0 #define MIN 6 0 x30 #define MAX 6 0 x85 #define USB SLEEP TIME 100000

i n t tmp x , tmp y , tmp i , tmp j ; struct p l o p { int usb ; unsigned char b u f [ SIZE ] ; unsigned char b u f [ SIZE ] ; unsigned char n o i s e [ YUV WIDTH YUV HEIGHT 2 ] ; int l o c k ; i n t oldx , o l d y ; // l i g h t int tab1 [ YUV WIDTH ] [ YUV HEIGHT ] ; int t a b 1 g b [ YUV WIDTH / 2 ] [ YUV HEIGHT / 2 ] ; // greenb l u e int t a b 1 g r [ YUV WIDTH / 2 ] [ YUV HEIGHT / 2 ] ; // greenr e d int G h [ YUV WIDTH ] [ YUV HEIGHT ] ; // hor e d g e s int G v [ YUV WIDTH ] [ YUV HEIGHT ] ; // v e r e d g e s float S [ YUV WIDTH ] [ YUV HEIGHT ] ; // s t r e n g t h float P [ YUV WIDTH ] [ YUV HEIGHT ] ; // p h a s e int e f i l t e r ; int f i l t e r g b ; int f i l t e r g r ; int n o i s e f ; i n t view ; int s t ; int f l a g ; i n t R[ R RES ] [ R WIDTH ] ; // RT a b l e i n t R v a l i d [ R RES ] [ R WIDTH ] ; // RT a b l e i n t R x [ R RES ] [ R WIDTH ] ; // RT a b l e D e l t a x // RT a b l e D e l t a y i n t R y [ R RES ] [ R WIDTH ] ; f l o a t R P [ R RES ] [ R WIDTH ] ; // p h a s e o f t h e e d g e i n t R W[ R RES ] [ R WIDTH ] ; // w i d t h i n t mouse x , mouse y ; i n t W[ YUV WIDTH ] [ YUV HEIGHT ] ; // w i d t h o f an e d g e . . . // i t i s t h e number o f p o i n t s b o t h e l e m e n t o f an e d g e and o f // t h e t a n g a n t o f an e d g e a t a g i v e n p o i n t . . . computed i n l e n g t h ( ) // i t i s an a p r o x i m a t i o n o f t h e rayon de c o u r b u r e [ i n f r e n c h ] int search ; // t e l l e i t h e r we must compute t h e HT int e d g e f a c t o r ; i n t r e s u l t [ YUV WIDTH ] [ YUV HEIGHT ] ; i n t show ; // I f o r g o t what t h i s i s f o r int s l e e p ; // s e t i f we must w a i t a f t e r t h e p i c i s shown } i2 ; struct p l o u f { i n t R[ R RES ] ; i n t R v a l i d [ R RES ] ; i n t R x [ R RES ] ; i n t R y [ R RES ] ; i n t R s [ R RES ] ; } plaf ; f l o a t c o s s [ PRECOS SIZE ] ; f l o a t s i n n [ PRECOS SIZE ] ; int f i l m ; // f i l t e r s i z e 1 #define F 4 #define H 3 // h a l f o f F +1 #define F 2 8 #define H2 6 int filter1 h [5][5]={

// s t r e n g t h o f e d g e f o r t h e RT a b l e

58

{0 , {0 , {0 , {0 , {0 , };

0, 1, 2, 1, 0,

0 , 0 , 0} , 1, 0 , 0 } , 2, 0 , 0 } , 1, 0 , 0 } , 0 , 0 , 0} ,

int

filter1 v [5][5]={ {0 , 0 , 0 , 0 , 0} , {0 , 1 , 2, 1 , 0} , { 0 , 1, 2, 1, 0 } , {0 , 0 , 0 , 0 , 0} , {0 , 0 , 0 , 0 , 0} ,

}; int filter2 v [5][5]={ {0 , 0 , 0 , 0 , 0} , { 1 , 2 , 3 , 2 , 1} , { 1 , 2, 3, 2, 1} , {0 , 0 , 0 , 0 , 0} , { 0 , 0 , 0 , 0 , 0}

}; int filter2 { 0 , 1, { 0 , 2, { 0 , 3, { 0 , 2, { 0 , 1, h [5][5]={ 1 , 0 , 0} , 2 , 0 , 0} , 3 , 0 , 0} , 2 , 0 , 0} , 1 , 0 , 0} ,

};

/ / #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include <s t d i o . h> < s t d l i b . h> < s t r i n g . h> <e r r n o . h> <u n i s t d . h> <s y s / i p c . h> <s y s /shm . h> <s y s /mman . h> <s y s / i o c t l . h> <s y s / time . h> <s y s / r e s o u r c e . h> < f c n t l . h> <X11/ X l i b . h> <X11/ X u t i l . h> <X11/Xatom . h> <X11/ e x t e n s i o n s /Xv . h> <X11/ e x t e n s i o n s / X v l i b . h> <X11/ e x t e n s i o n s /XShm . h>

#include <l i n u x / v i d e o d e v . h> #define GUID I420 PLANAR 0 x30323449 #include <math . h> #include <s y s / t y p e s . h> #include <s y s / s t a t . h> #include <s y s / w a i t . h> //#i n c l u d e <time . h> #include <s c h e d . h> #include <png . h> extern i n t XShmGetEventBase ( D i s p l a y ) ; extern XvImage XvShmCreateImage ( D i s p l a y , XvPortID , int , char , int , int , XShmSegmentInfo ) ; int grab fd , g r a b s i z e ; s t a t i c struct video mmap g r a b b u f ; i n t yuv width = YUV WIDTH ; i n t y u v h e i g h t = YUV HEIGHT ;

59

s t a t i c unsigned char g r a b d a t a = NULL; s t a t i c unsigned char g r a b d a t a t m p = NULL; f l o a t t i m e d i f f ( struct t i m e v a l s t a r t , struct t i m e v a l end ) { // g i v e s t h e d i f f time i n ms . f l o a t r , s , us ; s =( f l o a t ) ( end>t v s e c s t a r t >t v s e c ) ; us =( f l o a t ) ( end>t v u s e c s t a r t >t v u s e c ) ; r=s 1000+ us / 1 0 0 0 ; return ( r ) ; } void deinit () { // struct timeval t s t a r t , t end ; munmap( g r a b d a t a t m p , g r a b s i z e ) ; g r a b d a t a t m p = NULL; // g e t t i m e o f d a y (& t s t a r t ,NULL) ; close ( grab fd ) ; // g e t t i m e o f d a y (& t e n d ,NULL) ; // p r i n t f ( g r a b : %fms \n\n , t i m e d i f f (& t s t a r t ,& t e n d ) ) ; // g e t t i m e o f d a y (& t s t a r t ,NULL) ; c l o s e ( i 2 >usb ) ; // t h i s i s t h e c a l l which t a k e s 100% CPU when e x i t t i n g // g e t t i m e o f d a y (& t e n d ,NULL) ; // p r i n t f ( u s b : %fms \n\n , t i m e d i f f (& t s t a r t ,& t e n d ) ) ; // g e t t i m e o f d a y (& t s t a r t ,NULL) ; free ( i2 ); // g e t t i m e o f d a y (& t e n d ,NULL) ; // p r i n t f ( i 2 : %fms \n\n , t i m e d i f f (& t s t a r t ,& t e n d ) ) ; } void init () { struct v i d e o c a p a b i l i t y g r a b c a p ; struct v i d e o c h a n n e l g r a b c h a n ; struct v i d e o m b u f vid mbuf ; atexit ( deinit ); i f ( ( g r a b f d = open ( / dev / v i d e o 0 , O RDWR) ) == 1) { f p r i n t f ( s t d e r r , Couldn t open / dev / v i d e o 0 : %s \n , s t r e r r o r ( e r r n o ) ) ; e x i t ( 1); } i f ( i o c t l ( g r a b f d , VIDIOCGCAP, &g r a b c a p ) == 1) { p e r r o r ( i o c t l VIDIOCGCAP ) ; e x i t ( 1); } grab chan . channel = 0 ; i f ( i o c t l ( g r a b f d , VIDIOCGCHAN, &g r a b c h a n ) == 1) { p e r r o r ( i o c t l VIDOCGCHAN ) ; e x i t ( 1); } g r a b c h a n . norm = 0 ; i f ( i o c t l ( g r a b f d , VIDIOCSCHAN, &g r a b c h a n ) == 1) { p e r r o r ( i o c t l VIDIOCSCHAN ) ; e x i t ( 1); } grab grab grab grab b u f . f o r m a t = VIDEO PALETTE YUV420P ; b u f . frame = 0 ; b u f . width = yuv width ; buf . height = yuv height ;

i o c t l ( g r a b f d , VIDIOCGMBUF, &vid mbuf ) ; g r a b s i z e = vid mbuf . s i z e ; grab data tmp = mmap( 0 , g r a b s i z e , PROT READ | PROT WRITE, MAP SHARED, g r a b f d , 0 ) ;

60

i f ( ( g r a b d a t a t m p == NULL) | | (1 == ( i n t ) g r a b d a t a t m p ) ) { f p r i n t f ( s t d e r r , Couldn t mmap\n ) ; exit (1); } / U s e l e s s ? p r o b a b l y . / s e t p r i o r i t y (PRIO PROCESS , 0 , 2 0 ) ; nice (20); }

void grab image ( ) { int i = 0 ; fd set fds ; FD ZERO(& f d s ) ; FD SET ( 0 , &f d s ) ; FD SET( g r a b f d , &f d s ) ; / Maybe a l i t t l e n i c e r t o t h e cpu ? / s e l e c t ( g r a b f d + 1 , &f d s , NULL, NULL, NULL ) ; i f ( i o c t l ( g r a b f d , VIDIOCMCAPTURE, &g r a b b u f ) == 1) { p e r r o r ( i o c t l VIDIOCMCAPTURE ) ; return ; } i f ( i o c t l ( g r a b f d , VIDIOCSYNC, &i ) == 1) { p e r r o r ( i o c t r l VIDIOCSYNC ) ; return ; } return ; } int init2 () { int fd ; int i ; g r a b d a t a=m a l l o c ( YUV WIDTH YUV HEIGHT 2 ) ; i 2=m a l l o c ( s i z e o f ( struct p l o p ) ) ; i f ( i 2==NULL) { p r i n t f ( Couldnt a l l o c a t e memory . \ n ) ; exit (1); } // i 2 >u s b = open (/ dev / h c 0 8 0 0 , O RDWR) ; // f o r 2 . 4 i 2 >usb = open ( / dev / h c 0 8 0 , O RDWR) ; // f o r 2 . 6 i f ( i 2 >usb <0) { p e r r o r ( Could not open hc08 d e v i c e f i l e . ) ; e x i t ( 1); } memset ( i 2 >buf , 0 x00 , SIZE ) ; i f ( w r i t e ( i 2 >usb , i 2 >buf , SIZE )!= SIZE ) p r i n t f ( Could not m r i t e b y t e s t o hc08 \n ) ; r e a d ( i 2 >usb , i 2 >b u f , SIZE ) ; u s l e e p ( USB SLEEP TIME ) ; i 2 >b u f [ 5 ] = 0 x80 ; i 2 >b u f [ 6 ] = 0 x70 ; i 2 >b u f [ 7 ] = 0 x68 ; i f ( w r i t e ( i 2 >usb , i 2 >buf , SIZE )!= SIZE ) p r i n t f ( Could not m r i t e b y t e s t o hc08 \n ) ; r e a d ( i 2 >usb , i 2 >b u f , SIZE ) ; u s l e e p ( USB SLEEP TIME ) ; i 2 >l o c k =0; // i 2 >t a b 1 =( i n t ) m a l l o c ( y u v w i d t h y u v h e i g h t s i z e o f ( i n t ) ) ; // i 2 >t a b 2 =( i n t ) m a l l o c ( y u v w i d t h y u v h e i g h t s i z e o f ( i n t ) ) ; i 2 > e f i l t e r =1; i 2 > f i l t e r g b =0; i 2 > f i l t e r g r =0; i 2 >view =0; i 2 >n o i s e f =0;

61

i 2 >s t =20; i 2 >s l e e p =0; i 2 >s e a r c h =0; i 2 >e d g e f a c t o r =2; i 2 >f l a g =0; i 2 >show =0; f o r ( i =0; i <PRECOS SIZE ; i ++) { c o s s [ i ]= c o s f ( ( f l o a t ) i ) ; s i n n [ i ]= s i n f ( ( f l o a t ) i ) ; } // c r e a t i n g b l a c k n o i s e p i c t u r e f d=open ( BLACK FILE ,O RDONLY ) ; i f ( fd <0) { f p r i n t f ( s t d e r r , Could not open b l a c k n o i s e f i l e . . . d i d you c r e a t e i t ?\ t o c r e a t e i t j u s t add t h e b o p t i o n a s f i r s t argument . \ n ) ; exit (1); } i f ( r e a d ( fd , i 2 >n o i s e , 3 2 0 4 8 0 ) ! = 3 2 0 4 8 0 ) { f p r i n t f ( s t d e r r , Black n o i s e f i l e c o r r u p t r e c r e a t e i t ! \ n ) ; exit (1); } f i l m =0; p r i n t f ( \n\n\ A l l r e f e r e d k e y s a r e fr FR keyboard ; \ n\ P r e s s on Space t o a c t i v a t e t h e mouse , and on 12 t o change t h e f i l t e r \ ( edge d e t e c t o r width ) . \ n\ p r e s s A t o g e t o r i g i n a l p i c , Z f o r s t r e n g t h o f e d g e s , E f o r phase \n\ Q f o r s w i t c h i n g Green / Blue f i l t e r , \ n\ S f o r s w i t c h i n g Green /Red f i l t e r . \ n\ W f o r s w i t c h i n g b l a c k n o i s e computation \n\ X f o r showing t h e c u r r e n t s e a r c h s t a t i s t i c s \n\ M f a r s w i t c h i n g t h e f l a g \ n\ H f a r s w i t c h i n g t h e Hough f l a g \n\ K f o r t a k i n g one s h o t \n\ J a l l o w d i s a l l o w t o t a k e a s h o t e v e r y frame \n\ p r e s s a r r o w s t o change t h e edge d e t e c t o r f a c t o r \n\n\ Pass argument b t o c r e a t e b l a c k n o i s e image . \ n\ P l e a s e w a i t 5 s b e f o r e t h e stream s t a r t . . . \ n\n ) ; return ( 0 ) ; } int conv1 ( i n t x , i n t y ) // make c o n v o l u t i o n w i t h 3 p i x e l s f i l t e r s { int i , j ; i n t sum h ; i n t sum v ; sum h =0; sum v =0; f o r ( i =0; i <= F ; i ++) f o r ( j =0; j<= F ; j ++) { sum h+=f i l t e r 1 h [ i ] [ j ] i 2 >tab1 [ x+i H ] [ y+j H ] ; sum v+=f i l t e r 1 v [ i ] [ j ] i 2 >tab1 [ x+i H ] [ y+j H ] ; } i 2 >G h [ x ] [ y]= sum h ; i 2 >G v [ x ] [ y]= sum v ; i f ( i 2 > f i l t e r g b ) { sum h =0; sum v =0; f o r ( i =0; i <= F ; i ++) f o r ( j =0; j<= F ; j ++) { sum h+=f i l t e r 1 h [ i ] [ j ] i 2 >t a b 1 g b [ ( x+i )/2 H ] [ ( y+j )/2 H ] ; sum v+=f i l t e r 1 v [ i ] [ j ] i 2 >t a b 1 g b [ ( x+i )/2 H ] [ ( y+j )/2 H ] ; } i=i 2 >G h [ x ] [ y ] ; j=i 2 >G v [ x ] [ y ] ; i 2 >G h [ x ] [ y]+=abs ( sum h ) ( i ? i / ( abs ( i ) ) : 1 ) ; i 2 >G v [ x ] [ y]+=abs ( sum v ) ( j ? j / ( abs ( j ) ) : 1 ) ; } i f ( i 2 > f i l t e r g r )

62

{ sum h =0; sum v =0; f o r ( i =0; i <= F ; i ++) f o r ( j =0; j<= F ; j ++) { sum h+=f i l t e r 1 h [ i ] [ j ] i 2 >t a b 1 g r [ ( x+i )/2 H ] [ ( y+j )/2 H ] ; sum v+=f i l t e r 1 v [ i ] [ j ] i 2 >t a b 1 g r [ ( x+i )/2 H ] [ ( y+j )/2 H ] ; } i=i 2 >G h [ x ] [ y ] ; j=i 2 >G v [ x ] [ y ] ; i 2 >G h [ x ] [ y]+=abs ( sum h ) ( i ? i / ( abs ( i ) ) : 1 ) ; i 2 >G v [ x ] [ y]+=abs ( sum v ) ( j ? j / ( abs ( j ) ) : 1 ) ; } return ( 0 ) ; } int conv2 ( i n t x , i n t y ) { // make c o n v o l u t i o n w i t h 5 p i x e l s f i l t e r s int i , j ; i n t sum h ; i n t sum v ; sum h =0; sum v =0; f o r ( i =0; i <= F ; i ++) f o r ( j =0; j<= F ; j ++) { sum h+=f i l t e r 2 h [ i ] [ j ] i 2 >tab1 [ x+i H ] [ y+j H ] ; sum v+=f i l t e r 2 v [ i ] [ j ] i 2 >tab1 [ x+i H ] [ y+j H ] ; } i 2 >G h [ x ] [ y]= sum h ; i 2 >G v [ x ] [ y]= sum v ; i f ( i 2 > f i l t e r g b ) { sum h =0; sum v =0; f o r ( i =0; i <= F ; i ++) f o r ( j =0; j<= F ; j ++) { sum h+=f i l t e r 2 h [ i ] [ j ] i 2 >t a b 1 g b [ x/2+ i H ] [ y/2+ j H ] ; sum v+=f i l t e r 2 v [ i ] [ j ] i 2 >t a b 1 g b [ x/2+ i H ] [ y/2+ j H ] ; } i=i 2 >G h [ x ] [ y ] ; j=i 2 >G v [ x ] [ y ] ; i 2 >G h [ x ] [ y]+=abs ( sum h ) ( i ? i / ( abs ( i ) ) : 1 ) ; i 2 >G v [ x ] [ y]+=abs ( sum v ) ( j ? j / ( abs ( j ) ) : 1 ) ; } i f ( i 2 > f i l t e r g r ) { sum h =0; sum v =0; f o r ( i =0; i <= F ; i ++) f o r ( j =0; j<= F ; j ++) { sum h+=f i l t e r 2 h [ i ] [ j ] i 2 >t a b 1 g r [ x/2+ i H ] [ y/2+ j H ] ; sum v+=f i l t e r 2 v [ i ] [ j ] i 2 >t a b 1 g r [ x/2+ i H ] [ y/2+ j H ] ; } i=i 2 >G h [ x ] [ y ] ; j=i 2 >G v [ x ] [ y ] ; i 2 >G h [ x ] [ y]+=abs ( sum h ) ( i ? i / ( abs ( i ) ) : 1 ) ; i 2 >G v [ x ] [ y]+=abs ( sum v ) ( j ? j / ( abs ( j ) ) : 1 ) ; } return ( 0 ) ; } / / / / / / // int random ( i n t x , i n t y ) { int i , j , z ; Random GHT

63

z =0; while ( 1 ) { i=rand ()% YUV WIDTH ; j=rand ()% YUV HEIGHT ; i f ( i 2 >S [ i ] [ j ] ) break ; i f ( z> MAX EDGE ERROR ) return ( 1 ) ; } x=i ; y=j ; return ( 0 ) ; } int RGHT( ) { int i ; i n t x1 , y1 , x2 , y2 ; f l o a t sx , sy ; f l o a t ox , oy ; p r i n t f ( PLEASE UPDATE RGHT( ) BEFORE USING IT\n\ READ THE HEADER OF EGHT PART\n ) ; exit (1); f o r ( x1 =0; x1<yuv width ; x1++) f o r ( y1 =0; y1<y u v h e i g h t ; y1++) i 2 >r e s u l t [ x1 ] [ y1 ] = 0 ; f o r ( i =0; i < YUV WIDTH YUV HEIGHT 1 ; i ++) { i f ( random (&x1 ,& y1 ) | | random (&x2 ,& y2 ) ) return ( 1 ) ; // good l u c k t o t h e one who t r i e s t o u n d e r s t a n d what I w r o t e : sx =( f l o a t ) ( x2x1 ) / ( f l o a t ) ( i 2 >R x [ ( i n t ) ( i 2 >P [ x1 ] [ y1 ] R RES / PI )] i 2 >R x [ ( i n t ) ( i 2 >P [ x2 ] [ y1 ] R RES / ( 2 PI ) ) ] ) ; sy =( f l o a t ) ( y2y1 ) / ( f l o a t ) ( i 2 >R y [ ( i n t ) ( i 2 >P [ x1 ] [ y1 ] R RES / PI )] i 2 >R y [ ( i n t ) ( i 2 >P [ x2 ] [ y1 ] R RES / ( 2 PI ) ) ] ) ; // ox =(( f l o a t ) x1+s x i2 >R x [ ( i n t ) ( i2 >P [ x1 ] [ y1 ] R RES /(2 PI ) ) ] + // t o u p d a t e // ( f l o a t ) x2+s x i2 >R x [ ( i n t ) ( i2 >P [ x2 ] [ y1 ] R RES /(2 PI ) ) ] ) / 2 ; // oy =(( f l o a t ) y1+s y i2 >R y [ ( i n t ) ( i2 >P [ x1 ] [ y1 ] R RES /(2 PI ) ) ] + // t o u p d a t e // ( f l o a t ) y2+s y i2 >R y [ ( i n t ) ( i2 >P [ x2 ] [ y1 ] R RES /(2 PI ) ) ] ) / 2 ; // i f you r e a l l y t h i n k you g e t i t , j u s t e m a i l me t o l e t me know . . . if ( // i s O i n s i d e t h e p i c t u r e ? ( ( ox>=0)&&(ox< YUV WIDTH)&&(oy>=0)&&(oy< YUV HEIGHT ) ) && // i s t h e s c a l i n g r a t i o a c c e p t a b l e ? ( ( sx >0.5)&&( sx <2)&&(sy >0.5)&&( sy <2)) ) i 2 >r e s u l t [ ( i n t ) ox ] [ ( i n t ) oy ]=0xFF ; } return ( 0 ) ; } / / / / / / // // // // // // E x h a u s t i v e GHT when t h i s p a r t was f i r s t w r i t t e n , i 2 . R t a b l e o n l y had one p o i n t p e r r a d i u s ( [ R WIDTH ] d i d n o t e x i s t y e t ) i f you now want t o u s e t h i s , you have t o modify c o m p u t e l o c a l r ( ) e i t h e r t o u s e i 2 . R Table w i t h a l l t h e width , or make t h e l o c a l Rt a b l e u s e o n l y t h e f i r s t p o i n t s o f i 2 . Rt a b l e

int m o y l o c a l ( i n t x , i n t y ) // p u t t h e c u r s o r a t t h e a v e r a g e c e n t e r o f t h e o b j e c t { int i ; f l o a t m, n ; m=0; n=0; f o r ( i =0; i < R RES ; i ++) { // m +=i2 >R x [ i ] ; // t o u p d a t e // n+=i2 >R y [ i ] ; // t o u p d a t e } m/= R RES ;

64

n/= R RES ; i f ( tmp i ) printf ( x += m; y += n ; return ( 0 ) ; }

m =%f n=%f \n ,m, n ) ;

int c o m p u t e l o c a l r ( int { int r ; int i , j ; i n t count ; float x , y ; int o ; int e x i t ; int plop ;

x , int

y)

// number o f v a l i d

f i e l d s i n t h e RT a b l e

// r a d i u s when h a l f o t p o i n t s have been found

p r i n t f ( PLEASE UPDATE c o m p u t e l o c a l r ( ) BEFORE USING IT\n\ READ THE HEADER OF EGHT PART\n ) ; return ( 1 ) ; // e x i t ( 0 ) ; f o r ( i =0; i < R RES ; i ++) { p l a f . R valid [ i ]=0; p l a f . R s [ i ]=0; p l a f .R[ i ] = 0 ; } r =0; count =0; e x i t =0; p l o p= YUV WIDTH ; while ( ! e x i t ) { r ++; // i n c r e a s e c i r c l e f o r ( o =0; o< R RES ; o++) // w a l k a l o n g t h e w h o l e c i r c l e { // compute a b s o l u t e c o o r d i n o t e s or c u r r e n t p o i n t x= x +( f l o a t ) r c o s ( ( f l o a t ) o ) ; y= y +( f l o a t ) r s i n ( ( f l o a t ) o ) ; // x= x +( f l o a t ) r c o s s [ o ] ; // y= y +( f l o a t ) r s i n n [ o ] ; i f ( x<0 | | y<0 | | x> YUV WIDTH | | y >240) continue ; i f ( ( ( i 2 > ( i n t ) x ] [ ( i n t ) y ] >( i 2 >e d g e f a c t o r +2))&&( p l a f . R v a l i d [ o]<=( r / 4 ) ) ) | | W[ ( ( i 2 > ( i n t ) x ] [ ( i n t ) y]> i 2 >e d g e f a c t o r ) && ( ! p l a f . R v a l i d [ o ] ) ) W[ ) // compare c u r e n t e d g e w i t h t h e one j u s t n e a r e r t h e c e n t e r { p l a f .R[ o ]= r ; i f ( ! plaf . R valid [ o ] ) count++; p l a f . R v a l i d [ o ]+=5; p l a f . R s [ o ]= i 2 >S [ ( i n t ) x ] [ ( i n t ) y ] ; p l a f . R x [ o ]=( i n t ) x x ; p l a f . R y [ o ]=( i n t ) y y ; } i f ( ( i 2 > ( i n t ) x ] [ ( i n t ) y]> i 2 >e d g e f a c t o r ) && ( p l a f . R v a l i d [ o ] ) ) W[ p l a f . R v a l i d [ o ]++; } i f ( ( count >=( R RES 0 . 7 ) ) & & ( ! p l o p ) ) p l o p=r ; i f ( ( count >( R RES 0 . 9 ) ) | | ( r > 1 3 0 ) | | ( r>p l o p 3 ) ) e x i t =1; } return ( 0 ) ; } int compare local to R () { int i , p ; float r , s ; r =0; p=0; f o r ( i =0; i < R RES ; i ++)

65

{ i f ( p l a f . R v a l i d [ i ] && i 2 >R v a l i d [ i ] ) { // c h e c k i f p o i n t i s member o f t h e e d g e // s =100(1( f l o a t ) p l a f .R[ i ] / ( f l o a t ) i 2 >R[ i ] ) ; // t o u p d a t e i f ( s <0) s=s ; i f ( s< ACCEPTED RATE ) { r+=s ; p++; } } } i f ( 1 0 0 ( f l o a t ) p / ( f l o a t ) R RES < ACCEPTED RATE ) return ( 9 9 9 ) ; r /=( f l o a t ) p ; return ( r ) ; } // ///////////////////// TO USE EGHT, PUT R RES = 10

int EGHT( ) // e x h a u s t i v e GHT ( we scan t h e p i c t u r e ) { // I run o v e r t h e w h o l e p i c t u r e and c o n s i d e r s u c c e v l y each p o i n t as // an anchor p o i n t , and t h e n compute t h e l o c a l RT a b l e . . . i f i t // matches THE i2 >RT a b l e t h e n we g e t t h e good p o i n t . int i , j ; int x , y ; i n t tmp ; f o r ( i =0; i <yuv width ; i ++) f o r ( j =0; j <y u v h e i g h t ; j ++) i 2 >r e s u l t [ i ] [ j ] = 0 ; f o r ( i= H2 ; i <yuv width H2 ; i ++) f o r ( j= H2 ; j <y u v h e i g h t H2 ; j ++) { i f ( i 2 >S [ i ] [ j ] ) i 2 > i ] [ j ]= l e n g t h ( i , j ) ; W[ else i 2 > i ] [ j ] = 0 ; W[ } f o r ( i =0; i < YUV WIDTH ; i +=2) { i f ( ! ( i %2)) p r i n t f ( i =%3.1 f \%\n , 1 0 0 ( f l o a t ) i / ( f l o a t ) YUV WIDTH ) ; f o r ( j =0; j < YUV HEIGHT ; j +=2) { i f ( ( abs ( xtmp x)<4)&&( abs ( ytmp y ) <4)) t m p i =1; else t m p i =0; compute local r ( i , j ) ; // compute R t a b l e o f cur o b j e c t x=i ; y=j ; // m o y l o c a l (&x ,&y ) ; // f i n d a v e r a g e c e n t e r // c o m p u t e l o c a l r ( x , y , R, R v a l i d , R x , R y ) ; // recompute R t a b l e tmp=c o m p a r e l o c a l t o R ( ) ; i f ( tmp< ACCEPTED RATE ) i 2 >r e s u l t [ x ] [ y ]=0xFF ; } } i 2 >show =0; return ( 0 ) ; } / / / / / / // int Phase and e d g e b a s e d GHT

66

PGHT( i n t xx , i n t yy ) { int i , j ; int x , y ; i n t tmp ; int p ; float o p , o w ; int mat [ 7 ] [ 7 ] = { { 0 , 0, 0, { 0 , 1, 1, { 0 , 1, 2 , { 0 , 1, 2 , { 0 , 1, 2 , { 0 , 1, 1, { 0 , 0, 0,

// p h a s e s h a p e GHT

0, 1, 2, 4, 2, 1, 0,

0, 1, 2, 2, 2, 1, 0,

0, 1, 1, 1, 1, 1, 0,

0} , 0} , 0} , 0} , 0} , 0} , 0} ,

}; i n t r e s u l t [ YUV WIDTH ] [ YUV HEIGHT ] ; i n t max r , max x , max y ; // l o c a t i o n where t h e f i l t e r w i l l d e t e c t a maxima f o r ( i =0; i <yuv width ; i ++) f o r ( j =0; j <y u v h e i g h t ; j ++) i 2 >r e s u l t [ i ] [ j ] = 0 ; f o r ( i= H2 ; i <yuv width H2 ; i ++) f o r ( j= H2 ; j <y u v h e i g h t H2 ; j ++) { i f ( i 2 >S [ i ] [ j ] ) i 2 > i ] [ j ]= l e n g t h ( i , j ) ; W[ else i 2 > i ] [ j ] = 0 ; W[ } f o r ( i =0; i < YUV WIDTH ; i +=1) f o r ( j =0; j < YUV HEIGHT ; j +=1) { int w; i f ( ! i 2 >S [ i ] [ j ] ) continue ; o p=i 2 >P [ i ] [ j ] ; o w=i 2 > i ] [ j ] ; W[ f o r ( p=0;p< R RES ; p++) f o r (w=0;w< R WIDTH ; w++) { i n t x = i i 2 >R x [ p ] [ w ] ; i n t y = j i 2 >R y [ p ] [ w ] ; i f ( ! i 2 >R v a l i d [ p ] ) continue ; // t e s t i f p h a s e i n p i c i s c l o s e t o any p h a s e i n RT a b l e // i f ( a b s ( o pi2 >R P [ p ] [ w]) <5) // i f ( a b s ( o wi2 >R W[ p ] [ w]) <5) // i f ( ( i i2 >R x [ p ] [ w]>=0)&&( i i2 >R x [ p ] [ w]< YUV WIDTH)&& // ( j i2 >R y [ p ] [ w]>=0)&&( j i2 >R y [ p ] [ w]< YUV HEIGHT) ) // i 2 >r e s u l t [ i i2 >R x [ p ] [ w ] ] [ j i2 >R y [ p ] [ w]]+=0 x1 ; if ( x >= 0 && x < YUV WIDTH && y >= 0 && y < YUV HEIGHT && abs ( o p i 2 >R P [ p ] [ w ] ) < 5 && abs ( o w i 2 >R W[ p ] [ w ] ) < 5 ) ++i 2 >r e s u l t [ x ] [ y ] ; } } max r =0; // p o i n t d e t e c t o r f i l t e r f o r ( i =3; i < YUV WIDTH3; i +=1) // a l o n g t h e p i c t u r e f o r ( j =3; j < YUV HEIGHT3; j +=1) { tmp=0; f o r ( x =0;x <7; x++) // a l o n g t h e m a t r i x f o r ( y =0;y <7; y++) // ( t h e Matrix has you ) { tmp+=i 2 >r e s u l t [ i+x 3 ] [ j+y 3]mat [ x ] [ y ] ; }

67

r e s u l t [ i ] [ j ]=tmp ; i f ( tmp>max r ) { max r=tmp ; max x=i ; max y=j ; } } xx=max x ; yy=max y ; return ( 0 ) ; } // End o f GHT a l g o s

/ / / / / / int run2 ( long f r a m e s ) { int x , y , w; int i , j ; float r ; i f ( i 2 >n o i s e f ) f o r ( x =0;x< YUV WIDTH YUV HEIGHT+3201202; x++) { i f ( g r a b d a t a [ x]> i 2 >n o i s e [ x ] | | i 2 >f l a g ) g r a b d a t a [ x]= i 2 >n o i s e [ x ]+( x>= YUV WIDTH YUV HEIGHT?0 x80 : 0 ) ; else g r a b d a t a [ x]= i 2 >n o i s e [ x] g r a b d a t a [ x ]+( x>= YUV WIDTH YUV HEIGHT?0 x80 : 0 ) ; } f o r ( x =0;x<yuv width ; x++) // g e n e r a t e l i g h t a r r a y f o r ( y =0;y<y u v h e i g h t ; y++) i 2 >tab1 [ x ] [ y ]=( i n t ) g r a b d a t a [ x+y yuv width ] ; f o r ( x =0;x<yuv width / 2 ; x++) // c o l o r o r r a y s f o r ( y =0;y<y u v h e i g h t / 2 ; y++) { i 2 >t a b 1 g b [ x ] [ y ]=( i n t ) g r a b d a t a [ x+y yuv width /2+ YUV WIDTH YUV HEIGHT ] ; i 2 >t a b 1 g r [ x ] [ y ]=( i n t ) g r a b d a t a [ x+y yuv width /2+320240+160120]; } switch ( i 2 > e f i l t e r ) // a p p l y e d g e f i l t e r { case 1 : f o r ( x= F ; x<yuv width F ; x++) f o r ( y= F ; y<y u v h e i g h t F ; y++) conv1 ( x , y ) ; break ; case 2 : f o r ( x= F ; x<yuv width F ; x++) f o r ( y= F ; y<y u v h e i g h t F ; y++) conv2 ( x , y ) ; break ; } f o r ( x =0;x<yuv width ; x++) f o r ( y =0;y<y u v h e i g h t ; y++) { // compute s t r e n g t h and p h a s e i 2 >S [ x ] [ y ] = 0 ; r=s q r t ( ( f l o a t ) ( i 2 >G h [ x ] [ y ] ) ( f l o a t ) ( i 2 >G h [ x ] [ y ])+ ( f l o a t ) ( i 2 >G v [ x ] [ y ] ) ( f l o a t ) ( i 2 >G v [ x ] [ y ] ) ) ; i f ( r>i 2 >s t ) { i 2 >S [ x ] [ y]= r ; r=a t a n f ( ( f l o a t ) ( i 2 >G v [ x ] [ y ] ) / ( f l o a t ) ( i 2 >G h [ x ] [ y ] ) ) + 1 PI / 2 ; } i f ( r <0) r+=PI ; // i f ( r>=PI ) // r=PI ; i 2 >P [ x ] [ y]= r ; } switch ( i 2 >view )

68

//

// //

{ case 0 : // show r e a l image break ; case 1 : // shom s t r e n g t h f o r ( x =0;x<yuv width ; x++) // t h i s i s t h e shown p i c t u r e f o r ( y =0;y<y u v h e i g h t ; y++) { r=i 2 >S [ x ] [ y ] ; i f ( r <0x10 ) r =0; i f ( r >0xFF ) r=0xFF ; g r a b d a t a [ x+y yuv width ]=( unsigned char ) r ; } break ; case 2 : // show p h a s e f o r ( x =0;x<yuv width ; x++) // t h i s i s t h e shown p i c t u r e f o r ( y =0;y<y u v h e i g h t ; y++) { r =1; i f ( i 2 >S [ x ] [ y ] ) r=i 2 >P [ x ] [ y ] 2 5 5 / ( 2 PI ) ; i f ( r <0) r =0; i f ( r >0xFF ) r=0xFF ; g r a b d a t a [ x+y yuv width ]=( unsigned char ) r ; } break ; case 3 : // show W f o r ( x =0;x<yuv width ; x++) // t h i s i s t h e shown p i c t u r e f o r ( y =0;y<y u v h e i g h t ; y++) { r=i 2 > x ] [ y ] ; W[ r=r 2 5 5 / ( H2 2 + 1) ; g r a b d a t a [ x+y yuv width ]=( unsigned char ) r ; } break ; case 4 : // show t h e Hough s p a c e { // show t h e Hough t a b l e int x , y ; i n t r e s =100; i n t tab [ ( i n t ) ( r e s PI 2 ) ] ; i n t c [ ( i n t ) ( r e s PI 2 ) ] ; i n t max , v a l ; i n t max , cur ; u n s i g n e d c h a r cur , max ; v a l =0; max=0; f o r ( x =0;x< YUV WIDTH ; x++) { tab [ x ] = 0 ; c [ x ]=0; f o r ( y =0;y< YUV HEIGHT ; y++) g r a b d a t a [ x+y yuv width ] = 0 ; } f o r ( x =0;x< YUV WIDTH ; x++) f o r ( y =0;y< YUV HEIGHT ; y++) { float k ; float a , a , r ; i n t tmp1 , tmp2 ; i f ( i 2 >S [ x ] [ y]==0) continue ; a =a t a n f ( ( f l o a t ) y /( f l o a t ) x ) ; f o r ( k =0; k <320; k++) { a=k PI/320+ a ; a=(a+PI / 2 ) 3 2 0 / ( 3 2 PI ) ; i f ( a<0) p r i n t f (% f \n , a ) ; i f ( a >320) p r i n t f (% f \n , a ) ; r=s q r t f ( ( f l o a t ) ( x x+y y ) ) c o s f ( i2 >P [ x ] [ y ] ) / 2 ; i f ( g r a b d a t a [ ( i n t ) ( a+r y u v w i d t h )] <254) g r a b d a t a [ ( i n t ) ( a+r y u v w i d t h )]++; } /

69

//

p r i n t f ( p=%f s=%f \n , i 2 >P [ x ] [ y ] , i 2 >S [ x ] [ y ] ) ; grab data [ ( int ) ( i 2 >P [ x ] [ y ] r e s ) + ( int ) ( i 2 >S [ x ] [ y ] 2 ) yuv width ]+=10; tmp1=( i n t ) ( i 2 >P [ x ] [ y ] r e s ) ; tmp2=( i n t ) ( i 2 >S [ x ] [ y ] s q r t f ( i 2 >S [ x ] [ y ] ) ) ; i f ( tmp1>( i n t ) ( r e s PI 2 ) ) continue ; tab [ tmp1]+=tmp2 ; c [ tmp1]++; / i f ( ( v a l > 3 0 0 0 ) | | ( tmp2 > 1 0 0 ) | | ( tmp1 >400)) { p r i n t f ( x=%i y=%i tmp1=%i tmp2=%i P=%f S=%f max=%i v a l=%i \n , x , y , tmp1 , tmp2 , i2 >P [ x ] [ y ] , i2 >S [ x ] [ y ] , max , v a l ) ; f o r ( x =0; x <320; x++) p r i n t f (%3.3 i :%5.5 i /%3.3 i , x , t a b [ x ] , c [ x ] ) ; exit (1); } / i f ( tab [ tmp1]> v a l ) { max=tmp1 ; v a l=tab [ max ] ; } // grab data [( int )( // ( i2 >P [ x ] [ y ] YUV WIDTH/PI ) // + // ( i2 >S [ x ] [ y ] YUV HEIGHT/56) y u v w i d t h // )]+=10; } // p r i n t f (max=%i v a l=%i \n ,max , v a l ) ; g r a b d a t a [ max]=0xFF ; g r a b d a t a [ max/2+(0 yuv width /2)+320240+160120]=0xFF ; // f o r ( x =0; x< YUV WIDTH; x++) // f o r ( y =0;y< YUV HEIGHT ; y++) } break ; case 5 : i f ( ! i 2 >s l e e p ) // show t h e a c c u m u l a t e d r e s u l t s // e x e p t i f j u s t s e l e c t e d t h e o b j e c t ( ? ? ? ) { f o r ( x =0;x<yuv width ; x++) f o r ( y =0;y<y u v h e i g h t ; y++) { g r a b d a t a [ x+y yuv width ]=( unsigned char ) i 2 >r e s u l t [ x ] [ y ] ; // i f ( i2 >r e s u l t [ x ] [ y ] ) // { // g r a b d a t a [ x+y y u v w i d t h ]=0xFF ; // g r a b d a t a [ x/2+y y u v w i d t h /4+320240]=0xFF ; // } // else // g r a b d a t a [ x+y y u v w i d t h ]=0 x00 ; } } break ; default : p r i n t f ( The s w i t c h s e l e c t o r o f t h e show p a r t went wrong \n ) ; break ; } i f ( i 2 >s l e e p ) // draw i n r e d t h e s e l e c t e d o b j e c t o n l y once { f o r ( x =0;x< R RES ; x++) f o r (w=0;w< R WIDTH ; w++) { i f ( i 2 >R v a l i d [ x ] ) grab data [ ( i 2 >R x [ x ] [ w]+ i 2 >mouse x )/2+ ( ( i 2 >R y [ x ] [ w]+ i 2 >mouse y ) / 2 ) yuv width /2+ 320240+160120 ]=0xFF ; } scrot (); //

70

} e l s e i f ( i 2 >s e a r c h ) // o t h e r w i s e s e a r c h t h e p a t t e r n i n t h e p i c t u r e . { struct t i m e v a l t s t a r t , t e n d ; // c a l l one GHT g e t t i m e o f d a y (& t s t a r t ,NULL ) ; // RGHT( ) ; // EGHT( ) ; PGHT(&x ,&y ) ; g e t t i m e o f d a y (& t e n d ,NULL ) ; p r i n t f ( PGHT: %fms \n , t i m e d i f f (& t s t a r t ,& t e n d ) ) ; p r i n t f ( x=%3.3 i y=%3.3 i \n , x , y ) ; f o r ( i =1; i <=1; i ++) f o r ( j =1; j <=1; j ++) { if ( i || j ) g r a b d a t a [ x/2+ i +(( y/2+ j ) yuv width /2)+320240+160120]=0xFF ; else g r a b d a t a [ x/2+ i +(( y/2+ j ) yuv width /2)+320240+160120]=0 x00 ; } g r a b d a t a [ x/2+y yuv width /4+320240]=0xFF ; i 2 >b u f [5]+=( YUV WIDTH/2x ) / 4 0 ; i 2 >b u f [6] =( YUV HEIGHT/2y ) / 4 0 ; i f ( i 2 >b u f [ 5 ] < MIN 5 ) i 2 >b u f [ 5 ] = MIN 5 ; i f ( i 2 >b u f [ 5 ] > MAX 5 ) i 2 >b u f [ 5 ] = MAX 5 ; i f ( i 2 >b u f [ 6 ] < MIN 6 ) i 2 >b u f [ 6 ] = MIN 6 ; i f ( i 2 >b u f [ 6 ] > MAX 6 ) i 2 >b u f [ 6 ] = MAX 6 ; i f ( w r i t e ( i 2 >usb , i 2 >buf , SIZE )!= SIZE ) p r i n t f ( Could not m r i t e b y t e s t o hc08 \n ) ; r e a d ( i 2 >usb , i 2 >b u f , SIZE ) ; u s l e e p ( USB SLEEP TIME ) ; } return ( 0 ) ; } int le ng th ( int x , int y ) { // h e r e I d e t e r m i n e t h e s h a p e o f t h e p a r a l l e l l i n e c r o s s i n g t h e // c u r r e n t p o i n t , and c o u n t how many p o i n t s have a c l o s e e d g e s t r e n g t h int r ; int i ; r =0; f o r ( i= H2 ; i < H2 ; i ++) // w a l k a l o n g 10 p i x e l s i f ( abs ( i 2 >S [ x+( i n t ) ( i ( s i n f ( i 2 >P [ x ] [ y ] (2 PI ) / 2 ) ) ) ] [ y+( i n t ) ( i ( c o s f ( i 2 >P [ x ] [ y ] (2 PI ) / 2 ) ) ) ] ( ( i n t ) i 2 >S [ x ] [ y ] ) ) <10) // q u a l i t y f a c t o r : l o w e r makes f i l t e r more r e s t r i c t i v e r ++; return ( r ) ; } int moy r ( ) // p u t t h e c u r s o r a t t h e a v e r a g e c e n t e r o f t h e o b j e c t { int i , w; f l o a t m, n ; m=0; n=0; f o r ( i =0; i < R RES ; i ++) f o r (w=0;w< R WIDTH ; w++) { m +=i 2 >R x [ i ] [ w ] ; n+=i 2 >R y [ i ] [ w ] ; } m/= R RES ; n/= R RES ; i 2 >mouse x+= m;

71

i 2 >mouse y+=n ; return ( 0 ) ; } int e x t r a c t R ( i n t mouse x , i n t mouse y ) { int r , w; // s t r e n g t h o f e d g e f o r t h e RT a b l e i n t R s [ R RES ] [ R WIDTH ] ; int i , j ; i n t count ; // number o f v a l i d f i e l d s i n t h e RT a b l e float x , y ; int o ; int e x i t ; int plop ; // r a d i u s when h a l f o t p o i n t s have been found i 2 >mouse x=mouse x ; i 2 >mouse y=mouse y ; tmp x=mouse x ; tmp y=mouse y ; f o r ( i =0; i < R RES ; i ++) f o r (w=0;w< R WIDTH ; w++) { i 2 >R v a l i d [ i ] [ w] = 0 ; R s [ i ] [ w] = 0 ; i 2 >R[ i ] [ w] = 0 ; } f o r ( i= H2 ; i <yuv width H2 ; i ++) f o r ( j= H2 ; j <y u v h e i g h t H2 ; j ++) { i f ( i 2 >S [ i ] [ j ] ) i 2 > i ] [ j ]= l e n g t h ( i , j ) ; W[ else i 2 > i ] [ j ] = 0 ; W[ } r =0; count =0; e x i t =0; p l o p= YUV WIDTH ; while ( ! e x i t ) { r ++; // i n c r e a s e c i r c l e f o r ( o =0; o< R RES ; o++) // w a l k a l o n g t h e w h o l e c i r c l e { // compute a b s o l u t e c o o r d i n a t e s or c u r r e n t p o i n t x=mouse x+( f l o a t ) r c o s f ( ( f l o a t ) o ) ; y=mouse y+( f l o a t ) r s i n f ( ( f l o a t ) o ) ; i f ( x<0 | | y<0 | | x> YUV WIDTH | | y >240) continue ; // d e t e r m i n e l o c a t i o n o f f i r s t a v a i l a b l e c e l l f o r (w=0;(w< R WIDTH )&&( i 2 >R v a l i d [ o ] [ w ] ! = 0 ) ; ) w++; i f (w >= R WIDTH ) continue ; i f ( ( ( i 2 > ( i n t ) x ] [ ( i n t ) y ] >( i 2 >e d g e f a c t o r +2))&&( i 2 >R v a l i d [ o ] [ w]<=( r / 4 ) ) ) | | W[ ( ( i 2 > ( i n t ) x ] [ ( i n t ) y]> i 2 >e d g e f a c t o r ) && ( ! i 2 >R v a l i d [ o ] [ w ] ) ) W[ ) // i f ( ( ( i2 >S [ ( i n t ) x ] [ ( i n t ) y]> R s [ o ])&&(! i 2 >R v a l i d [ o ] ) ) | | // ( i2 >S [ ( i n t ) x ] [ ( i n t ) y ] >(( R s [ ( i n t ) o ] ) + 2 ( ri2 >R[ ( i n t ) o ] ) ) ) ) // compare c u r e n t e d g e w i t h t h e one j u s t n e a r e r t h e c e n t e r { i 2 >R[ o ] [ w]= r ; i f ( ! i 2 >R v a l i d [ o ] [ w ] ) count++; i 2 >R v a l i d [ o ] [ w]+=5; R s [ o ] [ w]= i 2 >S [ ( i n t ) x ] [ ( i n t ) y ] ; i 2 >R x [ o ] [ w]=( i n t ) xmouse x ; i 2 >R y [ o ] [ w]=( i n t ) ymouse y ; } i f ( ( i 2 > ( i n t ) x ] [ ( i n t ) y]> i 2 >e d g e f a c t o r ) && ( i 2 >R v a l i d [ o ] [ w ] ) ) W[ i 2 >R v a l i d [ o ] [ w]++; i 2 >R P [ o ] [ w]= i 2 >P [ ( i n t ) x ] [ ( i n t ) y ] ; i 2 >R W[ o ] [ w]= i 2 > ( i n t ) x ] [ ( i n t ) y ] ; W[ } i f ( ( count >=( R RES R WIDTH 0 . 7 ) ) & & ( ! p l o p ) ) p l o p=r ; i f ( ( count >( R RES R WIDTH 0 . 9 ) ) | | ( r > 1 3 0 ) | | ( r>p l o p 3 ) ) e x i t =1; }

72

i 2 >s l e e p =1; i 2 >s e a r c h =1; i 2 >show =0; return ( 0 ) ; } int save black noise () { int fd ; long i n t i , j ; i n t tab [ 3 2 0 4 8 0 ] ; unsigned char av [ 3 2 0 4 8 0 ] ; f d=open ( BLACK FILE ,O WRONLY | O CREAT | O TRUNC, S IRUSR | S IWUSR ) ; i f ( fd <0) { f p r i n t f ( s t d e r r , Could not open %s . \ n , BLACK FILE ) ; return ( 1 ) ; } p r i n t f ( R e c o r d i n g b l a c k n o i s e p i c t u r e . . . t a k e s about 8 s \n ) ; f o r ( i =0; i <30; i ++) // w a i t u n t i l t h e camera s e l f c a l i b r a t e d { grab image ( ) ; } f o r ( j =0; j <320480; j ++) tab [ j ] = 0 ; f o r ( i =0; i <10; i ++) // w a i t u n t i l t h e camera s e l f c a l i b r a t e d { grab image ( ) ; f o r ( j =0; j <320480; j ++) tab [ j ]+=(long i n t ) g r a b d a t a t m p [ j ] ; } f o r ( j =0; j <320480; j ++) av [ j ]=( unsigned char ) ( tab [ j ] / 1 0 ) ; w r i t e ( fd , av , 3 2 0 4 8 0 ) ; p r i n t f ( Black n o i s e p i c t u r e t a k e n . Dont t a k e c a r e about t h e n e x t s e g f a u l t . \ n ) ; return ( 0 ) ; } int child () { i f ( i 2 >s e a r c h ) usleep (451000); g e t t i m e o f d a y (& t s t a r t ,NULL) ; grab image ( ) ; g e t t i m e o f d a y (& t e n d ,NULL) ; p r i n t f (GRAB: %fms \n , t i m e d i f f (& t s t a r t ,& t e n d ) ) ; return ( 0 ) ;

// // // } int scrot () {

unsigned char ( data ) [ 3 ] ; i n t w, h ; char fname [ 1 0 2 4 ] ; FILE f p ; p n g s t r u c t p p n g p t r = NULL; p n g i n f o p i n f o p t r = NULL; p n g b y t e p r o w p o i n t e r s = NULL; int j ; time t t ; struct tm tm ; w= YUV WIDTH ; // Width o f t h e p i c t u r e from c o n s t a n t e s h= YUV HEIGHT ; // H e i g h t o f t h e p i c time (& t ) ; tm=l o c a l t i m e (& t ) ; s t r c p y ( fname , xvcamsh o t YYYY MM DD HH MM SS . png \0 ) ; s p r i n t f (&fname [ 1 1 ] , %4.4 i %2.2 i %2.2 i %2.2 i %2.2 i %2.2 i . png \0 , tm>t m y e a r +1900 ,tm>tm mon+1 ,tm>tm mday , tm>tm hour , tm>tm min , tm>t m s e c ) ; p r i n t f ( The s h o t w i l l be s t o r e d i n %s \n , fname ) ;

73

f p=f o p e n ( fname , wb ) ; i f ( f p == NULL) { p r i n t f ( Could not c r e a t e s h o t f i l e i n c u r r e n t d i r e c t o r y ) ; return ( 1 ) ; } ( data ) = m a l l o c (w h s i z e o f data ) ; i f ( data == NULL) { p r i n t f ( Could not a l l o c a t e memory f o r PNG e x p o r t . \ n ) ; return ( 1 ) ; } { int i , j ; unsigned char c ; f o r ( i =0; i <w ; i ++) f o r ( j =0; j <h ; j ++) { float y , u , v , r , g , b ; // g r a b d a t a [ ] i s t h e p l a c e where i s s t o r e d YUV p i c t u r e y=g r a b d a t a [ i+j w ] ; u=g r a b d a t a [ i /2+( j / 2 ) w/2+wh ] 1 2 8 ; //u=0; v=g r a b d a t a [ i /2+( j / 2 ) w/2+wh+wh / 4 ] 1 2 8 ; // v =0; r=y+(v ) 1 . 4 0 2 3 ; g=y(u ) 0 . 3 4 3 7 ( v ) 0 . 7 4 8 ; b=y+(u ) 1 . 7 7 3 4 ; i f ( r <0) r =0; i f ( g <0) g =0; i f ( b<0) b=0; i f ( r >255) r =255; i f ( g >255) g =255; i f ( b>255) b =255; ( ( ( data )+1( j w+i ) ) ) [ 0 ] = r ; ( ( ( data )+1( j w+i ) ) ) [ 1 ] = g ; ( ( ( data )+1( j w+i ) ) ) [ 2 ] = b ; } } p n g p t r = p n g c r e a t e w r i t e s t r u c t (PNG LIBPNG VER STRING , 0 , 0 , 0 ) ; i f ( p n g p t r == NULL) goto e r r p n g ; i n f o p t r = p n g c r e a t e i n f o s t r u c t ( png ptr ) ; i f ( i n f o p t r == NULL) goto e r r p n g ; i f ( s e t j m p ( png jmpbuf ( p n g p t r ) ) ) goto e r r p n g ; p n g i n i t i o ( png ptr , fp ) ; png set IHDR ( p n g p t r , i n f o p t r , w, h , 8 , PNG COLOR TYPE RGB, PNG INTERLACE NONE, PNG COMPRESSION TYPE DEFAULT, PNG FILTER TYPE DEFAULT ) ; p n g s e t c o m p r e s s i o n l e v e l ( png ptr , 1 0 ) ; p n g w r i t e i n f o ( png ptr , i n f o p t r ) ; r o w p o i n t e r s = png malloc ( png ptr , h sizeof r o w p o i n t e r s ) ; f o r ( j =0; j < h ; ++j ) r o w p o i n t e r s [ j ] = ( p n g b y t e ) data [ j w+ 0 ] ; png write image ( png ptr , r o w p o i n t e r s ) ; p n g f r e e ( png ptr , r o w p o i n t e r s ) ; p n g w r i t e e n d ( p n g p t r , NULL ) ; p n g d e s t r o y w r i t e s t r u c t (& p n g p t r , &i n f o p t r ) ; f c l o s e ( fp ) ; return 0 ; err png : i f ( r o w p o i n t e r s ) p n g f r e e ( png ptr , r o w p o i n t e r s ) ; p n g d e s t r o y w r i t e s t r u c t (& p n g p t r , &i n f o p t r ) ; f c l o s e ( fp ) ; return 2 ;

//

74

} int main ( i n t a r g c , char a r g v ) { i n t x v p o r t = 1, i , d , s c r e e n , CompletionType ; unsigned i n t ud , width , h e i g h t ; long f r a m e s ; unsigned i n t ver , r e l , req , ev , e r r , adapt ; long s t a c k [ 1 0 2 4 ] ; D i s p l a y dpy ; Window window , dw ; XSizeHints hint ; XSetWindowAttributes xswa ; XWindowAttributes a t t r i b s ; XVisualInfo vinfo ; XEvent e v e n t ; GC gc ; XvAdaptorInfo a i ; XvImage yuv image ; XShmSegmentInfo y u v s h m i n f o ; Atom wmDeleteWindow ; dpy = XOpenDisplay (NULL ) ; s c r e e n = D e f a u l t S c r e e n ( dpy ) ; XGetWindowAttributes ( dpy , DefaultRootWindow ( dpy ) , &a t t r i b s ) ; XMatchVisualInfo ( dpy , s c r e e n , a t t r i b s . depth , TrueColor , &v i n f o ) ; wmDeleteWindow = XInternAtom ( dpy , WM DELETE WINDOW , F a l s e ) ; i f ( a r g c !=1) i f ( ! strcmp ( a r g v [ 1 ] , b ) ) { init (); save black noise (); return ( 0 ) ; } f r a m e s =0; hint hint hint hint hint . x = 1; . y = 1; . width = yuv width ; . height = yuv height ; . f l a g s = PPosition | PSize ;

xswa . colormap = XCreateColormap ( dpy , DefaultRootWindow ( dpy ) , v i n f o . v i s u a l , AllocNone ) ; xswa . event mask = S t r u c t u r e N o t i f y M a s k | ExposureMask ; xswa . b a c k g r o u n d p i x e l = 0 ; xswa . b o r d e r p i x e l = 0 ; window = XCreateWindow ( dpy , DefaultRootWindow ( dpy ) , 0 , 0 , yuv width ASPECT RATIO, y u v h e i g h t ASPECT RATIO, 0 , v i n f o . depth , InputOutput , v i n f o . v i s u a l , CWBackPixel | CWBorderPixel | CWColormap | CWEventMask , &xswa ) ; // X S e l e c t I n p u t ( dpy , window , S t r u c t u r e N o t i f y M a s k ) ; X S e l e c t I n p u t ( dpy , window , S t r u c t u r e N o t i f y M a s k | ExposureMask | ButtonPressMask | PointerMotionMask | ButtonMotionMask | KeyPressMask | KeyReleaseMask ) ; X S e t S t a n d a r d P r o p e r t i e s ( dpy , window , xvcam , xvcam , None , NULL, 0 , &h i n t ) ; XSetWMProtocols ( dpy , window , &wmDeleteWindow , 1 ) ; XMapWindow( dpy , window ) ; i f ( XShmQueryExtension ( dpy ) ) CompletionType = XShmGetEventBase ( dpy ) + ShmCompletion ; else e x i t ( 1); i f ( S u c c e s s != XvQueryExtension ( dpy , &ver , &r e l , &req , &ev , &e r r ) ) f p r i n t f ( s t d e r r , Couldn t do Xv s t u f f \n ) ;

75

i f ( S u c c e s s != XvQueryAdaptors ( dpy , DefaultRootWindow ( dpy ) , &adapt , &a i ) ) f p r i n t f ( s t d e r r , Couldn t do Xv s t u f f \n ) ; f o r ( i = 0 ; i < ( i n t ) adapt ; i ++) { xv port = ai [ i ] . base id ; } i f ( adapt > 0 ) XvFreeAdaptorInfo ( a i ) ; gc = XCreateGC ( dpy , window , 0 , 0 ) ; yuv image = XvShmCreateImage ( dpy , x v p o r t , GUID I420 PLANAR , 0 , yuv width , y u v h e i g h t , &y u v s h m i n f o ) ; y u v s h m i n f o . shmid = shmget (IPC PRIVATE , yuv image>d a t a s i z e , IPC CREAT | 0 7 7 7 ) ; y u v s h m i n f o . shmaddr = ( char ) shmat ( y u v s h m i n f o . shmid , 0 , 0 ) ; yuv image>data = y u v s h m i n f o . shmaddr ; y u v s h m i n f o . readOnly = F a l s e ; if ( ! XShmAttach ( dpy , &y u v s h m i n f o ) ) { p r i n t f ( XShmAttach go boom boom ! \ n ) ; e x i t ( 1);

} init (); init2 (); memset ( yuv image>data , 0 x00 , 3 2 0 2 4 0 ) ; memset (320240+ yuv image>data , 0 x80 , 3 2 0 2 4 0 ) ; f o r ( f r a m e s =0; frames < SKIP ; f r a m e s++) { int a , b ; f o r ( a =0; a <255 f r a m e s / SKIP ; a++) f o r ( b =110;b <130; b++) yuv image>data [ a+b 320]=0 x80 ; XGetGeometry ( dpy , window , & dw , &d , &d , &width , &h e i g h t , &ud , &ud ) ; XvShmPutImage ( dpy , x v p o r t , window , gc , yuv image , 0 , 0 , yuv image>width , yuv image>h e i g h t , 0 , 0 , width , h e i g h t , True ) ; grab image ( ) ; } f r a m e s =0; while ( 1 ) { struct t i m e v a l t c u r , t o l d ; struct t i m e v a l t s t a r t , t e n d ; int r e s ; g e t t i m e o f d a y (& t c u r ,NULL ) ; i f ( frames ) p r i n t f ( L a s t l o o p t o o k %f ms . \ n , t i m e d i f f (& t o l d ,& t c u r ) ) ; t o l d . t v s e c=t c u r . t v s e c ; t o l d . t v u s e c=t c u r . t v u s e c ; g e t t i m e o f d a y (& t s t a r t ,NULL) ; grab image ( ) ; g e t t i m e o f d a y (& t e n d ,NULL) ; p r i n t f (GRAB: %fms \n , t i m e d i f f (& t s t a r t ,& t e n d ) ) ; i f ( r e s=c l o n e ( c h i l d ,& s t a c k [ 1 0 2 0 ] , SIGCHLD | CLONE FILES |CLONE VM,NULL)==1) { p r i n t f ( Could not f o r k . \ n ) ; exit (1); } memcpy ( g r a b d a t a , g r a b d a t a t m p , YUV WIDTH YUV HEIGHT 2 ) ; g e t t i m e o f d a y (& t s t a r t ,NULL ) ; run2 ( f r a m e s ) ; if ( film ) scrot (); g e t t i m e o f d a y (& t e n d ,NULL ) ;

//

// // // //

76

//

p r i n t f (RUN2: %fms \n , t i m e d i f f (& t s t a r t ,& t e n d ) ) ; g e t t i m e o f d a y (& t s t a r t ,NULL ) ; w a i t p i d ( r e s , NULL, 0 ) ; g e t t i m e o f d a y (& t e n d ,NULL ) ; p r i n t f ( w a i t : %fms \n\n , t i m e d i f f (& t s t a r t ,& t e n d ) ) ; memcpy ( yuv image>data , g r a b d a t a , yuv image>d a t a s i z e ) ; XGetGeometry ( dpy , window , & dw , &d , &d , &width , &h e i g h t , &ud , &ud ) ; XvShmPutImage ( dpy , x v p o r t , window , gc , yuv image , 0 , 0 , yuv image>width , yuv image>h e i g h t , 0 , 0 , width , h e i g h t , True ) ; f r a m e s ++; XPending ( dpy ) ; // i f ( i2 >s l e e p ) // sleep (2); i 2 >s l e e p =0;

//

// //

while ( XPending ( dpy ) ) { XNextEvent ( dpy , &e v e n t ) ; switch ( e v e n t . t y p e ) { case C l i e n t M e s s a g e : { i f ( e v e n t . x c l i e n t . f o r m a t == 32 && e v e n t . x c l i e n t . data . l [ 0 ] == ( signed ) wmDeleteWindow ) exit (0); }; break ; case B u t t o n P r e s s : { i n t b u tt o n ; b u tt o n =3; switch ( e v e n t . xbutton . b u tt o n ) { case Button1 : b u tt o n =1; i f ( i 2 >l o c k ) { i 2 >b u f [ 7 ] + + ; i f ( i 2 >b u f [ 7 ] < 0 x10 ) i 2 >b u f [ 7 ] = 0 x10 ; i f ( i 2 >b u f [ 7 ] > 0 xF0 ) i 2 >b u f [ 7 ] = 0 xF0 ; i f ( w r i t e ( i 2 >usb , i 2 >buf , SIZE )!= SIZE ) p r i n t f ( Could not m r i t e b y t e s t o hc08 \n ) ; r e a d ( i 2 >usb , i 2 >b u f , SIZE ) ; u s l e e p ( USB SLEEP TIME ) ; i 2 >s e a r c h =0; i 2 >show =0; } else { p r i n t f ( E x t r a c t i n g o b j e c t from l o c a t i o n (%i ,% i ) . . . \ n , event . xbutton . x , event . xbutton . y ) ; e x t r a c t R ( e v e n t . xbutton . x/ASPECT RATIO, e v e n t . xbutton . y/ASPECT RATIO ) ; moy r ( ) ; e x t r a c t R ( i 2 >mouse x , i 2 >mouse y ) ; } break ; case Button2 : b u tt o n =2; break ; default : // s h o u l d be b u t t o n 3 ( or 4/5 ?) i f ( i 2 >l o c k ) { i 2 >b u f [7] ; i f ( i 2 >b u f [ 7 ] < 0 x10 ) i 2 >b u f [ 7 ] = 0 x10 ; i f ( i 2 >b u f [ 7 ] > 0 xF0 ) i 2 >b u f [ 7 ] = 0 xF0 ; i f ( w r i t e ( i 2 >usb , i 2 >buf , SIZE )!= SIZE ) p r i n t f ( Could not m r i t e b y t e s t o hc08 \n ) ; r e a d ( i 2 >usb , i 2 >b u f , SIZE ) ; u s l e e p ( USB SLEEP TIME ) ;

77

i 2 >s e a r c h =0; i 2 >show =0; } break ; } / p r i n t f ( Button %i p r e s s e d a t (%i ,% i ) \ n , b u t t o n , e v e n t . x b u t t o n . x , e v e n t . x b u t t o n . y ) ; / }; break ; case M o t i o n N o t i f y : { // p r i n t f ( p o s i t i o n : (%i ,% i ) \ n , e v e n t . x b u t t o n . x , e v e n t . x b u t t o n . y ) ; i f ( i 2 >l o c k ) { // move t h e s e r v o s // x t o z a x i s i f ( abs ( e v e n t . xbutton . xi 2 >o l d x ) <10) i 2 >b u f [5]+=( e v e n t . xbutton . xi 2 >o l d x ) / 1 ; i f ( i 2 >b u f [ 5 ] < MIN 5 ) i 2 >b u f [ 5 ] = MIN 5 ; i f ( i 2 >b u f [ 5 ] > MAX 5 ) i 2 >b u f [ 5 ] = MAX 5 ; i 2 >o l d x=e v e n t . xbutton . x ; // y t o x a x i s i f ( abs ( e v e n t . xbutton . yi 2 >o l d y ) <10) i 2 >b u f [6] =( e v e n t . xbutton . yi 2 >o l d y ) / 1 ; i f ( i 2 >b u f [ 6 ] < MIN 6 ) i 2 >b u f [ 6 ] = MIN 6 ; i f ( i 2 >b u f [ 6 ] > MAX 6 ) i 2 >b u f [ 6 ] = MAX 6 ; i 2 >o l d y=e v e n t . xbutton . y ; i f ( w r i t e ( i 2 >usb , i 2 >buf , SIZE )!= SIZE ) p r i n t f ( Could not m r i t e b y t e s t o hc08 \n ) ; r e a d ( i 2 >usb , i 2 >b u f , SIZE ) ; u s l e e p ( USB SLEEP TIME ) ; i 2 >s e a r c h =0; i 2 >show =0; } } ; break ; case KeyPress : { switch ( e v e n t . xkey . k e y c o d e ) { case 9 : // echap p l e s s e d . . . exit (0); break ; case 6 5 : // s p a c e i 2 >l o c k =! i 2 >l o c k ; i 2 >o l d x=e v e n t . xkey . x ; i 2 >o l d y=e v e n t . xkey . y ; i 2 >s e a r c h =0; break ; // d v o r a k k e y s : a z e r t y : q w e r t y // case 19: // 0 // i 2 > f i l t e r =0; // break ; case 1 0 : // 1 : & : 1 i 2 > e f i l t e r =1; i 2 >s t =20; break ; case 1 1 : // 2 : : 2 e i 2 > e f i l t e r =2; i 2 >s t =50; break ; case 2 4 : // : a q i 2 >view =0; break ; case 2 5 : // , : z : w i 2 >view =1; break ; case 2 6 : // . : e : e i 2 >view =2; break ; case 2 7 : // p : r : r i 2 >view =3; break ;

78

// a : q : q i 2 > f i l t e r g b =! i 2 > f i l t e r g b ; break ; case 3 9 : // o : s : s i 2 > f i l t e r g r =! i 2 > f i l t e r g r ; break ; case 5 2 : // ; : w : z i 2 >n o i s e f =! i 2 >n o i s e f ; break ; case 5 3 : // q : x : x i 2 >view =5; break ; case 4 3 : // d : h : h i 2 >view =4; break ; case 4 4 : // h : j : j f i l m=1f i l m ; break ; case 4 5 : // t : k : k scrot (); break ; case 4 7 : // s :m: i 2 >f l a g =! i 2 >f l a g ; break ; case 1 0 0 : // l e f t arrow i 2 >e d g e f a c t o r ; p r i n t f ( The new edge f a c t o r i s %i \n , i 2 >e d g e f a c t o r ) ; break ; case 1 0 2 : // r i g h t arrow i 2 >e d g e f a c t o r ++; p r i n t f ( The new edge f a c t o r i s %i \n , i 2 >e d g e f a c t o r ) ; break ; default : p r i n t f ( Key %i :% i p r e s s e d a t (% i ,% i ) \ n , e v e n t . xkey . s t a t e , e v e n t . xkey . keycode , e v e n t . xkey . x , e v e n t . xkey . y ) ; } } ; break ; case KeyRelease : { } ; break ; default : break ; } } } return 0 ; }

case 3 8 :

79

Building the project

Here is a description on how to build the board and reproduce the project. For the last time, the reader is reminded that the project is designed to work under GNU/Linux. It uses the fact everything under UNIX is a le, ever hardware, and accesses the webcam using the Video4linux library.

B.1

building the board

The schematic of the board is shown on Fig 2. There is no special trick about it: just put the clock close to the chip, and try to keep the two USB lines close together and away from any noise source. Resistor R2 is optional; if you remove it, you must update the rmware to activate the internal pull-up. The low half part of the schematic and switches 5 to 8 are part of the programmer; it could be possible to build a programmer board and a reduced board, but we do not think any body will need to do so.

B.2

testing the board

If any thing comes wrong, remember to double check all connections and wires; it happens too often that one forget to plug a wire, cut a strip, or put upside down an active component. Also remember that all testing must be done in monitor mode, ie with switch 4 close. To test the board, you need to install the monitor software. Download it from [11]. After extracting (tar -xvzf monitor-68HC08 1.02.tgz) Compile it with make && make install If you endure any trouble, refer to the README. monitor-68HC08 -z 3000 -c 908JB8 -v will launch the program; the default serial port is /dev/ttyS0. key will send the default security key. If the serial line and the Maxim232 are OK, the monitor must at least receive the rst echo byte. If it receives also the second echo, then the chip works ( quartz OK and so on). If the lot fails after the third byte transmitted, then there must be frequency problem: -z option should take as argument half of the frequency of the quartz. If you use a dierent quartz than 6MHz, then the board is unlikely to use the USB port. If the key command fails, rst try a bigger value for -l, 2.5 or 4 for example. You can not do any thing with the board until the key command succeeds. It does not matter if the monitor reports that the last byte was neither NULL or BREAK. We have this warning on several of our 4 boards, and it still work ne afterward. After the key command, you can try a basic command: read 0x04 The return value does not matter, as long as the monitor does not complain. If the key command succeeded, this is very unlikely to fail. check key is very important: it will report whether the security key unlocked the FLASH memory or not. If it says the key was wrong, then you can read the RAM, 80

but not the FLASH. You can still erase it with: flash erase FFFF m Do not forget to POWER RESET the chip after that; a simple push on the reset button is not enough. If you have to perform any reset, you must re-hit the key command, but you do not need to restart the monitor. The monitor can do everything in the same cession. After a ash erase and a reboot, check key MUST report a good key. If for any reason you need to send a key dierent than FF-FF-FF-FF-FF-FF-FF-FF, you must restart the monitor and use the option -k. Exit the monitor before proceeding. Note that if inside the monitor you bypass the security key, and exit it, re-launching it will let you chat with the chip at once: there is no need to re-authenticate.

B.3

Uploading the rmware

Download the sources of the rmware from [16]. Extract the archive, enter the directory, and power reset the chip. make program will do everything for you: compile the rmware using ASL, and upload it in the board. Note that the chip must be previously blank; to make sure it is blank, just do: make erase and power reset the chip.

B.4

Installing the driver

Open switch 4 of the board. When plugging the board ( in NON monitor mode), the syslog must report something like: Aug 25 21:11:27 pluton kernel: Manufacturer: DEMAINE Benoit-P Aug 25 21:11:27 pluton kernel: Product: USB Servo adaptor. with possibly dierent host name and date. Under MS-Windows, this board is partly compatible with the MCT driver. Syslog shall also report a line saying something like device not claimed by any active driver This means the chip works ne, the rmware was properly uploaded, the 27R resistors are in place, and the pull-up is OK ( either internal or external). If it does not work, but you could upload the rmware, double check those three resistors. Maybe the default setting of the rmware was to use internal pull-up; if so, remove R2. A message like device did not accept address means that either the rmware was not properly installed, or the USB link is faulty (highly probably bad pull-up resistor ( none or two), or bad impedance equilibration). Download the driver from [19], or the snapshot from [16]. Refer to the README s, compile and install the driver. You do not need to reboot the computer, but you need to be root, and also need the headers ( or full sources) of the running kernel. If you use the pre build kernel of your distribution, nd out the kernel-header package.

81

The procedure with our snapshot is : make cp 2.4/hc08.o /lib/module/2.4.25/misc depmod -a modprobe hc08 Adapt that for your own computer ( version of kernel and so on). Two new lines should appear in syslog: pluton kernel: hc08.c: USB hc08 driver now attached to hc08_0 pluton kernel: usb.c: hc08 driver claimed interface cea55ac0

B.5

The pivot

Once the board works ne, you must build the support of the webcam. It is composed of three servo-motors. We used the model called X-2, which was the second smallest one available at the model making shop. They are still far too powerful for a small webcam. If you use big servos, do not forget to put a large capacitor on the power line of the board. we put a 1mF capacitor; remember that the servo uses ( at least) 550mA continuous each, with square peaks !!! It looks like they do not require peaks all at the same time. Since our laptop can deliver 1.2A ( short circuit current), the lot nearly work ne. You may want to add some resistors and diode in series with the supply of the servo to avoid reseting the whole bus on each peak. Anyway, when we plug the board with the big capacitor, the whole bus get reseted. we can not plug the board on a non supplied USB hub. To make things work, we plug rst the board directly on the laptop, and then the small hub on the second port. Then we can plug the other devices ( mouse, webcam, key, disk ... ) on the hub. we advise to unmount any USB storage device before plugging the board7 . The servo shall be connected to pins A[5] to A[7]. In the order, we put on A[5] the Z axis, on A[6] the X axis, and on A[7] the Y axis. The camera is directly glued on the third servo. The two rst are glued together. Do not use glue containing diluent, otherwise it could damage the board, or the camera, or the servo them self, EVEN IF THE GLUE DO NOT TOUCH DIRECTLY ANY CHIP. An idea of the lot can be seen on g 4 p.19.

B.6

The image recognition

Download from [16] the sources of the patched xvcam. In the tree, just run make. This may require some extra development libraries. If you endure any trouble, rst read the header of xvcam.c; if you do not nd any answer to your problem, then grab the source from Enlightenment CVS ( see the directory Enlightenment/misc/camE/ ), and try to compile the original one. As long as the original xvcam does not compile, there is no hope to compile our version. Once xvcam from E.CVS works, keep in mind that our patched version is dierent. It shall compile properly, but may not run; it adds the following features:
7

A reset of the whole bus could cause harm to data in a memory key or disk if it is mounted

82

 a working and congured 68hc908 board ( see section Building the board) (Note that the device name is hard-coded in the source of xvcam.c - we put for me /dev/hc08 00 in the function init2() ... but you probably want to change that for the device created in /dev or /devfs depending on how the hc08 driver works for you) a compatible and congured USB webcam ( it works with ov511, but not with CPIA). By browsing GNU/Linux website you can easily nd out which cams are ov511 compatible. Note that it might work with non ov511 ... YOU MUST have a camera compatible with xvcam. First try the original xvcam. If it complains, then your camera is not compatible. The rst point is easy to correct: remember that the device name depends on the driver: just refer to syslog to know to which device le the board is attached, and modify by hand the source of xvcam ( line 341 in our version). On 2.4 kernel, we use static /dev. It worked ne. On 2.6, we now use udev. The name of the device le for the board is dierent. It is also possible to use devfs if you prefer, it is supported by the driver, but not maintained any more at the kernel level. Before running the software for the rst time, you must create the black noise archive. Just run ./xvcam -b. The software will show a progress bar for 8s. The console will show a short reminder of shortcuts. The key must be pressed when the view window is selected, not when the console is selected. The program only reads the scan-codes; keys are NOT translated into letters. The console shows bindings for the French ( fr ) layout. It diers from US and GB only for A,Z,W,Q. The advantage of scan-code is that whatever the layout is, the location of switches remains the same. The source of xvcam shows at the bottom the list of keys in the format dvorak:french:qwerty Using the french layout : press a to view normal view; z for edges, e for phase space. In edge space, have a look at color correction ( q and s); by default, it is o. Also try the black noise correction. Keep the settings which show fewest noise. To move the camera around, press space ( should be the same for all layouts ), and move VERY SLIGHTLY the mouse. Mouse move are interpreted very fast, but a bug in current driver forced me to slow down moves: a large move will take a lot of time. If an update of the USB drive is said to correct faults with low latency, then change the value of USB SLEEP TIME for a lower one. It is by default 100ms, but a better driver should support soon 10ms or less. Y axis is controlled by left/right clicks. Press space again to release the mouse. In edge space, identify an object with strong edges. Left click in the middle of it. Enjoy. Press space ( twice) to stop tracking. During tracking, you can view the Hough Space by pressing x. When clicking to select, a shot in recorded. Additional shots can be take, one at a time with k, or at every frame with j ( until j is pressed again).

B.7

FAQ

A FAQ is maintained on the HTML page of the project.

83