Report on Vision-Based Robot Control

Submitted to Dr. Ray Gosine Term 8 Project Supervisor C-CORE

April 10, 2001

TABLE OF CONTENTS LIST OF FIGURES LIST OF TABLES ABSTRACT I. II. INTRODUCTION DESCRIPTION OF COMPONENTS Vision System Hardware Software Mobility System Arm System Intelligence System State Machine Scavenger State Retrieval State Delivery State III. IV. OPERATING CYCLE OF THE SYSTEM CONCLUSION iii iii iv 1 1 4 4 6 7 11 13 13 14 14 15 15 16 17 18

LIST OF REFERENCES APPENDIX

LIST OF FIGURES

Figure 1. Sambuca Trash Collecting Robot 2. Overall System Block Diagram 3. Hardware Realization in Software 4. Camera Distance Calibration 5. Pronto R2 Electric Wheelchair Base 6. Robot Maneuvering via Drive Wheel Rotations 7. Analog Voltage Control of Drive Motors 8. Communication between PC and Mobility System 9. Different methods of grasping a can 10. State Diagram of Intelligence System

Page 2 3 4 6 8 9 10 11 12 14

LIST OF TABLES 1. Communication Protocol for Sambuca Motor Control 11

iii

ABSTRACT The Vision-Based Robot Control system is a robot control system developed by Puffin Software Solutions that uses vision as its primary sensor. Can and bottle collection in a recycling plant is used as an environment to test the system. The robot looks around its environment for cans and bottles. When it finds either, it maneuvers close to it, picks it up with an attached arm, and deposits it into an appropriate bin. The robot is comprised of four components: (1) a vision system, (2) a drive system, (3) an arm system, and (4) an intelligence system. The first three components are utilized by the fourth component to achieve its goal, namely, picking up cans and bottles.

iv

Report on VISION-BASED ROBOT CONTROL I. INTRODUCTION

Many recycling plants employ people to separate bottles and cans. The process consists of putting all the cans in a can-bin and all the bottles in a bottle-bin. It is proposed that throughput of the plant can be improved greatly if robots did this long tedious task. This would give the humans more valuable time to work at other things in the plant. For this reason, a Vision-Based Robot Control system has been developed by Puffin Software Solutions. This robot system has the ability to search for cans and bottles, collect them using its RT200 Robotic Arm and deposit them into the appropriate bin. While this is being executed, plant personnel can be doing other, less tedious, things.

II.

DESCRIPTION OF COMPONENTS

The main components of the system are the vision system, the arm system, the mobility system and the intelligence system. The physical implementation of the system, is named Sambuca, and is shown in Figure 1. The mobility system is poared by two 24 volt

Figure 1. Sambuca Trash Collecting Robot batteries, whereas the arm system and all other electronic devices used are poared using an extension cord into a 120 volt outlet. The intelligence system uses the other systems to gather information, plan and execute its next move. The vision system provides information about Sambucas environment (ie if there are any cans/bottles around it). Based on this information, and Sambucas previous state (ie was it looking for a can/bottle? Did it already find a can/bottle and was trying to pick it up? Does it have a can/bottle in its grasp?), the next move is planned. The next move may involve the arm and/or the mobility system. The overall block diagram of the relationship between each of the systems is shown in Figure 2.

Figure 2. Overall System Block Diagram The great thing about controlling a physical system made up of different components in software is that Object-Oriented Programming (OOP) is ideally geared towards this. This makes the software easier to understand and to write. In software, Sambuca readily maps to OOP concepts. Sambuca has three physical objects: an arm, a camera, and a mobile base. The software therefore should have three objects: an arm object, a vision system object, a mobility system object. All of the physical objects are coordinated via the intelligence system. The OOP implementation of Sambuca is shown in Figure 3.

Figure 3. Hardware Realization in Software Vision System The Vision System consists of a video camera, and software to grab an image from the camera and process it. Hardware There is much debate about the choice of video camera for this project. Available are expensive video camera systems costing upwards of a thousand dollars, which offered very clear high-resolution images and is very fast. The main problem with these is that they are so expensive and this project didnt really need all the fancy options that they offered. Also, special hardware restrictions are placed on the setup if these are used. Namely, a desktop PC would have to be used instead of a laptop, because the camera system required access to PCI slots. The philosophy is that a can at 640x480 is still a can at 320x240. Plus, the system being developed didnt need a high frame rate. So, the focus turned to webcams. A quick 4

browse at Futureshop revealed very good quality cameras with frame-rates comparable to the expensive systems. The webcams use USB ports, so they could be used on a laptop or a desktop PC. In general, most vision-systems require lowpass filtering of images before they use them, so it didnt matter that the resolution is less and the quality of the image is less since it all got smoothed out anyway (blurred). In the end, a $200 camera from 3com was chosen. The camera is very small, so it could be mounted anywhere, and the only attachment to the camera is a cord to the USB port. Another great thing about the webcam is that all that has to be done is plug it in and it works. Windows2000 automatically installed the drivers for it! This is a great advantage to the hour or more installation time of an expensive camera system. Positioning of the camera is also of high importance, since this is the main sensor of the robot and we want to get as much information from it as possible. The idea is to position the camera such that it would see cans/bottles as easy as possible, but see as little as possible, the background area of the environment. Also, from an exact position of the camera, distances could be calculated. The camera has a horizontal field of view of 24 degrees from center, and a vertical field of view of 17 degrees from center. The camera is placed 20 degrees with respect to horizontal, at a distance of 15 inches from the floor. This gave a back-to-front viewing distance from 20 inches in front of the robot to 50 inches in front of the robot, although objects could only be reliably detected up to 40 inches, since they appear too small at distances greater than that, and are considered noise by the vision system. See Figure 4. Left-to-right distances arent important, since the robot is made to align itself directly facing the object. Back-to-front distances are important since the robot needs to know how far to move the arm to pick up a can/bottle.

Figure 4. Camera Distance Calibration. Software Instead of developing low-level image processing algorithms, it was decided to use readily available resources. These resources include: Matrox Imaging Library (MIL) and OpenCV. Both are freely available. MIL is obtained from a demo disk, and OpenCV is available on the internet. OpenCV is used to grab images from the webcam. It has a lot of other image processing functions built into it, but it takes a while to learn how to use them. Familiarity with MIL was much greater than with OpenCV, so it was used for all the image processing after an image is grabbed.

To make the webcam easy to implement in software and for future uses, it was decided to encapsulate the webcam in a class, and use object-oriented programming for development. Then to use the webcam, all that one has to do is to create an instance of it. Everything that had to do with the vision system (ie image grabbing, image processing), is grouped into a class which is called FrameGrabber. Once the software for this class is written, all other software that used it would be greatly simplified. For example, to create a vision system, all that has to be done is to create an object of type FrameGrabber: FrameGrabber visionSystem; Now one could grab an image using: visionSystem.grabImage(); and create a binary image from the original using: visionSystem.binarizeImage(); Mobility System The Mobility System consisted of a Pronto R2 Electric Wheelchair Base, a MAD card, and communication software. The Pronto R2 Electric Wheelchair Base is a generic electric wheelchair base without the chair. See Figure 5.

Figure 5. Pronto R2 Electric Wheelchair Base. The base is controlled via analog voltage output to two drive motors. The base has four wheels: two are used for driving (Drive Wheels) and two are free to spin and rotate (Free Wheels - just like the front wheels on a shopping cart). The robot turns by making the drive wheels rotate in opposite directions. See Figure 6.

Figure 6. Robot Maneuvering via Drive Wheel Rotations. Arrows represent Drive Wheel rotation direction. To Get the Drive Wheels to move with variable speed, an analog voltage between zero and five volts has to be sent to the appropriate motor. Two and a half volts represents zero wheel rotation. To get the wheel to rotate forwards, the voltage to the motor has to be higher than 2.5 volts (ie between 2.5 and 5 volts), and to get the wheel to rotate backwards, the voltage to the motor has to be lower than 2.5 volts (ie between 2.5 and 0 volts). See Figure 7.

Figure 7. Analog Voltage Control of Drive Motors. The average pc/laptop doesnt output analog voltages, so a DAC is needed to convert a digital signal from the pc to an analog signal that the mobile base can use. See Figure 8. The immediate solution (proposed and implemented by Jagan Sheshadri) is to use the MAD card. The MAD card is developed primarily by Lloyd Smith. The MAD card (MUNs A to D card) is a little more complex than a simple DAC. The MAD card has an on-board microcontroller that can communicate with a pc via an RS232C serial port. A simple communication protocol is developed by Jagan, and is shown in Table 1. For example, to move the robot forward at half speed (127/2 = 64), the PC would send f@ to the MAD card. The ascii symbol for ascii value 64 is @. The MAD card would then interpret this and output 3.75 volts to both Drive Motors. This would cause the robot to move forward. Another nice feature of the MAD card is that it will set the voltage of the Drive Motors to 2.5 (ie stop), 250 milli-seconds after every command is received from the pc. This means that to keep the robot moving, the pc has to continually send it commands. In the event of a computer crash the robot will just stop.

10

Figure 8. Communication between PC and Mobility System. Table 1. Communication Protocol for Sambuca Motor Control Command from PC: b, <ASCII 0-127> f, <ASCII 0-127> l, <ASCII 0-127> r, <ASCII 0-127> s, <ASCII 0-127> x, no argument necessary Description: Moves robot backwards Moves robot forwards Turns robot left Turns robot right Set maximum allowable speed Stop

The software that is used to talk to the MAD card through the serial port is simply Windows2000 API software (provided with Microsoft Visual C++ 6.0) encapsulated in a class to make it easier to use. Arm System The Arm System consists of the RT200 Robotic Arm developed by Oxford Intelligent Machines, and communication software.

11

The RT200 Robotic Arm is an industrial robot arm. It has a two-finger gripper attached to it, which enables it to grasp objects. This is used to pick-up cans and bottles. Originally, a shovel is going to be used as an extension to the arm to scoop up the cans/bottles, as the arm is too short to grasp the cans/bottles side-on. This is found to be very unreliable as about 90% of the time it would just knock the can/bottle over. In the end, the shovel is removed and cans/bottles are grasped, from above, by the arm. This has its disadvantages as well, since the arm had to be more precise to grasp a can/bottle. See Figure 9.

Figure 9. Different methods of grasping a can. The arm communicated with the pc via a RS232C serial port. Programming the arm is very difficult, primarily because the arm works on a command-acknowledge basis. By this, I mean that the pc sends the arm a command (ie move to position), and the pc has to

12

wait for the arm to acknowledge that it has completed that command (ie arm is at position). Intelligence System The Intelligence System is software running on a PC. This is one of the more challenging parts of the robot to design, due to the complexity of the task. Many questions had to be ansared to solve this problem. What to do if it sees an object? How far away is the object? Is the object on the left or right side of the robot? What if the robot sees multiple objects? How will the robot get to the object? To reduce the complexity of the Intelligence System, the robots task is decomposed into three steps and implemented as a state machine. State Machine To implement the robots Intelligence System as a State Machine, three states are needed. They are (1) Scavenger State, (2) Retrieval State, and (3) Delivery State. See Figure 10.

13

Figure 10. State Diagram of Intelligence System. Scavenger State As soon as the robot is started it enters the Scavenger State. In this state, the robot scavenges around looking for cans and bottles. Once it detects a can or bottle (DETECT), the robot then moves into State Two. Retrieval State Now that a can or bottle has been detected, the robot must move close enough to it so that the arm can reach it. This is done in the Retrieval State. For example, if a can is spotted to the left of the robot, then the robot will turn left and then drive towards it, until it gets close enough to deploy the arm (GRAB). Then the third state is entered.

14

Delivery State Now that the robot is close enough to the can/bottle that the arm can reach it, the Delivery State is reached. Here the Intelligence System calculates are the can/bottle is and sends the arm to that point, grabs the can/bottle and dumps it in an appropriate bin. This signals RELEASE and the robot goes into the Scavenger State again searching for cans/bottles. III. OPERATING CYCLE OF THE SYSTEM

The system is driven by a timer (5 times per second). The system initially starts in the Scavenger State. The following is executed when the timer is tripped. 1. Grab an image from Camera 2. Do image processing on the image to identify objects 3. If the system is in State: a) Scavenger State Move around until a can or bottle is identified b) Retrieval State Go to identified object c) Delivery State Grasp identified object with Arm and deposit into bin The interesting state here is the Delivery State. When the robot enters the Delivery State, it disables the timer, sends a command to the arm, waits for the arm to acknowledge that it completed the command, sets the State to Scavenger, and then re-enables the timer.

15

IV.

CONCLUSION

The robot worked very well considering that it only took 4 months to build (while taking 4 other courses + working). At the demo, it looked for cans and bottles, identified them and tried to pick them up. Much of the time, however, the robot didnt succeed in picking up the objects. This is due to two problems: (1) the arm control is open-loop, and (2) the arm sticks. What I mean by open-loop this is that, the arm is told to grab the object and deposit it in a bin without any feedback adjusting where it thought that the object is. But implementing this is a project in itself. Also, the arm sticking was an unfortunate, but unforeseeable event. The author believes that if the original plan of approaching an object side-on could have been implemented, then the open-loop arm control would have worked much better. Having touch sensors on the gripper would have made the system more robust. It isnt absolutely necessary to the success of the project, but it would be something that is suggested to be looked into for the next generation of the product.

16

LIST OF REFERENCES 1) Jain, A. K., Fundamentals of Digital Image Processing, Prentice-Hall, New Jersey, 1989 2) Shapiro, L. G., and Haralick, R. M., Computer and Robot Vision, Vol II, Addison-Wesley, New York, 1993 3) Sinha, N. K., Gupta, M. M., Intelligent Control Systems: Theory and Applications, IEEE Press, New York, 1996 4) Universal Machine Intelligence Limited, Inside RTX, Baddeley Associates Limited, Cambridge, 1986 5) 6) Nebel, B., Advances in Artificial Intelligence, Springer, New York, 1994 Veloso, M., Pagello, E., RoboCup-99: Robot Soccer World Cup III, Springer, New York, 2000

17

APPENDIX

18