Multilevel Classification Scheme for AGV Perception

Naeem ur Rehman
Center of Research in Data Engineering (CORDE) Mohammad Ali Jinnah University Islamabad, Pakistan
naeem@corde.org.pk

Dr. Sohail Asghar

Center of Research in Data Engineering (CORDE) Mohammad Ali Jinnah University Islamabad, Pakistan
sohail.asghar@Jinnah.edu.pk

Shahzad Rafiq Irfan

Center of Research in Data Engineering (CORDE) Mohammad Ali Jinnah University Islamabad, Pakistan shahzad@jinnah.edu.pk

Simon Fong
Faculty of Science and Technology University of Macau Macau SAR ccfong@umac.mo

AbstractAn Autonomous Ground Vehicle (AGV) should be capable of self-navigating through various terrains based on priori data as well as self-configuring and optimizing its motion on the basis of sensed data. Research has been in progress in this domain to improve terrain perception for planning, execution, and control of desired motion of an AGV. There involve certain processes to achieve these goals. During the perception phase multiple classification techniques such as Bayesian Inference, K-Mean clustering, Artificial Neural Network and many others are used depending on underlying sensing technology for example LADAR and RGB Camera. This paper proposes a multilevel classification scheme for terrain identification and obstacle detection to improve selforganization according to the known terrain type. As a result the computation cost is reduced because of the use of multiple sensors. Keywords- Autonomous Ground Vehicle, Classification

I.

INTRODUCTION

A lot of research in robotics have been conducted in the past years with a fundamental objective of relinquishing human from performing certain roles such as piloting, driving, or operating a machine in hazardous environments. Many instances to name, those include but not limited to dismantling bombs, patrolling a vehicle in battle zones or critical missions that risk radioactive exposures. There has been a significant contribution of Artificial Intelligence (AI) in similar fields and Autonomous Ground Vehicle (AGV) is one of the hottest issues nowadays in AI research community. An AGV has the auto-ability of navigating along a path based on its real-time sensing abilities, decision- making, and intelligence applications. Research is in progress on ever improving the abilities of autonomous ground vehicle in all areas, generation after generation of its prototypes. Pattern classification is indeed one of the important phases in AGVs operation. There are certain pattern classification techniques proposed by researchers based on Artificial Neural Network [1], Bayesian classification, and many more applied during perception phase of autonomous ground vehicle system. High-level architecture of an AGV proposed by [2] has four components defined as:

Planning element acts as a repository for priori data like road, etc. 2. Control element performs closed-loop control in order to keep vehicle on a specified path. 3. Perception element performs the sensing task required to locate obstacles and to evaluate the smoothness of terrain. 4. Intelligence element determines the best path among the many. A concept named Smart Sensor unifies the formatting and distribution of perception data among the components that produce or consume it [2]. According to this concept each sensor processes its data independently of others. During perception a great deal of calculation is involved regarding pattern classification. Perception element should constitute reliable sensors for sensing the environment, building environment models and following the right path [3]. Also environmental features used for building models are RGB color space, range data, and texture and any of these sensors may be preferable depending on the environment [3]. Some researchers even contributed towards GPS position estimates but when it comes to urban environments a number of complications arise. In [4], the idea of vision based lane detection is proposed, which is then used to correct the vehicles position in a global reference frame. During vision based lane detection several image processing and transformation techniques are used to localize road lane demarcations [4]. Different researchers work on different aspects of perception [5, 3, 4, 3, 7, 8]. Humans easily perceive terrain conditions from visual images just as they easily recognize human faces [9]. To cater for the issue of pattern classification of terrain conditions from imaging, we propose a multilevel classification approach to cover the limitations of current classification techniques. The rest of the paper is organized as follows. Sections 2 takes a literature review regarding research contributions in the domain of autonomous ground vehicle. Section 3 explains the proposed model i.e. Dynamic Classification. Section 4 discusses over our experiments regarding Dynamic Classification and shows that it worked better than Bayesian etc. Finally section 5 concludes. 1.

II.

LITERATURE REVIEW

The primary objective of this section is to review the literature on existing techniques and methodologies for data classification that is used during the perception phase of autonomous systems working. Lorigo [10] considers the problem of developing a mobile robot to avoid obstacles while traveling in an unstructured environment. There is no strong prior knowledge of the appearance of the ground or location or even appearance of the obstacles. Lorigo in [10] suggests an independent module based on Brightness gradients, RGB and HSV (Hue, Saturation and Value) and by using ground sensory input with stereo-vision and its variations, monocular vision, and the combination of vision with other sensor (s). The pattern classification proposed by [10] uses brightness, gradient magnitude normalized RGB and HSV color. Sofman [5] presented techniques to achieve accurate terrain classification by utilizing high-density, colorized, three-dimensional laser data. The approach for classifying the image is RGB to HSV. According to their approach for image classification, firstly the sensor data was discretized into grid cells such that each cell holds data values, which correspond to the average values of all data points. Secondly, reflectance approach was introduced which converts RGB data into HSV. In this approach the laser signal reflected from the ground to the helicopter's sensors, which indicates the relative orientation to the incident angle of the surface materials. Jean-Frabcois [6] examines the problem of natural terrain classification based on 3-D point cloud statistics used to compute saliently features that capture the spatial distribution of the points in a local neighborhood. Proposed technique in [6] composed of three main steps: i) classification ii) segmentation and iii) semantic interpretation. Point wise classification is based in local point cloud geometric analysis. The 3-D point cloud produced incrementally as the robot traverses the environment, which is segmented into three classes: surface, linear structures and porous volumes. Those three correspond to three low-level geometric primitives that can capture the variability of natural environments. Velat [4] addresses the problem created by fluctuations in GPS position estimation and suggests a Vision Based position correction for urban environment (Developed in 2007 DARPA urban challenge) using Kalman Filter Estimation. This is an image processing technique in which geometrical shapes and arbitrary shapes are identified from the image. Then most dominant lines are extracted from this image subsequently. A 2D virtual plan is designed that depicts the area around the vehicle, on its bases the estimation of current position of vehicle is performed along with the vehicles current lane its boundaries and its relative position is also estimated. Then this information with prior information is used to calculate the error in position. This error is used to map the road network with corrected position, which can be used to map additional sensors. Their classification involves two types of roads i.e. Painted lanes

and Unpainted lanes. The whole process is divided into four parts. That is edge filtering, color segmentation, Hough transformation, and global map generation. Crane [3] worked on the development of integrated sensor system for obstacle detection and terrain evaluation by suggesting a group of sensors. There are two groups of sensors i.e. Obstacle detection sensors involving one moving rotating 3D Ladar, and one stereo vision system and Terrain sensors involving fixed stick Ladar, monocular camera vision processing system. The underlying classification involves 3D spinning ladar, stereo-vision, and stationary ladar. 3D Spinning ladar is a laser range finder sensor generates a 3D image that is 3600 by 1800 sensor is mounted on front facing of vehicle. It scans a hemi-sphere region. Initial image is 3D point cloud image that is then orthogonally projected to 2D space image. Stereo-vision uses two fixed three dimensional auto iris cameras that give images just like human perception, both cameras are horizontally aligned. Both cameras take pictures at exactly the same time and they give images of same location with different viewpoints; later these pictures are matched by pixels there is no vertical distance between pixels but only horizontal distance is presented. Soto [7] examines the problem of Dynamic and Distributed Reconnaissance using all terrain Unmanned ground vehicles (UGVs). The visual perception system consists of 5 color cameras: a panning stereo pair located at the front center, and three pan/tilt color cameras at rear. Although the stereo pair is mainly used for obstacle avoidance and mapping, and the pan/tilt cameras for surveillance, the architecture is totally flexible. A task located on any of the processors can request access to any of the cameras or pan/tilt systems. This is achieved by using a custom-built connection box. The color segmentation technique in [7] is based on a HSI model. The main advantage of HSI over RGB-based techniques is that HSI allows a nice decoupling of the color and intensity information. While the Hue component conveys the color information, the intensity component contains information of brightness, and the saturation indicates how pure the color is. During stereo correlation two pre-processor techniques are used for the stereo correlation. First the images are rectified in order to align the epipolar lines with the horizontal scan lines of the video cameras. Second, the images are convolved with a Laplacian of a Gaussian (LoG) filter. This enhances the texture of the image. Crane [8] discusses Team Gator Nations Autonomous Vehicle Development. They classified sensors with respect to their functions. Long-range obstacle detection sensors collect the information up to the range of 300m, at the rate of 15Hz and, resolution at a meter distance. Sensors are vertically offset to allow different perceptions. It also prevents flashing of sensors by each other. The data from long-range sensors are also used for medium and short-range calculation that is aided by 6 other sensors. Two of them are mounted in planar fashion located on front and rare bumpers allow being pitched 900 to 200 above the plane of vehicle. This accomplishes both enhanced terrain classification and maintains parallelism with road surface. Two others are

mounted on overhanging wings; these sensors are configured as vertical fan orientation that plays a major role in active measurement of environment surrounding. This information is consumed by many computer-nodes known as smart sensors and they generate high-level information about environment, traversability obstacle detection, and vehicle localization. Smart Sensors unify data of each sensor and output is used for generation, analysis, and transfer of sensor information. Broten [9] tries to resolve the problem raised by different environmental changes. This is done by classifying different types of terrains and to build an adaptive learning based system that learn from testing samples and become able to classify different types of terrains for example mud, vegetation and snow. This approach is divided into two parts. First part is about gathering images and dividing images through Eigen images technique and creating data and samples from it. Then in second portion the data is used for learning purposes and test samples are used for testing the performance. The source images are collected and each image is taken in a mathematical representation of vector. Its dimensionality is reduced in such a way that retains as much variation as possible. For this purpose principle component analysis is applied on images. Later these principle components are arranged in such a way that first few have maximum variation from the original values. It does not deal with different light situations and also pay huge computation cost in shape of adaptive learning, which in case of an AGV is not useful and creates speed limitation. In our proposed work a new multilevel classification scheme is introduced using a combination of underlying classification techniques for various terrains in published research. The proposed scheme focuses on image based data classification only as we believe that more sensors involve higher computational complexity which is not an efficient approach for high speed moving vehicle to keep system live in real operational environment; it should be light weight in term of sensor inputs. Our scheme will work on two aspect of an image, i.e. firstly, identification of terrain type, and secondly diction of Obstacle for better planning of motion for AGV. Techniques and methods discussed in the above works are designed for specific environment with specific attributes. However, our proposed method can be applied dynamically by choosing a classification as per given type of terrain. III. PROPOSED APPROACH In real world there is a great deal of variations in terrain types, backgrounds and locations. Each terrain comes with its own constraints such as environment, sight, ground situation and light. Further, environmental factors like cloud, light, etc also contribute to constructing the view of a terrain. This variation sometimes causes a classification approach to fail unless the system is able to identify a particular terrain dynamically. Humans make inferences about terrain conditions using simple 2D images [9]. They have learnt to recognize terrains types by visual inspection [9]. For an AGV it becomes important to learn how to identify a

particular terrain when its image is provided (using neural network) just like humans as they identify human faces [9]. The capability to identify and categorize terrain conditions is an advantageous skill for an AGV since it supplies additional information to the path planning process [9]. That is why terrain identification plays an important part in Multilevel Classification Scheme proposed. There are terrains like snowy, muddy, urban, and many more. For each terrain type mentioned above, we require a different classification model for the execution and intelligence phase to take go/un-go decision. It is intuitive if an AGV has the option of traversing trafficable terrain that is the least rough then it should be possible to maximize velocity of traversal [9]. The challenge is of terrain identification at runtime. Multilevel Classification Scheme uses Neural Network to identify a particular terrain and then it applies its respective classification technique.

Figure 1. Multilevel Classification Scheme (Proposed Approach).

This scheme goes through three main phases, data perception, pre-processing, and classification. Data perception phases output becomes the input for the preprocessing phase and pre-processing phases output becomes the input for the classification phase. In Data Perception, a sensing device (usually camera) acquires multiple images of what it sees up-front and passes some of them to pre-processor using selection based on mean, however other mechanism can be used.

Pre-processing phase starts with segmentation of the selected images into horizon and terrain. It takes the terrain image and corrects it, thins it, and centers it according to a particular threshold so that classifier identifies noise and terrain type easily. Selection passes this corrected terrain image to a neural network, already trained at identifying different types of terrain images. Based on image attributes our proposed classifier suggests us a terrain type. Depending on the terrain type a particular classification model is applied. Advantage of using multilevel classification approach is that the Planning Element always gets the same simplified image no matter what the terrain type is, because multilevel classification has applied respective classification method to transform it into the same simplified image. Multilevel classification can classify every image and can transform it into a simplified image if our neural network module is trained at recognizing that image type and we have a respective classification method to deal with it. This advantage makes the Dynamic Classification method extensible. The algorithm for this task is shown as follow: Step 1. Capture image frames using image perception device Step 2. Select a frame by applying selection method (like mean, average or any other) Step 3. Segment the selected frame into horizon and terrain and exclude horizon part Step 4. Adjust terrain frame to center Step 5. Adjust thinness of the image so that the terrain can be clearly identified Step 6. Check the classifiers mode. If it is in Training Mode go to step 10, else go to step 7 Step 7. Pass the terrain image to classifier to know terrain type Step 8. Depending on the terrain type suggested by classifier apply respective classification method to get a classified image Step 9. Pass the classified image to planning module of the AGV and go to step1 Step 10. Train the classifier according to the given terrain type and data available then go to step 1 IV. EXPERIMENTS AND RESULTS

such as urban, rural, snowy, muddy, and more in order to identify a terrain type. The images that we provided to classify during experiments were of rural and urban terrain types. Our classification module did identify the images as rural and urban type as expected. Classification was performed by applying the same formula for the two types of terrain and calculated the threshold value for each classification model. Formula used for this purpose is the following: 0.600 * Red + .333 * Green + .333 * Blue (1)

The images before and after the application of above formula are given in Figure 2 (left) and Figure 2 (right) respectively. Before After

Figure 2. Results of Classification on Rural Terrain.

In the proposed scheme we considered the rural image where road color was muddy as shown in Figure 2. Each pixel of the image was examined and converted according to the following rule: If the value of the pixel was above 127 (RGB) then it was converted to 255 (White). If the value of the pixel was below 127 (RGB) then it was converted to 0 (Black) Before After

Experiments were performed on different types of terrain each with different environmental attributes. We used camera as a sensing device, which was capable of capturing images at the speed of 10 frames/second. We labeled these frames as F1, F2, F10. Out of these frames some of them were selected by taking their mean. These selected images were taken and segmented into horizon and terrain. The horizon parts of the images were ignored (as we didnt need the horizon parts) and terrain parts of the images were selected for correction, thinning, and centering according to a particular threshold. These methods were applied to avoid characteristics of the images resulting in noise. These images were then passed to the classifier for classification. The classifier identifies a particular type of terrain. For this purpose we kept on providing the classifier different terrains

Figure 3. Results of Classification on Asphalt Concrete Road (Urban Terrain)

Next classification approach was with urban terrain type where we had black colored roads as shown in Figure 3. For that case each pixel of the image was examined and converted according to the following rule: If the value of the pixel was below 127 (RGB) it was converted to 255(White). If the value of the pixel was above 127(RGB) it was converted to 0 (black)

Its evident from the results that in both type of terrain we got the classified images after applying our proposed scheme. In both of the resulted images we obtained the same color for roads, which ensures the dynamicity for proposed scheme. V. CONCLUSION

Classification of terrain type during perception phase of autonomous system working is an important sub task. There are a number of techniques and methods available for this purpose. The problem with these available techniques and methods is that they can only be used in a particular environment. We focused on a multilevel classification technique that regardless of the terrain type it is exposed to, it provides accurate results by applying a respective classification method. These experiments investigate the use of multilevel classification to identify dynamically the terrain in which vehicle is moving around. The images are then used for classifying terrains. The addition of color segmentation was found to be only marginally useful than grayscale, and it did not improve the classification rates further from the acceptable levels. Further investigations were performed with pairs of terrain types and it was found that the dynamic classification could differentiate between most pairs of terrain types. Still some limitations were found during experiments while classification in abnormal color situations like break lights of cars in dark etc. Further work is under process for improving the speed of classification to get better results while moving in fast scenarios like moving up to 50kms per hour. Multi-level classification technique proposed here currently addresses clear images of plain roads. Fuzzy and noisy images of roads will be considered in future research. ACKNOWLEDGMENT Thanks to Inam ul Haq, Arman Bari, Badr u Zaman and Muhammad Ali for the kind guidance in understanding the research problem and in writing this research article. Thanks to Miss. Sonia Hameed for her help in reviewing and improving the presentation of this paper. REFERENCES
[1] Anthony Stentz, Alonzo Kelly, Peter Rander, Herman Herman, Omead Amidi: Real Time, Multi Perspective Perception for Unmanned Ground Vehicles, National Robotics Engineering Consortim, Robotics Institute, Carnegie Mellon University. Proc. of AUVSI Unmanned Systems Symposium 2003, July, 2003 [2] Carl D. Crane III, David G. Armstrong II, Robert Touchton, Tom Calluzzo, Sanjay Solanki, Jaesan Lee, Daniel Kent, Maryum Ahmed, Roberto Montane, Shannon Ridgeway, Steve Velat, Greg Garcia, Micheal Griffs, Sarah Gray, John Washburn, Gregory Routson: "Team CIMAR's NaviGator: An Unmanned Ground Vehicle for Application to the 2005 DARPA Grand Challenge", University of Florida, Center of Intelligent Machines and Robotics, Gainesville, Florida, USAJ. U. Duncombe, Infrared navigationPart I: An assessment of feasibility (Periodical style), IEEE Trans. Electron Devices, vol. ED-11, pp. 3439, Jan. 1959

[3] Carl D.Crane, David G.Armstrong, Maryam Ahmed, Sanjay Solanki, Donald MacArthur, Erica Zawodny, Sarah Gray, Thomas Petroff, Mike Grifis, Carl Evans: Development of and Integrated Sensor System for Obstacle Detection and Terrain Evaluation for Application to Unmanned Ground Vehicles, University of Florida, Center of Intelligent Machines and Robotics, Gainesville, Florida, USA E. H. Miller, A note on reflector arrays (Periodical styleAccepted for publication), 2005 [4] Steven J, Velat, Jaesang Lee, Nicholas Johnson, Carl D. Crane: Vision based vehicle localization for autonomous navigation. in Robotics and Automation, 2007. CIRA, 2007 [5] Boris Sofman, J.Andrew Bagnell, Anthony Stentz, Nicolas Vandapel: Terrain Classification from Aerial data to support Ground Vehicle Navigation, Robotics Institute Carnegie Mellon University, USA. 2005. [6] Jean-Francois Lalonde, Nicolas Vandapel, Daniel F. Huber, Martial Herbert: Natural Terrain Classification using Three-Dimensional Ladar Data for Ground Robot Mobility, Carnegie Mellon University, USA. Journal of Field, 2006 [7] Alvaro Soto, Mahesh Saptharishi, Ashitey Trebi Ollennu, John Dolan, Pardeep Khosla: CYBER-ATVs: Dynamic and distributed reconnaissance and surveillance using all terrain UGVs .IEEE Transactions on Robotics and Automation, volume 18, 2002 [8] Carl Crane, David Armstrong, Antonio Arroyo, Antoin Baker, Doug Dankel, Greg Garcia, Nicholas Johnson, Jaesang Lee, Shannon Ridgeway, Eric Schwartz, Eric Thorn, Steve Velat, Ji Hyun Yoon: Team Gator Nations Autonomous Vehicle Development for the 2007 DARPA Urben Challange, University of Florida, Gainesville, USA. 2007 [9] G.S. Broten and B.L. Digney: A learning system approach for terrain perception using Eigen images, Defense R&D, Canada. 2008. [10] Liana M. Lorigo, Rodney A. Brooks, W. E. L. Grimson. VisuallyGuided Obstacle Avoidance in Unstructured Environment. Proceedings of IROS, 1997