A Semantic Approach for the Inspection of Complex Environments with Autonomous Service Robots

M. ZIEGENMEYER, K. UHL, S. SAYLER, J. M. ZLLNER and R. DILLMANN FZI Forschungszentrum Informatik, Interactive Diagnosis and Service Systems, Haid-und-Neu-Str. 10-14, D-76131 Karlsruhe, Germany, {ziegenmeyer, uhl, sayler, zoellner, dillmann}@fzi.de, www.fzi.de
Abstract. The inspection of complex environments is a challenging task for autonomous service robots. An autonomous inspection robot should actively examine entities of interest (EOIs), e.g. defects, and should take additional inspection actions if the data analysis results are uncertain. The selection of these actions should be driven by the assessment of the individual circumstances. In this paper a semantic approach for inspection planning, assessment of the data analysis results, decision making and replanning is investigated. For the experimental evaluation of this approach the detection and classication of waste on irregular terrains with the hexapod walking machine L AURON is chosen. First preliminary simulation results are presented. Keywords. autonomous inspection, inspection ontology, inspection planning, decision making

1. Introduction The inspection of complex environments like sewers, pipelines, power transmission lines or dams is a challenging task for autonomous service robots. Recently, there has been a lot of research in this area. The approaches can be roughly categorized into two categories. First, the hardware design and the control of the inspection robot itself are considered, e.g. Birkenhofer et al. [1], Nassiraei et al. [9]. Second, appropriate sensor systems, their automatic placement and the corresponding data analysis components are examined, e.g. Duran et al. [2], Pedersen et al. [11]. However, there exist only few integrated approaches which aim at fully autonomous inspection systems. Streich and Adria [12] describe a hierarchical software approach for the sewer inspection robot MAKRO. It consists of four levels of abstraction: low-level control, base machine, action controller and planner. For the planning level the FF planer, a forward chaining heuristic state space planner, is used. Zhou et al. [16] present a three layer robot control system for an inspection robot for power transmission lines. They distinguish actuation layer, planning layer and supervision & management layer. For the planning layer, which is located in the inspection robot, and the supervision & management layer, which is located in the ground station, a distributed expert system based on CLIPS is proposed.

Palomeras et al. [10] introduce a mission control system for dam inspection with an AUV. The control system consists of the vehicle level, the task level and the mission level. For the mission level Petri Nets are employed. In all these approaches very little is reported on the on-line assessment of the inspection data analysis results as a basis for intelligent decision making and inspection planning. Nevertheless, this is a key aspect for generating truly intelligent system behavior. An autonomous inspection robot should actively examine entities of interest (EOIs), e.g. defects. If the data analysis results are uncertain, additional inspection actions, e.g. activating a special sensor, approaching the EOI from a different perspective or employing a different data analysis algorithm, should be taken to increase the condence of the results. The selection of these actions should be driven by the assessment of the individual circumstances. Human experts usually resort to experience and expert knowledge to solve this task. In this paper we investigate a semantic approach for inspection planning, assessment of the data analysis results given the individual inspection situation, decision making and replanning. The main idea is to incorporate human expert knowledge via a semantic inspection model to ll the above mentioned gap. Today, there exist only few semantic approaches regarding inspection tasks. Most of them are targeted at semantic image interpretation [5], cognitive vision [8] and methods to develop computer vision applications [7]. Nevertheless, a semantic inspection approach offers several advantages. On the one hand, easy system extensibility and maintenance is achieved by the explicit separation of knowledge representation and execution control. On the other hand, the human comprehension of the system decisions is improved signicantly. Moreover, the usability of the system is increased by allowing the user to communicate with the system on a semantic level. The paper is organized as follows. After the introduction, section 2 describes the proposed semantic inspection approach. The realization of the proposed approach is outlined in section 3. First preliminary simulation results aimed at validating the proposed semantic inspection approach are presented in section 4. Finally, section 5 concludes the paper.

2. The Semantic Inspection Approach The semantic inspection approach comprises a mission control architecture which has been described in a preceding paper [14]. At the core of this mission control architecture a knowledge base containing all knowledge relevant to the execution of inspection missions with autonomous service robots is located. It consists of several ontologies [4] which model the concepts and contexts required for the semantic inspection control in a general form (terminological box, T-Box), and concrete instances of concepts and relations which represent the current state of the world (assertional box, A-Box). The terminological box (T-Box) of the knowledge base is organized in three abstraction layers (cf. Fig. 1): a basic ontology, a core ontology and a specic domain ontology. The basic ontology contains fundamental concepts like parameter, timestamp, condition, function, and data type. The core ontology includes robot, environment, inspection, navigation and mission subontologies modeling the central concepts and relations of the particular elds. The domain ontology contains application specic subontologies, e.g. model-

Figure 1. The structure of the knowledge base.

ing the hexapod walking machine L AURON IVc, irregular terrains and the detection and classication of waste. The design of the knowledge base followed a middle out approach [15]. At rst, the core concepts of the particular subelds were identied (core ontology). Based on these, more specic (domain ontology) and more general concepts (basic ontology) were developed. For this paper, the description of the knowledge base concentrates on the mission and inspection subontologies. Then the inspection planning and the decision making processes are described. A detailed treatment of the other parts of the knowledge base and the corresponding reasoning mechanisms shall be left for a future paper. 2.1. The Mission Subontology The core concept of the mission subontology is the plan (cf. Fig. 2). The structure of plans is modeled after so-called exible programs [6]. A plan is represented as a tree of nodes, which is executed by a depth-rst strategy. There are three types of nodes: branching nodes, action nodes and planning nodes. Each node contains a rating function, a precondition, a runtime condition, a postcondition and a success measure. These are needed for the control of the plan execution. All inner nodes of a plan are branching nodes. They structure the plan into sequential and parallel parts. Therefore, they contain parallel groups which contain sequential seats. For each seat there are several candidate nodes. The selection of a node takes place by checking the preconditions and evaluating the rating functions of the respective candidates. Both the precondition checks and the evaluation of the rating functions are based on the current situation stored in the assertional box (A-Box) of the knowledge base. The leaf nodes of a plan are either action or planning nodes. Action nodes trigger elementary actions which are executed until the postcondition is reached or the runtime condition is no longer satised. Planning nodes contain subgoals which initiate replanning processes.

Figure 2. The plan concept of the mission subontology in UML representation.

2.2. The Inspection Subontology The inspection subontology consists of the available inspection actions and the relevant EOI classes. The inspection actions are a subconcept of actions and have additional input and output data types. Based on these data types the inspection planner can decide which actions can be combined. The inspection actions are subdivided into actions for acquiring sensor measurements, for preprocessing sensor measurements, for sensor fusion, for segmentation of measurements into EOIs, for feature computation, for classication and for the fusion of classication results. The inspection actions are used to detect and classify EOIs. To achieve this, the knowledge base contains information about which sensors and inspection actions are appropriate for each EOI class. This background knowledge is used to compute plans which gradually increase the classication condence of an EOI. The plans can also contain planning nodes with navigation goals, which are used to change the robot position or to reposition sensors. 2.3. Inspection Planning The course of the data analysis during the inspection of a single EOI usually follows a generic scheme: sensor measurement, segmentation, feature computation, classication. Of course, there can be more data processing steps in-between and there can be several different iterations of the data analysis process (e.g. after activation or repositioning of a sensor or due to a different data analysis algorithm being used), but in general every inspection of a single EOI follows a scheme like this. Having this in mind, we decided to choose an hierarchical approach for inspection planning: Complex inspection goals are recursively decomposed into simpler subgoals until the subgoals can be solved with elementary actions. The knowledge necessary for decomposing goals into subgoals is stored within the knowledge base in form of the available exible program nodes. For each goal to be achieved a corresponding root node for a exible program is selected. Based on the current situation stored in the knowledge base the inspection planner than decomposes this root node into an executable exible program. For that purpose it keeps track of the currently valid conditions in each step of the planning process. This is necessary because the situation stored in the knowledge base is only updated upon the execution of a node.

If the given node is a branching node, the corresponding groups and seats are added to the exible program and the preconditions of the candidates are checked. If the selection of the candidates has to be made at runtime all applicable candidates are recursively decomposed and added to the exible program. If the selection of the candidates can be made during the planning phase, only the applicable candidates with the highest rating functions are decomposed and added to the exible program. If the given node is an action node, the availability of the corresponding action is checked. If it is available, the node is added to the exible program. If the given node is a replanning node, the node is directly added to the exible program. If the preconditions of a node are not satised during the planning phase or an action is not available the corresponding node cant be executed. In such cases the recursion terminates and returns an empty exible subprogram. By this the inspection planner can detect when a subtree is not available. If there are no alternative candidates the parent node is not executable as well and and an empty exible program is returned for the parent node as well. 2.4. Decision Making In case of uncertainty regarding the data analysis results of an EOI a decision has to be made whether and how to proceed with the inspection of the EOI. Here we use a probabilistic approach in form of a Bayesian decision network. The structure of the decision network and the conditional probability tables are stored in the knowledge base. The values of the random variables within the network can be directly determined from the current situation stored in the knowledge base. This is possible because of a direct mapping between the random variables and the modeled concepts in the knowledge base. Moreover, the available decision options correspond to goals stored in the knowledge base as well.

3. Realization To be able to conduct experiments and evaluate the proposed semantic approach an appropriate robotic platform and inspection scenario has to be chosen. For this purpose the hexapod walking machine L AURON IVc (cf. Fig. 3) is used. L AURON IVc is equipped with appropriate sensors for localization, navigation and perception of its environment, e.g. a stereo camera system and a small laser line scanner on a pan-tilt unit. Moreover, an extensive behavior repertoire for locomotion and navigation exists [3]. As inspection scenario the detection and classication of different kinds of waste on irregular terrains like river and channel banks, seashores, countryside areas such as dunes or forests, or areas along the highways, is chosen. The vision is to equip the front legs of the next L AU RON generation with simple waste-grippers, extend the working area of the legs by an additional degree of freedom and place a garbage container on the back of the machine. The mission control architecture [14] of the semantic inspection approach has been implemented with MCA2 [13] - a modular, network-transparent and real-time capable C++ framework for controlling robots. The mission control system uses the existing behavior repertoire of L AURON IVc and contains an inspection data analysis component for the image based detection and classication of different waste classes, which is currently

Figure 3. Left: The six-legged walking machine L AURON IVc. Middle: Pan-tilt unit with stereo camera system and 2D laser scanner. Right: Picture of a cup lying around taken with the robot camera.

Figure 4. A generated exible subprogram for the feature computation, classication and decision steps in a waste inspection scenario with a camera and a laser scanner. For the actual decision on how to proceed the Bayesian decision network is used at runtime.

under development. Moreover, it comprises a semantic navigation approach, which will be presented in a future paper. The ontologies of the knowledge base are realized with OWL-DL1 and modeled with Protg2 . As framework for managing the ontologies and for reasoning processes regarding the ontologies KAON23 is used, which supports the SHIQ(D) subset of OWLDL. The inspection planer is implemented in Java (as KAON2 is written in Java) and is incorporated into the mission control architecture via JNI 4 . The Bayesian decision network is modeled with GeNIE5 and is accessed via SMILE.
1 OWL-DL: 2 Protg:

http://www.w3.org/TR/owl-features/ http://protege.stanford.edu/ 3 KAON2: http://kaon2.semanticweb.org/ 4 JNI: http://java.sun.com/docs/books/jni/ 5 GeNIE and SMILE: http://genie.sis.pitt.edu/

Figure 5. The Bayesian decision network used for computing the expected utility of several decision options in case of uncertainty regarding the data analysis results.

4. Preliminary Results To validate the proposed semantic inspection approach several simulation experiments were conducted. First, the functionality of the inspection planner and the suitability of the expert knowledge dened in the knowledge base were veried. Therefore, different inspection goals were passed to the inspection planner for decomposition (cf. Fig. 4 for an example of a generated exible program). Second, the execution of the exible programs in case of errors (e.g. malfunction of a sensor) was analyzed and the replanning process was tested. Third, the decision making by means of the Bayesian decision network was validated. For that purpose different inspection situations and data analysis results were simulated. The results of these rst functional tests were promising and showed the principal feasibility of the proposed semantic inspection approach. Nevertheless, more simulation and especially real-world experiments have to be done to detect possible limitations or shortcomings of the proposed approach.

5. Conclusion and Future Work In this paper a semantic approach for inspection planning, assessment of the data analysis results, decision making and replanning was presented. For the experimental evaluation of the proposed approach the detection and classication of waste on irregular terrains

with the hexapod walking machine L AURON was chosen. The realizations of the knowledge base, the inspection planning and the decision making processes were outlined and rst preliminary simulation results were presented. Future work will focus on the further realization of the mission control system for the proposed inspection scenario to allow for real eld tests. Moreover, an appropriate user interface for semantic interaction with the inspection control system will be developed. Finally, learning capabilities for self optimizing the sensor usage, the data analysis process, the planning process and the decision process will be added.

References
[1] C. Birkenhofer, K. Regenstein, J. M. Zllner, and R. Dillmann. Architecture of multi-segmented inspection robot KAIRO-II. In Krzysztof Kozlowski, editor, Sixth International Workshop on Robot Motion and Control (Romoco07), volume 360 of Lecture Notes in Control and Information Sciences, pages 381388. Springer, 2007. O. Duran, K. Althoefer, and L. D. Seneviratne. Automated pipe defect detection and categorization using camera/laser-based proler and articial neural network. In Automation Science and Engineering, IEEE Transactions on, volume 4, pages 118126, January 2007. B. Gamann. Modellbasierte, sensorgesttzte Navigation von Laufmaschinen im Gelnde. PhD thesis, Fakultt fr Informatik, Universitt Karlsruhe (TH), 2007. T. R. Gruber. Towards Principles for the Design of Ontologies Used for Knowledge Sharing. In N. Guarino and R. Poli, editors, Formal Ontology in Conceptual Analysis and Knowledge Representation, Deventer, The Netherlands, 1993. Kluwer Academic Publishers. C. Hudelot, N. Maillot, and M. Thonnat. Symbol grounding for semantic image interpretation: From image data to semantics. In Computer Vision, 2005. Tenth IEEE International Conference on, page 1875, 2005. S. Knoop, S. R. Schmidt-Rohr, and R. Dillmann. A exible task knowledge representation for service robots. In The 9th International Conference on Intelligent Autonomous Systems (IAS-9), 2006. N. J. J. P. Koenderink, J. L. Top, and L. J. van Vliet. Supporting knowledge-intensive inspection tasks with application ontologies. International Journal of Human-Computer Studies, 64(10):974983, 2006. N. Maillot, M. Thonnat, and A. Boucher. Towards ontology-based cognitive vision. Machine Vision and Applications, 16(1):3340, December 2004. A. A. F. Nassiraei, Y. Kawamura, A. Ahrary, Y. Mikuriya, and K. Ishii. Concept and design of a fully autonomous sewer pipe inspection mobile robot "KANTARO". In Robotics and Automation, 2007 IEEE International Conference on, pages 136143, Roma, April 2007. N. Palomeras, M. Carreras, P. Ridao, and E. Hernandez. Mission control system for dam inspection with an AUV. In Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on, pages 25512556, Beijing, October 2006. L. Pedersen, D. E. Smith, M. Deans, R. Sargent, C. Kunz, D. Lees, and S. Rajagopalan. Mission planning and target tracking for autonomous instrument placement. In Aerospace, 2005 IEEE Conference, pages 3451, March 2005. H. Streich and O. Adria. Software approach for the autonomous inspection robot MAKRO. In Robotics and Automation, 2004. Proceedings. ICRA 04. 2004 IEEE International Conference on, volume 4, pages 34113416, April/May 2004. K. Uhl and M. Ziegenmeyer. MCA2 an extensible modular framework for robot control applications. In The 10th International Conference on Climbing and Walking Robots (CLAWAR 2007), 2007. K. Uhl, M. Ziegenmeyer, B. Gamann, J.M. Zllner, and R. Dillmann. Entwurf einer semantischen Missionssteuerung fr autonome Serviceroboter. In Autonome Mobile Systeme (AMS 2007). Springer, 2007. M. Uschold and M. Gruninger. Ontologies: principles, methods, and applications. In Knowledge Engineering Review, volume 11, pages 93155, 1996. F. Y. Zhou, J. D. Wang, Y. B. Li, J. Wang, and H. R. Xiao. Control of an inspection robot for 110KV power transmission lines based on expert system design methods. In Control Applications, 2005. CCA 2005. Proceedings of 2005 IEEE Conference on, pages 15631568, Toronto, Ont., August 2005.

[2]

[3] [4]

[5]

[6] [7] [8] [9]

[10]

[11]

[12]

[13] [14]

[15] [16]